T h e J o u r n a l o f t h e R e l i a b i l i t y A n a l y s i s C e n t e r S e c o n d Q u a r t e r - 2 0 0 0 4 [From time to time, we find and reprint articles in other journals that we believe would be of interest to our readers. The following article was originally pub- lished in the Winter 1999 issue (Volume XXIII, Number 4) of the Air Force Journal of Logistics and is reproduced in its entirety with permission. The Editor] Commercial-Off-the-Shelf - or COTS - has become a byword for acquisition reform, but there are significant risks associated with the use of COTS prod- ucts in military systems. These risks are especially acute for aviation systems. To take advantage of the fast pace of technological advances in industry, the Department of Defense (DoD) is acquir- ing commercial products and compo- nents for use in military systems. COTS items provide the Department of Defense with numerous potential benefits. Primarily, they allow incorporation of new technology into military systems more quickly than typical developmental programs. COTS can also reduce research and development costs. Even more important, the Department of Defense has looked to COTS purchases to help reduce operations and support costs for military systems. Figure 1 shows why this is highly desired: the cost of operations and support is almost three- quarters the overall cost of a typical sys- tem. With this in mind, what could be the worst misfortune to befall an item procured as COTS? Could it be that the item changed and the original was no longer available commercially? What if the commercial replacement would no longer work in the military system for which it was procured? The very worst misfortune, which incorporates both of these problems, would be if the item were to suddenly become government uniqueno replacement available com- mercially. Becoming government unique would not entirely defeat the purpose of a COTS acquisition, but it would signifi- cantly affect support - the longest tail and, as shown in Figure 1, the greatest cost in the acquisition life cycle. This misfortune could never affect our COTS procurement - or could it? In any COTS acquisition, the acquirer needs to have already planned for this eventuality. Government unique is the conceptual opposite of COTS. An item is govern- ment unique when the only source or user of the item is the government. An item is a discrete unit that can be individ- ually acquired for the logistical support of a system. A system, in this definition, is the higher level mission component for which the item is procured. For example, an aircraft and its support equipment are a system, but a radio installed in the air- craft is an item. Whenever a manufac- turer discontinues or makes a change to a COTS item, the item can become gov- ernment unique. When the manufacturer changes the item, if the government does not either acquire the variant or reflect the change in the systems incorporating the item and the systems documentation, the original becomes government unique. After a manufacturer makes a change to an item, the government might be able to purchase and use the new variant without any negative effect to the system. In this case, though the original item is now government unique, the change would not affect the form, fit, interface, or mis- sion characteristics of the device. Unfortunately, manufacturers changes routinely affect these characteristics, and the effects of these COTS item changes for systems incorporating them are sig- nificant. The problems of changing form, fit, and interface should be obvi- ous; if the variant item is to be installed and operate correctly, these characteris- tics generally cannot change. To accom- modate form, fit, and interface changes, the acquirer must usually make modifi- cations to the system. Modifications are costly and usually result in the original item becoming obsolete. Changes to mission characteristics do not necessarily result in system modifications. However, if they affect the overall per- formance or capability of the system, they can cause significant problems. For example, if the new item has an operat- ing temperature range less than that of the original, the system could fail when used in an environment where tempera- tures exceed operating limits. Although configuration changes can cause create [sic] in a logistics program, the most devastating cause of govern- ment uniqueness occurs when a manu- facturer discontinues an item. Figure 2 shows that, for a large number of COTS acquisitions, this is inevitable. The life The Problem with Aviation COTS Lieutenant Colonel L. D. Alford 100 Life-Cycle Cost Operation & Support System Acquisition Production Research & Development 8% 20% 72% 0 I II III Milestones 90 80 70 60 50 40 30 20 10 0 Figure 1. Typical Cost Distribution1 T h e J o u r n a l o f t h e R e l i a b i l i t y A n a l y s i s C e n t e r S e c o n d Q u a r t e r - 2 0 0 0 5 of a typical military acquisition exceeds 20 years, yet the life of a typical civil product, especially electronics, is much less. From our own experience, we know it is almost impossible to purchase an ancient Z80-based computer, but right now, the Z80 lives on in the Air Forces AP-102 computer. This problem is not isolated to the electronics industry. For example, electronic gauges are replacing aviation steam gauges, the mechanical gauges on instrument panels. As a result, sources for mechanical components are becoming scarce, and they are difficult to obtain. The concepts outlined provide the defin- itive framework under which COTS must be understood. Without notice, the man- ufacturer is free to make changes to or discontinue production of the COTS item. As long as the manufacturers item changes do not affect characteristics or logistics supply, the acquirer has no problem. When changes do affect form, fit, interface, mission characteristics, or logistics supply, these changes become a significant problem for any COTS acqui- sition. This is especially true for aviation COTS. Two specific difficulties, airworthiness and forced modifications, result from manufacturers changes to aviation COTS. Airworthiness is the primary safety characteristic of any aircraft. It is the primary element proven in the testing of the aircraft. The Federal Aviation Administration (FAA) certifies the air- worthiness of most COTS items for air- craft, and these items must be certified in the system as well as individually. Military system certification, except for FAA-certified aircraft, is done wholly by the aircrafts configuration management (CM) authority. In the Air Force this authority is the single manager. This means that a simple change in mission characteristics, including improved func- tionality, will always drive a recertifica- tion of the aircraft. This recertification can range from a paper review to full flight test. The rate of change in COTS items is significant. This is especially true for aviation COTS. Considering the rate of change in COTS items, frequent recertification is a daunting prospect for the CM authority. In addition, COTS item changes can also drive changes to the specifications and technical data of any system on which these items are installed. The other difficulty for aviation COTS, which also affects any system, is forced modifications. A forced modification is a systems modification caused by the change of form, fit, interface, function, mission characteristic, or logistics sup- ply. When logistics supply is affected, the acquirer must support the discontin- ued item or find a replacement. The lat- ter may force a modification. More common in aviation COTS is an FAA- directed (airworthiness directive [AD]) change to an item.3 These directives are FAA regulation-based orders that man- date a change to an aviation item or sys- tem. Airworthiness directives are regula- tory in nature, and no person may operate a product to which an airworthi- ness directive applies except in accor- dance with the requirements of that air- worthiness directive.4 The manufactur- er has two choices in implementing the AD: discontinue the product or make the required change. The user of the item also has two choices: get a replacement product, if available, or make the changes required by the directive. When the change affects the form, fit, or inter- face of the item, an AD forces a modifi- cation to the system. For FAA-certified aircraft, the system must also receive FAA flight certification. For government certified aircraft, the CM authority must modify the system and certify airworthi- ness. However, the government is under no obligation to change its COTS items to accommodate an AD. If the govern- ment does not change a COTS item to comply with an AD, the item becomes government unique. Because the gov- ernment self-certifies, commonly, non- FAA certified government aircraft do not make AD directed changes. Further, because in many cases, the government does not subscribe to technical changes from manufacturers, the CM authority may not be aware of ADs that pertain to a systems components. This problem is exacerbated when the CM has estab- lished a depot for a COTS acquisition and is, in that case, supporting the com- ponent without knowledge of or real commonality with the original item. Usually ADs are issued more than once a year affecting well-established air vehi- cles; however, thousands of ADs may affect a single aircraft model. All this boils down to the fact that, for aviation, a COTS item will become gov- ernment unique in a very short period of time from a few months to a year after the acquisition of the item. Government (continued on page 8) Figure 2. COTS Obsolescence2 T h e J o u r n a l o f t h e R e l i a b i l i t y A n a l y s i s C e n t e r S e c o n d Q u a r t e r - 2 0 0 0 8 probability that a randomly selected value of z would be between -1.645 and 1.645. This, in turn, implies that it is 90% probable that p would be in a range defined by Equations 6 and 7. (6) (7) For example, if we took a sample of 1000 parts and found 500 defective, p' =.5, and we would be 90% sure that p, the propor- tion defective in the parent population, was actually in the range: = (.5 - 1.645 [.0158]) to (.5 + 1.645 [.0158]) = (.5 - .026) to (.5 + .026) = .474 to .526 (8) Thus, 1000 samples allow us to determine the value of p to be .5 plus or minus .026 which is less than a 3% error. Note that this is 3% of the possible range of values for p, not 3% of the esti- mated value of p. If, instead of counting defectives, we were polling voters and found 500 out of 1000 polled in favor of some government action, we would report the results as 50% in favor with a margin of error of 3%. While the measured value p' will affect the margin of error, the biggest influence on it is the sample size. A sample size of 100 with p' measured at .5 will result in a margin of error, obtained by using Equations 6 and 7, of .08225, or just over 8%. Similarly, a sample size of 10,000 will reduce the margin of error to .00825 or less than 1%, for a 90% confidence. The user can trade-off between confidence, margin of error, and sample size, based on a normal distribution of sample means, no matter what the underlying distribution of the parameter of interest, thanks to the central limit theorem. For a user-friendly introduction to statistics and its application to reliability engineering, try Practical Statistical Tools for the Reliability Engineer, RAC Product code STAT. ( ) [ ] N/p'-1p' 1.645 - p' range oflimitLowerLL == ( ) [ ] N/p'-1p' 1.645 p' range oflimit Upper UL +== ( ) [ ] ( ) ( ) [ ] ( ), 1000/ .5-1.5 1.645 .5 UL to1000/ .5-1.5 1.645 - .5 LL += The Problem with Aviation COTS (continued from page 5) uniqueness means forced review, modification, support changes, and recertification when the change is recognized - or blissful ignorance and risk if the change is not recognized. COTS Support Strategies What can be done to prevent these problems for aviation systems specifically and all systems generally? One solution has been mentioned, and this solution has been accomplished with vary- ing degrees of success since the first acquisition of COTS items. Depot. This approach is the acknowledgment of an items potential government uniqueness before the manufacturer makes any changes. In this strategy, the acquirer purchases spares and builds a government depot activity to support the item. This solution does take advantage of the COTS item commercial development, but the overall cost savings may not be significant because the longest tail, the support tail, is at least as long as any normal government item develop- ment. In fact, the support tail may be costlier because the government has not been involved in the item development. Many programs use this strategy; the C-130 improved aux- iliary power unit program is one example. Lifetime Spares. Another similar solution is to purchase enough spares for the total life of the system and item. The AP-102 computer program used this strategy to ensure suf- ficient Z80 chips to support the life of the system. Again, this is not an optimum solution because it usually increases the items logistics tail. In this case, if the items life expectancy is less than predicted or the items life is extend- ed, the government has no other recourse than to entirely replace the item or to develop a support capability. These two solutions, government depot and lifetime spares buy, prevent forced modifications and subsequent airworthiness certification requirements. They can also introduce risk. In addition, they defeat two major potential advantages of COTS: the ability to reduce the support tail and the ability to take advantage of future commercial developments in the item. There are four other solutions to these problems that do take full advantage of the possibilities of COTS acquisition, but each is fraught with its own risk. Each of these solutions is a variant of what is commonly known as contractor logistic support (CLS). Purchase Technical Information. In the first alternative, the acquirer can purchase the servicing information support of the manufacturer. This allows the CM authority to make decisions based on changes to the item. If the CM authori- ty knows of a manufacturers changes to an item, the CM can choose to acquire a replacement or modify the system as required to allow continued use of the variant item. The CM has three options. First, when an item changes and the decision is made to replace the item, the CM must acquire ( ) ( ) .00025 1.645 .5 to.00025 1.645 - .5 += T h e J o u r n a l o f t h e R e l i a b i l i t y A n a l y s i s C e n t e r S e c o n d Q u a r t e r - 2 0 0 0 9 and certify the new item. Second, if the item is retained with changes, the CM must certify and possibly change the system. And third, if a decision is made to not make any changes to the item, the CM must set up government-unique support. The advantages of retention or replacement (options 1 and 2) are the continued COTS logistics tail and guaranteed item certification. The CM must still recertify the system. If the item is retained in its original configura- tion (option 3), the decision to support a government-unique item leads to a typical high-cost government logistics tail. This pick-and-choose method of systems support probably has not been used intentionally. However, after a manufac- turer has made unexpected changes to a COTS component, many programs have found themselves in this situation. Purchase Manufacturer Support. The second alternative is the acquirer can purchase manufacturer support for the item. The risks in this are similar to that of purchasing serv- icing information support; however, the manufacturer has more incentive to keep the item within form, fit, and inter- face configuration for the system. When changes in the sys- tem are required to support changes in the item, the manu- facturer can aid the CM authority. This is a very common method used to support COTS. Purchase Manufacturer Modification Support. In the third alternative, the acquirer can purchase the full, integrat- ed support of the manufacturer. This allows the manufac- turer to make changes to the system, along with changes to the item. The contractor may have some Total System Performance Responsibility (TSPR), but the CM authority must still recertify the system. The AC-130U is using this method to manage COTS in its new Integrated Weapon System Support program. This is the most common method used today to support COTS items and systems through CLS. Purchase Full Manufacturer Support. Fourth, the acquir- er can purchase the full system support that would allow an integrator to automatically make changes to the system nec- essary to accommodate any item changes. In this scenario, the contractor would have TSPR and certify the weapon sys- tem. This fourth option is used primarily to support FAA- certified government aircraft. It could potentially be used to support any government aircraft or system incorporating COTS items. The message should be plain. COTS acquisitions lead the acquirer down two support paths: government-unique, high-cost logistics and COTS manufacturer support. Both of these paths involve risk and guarantee future costs for any system incorpo- rating COTS items. The potential of COTS acquisitions is embodied in a lower cost development, initial acquisition, and support costs. That potential must be balanced with the knowl- edge that COTS acquisitions will either force modifications and recertifications or lead to a typical government-unique logistics tail. COTS for aviation is a viable method of aircraft and aviation acquisition, but it is not a simple solution. It requires careful planning and forethought that must be incorporated into any pro- gram contemplating a COTS acquisition. Notes 1. John F. Phillips, DUSD (L) September 1996, and John W. Jones, Ed., Integrated Logistics Support Handbook, 2nd Ed., 1995. 2. Joint Stars AFPEO/C3 briefing. 3. Federal Aviation Administration, Part 39Airworthiness Directives, Federal Aviation Regulations. Washington, DC: Government Printing Office, February 1996. 4. Ibid. Lieutenant Colonel Alford is an aeronautical test policy manag- er at the Air Force Materiel Command. Editor's Note: Another article on COTS appeared in the March 2000 issue of National Defense. The article, Commercial-Off- the-Shelf Military Systems: Myth vs. Reality, discusses the risks associated with over-reliance on COTS technology. The "myth" is that users expect that systems will be fielded much more quickly when COTS is used. Philip E. Coyle, the Pentagon's director of operational test and evaluation, cautions that military equipment "is never completely off-the-shelf." Many of the issues raised by Lieutenant Colonel Alford and Mr. Coyle are addressed in a forthcoming handbook from the RAC, titled Supporting Commercial Products in Military Applications. Now available from the RAC is SELECT, a software based tool to assist in evaluating COTS for military applications.