Maximize the Menu
Impacts on Reliability of Recent Changes in DoD Acquisition Reform Policies

Introduction

The Department of Defense (DoD) has been pursuing Acquisition Reform (AR) for over 25 years, so it is therefore not a new concept. However,with the release of what is now simply called "The Perry Memo", AR has taken on a whole new meaning. "The Perry Memo" is the June 1994 four-page memorandum on "Specifications & Standards - A New Way of Doing Business" signed by Secretary of Defense William J. Perry [1]. One of the main thrusts of this memo was changing the emphasis on an acquisition process that relied heavily on militarystandards and specifications and systems built uniquely for the defense customer, to one that decreases reliance, "to the maximum extent practicable", on such standards and specifications. Further, the thrust is now on developing systems based on performance and commercial based specifications and standards with a concerted effort to incorporate commercial off-the-shelf (COTS) and non-developmental item (NDI) equipments into systems acquisition programs.

The objective of this START sheet is to provide a brief overview on where recent changes in AR policy are affecting Reliability and Maintainability (R&M) and to provide the reader with additional references that should enable tracking future changes and impacts. More specifically, the following areas will be discussed:

  • R&M Specifications and Standards
  • Use of COTS in Military Systems
  • On-going Efforts
  • References


R&M Specifications and Standards

Possibly the largest and most far reaching impact to date on R&M has been the Defense Standards Improvement Council (DSIC) decisions to cancel or convert many well known R&M specifications and standards tohandbooks. The current status of those specifications and standards related to R&M is documented in Table 1. Table 1 was derived from the latest status on military specifications and standards obtained from the Standardization Program Division Homepage on the Internet [11].

Table 1: Current Status of R&M Specifications and Standards
Document Title Decision/Action Completed
MIL–HDBK–217F Reliability Prediction of Electronic Equipment Cancel upon development of suitable NGS.
MIL–HDBK–338 Electronic Reliability Design Handbook Retain as a guidance only document.
MIL–STD–470B Maintainability Program for Systems and Equipment Short term: Cancel and redesignate as a handbook. Consolidate with MIL–STD–470 and publish as a new handbook by FY'97.
MIL–STD–471A Maintainability Demonstration Short term: Cancel and redesignate as a handbook. Consolidate with MIL–STD–471 and publish as a new handbook by FY'97.
MIL–STD–781D Reliability Testing for Engineering Cancel after incorporating appropriate sections into MILHDBK– 781.
MIL–STD–785B Reliability Program for Systems Equipment Cancel after publication of a suitable NGS. Development and Production Cancel after publication of a suitable NGS.
MIL–STD–790E Product Assurance Program for Electronic and Fiber Optic Parts Specifications MIL–STD–790F dated 1 August 1995 as a Standard Practice.
MIL–STD–1543–B Reliability Program Requirements for Space and Launch Vehicles Cancel after publication of a suitable NGS.
MIL–STD–1629A Procedures for Performing a Failure Mode, Effects and Criticality Analysis Cancel after publication of a suitable NGS. DSIC Chair will write to the Society of Automotive Engineers (SAE) requesting they attempt to publish an industry STD by June 96 (5/17/95).
MIL–STD–1843 Reliability–Centered Maintenance for Aircraft, Engines & Equipment Canceled w/o replacement by Notice 1 of 9 August 1995.
MIL–STD–2084 General Requirements for Maintainability of Avionics and Electronic Systems and Equipment Notice 3 of 31 July 1995 redesignated MIL–STD–2084(AS) as a HDBK. MIL–HDBK–2084 was dated 31 July 1995.
MIL–STD–2164 Environmental Stress Screening Process for Electronic Equipment Cancel via publication within 30 days of a Notice redesignating as a HDBK (6/7/95).
MIL–STD–2165A Testability Program for Systems & Equipment Redesignated as a HDBK (MIL–HDBK–2165)

In reviewing the table, two facts come to light. Many of the standards and specifications are being redesignated as handbooks, and many are to be canceled after publication of suitable Non-Government Standards (NGS). Further, not all of the reviewed documents were standards and specifications. Some handbooks have also been reviewed and acted upon. Because suitable NGSs are not yet available, many of these documents are not yet canceled, and are currently active. If that is the case, how does this affect R&M? The answer lies once again in Secretary Perry's memo. A primary directive of this memo is that performance and commercial specifications and standards are to be used in lieu of military specifications and standards, unless no practical alternative exists to meet the user's needs. Even the use of military handbooks, such as those listed in Table 1, cannot be imposed as a requirement. Handbooks and standards, if listed at all, are to be referenced as guidance documents only. The only way to still require the use of any of the R&M documents listed in Table 1 is to request a waiver, but only as a last resort in those cases where there is no acceptable NGS, or the use of a performance specification or non-government standard is not cost-effective. The reader should note that each time a waiver has been requested to use MIL-STD-785, the primary reliability program requirements document, it has been denied.

There are several documents that have been "exempted" from the waiver process within each of the three military departments. Exempted documents are those military and federal specifications and standards which do not require a waiver to be cited as a requirement by the military department which granted the exemption. This list, which is available on the WWW [2], does not contain any R&M documents.



Performance Specifications

The DSIC-approved definition of a performance specification is:


"A performance specification states requirements in terms of the required results with criteria for verifying compliance, but without stating the methods for achieving the required results. A performance specification defines the functional requirements for the item, the environment in which it must operate, and interface and interchangeability characteristics."

Requirements for R&M in terms of what is to be achieved, such as a Mean Time Between Failure (MTBF) value, must be defined without telling the contractor how to achieve them. Here in lies the biggest change, for both the government and the contractor. No longer can the government require a reliability prediction be performed per MIL-HDBK-217, or that a reliability program plan be developed per MIL-STD-785,or that any of the tasks contained in MIL-STD-785 or MIL-STD-470, etc., be performed and documented as a deliverable data item. None of those standards, specifications or handbooks are performance-based specifications. These, and many other military documents have been determined to be non-value added and too costly to implement. This is not to imply that for the last 25 years the government has been intentionally requiring work that is costly and "non-value added". However, as recently pointed out in an article written on standards reform [3], ". . . no one intentionally includes requirements that do not add value. But program offices, like many other offices, have more work than they can do. So they develop requests for proposals (RFPs) at the photocopy machine -- if it worked for the last contract, it will surely be good enough for this contract."

Many, if not all, of the R&M documents listed in Table1 are what is known as "tailorable" documents, meaning that only those tasks absolutely required to meet the users objectives should be cited as a requirement for system development. However, the time required to determine which tasks, or even which standards, are most applicable was often a commodity many program offices did not have. As a result, blanket requirements have been used for R&M, often resulting in R&M work being performed that may have provided excellent information, but added no value in terms of achieving a system that meets the user's R&M needs. Note that the government can still require that a stated reliability be demonstrated. Demonstration plans and specific methods, however, cannot be stated. It is now left up to the contractor to propose the means by which demonstration will be performed, which brings us to the next point of discussion.



R&M Challenges

Given that solicitations for new procurements or major modification scan only state the R&M performance requirements, and not how those requirements are to be achieved, there exist two challenges to those responsible for R&M. First, the contractor has to decide what needs to be done to cost effectively achieve a stated R&M capability. Second, the government must evaluate which one of potentially many different approaches provides the best value. Note that contractors are not prohibited from proposing to use military specifications and standards. Therefore,for those organizations that perform R&M activities as defined in the historical military R&M standards, a complete reengineering of that process need not take place. However, the onus is still on the contractor to perform proper tailoring of these documents to achieve an effective balance between cost and R&M. On the flip side, the government must also have a better feel for what activities are truly needed to achieve stated R&M performance to be able to effectively evaluate which approach provides best value. While this means that the days of comparing "apples to apples" is gone, it also means that true tailoring of R&M development and demonstration tasking is now achievable. This will result in more efficient use of engineering resources and better planning up-front in the acquisition process. Note further that the R&M program stated in the winning proposal will become a contractual statement of work, meaning that the contractor will be held responsible for providing what they said is necessary to achieve the R&M performance.



The Use of COTS in Military Systems

Another challenge resulting from recent AR changes is the emphasis on use of COTS equipments in military systems. The increased emphasis oncommercial products and practices has occurred for a number of reasons. First, the decrease in military spending over the last decade has resulted in an erosion in the industrial base that existed to support development of weapon systems. Second, while technology was driven primarily by the DoD in the past, this is no longer the case. Third, many technologies (e.g., electronics, information, communications) are advancing at such a rapid pace that the government can no longer afford an acquisition process that has historically required at least a 2-3 year cycle to develop, test, and field a system.

When considering the use of COTS equipment, much work needs to be done up front in terms of market research and development of minimum requirements. This means that the procuring offices must work closely with the end user to define the minimum acceptable performance specifications for R&M. Market research then needs to be performed to see what COTS equipment exists that has the potential of meeting defined requirements at an affordable price. The challenge to the market research is in obtaining R&M data on COTS equipment. Many COTS vendors are not likely to have the kinds of data that exist in military R&M data collection systems. This puts a greater emphasis on the need to validate COTS equipment through testing. Testing will be required to overcome the lack of R&M data and to provide some assurance that COTS equipments will meet stated R&M requirements. This includes such testing as Environmental Stress Screening (ESS) and qualification and verification testing. If design documentationis available, specific R&M tasks, such as predictions and failure mode and effects analysis (FMEA), may be part of the COTS evaluation process. Because the prime military contractor is not likely to be the COTS vendor in this case,boththegovernment and the prime will need to perform the evaluation. In some cases, a cooperative effort should be performed between the twoparties. Infact,there are documented cases where this has already occurred [4].



On-Going Efforts

As stated previously,AR is not new to the DoD. However, many past reforms, while encouraged, were not funded as they now are. These monies have been allocated over the next five years to assist in implementing reforms outlined by Secretary Perry. This funding has generated a number of different efforts in the R&M community to assist in helping with the transition to a more commercial and performance - based way of doing business. The US Air Force Rome Laboratory (RL) and the Reliability Information Analysis Center (RIAC) are undertaking a number of initiatives in the area of COTS/NDI usage, and in the area of standards and specifications.

In the COTS area, RL has initiated efforts to look at the impact of using Plastic Encapsulated Microcircuits (PEMs) in military applications and is sponsoring an effort being performed by the IIT Research Institute (IITRI) to develop a software tool to help compare and select COTS hardware for military applications. One effort in the PEMs area is the preparation of a handbook entitled, "Commercial Parts and Processes for Military Applications," for the Joint Tactical Information Distributed System (JTIDS) program office. This handbook contains valuable case studies and lessons learned that relate commercial products and processes to military applications involving harsh environments. See reference [5] for further information on RL initiatives in the PEM area.].

The COTS software tool effort is focused on the equipment level and will address such items as computers, signal processors, displays, power supplies, radio/communication systems, data storage devices and video units. A primary task of this effort is to collect and analyze data on fielded COTS equipment to determine failure histories and susceptibilities, and then to use this data to develop translation models that estimate the reliability and readiness of COTS equipment in adverse military environments.

In the area of R&M specifications, standards and documents, a joint RL/RIAC effort has resulted in the release of the popular "Reliability Toolkit: Commercial Practices Edition" [6]. The RIAC is also developing a number of "Blueprints for Product Reliability." The RIAC Blueprints are designed for use in both the government and industry to provide guidance in the successful definition and implementation of are liability program, and address products ranging from completely new commercial consumer products to highly specialized military systems.

RL is further sponsoring the development of a new handbook on maintainability to replace MIL-STD-470 and -471, and the development of a compendium of US and international commercial and military R&M hardware standards. The maintainability handbook effort is being performed by RIAC with assistance from McDonnell Douglas and a draft is available on the RIAC's WWW homepage [8]. The compendium provides information on a number of standards, handbooks and documents that could be considered for use in lieu of equivalent military documents. It will also contain information on standards bodies, and any on-going efforts to develop R&M standards. IITRI is also revising MIL-HDBK-338, which will consolidate valuable information contained in R&M standards that have been or are to be canceled.

As a final note, a group known as the Partnership in Reliability, Maintainability and Supportability (RMS) Standards has been formed to ensure that the US continues to develop and maintain world class standards in the area of RMS. This group comprises professionals from government, industry and academia. The group meets on a periodic basis and publishes a newsletter that is available on the WWW [11]. The newsletter keeps readers informed on the current issues facing RMS standardization and on current efforts to develop replacement RMS standards. Further references are provided below.



References

  • [1] "Specifications & Standards - A New Way of Doing Business", Memorandum from the Secretary of Defense to various military departments, Washington DC, June 1994.

  • [2] List of exempted military documents (i.e., not requiring a waiver).

  • [3] Bergmann,WalterB.,II,"StandardsReform",Partnership in RMS Standards Newsletter, Volume 1, Number 3, March 1996.

  • [4] SD-2, "Buying Commercial & Nondevelopmental Items: A Handbook", Office of the Under Secretary of Defense for Acquisition and Technology, April 1996.

  • [5] Caroli, Joseph A., et. al., "R&M In An Era Of Acquisition Reform", Proceedings of the Annual Reliability and Maintainability Symposium, January 1996, Las Vegas, Nevada.

  • [6] "Reliability Toolkit: Commercial Practices Edition: A Practical Guide For Commercial Products and Military



About the Author

* Note: The following information about the author(s) is same as what was on the original document and may not be correct anymore.

C. Richard Unkle is a research engineer for IIT Research Institute and provides engineering and management expertise in reliability, maintainability and testability to the Reliability Analysis Center. At IITRI he has been active in R&M program planning, reliability modeling and analysis, life extension analysis, testability analysis of electronic systems, and development of test automation tools and processes. Currently, Mr. Unkle is supporting RAC efforts in the development of a new military handbook on maintainability to replace MIL-STD-470 and 471. He is also the principal author of the compendium of US and international commercial and military R&M hardware standards being prepared under the sponsorship of Rome Laboratory.

Prior to joining IITRI, Mr. Unkle was a principal engineer for ARINC Research Corporation where he supported the DOE and the electric power industry in the areas of R&M management, availability analysis, testability and development of intelligent, portable, maintenance aids.

Mr. Unkle holds a bachelor of science degree in electrical and electronics engineering from Rochester Institute of Technology and has authored over 12 papers on testability analysis and test automation tools and techniques.