Maximize the Menu
New DoD RAM Guide

By: Ned H. Criscimagna, Alion Science and Technology and Michael Zsak, Decisive Analytics Corporation

Introduction

A new DoD Guide on Reliability, Availability, and Maintainability (RAM) is undergoing final coordination and will be released in the very near future. The Reliability Analysis Center served as the facilitator and lead developer for the teambased effort, which involved participants from DoD, the military services, and industry. The Introduction to the new guide succinctly states the purpose of the document:

The primary objective of Department of Defense (DoD) acquisition is to acquire quality products that satisfy user needs with measurable improvements to mission capability and operational support in a timely manner, and at a fair and reasonable price (Reference 1). This guide supports that objective. It addresses reliability, availability, and maintainability (RAM) as essential elements of mission capability. It focuses on what can be done to achieve satisfactory levels of RAM and how to assess RAM.



Addressing a Fundamental Reliability Issue

Poor reliability in defense systems negatively affects the operational readiness rates required by commanders in the field. It also leads to larger logistics footprints that detract from mobility and increases operating and support costs that divert resources from necessary modernization. Reliability issues continue to cause problems for the Department. In 1998, the National Research Council wrote:

The Department of Defense and the military services should give increased attention to their reliability, availability, and maintainability data collection and analysis procedures because deficiencies continue to be responsible for many of the current field problems and concerns about military readiness. (Reference 2)

The General Accounting Office (GAO) observed that in FY 2003, DoD asked for approximately $185 billion to develop, procure, operate, and maintain its weapon systems--an increase of 18% over FY 2001. A significant portion of these funds are needed to cover spare parts and support systems to meet required readiness levels, and the high cost of maintaining systems has limited DoD's ability to modernize and invest in new weapons. This led Government Accounting Office (GAO) to issue a report to the Senate Armed Services Subcommittee on Readiness Management Support in which one recommendation to DoD was to revise acquisition regulations to require a firm estimate of component and subsystem reliability at the production decision (Reference 3).



Latest Changes in Acquisition Policy Help

New Defense Acquisition Policy guidance, in the DoD 5000 series regulations, addresses part of the problem. For example, consider the following provisions.

  • The new policy requires that acquisition programs be managed through the application of a systems engineering approach that optimizes total system performance, and minimizes total ownership costs.

  • Entrance into the System Development and Demonstration phase depends on technology maturity (including software), approved requirements, and funding. Unless some other factor is overriding in its impact, the maturity of the technology shall determine the path to be followed.

  • Reliability in the context of overall system performance, supportability, and affordability must be considered before proceeding into full rate production and deployment.

  • New performance-based logistics strategies will optimize total system availability while minimizing cost and logistics footprints.
Policy alone is not enough. Processes and procedures are needed to help change the behavior of the acquisition system to improve up-front design. Finally, technical guidance must be made readily available on how to "do the job right."



Providing Better Technical Guidance

Under acquisition reform, many military specifications, standards, and handbooks were cancelled to force the DoD to rely more on the commercial world and thereby accelerate the adoption of commercial best practices and technologies which would improve capability at lower cost. Unfortunately, in the RAM area, a void formed. The National Research Council (NRC) recognized this situation in the aforementioned 1998 report where it recommended:

Military reliability, availability, and maintainability testing should be informed and guided by a new battery of military handbooks containing a modern treatment of all pertinent topics in the fields of reliability and life testing, ...

As a result of this void, many programs do not execute RAM tasks effectively. Recent studies concluded that defense contractor reliability design practices may be inconsistent with best commercial practices for accelerated testing, simulation guided testing, and process certification/control. Other studies found componentand system-level testing is inadequate, testing time is limited, and sample sizes are too small.

Component stress testing was frequently missing or inadequate. Proper accelerated life testing was rarely accomplished. Early system level modeling and simulation was inadequate to enter tests with acceptable levels of reliability. This situation leads to inefficiencies e.g., reinventing the wheel, errors, greater costs, stovepipes, inadequate data, limited data sharing, and reuse.

As part of a call to action, the NRC Committee on National Statistics convened a workshop on reliability issues for DoD systems in June 2002 (Reference 4). The workshop focused on seven topics.

  • Global reliability test designs to direct defense system development

  • Recent developments in reliability growth modeling

  • Use of statistical modeling for combining information, with applications to developmental and operational testing

  • Methods for estimating reliability from field performance data

  • Modeling of fatigue

  • Reliability of software-intensive systems

  • Assessment of life-cycle costs through the use of reliability analysis
The broad goal of the workshop was to foster greater interaction between the academic and defense acquisition communities to:

  • Generate greater interest from the academic community in reliability issues facing defense acquisition.

  • Inform decision makers about ways of dealing with constraints that may hamper the application of statistical techniques in obtaining test and field data and in the use of testing for the development of reliable systems.

  • Acquaint the defense community with state-of-the-art techniques applicable to the problems they face.
Accordingly, the NRC workshop recommended that the cancelled DoD Handbook, DoD 3235.1-H Test and Evaluation of Systems Reliability, Availability, and Maintainability: A Primer (RAM Primer), last updated in 1982, be upgraded and replaced. The objective of such an effort was to improve communication on the most readily applied and broadly applicable statistical techniques between (1) those in DoD who have the responsibility for reliability of defense systems and (2) academic researchers with significant expertise in such statistical techniques.

Specifically, the NRC workshop determined that the treatment of the following subjects in the revised RAM Primer should be updated with more modern approaches.

  • Methods for combining information across test environments

  • Methods for incorporating subjective assessments

  • Fatigue modeling

  • Statistical methods for software engineering

  • Wider use of nonparametric methods, specifically for reliability growth

  • Alternative methods for modeling reliability growth
In addition, the workshop indicated that the material on the following subjects in the RAM Primer was deficient.

  • Discussion of physics of failure models

  • Robust methods

  • Variance estimation

  • Stress testing

  • Accelerated testing

  • Decision analytic issues

  • Repair and replacement policies

  • Methods for dependent components

  • Current methods in experimental design

  • Baysian approaches
The National Research Council recommendations were provided to a multi-organizational team chartered to develop the RAM Guide.



The New Guide

The new RAM guide provides information on reliability, availability, and maintainability throughout the system lifecycle. The content of the guide is organized into six chapters and four appendixes. The chapter and appendix titles are:

  • Chapter 1: RAM and the Department of Defense
  • Chapter 2: Achieving RAM in Military Systems
  • Chapter 3: Understand and Document User Needs and Constraints
  • Chapter 4: Design and Redesign for RAM
  • Chapter 5: Produce Reliable and Maintainable Systems
  • Chapter 6: Monitor Field Performance
  • Appendix A: Proposals and Contracts
  • Appendix B: Software Reliability
  • Appendix C: Reliability Growth Management
  • Appendix D: Field Assessment and System Trending
Chapters 3 through 6 develop in detail a four-step model for achieving RAM. The model, introduced in Chapter 1 and overviewed in Chapter 2, is shown in Figure 1. A brief explanation of the key steps shown in the figure follows.

  • Before a system can be designed the needs and constraints of the user must be understood and documented. Therefore this first step is the foundation required to achieve RAM performance for a system.

  • After the user needs and constraints are accounted for the acquisition process shifts to Step 2 which focuses on ensuring RAM requirements are "built-in" the system first in the design phase and then improved during the redesign phase for the system. RAM requirements are balanced against other system performance requirements.

  • After the needs and constraints of the user are addressed through design and redesign, the system must be manufactured in such a manner that the designed-in RAM performance remains intact throughout production. Step 3 ensures that a reliable and maintainable system is produced.

  • Often, the final step of the process, which is monitoring field experience, is overlooked, with adverse results. Since the cost to operate and support systems is so high in Defense systems, Step 4 is necessary because without monitoring field performance, the strong RAM foundation formed during the first three steps may degenerate. Field experience can be used to improve maintenance, identify necessary improvements to the system, and provide much needed "lessons learned" for future systems.

Figure 1. Model Showing the Key Steps to Developing Reliable, Available, and Maintainable Systems in the Context of Current Acquisition Policy. The four-step model was conceptualized by Dr. E. Seglie, OSD/OT&E.
Figure 1. Model Showing the Key Steps to Developing Reliable, Available, and Maintainable Systems in the Context of Current Acquisition Policy. The four-step model was conceptualized by Dr. E. Seglie, OSD/OT&E. (Click to Zoom)


These steps are consistent with robust system engineering practices and are compatible with any general acquisition process. Each step consists of five elements, each of which is needed to be successful: (1) a clear goal for the step; (2) the right organizations and people involved; (3) adequate supporting information; (4) available tools, funds, and time to support the appropriate activities for that step; and (5) a good record of the results. The guide focuses on how the four steps of the model apply within the DoD acquisition framework.

The RAM Guide supports the DoD acquisition objective of acquiring quality products that satisfy user needs with measurable improvements to mission capability and operational support in a timely manner and at a fair and reasonable price. It certainly is consistent with and supports the stated imperatives of Undersecretary of Defense (Acquisition, Technology and Logistics) [USD(AT&L)] to:

  • Provide a context within which I [USD(AT&L)] can make decisions about individual programs.

  • Achieve credibility and effectiveness in the acquisition and logistics support processes.

  • Help drive good systems engineering practice back into the way we do business.
The new RAM Guide does not seek, in one publication, to address all the deficiencies that the NRC identified. It places RAM in context as an important system performance parameter, it provides insight into what can be done to achieve satisfactory levels of RAM and it identifies how to assess RAM. It provides insight into issues that system developers need to consider and address. It introduces and explains topics, places them within the most recent Defense Acquisition framework, and provides the reader with further references for more detailed coverage. If there is sufficient demand, more detailed subsequent volumes may be developed to address in greater depth the issues identified by the NRC.

The RAM Guide will be available on the Systems Engineering Web Site, http://www.acq.osd.mil/ds/se/index.html. Feedback on the utility of the RAM Guide and suggestions for improvements should be provided to the Office of Primary Responsibility, OUSD(AT&L)DS/SE/ED via E-mail atled@osd.mil.



References

  1. DoD Directive 5000.1, The Defense Acquisition System, May 12, 2003, Paragraph 4.2, page 2.

  2. National Research Council, Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements, Panel on Statistical Methods for Testing and Evaluating Defense Systems, Michael L. Cohen, John B. Rolph, and Duane L. Steffey, Editors, Committee on National Statistics; Washington D.C., National Academy Press, 1998.

  3. GAO Final Report, Best Practices: Setting Requirements Differently Could Reduce Weapon Systems' Total Ownership Costs, February 11, 2003, (GAO Code 120092/GAO-03-057).

  4. National Research Council; Reliability Issues for DoD Systems, Report of a Workshop; Francisco Samaniego and Michael Cohen, Editors.; Committee on National Statistics; Washington D.C., National Academy Press; 2002.