Defense Acquisitions

Measuring the Value of DOD's Weapon Programs Requires Starting with Realistic Baselines Gao ID: GAO-09-543T April 1, 2009

n 2008, the cumulative cost growth in the Department of Defense's portfolio of 96 major defense acquisition programs was $296 billion and the average delay in delivering promised capabilities to the warfighter was 22 months. These poor outcomes mean that other critical defense and national priorities may go unfunded and that warfighters may go without the equipment they need to counter the changing threats that they face. GAO has examined extensively the issues in DOD's weapon system programs from the perspective of best practices in product development, and believes that significant improvements are possible. Because the ability to measure knowledge, processes, and outcomes is critical to achieving these improvements, GAO has used metrics to review the management and health of these programs from within the framework of best practices. This testimony discusses: 1) "knowledge metrics," used to determine how well programs manage technology, design, and manufacturing risks; 2) outcome metrics--concerning cost, schedule, and capability--that serve as "health indicators" of how well programs are being executed in terms of predicted outcomes; and 3) the prerequisites that GAO believes must be met in order for a program's plans and goals to be realistic.

GAO employs a set of knowledge metrics to determine whether programs have attained the right knowledge at critical points over the course of a weapon system acquisition, and facilitate the identification of potential problems that could lead to cost, schedule, or performance shortfalls. In essence, knowledge supplants risk over time. Key knowledge points and metrics include 1) achieving a high level of technology maturity at the start of program development, 2) reaching design stability at the system-level critical design review, and 3) demonstrating that critical manufacturing processes are in control before starting production. By applying these metrics to selected programs in DOD's 2008 portfolio of major defense acquisitions, GAO found that most programs have started system development without mature technologies and moved into system demonstration with low levels of design stability. GAO has determined that programs with immature technologies and unstable designs have experienced significant cost and schedule growth. Program outcome metrics--quantitative measures of cost, schedule, and performance over time--provide useful indicators of the health of acquisition programs and whether they are meeting their intended goals. When assessed regularly for changes and the reasons that cause changes, these indicators can be valuable tools for improving insight into and oversight of individual programs as well as DOD's total portfolio of major defense acquisitions. The collective performance of the programs in DOD's portfolio is a key indicator of how well the acquisition system generates the return on investment that it promises to the warfighter, Congress and taxpayers. GAO recently reported that outcome metrics for DOD's 2008 major defense acquisition portfolio show worsening performance when compared to the department's 2003 portfolio. For example, total acquisition costs for programs in the 2008 portfolio increased 25 percent from first estimates compared to a 19-percent increase for programs in the 2003 portfolio. DOD is working with GAO and the Office of Management and Budget to develop a comprehensive set of outcome metrics to better assess its portfolio of programs. While knowledge and outcome metrics provide valuable information about the potential problems and health of programs, they are of limited value if DOD does not do a better job ensuring acquisitions begin with realistic plans and baselines prior to development start. GAO believes there is a clear set of prerequisites that must be met by each program's acquisition strategy before a measurement of the program's health will be of real value. These prerequisites include: 1) establishing an evolutionary, knowledge-based business case for each acquisition; 2) separating technology development from product development; 3) limiting time and requirements for product development to manageable levels; 4) employing systems engineering early on in the process to arrive at realistic cost and schedule estimates; 5) committing to fully funding a program once it is approved; and 6) setting priorities from the top to ensure that candidate programs are truly needed and have a solid plan for delivery.



GAO-09-543T, Defense Acquisitions: Measuring the Value of DOD's Weapon Programs Requires Starting with Realistic Baselines This is the accessible text file for GAO report number GAO-09-543T entitled 'Defense Acquisitions: Measuring the Value of DOD's Weapon Programs Requires Starting with Realistic Baselines' which was released on April 1, 2009. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Testimony: Before the Panel on Defense Acquisition Reform, Committee on Armed Services, House of Representatives: United States Government Accountability Office: GAO: For Release on Delivery: Expected at 7:30 a.m. EDT: Wednesday April 1, 2009: Defense Acquisitions: Measuring the Value of DOD's Weapon Programs Requires Starting with Realistic Baselines: Statement of Michael J. Sullivan: Director Acquisition and Sourcing Management: GAO-09-543T: GAO Highlights: Highlights of GAO-09-543T, a testimony before the House Committee on Armed Services Panel on Defense Acquisition Reform. Why GAO Did This Study: In 2008, the cumulative cost growth in the Department of Defense‘s portfolio of 96 major defense acquisition programs was $296 billion and the average delay in delivering promised capabilities to the warfighter was 22 months. These poor outcomes mean that other critical defense and national priorities may go unfunded and that warfighters may go without the equipment they need to counter the changing threats that they face. GAO has examined extensively the issues in DOD‘s weapon system programs from the perspective of best practices in product development, and believes that significant improvements are possible. Because the ability to measure knowledge, processes, and outcomes is critical to achieving these improvements, GAO has used metrics to review the management and health of these programs from within the framework of best practices. This testimony discusses: 1) ’knowledge metrics,“ used to determine how well programs manage technology, design, and manufacturing risks; 2) outcome metrics--concerning cost, schedule, and capability--that serve as ’health indicators“ of how well programs are being executed in terms of predicted outcomes; and 3) the prerequisites that GAO believes must be met in order for a program‘s plans and goals to be realistic. What GAO Found: GAO employs a set of knowledge metrics to determine whether programs have attained the right knowledge at critical points over the course of a weapon system acquisition, and facilitate the identification of potential problems that could lead to cost, schedule, or performance shortfalls. In essence, knowledge supplants risk over time. Key knowledge points and metrics include 1) achieving a high level of technology maturity at the start of program development, 2) reaching design stability at the system-level critical design review, and 3) demonstrating that critical manufacturing processes are in control before starting production. By applying these metrics to selected programs in DOD‘s 2008 portfolio of major defense acquisitions, GAO found that most programs have started system development without mature technologies and moved into system demonstration with low levels of design stability. GAO has determined that programs with immature technologies and unstable designs have experienced significant cost and schedule growth. Program outcome metrics”quantitative measures of cost, schedule, and performance over time”provide useful indicators of the health of acquisition programs and whether they are meeting their intended goals. When assessed regularly for changes and the reasons that cause changes, these indicators can be valuable tools for improving insight into and oversight of individual programs as well as DOD‘s total portfolio of major defense acquisitions. The collective performance of the programs in DOD‘s portfolio is a key indicator of how well the acquisition system generates the return on investment that it promises to the warfighter, Congress and taxpayers. GAO recently reported that outcome metrics for DOD‘s 2008 major defense acquisition portfolio show worsening performance when compared to the department‘s 2003 portfolio. For example, total acquisition costs for programs in the 2008 portfolio increased 25 percent from first estimates compared to a 19-percent increase for programs in the 2003 portfolio. DOD is working with GAO and the Office of Management and Budget to develop a comprehensive set of outcome metrics to better assess its portfolio of programs. While knowledge and outcome metrics provide valuable information about the potential problems and health of programs, they are of limited value if DOD does not do a better job ensuring acquisitions begin with realistic plans and baselines prior to development start. GAO believes there is a clear set of prerequisites that must be met by each program‘s acquisition strategy before a measurement of the program‘s health will be of real value. These prerequisites include: 1) establishing an evolutionary, knowledge-based business case for each acquisition; 2) separating technology development from product development; 3) limiting time and requirements for product development to manageable levels; 4) employing systems engineering early on in the process to arrive at realistic cost and schedule estimates; 5) committing to fully funding a program once it is approved; and 6) setting priorities from the top to ensure that candidate programs are truly needed and have a solid plan for delivery. View [hyperlink, http://www.gao.gov/products/GAO-09-543T] or key components. For more information, contact Michael J. Sullivan at (202) 512-4841 or sullivanm@gao.gov. [End of section] Mr. Chairman and Members of the Committee: I am pleased to be here this morning to discuss how best to measure and determine whether DOD's acquisition system is providing value to the warfighter. Earlier this week, we reported that the cumulative cost growth in DOD's portfolio of 96 major defense acquisition programs was $296 billion and the average delay in delivering promised capabilities to the warfighter was 22 months. These outcomes mean that other critical defense and national priorities go unfunded and warfighters go without the equipment they need to counter ever changing threats that they face. This condition is unacceptable. We believe that significant improvement in the acquisition of weapon systems is possible and that the ability to measure knowledge, processes, and outcomes is critical to achieving that improvement. It is important to note that not one single metric or set of metrics is enough to monitor acquisitions and gain efficiencies. Today, we would like to break our discussion about how to measure the department's acquisitions into 3 basic sections: * First, we would like to present a set of metrics that we refer to as "knowledge metrics" and use to determine how well acquisition programs are managing and retiring predictable technology, design, and manufacturing risks and gaining knowledge. These metrics are valuable because they can predict problems and identify causes. * Second, we would like to discuss a set of outcome measures-- concerning cost, schedule, and capability--that serve as health indicators. These indicators measure how well programs are being executed and achieving predicted outcomes in terms of meeting original baselines for cost, schedule, and performance. These metrics have intrinsic value as simple measurements, just as a thermometer can warn a parent that a child has a fever. * Third, there are certain indicators that we look for--based on the work we have done examining best practices for product development-- that are, perhaps, more important than these knowledge and health metrics because they determine from the outset how realistic the acquisition plans and strategies of programs are. For the sake of today's discussion, we will refer to them as "prerequisite indicators." These prerequisites are most important because we question the value of ANY metric when measuring from an unrealistic baseline. We know that the knowledge and program health metrics we use to measure programs' progress and outcomes are valuable when used in realistic, market-driven product development environments. We also know that ALL of these metrics are important indicators for decision makers. Our extensive body of work examining world-class enterprises and the way they operate has validated their value for programs that must deliver a new product to market at a certain time and within a certain investment cost or suffer significant consequences. These metrics work because they are measuring realistic plans and goals that are supported by doable requirements, accurate cost and schedule estimates, and stable funding. The company developing the products suffers dire consequences, such as loss of market share, if these programs do not succeed. This statement draws from our extensive body of work on DOD's acquisition of weapon systems. A list of our key products is provided at the end of this statement. This work was conducted in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. "Knowledge" Metrics Identify Potential Problems and Their Likely Causes: We have conducted a body of work that examines weapon acquisition issues from a perspective that draws upon lessons learned from best practices in product development. Collectively, these practices comprise a process that is anchored in knowledge. Achieving the right knowledge at the right time enables leadership to make informed decisions about when and how best to move into various expensive acquisition phases. In essence, knowledge supplants risk over time. This building of knowledge consists of gathering information about technology, design, and manufacturing at three critical points over the course of a weapon system program (Figure 1). We have developed valuable "knowledge metrics" that measure this knowledge build and allow us to identify potential problems that could lead to cost, schedule, or performance shortfalls and their likely causes. The metrics can be described as: * Knowledge Point 1, evidenced by the balance between a product's required capabilities and the resources available to meet them. Focus should be on understanding technological and design implications and achieving a high level of technology maturity at the start of system development. This means that the critical technologies needed to meet essential product requirements must be demonstrated to work in their intended environment. The technology readiness level for each critical technology is the metric we use to measure technology maturity. [Footnote 1] * Knowledge point 2, evidenced by the development of engineering prototypes and the completion of engineering drawings for an integrated product at the system design review. This metric provides tangible evidence that the product's design is stable, meaning it has a high probability of meeting customer requirements, as well as cost, schedule, and reliability targets. A best practice is to achieve design stability at the system-level critical design review, usually held midway through development. Completion of at least 90 percent of engineering drawings is the metric we use to measure design stability. * Knowledge point 3, evidenced by the demonstration that critical manufacturing processes are in control--that is, they are repeatable, sustainable, and capable of consistently producing parts within the product's quality tolerances and standards--at the start of production. One hundred percent of critical manufacturing processes in control is the metric we use to evaluate manufacturing maturity. Figure 1: Knowledge-Based Acquisition Process: [Refer to PDF for image: illustration] Material development start: Technology Development (A); PDR: Knowledge point 1: Metric: technology readiness levels at 7 or higher (B); Product Development: - Integration; - CDR: Knowledge point 2: Metric: 90 percent of engineering drawings complete; - Demonstration; Production: (C): Knowledge point 3: Metric: 100 percent of critical manufacturing processes in control. Source: GAO analysis of commercial best practices. [End of figure] Each of these metrics gauges the point when the requisite level of knowledge has been attained for a product in relation to where that product is in its development. World-class firms we have visited work hard to establish metrics such as these and their decision makers are required to consider them before deciding to advance a program to the next level. Theses types of metrics also help decision makers gauge progress in meeting cost, schedule, and performance goals and ensure that managers will (1) conduct activities to capture relevant product development knowledge, (2) provide evidence that this knowledge has been captured, and (3) hold decision reviews to determine that appropriate knowledge has been captured before moving the product to the next phase. The result is a product development process that provides critical measurements of knowledge, holds decision makers accountable, and delivers the expected results in a predictable manner. Attachment 1 to this statement provides a detailed list of activities that would provide program managers with the requisite technology, design, and manufacturing knowledge at key points in time during development. We have used these metrics to identify problems on major weapon system acquisition programs and have found a strong correlation between each of them and cost and schedule outcomes. For example, for 47 weapon programs in DOD's 2008 portfolio of major defense acquisitions, we assessed the knowledge attained at key decision points in the acquisition process and found the following:[Footnote 2] * Most programs have started system development without mature technologies. Only 4 of the 36 programs that provided data on technical maturity at development start did so with fully mature critical technologies. Further, only 14 of 39 programs that provided data have or plan to have demonstrated all of their technologies in a realistic environment prior to system-level critical design review, at which point the system's design should be stable. The 5 newer programs--those initiated since 2003[Footnote 3]--have higher levels of technology maturity, with all 5 programs demonstrating their technologies in a relevant environment prior to development start, in accordance with DOD and statutory criteria. However only 1of these programs met the best practice standard of demonstrating critical technologies in an operational environment. Last year, we determined that programs with immature technologies at the start of system development experienced 44 percent higher cost growth than programs that began with mature technologies. * Programs that have held design reviews in recent years reported higher levels of design knowledge. However, designs, on average, are still far from stable. For the 24 programs in our assessment that held a critical design review since 2003, the average percentage of total expected design drawings releasable at this review was 65 percent, compared to a best practice standard of 90 percent. We have found that programs moving forward into system demonstration with low levels of design stability are more likely than other programs to encounter costly design changes and parts shortages that, in turn, cause labor inefficiencies, schedule delays, and quality problems. Attachment 3 represents our notional depiction of the problems and outcomes that can typically be expected when these knowledge metrics are followed versus when they are not. Generally speaking, programs that move forward without retiring technology, design, and manufacturing risk at appropriate junctures will encounter a cascade of problems beginning with design changes and continuing with parts shortages, changes to manufacturing processes, labor inefficiencies, and quality problems. All of these problems delay programs and add to their development costs. We have found, for example, that a significant portion--about 70 percent--of the total development cost growth in programs typically occurs after the design review. Outcome Metrics Provide Insight into the Health and Performance of Individual Weapon System Programs and DOD's Total Portfolio: Program outcome metrics--quantitative measures of cost, schedule, and performance, and changes in these factors over time--provide useful indicators of the health of acquisition programs and facilitate analyses of how well programs are meeting cost, schedule, and performance goals. When assessed regularly for changes and the reasons that cause changes, such indicators can be valuable tools for improving insight and oversight of individual programs as well as DOD's total portfolio of major defense acquisitions. Over the years we have reported cost, schedule and performance data--good and bad--on numerous weapon systems. Our work continues to identify systemic and program- specific causes for cost, schedule, and performance problems and has led us to designate, since 1990, DOD's management of major weapon system acquisitions as a high risk area. To improve acquisition performance and address the factors that keep weapon acquisitions on the high risk list, DOD is working with us and the Office of Management and Budget to develop a comprehensive set of outcome metrics to provide better, comprehensive, and consistent measures of program cost and schedule performance. Last year, this cooperative effort resulted in agreement to track trends and changes in programs from their original baselines, from 5 years ago, and from the previous year, for the following data points: * Development cost; * Procurement cost; * Total program cost; * Quantities to be procured; * Procurement unit costs; * Total program unit costs; * Cycle time from Milestone B to Initial Operational Capability: DOD initiated a pilot study of 7 major defense programs to assess the adequacy of the proposed metrics and their value in analyzing performance, and the results proved promising. DOD approved the outcome metrics and intends to collect and report such data on an annual basis. Efforts to develop similar metrics on schedule performance continue. We believe that the metrics DOD plans to use are valuable for providing insight into the performance of weapon system programs. We have used similar metrics for many years in assessing programs. For example, we recently reported that ten of DOD's largest acquisition programs, commanding about half the overall acquisition dollars in the department's 2008 portfolio of major programs, have experienced significant cost growth and have seen quantities reduced by almost a third (see table 1). The two largest programs--the Joint Strike Fighter and the Future Combat System--represent significant cost risk moving forward and will dominate the portfolio for years. Since these programs consume such a large portion of the funding that DOD spends on research and development and procurement, their performance also affects other major weapon acquisitions, smaller acquisition programs, and DOD's ability to fund and acquire other supplies and equipment as well. Table 1: Changes in Costs and Quantities for Ten of the Highest Cost Acquisition Programs: Fiscal year 2009 dollars in millions: Program: Joint Strike Fighter; Total cost: First full estimate: $206,410; Total cost: Current estimate: $244,772; Total quantity: First full estimate: 2,866; Total quantity: Current estimate: 2,456; Acquisition unit cost: Percentage change: 38. Program: Future Combat System; Total cost: First full estimate: $89,776; Total cost: Current estimate: $129,731; Total quantity: First full estimate: 15; Total quantity: Current estimate: 15; Acquisition unit cost: Percentage change: 45. Program: Virginia Class Submarine; Total cost: First full estimate: $58,378; Total cost: Current estimate: $81,556; Total quantity: First full estimate: 30; Total quantity: Current estimate: 30; Acquisition unit cost: Percentage change: 40. Program: F-22A Raptor; Total cost: First full estimate: $88,134; Total cost: Current estimate: $73,723; Total quantity: First full estimate: 648; Total quantity: Current estimate: 184; Acquisition unit cost: Percentage change: 195. Program: C-17 Globemaster III; Total cost: First full estimate: $51,733; Total cost: Current estimate: $73,571; Total quantity: First full estimate: 210; Total quantity: Current estimate: 190; Acquisition unit cost: Percentage change: 57. Program: V-22 Joint Services Advanced Vertical Lift Aircraft; Total cost: First full estimate: $38,726; Total cost: Current estimate: $55,544; Total quantity: First full estimate: 913; Total quantity: Current estimate: 458; Acquisition unit cost: Percentage change: 186. Program: F/A-18E/F Super Hornet; Total cost: First full estimate: $78,925; Total cost: Current estimate: $51,787; Total quantity: First full estimate: 1,000; Total quantity: Current estimate: 493; Acquisition unit cost: Percentage change: 33. Program: Trident II Missile; Total cost: First full estimate: $49,939; Total cost: Current estimate: $49,614; Total quantity: First full estimate: 845; Total quantity: Current estimate: 561; Acquisition unit cost: Percentage change: 50. Program: CVN 21 Nuclear Aircraft Class Carrier; Total cost: First full estimate: $34,360; Total cost: Current estimate: $29,914; Total quantity: First full estimate: 3; Total quantity: Current estimate: 3; Acquisition unit cost: Percentage change: -13. Program: P-8A Poseidon Multi-mission Maritime Aircraft; Total cost: First full estimate: $29,974; Total cost: Current estimate: $29,622; Total quantity: First full estimate: 115; Total quantity: Current estimate: 113; Acquisition unit cost: Percentage change: 1. Source: GAO analysis of DOD data. [End of table] While program outcome metrics are good measures of individual program performance, the collective performance of DOD's portfolio of major defense acquisition programs is a key indicator of how well the department's acquisition system generates the return on investment it promises to the warfighter, Congress, and the taxpayer. Portfolio metrics also provide senior leaders and Congress with a snapshot of the cumulative impact of current investment decisions and poor program performance on future budgets. In our annual assessment of selected weapon programs, we analyzed the performance of DOD programs at the portfolio level by comparing programs' initial cost, schedule, and quantity estimates to their current estimates, based on data obtained from the Selected Acquisition Reports. This year's cumulative results, reported earlier this week, [Footnote 4] are shown in table 2. Table 2: Analysis of DOD Major Defense Acquisition Program Portfolios[A]: Fiscal year 2009 dollars. Portfolio size: Number of programs; Fiscal Year 2003: 77; Fiscal Year 2007: 95; Fiscal Year 2008: 96. Portfolio size: Total planned commitments; Fiscal Year 2003: $1.2 trillion; Fiscal Year 2007: $1.6 trillion; Fiscal Year 2008: $1.6 trillion. Portfolio size: Commitments outstanding; Fiscal Year 2003: $724.2 billion; Fiscal Year 2007: $875.2 billion; Fiscal Year 2008: $786.3 billion. Portfolio indicators: Change to total RDT&E costs from first estimate; Fiscal Year 2003: 37 percent; Fiscal Year 2007: 40 percent; Fiscal Year 2008: 42 percent. Portfolio indicators: Change to total acquisition cost from first estimate; Fiscal Year 2003: 19 percent; Fiscal Year 2007: 26 percent; Fiscal Year 2008: 25 percent. Portfolio indicators: Total acquisition cost growth; Fiscal Year 2003: $183 billion; Fiscal Year 2007: $301.3 billion[B]; Fiscal Year 2008: $296.4 billion. Portfolio indicators: Share of programs with 25 percent increase in program acquisition unit cost growth; Fiscal Year 2003: 41 percent; Fiscal Year 2007: 44 percent; Fiscal Year 2008: 42 percent. Portfolio indicators: Average schedule delay in delivering initial capabilities; Fiscal Year 2003: 18 months; Fiscal Year 2007: 21 months; Fiscal Year 2008: 22 months. Source: GAO analysis of DOD data. [A] Data were obtained from DOD's Selected Acquisition Reports (dated December 2002, 2006, and 2007). In a few cases data were obtained directly from program offices. The number of programs reflects the programs with Selected Acquisition Reports; however, in our analysis we have broken a few Selected Acquisition Reports programs into smaller elements or programs. Not all programs had comparable cost and schedule data and these programs were excluded from the analysis where appropriate. Portfolio performance data do not include costs of developing Missile Defense Agency elements or the DIMHRS program. [B] The acquisition cost growth for the 2007 portfolio was $295 billion in 2008 constant dollars. [End of table] Analyzing the data and comparing metrics from different time periods provides unique insights into the relative health of the portfolio and trends. From 2003 to 2008, the number of programs in DOD's major defense acquisition portfolio has grown from 77 to 96. Total costs for these programs now total $1.6 trillion with almost one-half of this amount still to be spent. Outcome metrics for 2008 show worsening performance in all categories compared to the 2003 portfolio and mixed performance--some better, some worse--compared to the 2007 data. While DOD is committing substantially more investment dollars to developing and procuring new weapon systems, the total acquisition costs for the 2008 portfolio has grown by $296 billion over initial estimates and the average schedule delay in delivering capabilities to the warfighter averages 22 months. Implications for the future are obvious. Continued cost growth reduces DOD's buying power and results in less funding being available for other DOD priorities and programs. As program costs increase, DOD must request more funding to cover overruns, make trade- offs with existing programs, delay the start of new programs, take funds from other accounts, or reduce procurement quantities. Continued failure to deliver weapon systems on time delays providing critical capabilities to the warfighter and results in operating costly legacy systems longer than expected, finding alternatives to fill capability gaps, or going completely without the capability. Key Prerequisites Need to Be Met to Improve the Value of Metrics and Achieve Better Acquisition Program Outcomes: While the metrics discussed above can provide valuable knowledge about potential problems and additional information on the health of DOD's acquisition programs, metrics alone may not be sufficient if the department does not do a better job ensuring that acquisitions begin with realistic plans and baseline estimates for cost and schedules prior to development start. We believe there is a clear set of prerequisites that must be a part of any acquisition strategy before any measurement of the acquisition's health can be valuable. Otherwise, metrics measured against unrealistic estimates will do no good. These key prerequisites for obtaining realistic baselines include: * Establishing a clear, knowledge-based, evolutionary business case for the product. This business case must: validate that a need exists; determine that resources are available to develop a product that will meet the need; determine that the product developer has a knowledge- based plan and strategy to deliver the product; establish reasonable estimates for cost, delivery time and quantities; and ensure available funding for the product. All of these elements of the business case should also be agreed upon by major stakeholders across the requirements, funding, acquisition, and warfighting communities. * Separating technology development activities from product development activities. The process of developing technology culminates in discovery--the gathering of knowledge--and must, by its nature, allow room for unexpected results and delays. Leading firms do not ask their product managers to develop technology because they have learned the hard way that invention cannot be scheduled. When immature technologies are brought onto the critical path of product development programs too early, they often cause long delays in an environment where large workforces must be employed, complex tools, plants, and facilities must be operated, long and expensive supplier networks must be paid, and the product itself must sometimes be redesigned once the final form of the technologies is known. Successful programs give responsibility for maturing technologies to science and technology organizations, rather than the program or product development managers, because the science and technology environment is less costly. We have recommended in the past that DOD's risks should be taken in the science and technology arena and that more funding should be made available to this process to do so. * Limiting time and requirements for product development to manageable levels. Product developers should strive to deliver the best available capabilities within realistic timeframes and should expect to continue to develop new capabilities when they are technologically feasible. By limiting product development cycle times to 6 years or less, DOD could assimilate new technologies into weapon systems more frequently, accelerate delivery of new technology to the warfighter, hold program managers accountable, and make more frequent and predictable work in production, where contractors and the industrial base can profit by being efficient. Too many major acquisitions currently take the opposite approach by seeking to deliver a revolutionary "big bang" capability in one step. This means that programs are more risky, delivery takes as long as 15 years in some cases, and costs grow at exponential rates from the original baseline due to the risky nature of the acquisition strategy. We point to the private sector and some past defense acquisitions, such as the F-16 program, as models for this practice. * Employing early systems engineering discipline in order to develop realistic cost and schedule estimates prior to development start. Early systems engineering provides the knowledge a product developer needs to identify and resolve performance and resource gaps before product development begins either by reducing requirements, deferring them to the future, or increasing the estimated cost for the weapon system's development. Requirements that are too risky given the state of technology and design should not be allowed into this expensive environment. * Making a commitment to fully fund programs once they are approved. This would require the department to ensure that it does not have too many programs underway given the amount of available resources. * Setting priorities from the top to ensure that candidate programs are truly needed and have a solid plan for delivery. DOD will continue to experience poor acquisition outcomes until it begins making choices that reflect the most important needs of the joint warfighter and match requirements with available resources. The urge to accept all candidate programs and to go for the "big bang" capability without the knowledge to achieve it should be resisted. Only the best candidates--defined in terms of priorities, resource availability, and executability--should be approved. There is no doubt that the current state of the department's acquisition process is too expensive for the taxpayer and not timely enough for the warfighter. The following illustration reinforces this point. Figure 2: Cost Remaining Versus Annual Appropriations for Major Defense Acquisitions: [Refer to PDF for image: multiple line graph] Fiscal year: 1992; Annual RDTE and Procurement Appropriations: $100 billion; Costs Remaining for Major Defense Acquisitions: $324 billion. Fiscal year: 1993; Annual RDTE and Procurement Appropriations: $91 billion; Costs Remaining for Major Defense Acquisitions: $271 billion. Fiscal year: 1994; Annual RDTE and Procurement Appropriations: $79 billion; Costs Remaining for Major Defense Acquisitions: $253 billion. Fiscal year: 1995; Annual RDTE and Procurement Appropriations: $78 billion; Costs Remaining for Major Defense Acquisitions: $341 billion. Fiscal year: 1996; Annual RDTE and Procurement Appropriations: $78 billion; Costs Remaining for Major Defense Acquisitions: $345 billion. Fiscal year: 1997; Annual RDTE and Procurement Appropriations: $79 billion; Costs Remaining for Major Defense Acquisitions: $331 billion. Fiscal year: 1998; Annual RDTE and Procurement Appropriations: $82 billion; Costs Remaining for Major Defense Acquisitions: $280 billion. Fiscal year: 1999; Annual RDTE and Procurement Appropriations: $89 billion; Costs Remaining for Major Defense Acquisitions: $233 billion. Fiscal year: 2000; Annual RDTE and Procurement Appropriations: $94 billion; Costs Remaining for Major Defense Acquisitions: $309 billion. Fiscal year: 2001; Annual RDTE and Procurement Appropriations: $104 billion; Costs Remaining for Major Defense Acquisitions: $234 billion. Fiscal year: 2002; Annual RDTE and Procurement Appropriations: $111 billion; Costs Remaining for Major Defense Acquisitions: $586 billion. Fiscal year: 2003; Annual RDTE and Procurement Appropriations: $137 billion; Costs Remaining for Major Defense Acquisitions: $579 billion. Fiscal year: 2004; Annual RDTE and Procurement Appropriations: $148 billion; Costs Remaining for Major Defense Acquisitions: $722 billion. Fiscal year: 2005; Annual RDTE and Procurement Appropriations: $165 billion; Costs Remaining for Major Defense Acquisitions: $745 billion. Fiscal year: 2006; Annual RDTE and Procurement Appropriations: $157 billion; Costs Remaining for Major Defense Acquisitions: $841 billion. Fiscal year: 2007; Annual RDTE and Procurement Appropriations: $160 billion; Costs Remaining for Major Defense Acquisitions: $767 billion. Source: DOD (data); GAO (analysis and presentation). [End of figure] This figure depicts an investment strategy for major weapon systems that continues to increase the costs to develop our existing weapons well into the future while the funding available to retire those costs appears capped at a very low level. While costs continue to rise as the result of more and more risky programs being added to the portfolio, our ability to allocate funds for these costs appears to be, at best, capped at very low percentages of the total cost. We could measure the risk of these acquisitions much better than we have in the past if we set the appropriate prerequisites for their initiation, measure the knowledge that must be in place at various points, and continue to monitor their health in terms of cost and schedule. Concluding Remarks: Measuring the performance of weapon system programs both individually and collectively is critical for determining whether the warfighter and the taxpayer are receiving the promised return on investment. No single metric, however, can capture the whole picture of how well programs are performing. It is important to look at knowledge and outcome metrics. Knowledge metrics provide key information for determining whether programs have the requisite knowledge to move from one phase of development to the next and are at risk of cost and schedule overruns. Outcomes metrics are also needed to provide temperature checks on the health and status of individual programs and the portfolio of programs as a whole. These metrics are vital for informing program decision making and helping to manage programs. Metrics by themselves do not solve problematic acquisitions. Ultimately, DOD still needs to do a better job planning and executing programs to achieve better outcomes. Critical to achieving successful outcomes is establishing knowledge-based, realistic program baselines. Without realistic baselines, there is no foundation for accurately measuring the knowledge and health of programs. Over the past several years, our work has highlighted a number of underlying causes for why DOD does not effectively manage the acquisition of weapon system programs. DOD recently revised its acquisition policy to provide a better foundation for developing weapon systems, however, reform will not be achieved without fundamental changes to the overall acquisition culture and environment that exists in DOD. I would be pleased to discuss these causes and issues with the Committee at a future time. Mr. Chairman, this concludes my prepared statement. I would be happy to answer any questions you may have at this time. Contacts and Acknowledgements: For further information about this statement, please contact Michael J. Sullivan (202) 512-4841 or sullivanm@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals who made key contributions to this statement include Cheryl Andrew, Ridge Bowman, Bruce Fairbairn, Susan Neill, John Oppenheim, and Ron Schwenn. [End of section] Attachment 1: Knowledge-Based Activities: Knowledge Point 1: Start of product development activities: Best practice metric: Technology readiness level 7 (indicating technologies work in an operational environment): * Demonstrate technologies to high readiness levels. * Ensure that requirements for the product increment are informed by preliminary design using systems engineering process (such as prototyping of preliminary design). * Establish cost and schedule estimates for product on the basis of knowledge from preliminary design using system engineering tools (such as prototyping of preliminary design). * Constrain development phase (5 to 6 years or less) for incremental development. * Ensure development phase fully funded (programmed in anticipation of milestone). * Align program manager tenure to complete development phase. * Contract strategy that separates system integration and system demonstration activities. * Conduct independent cost estimate. * Conduct independent program assessment. * Conduct major milestone decision review for development start. Knowledge Point 2: Transition from system integration to system demonstration activities; Best practice metric: 90 percent of design drawings are complete by the critical design review: * Complete system critical design review. * Complete 90 percent of engineering design drawing packages. * Complete subsystem and system design reviews. * Demonstrate with system integration prototype that design meets requirements. * Complete the failure modes and effects analysis. * Identify key system characteristics. * Identify critical manufacturing processes. * Establish reliability targets and growth plan on the basis of demonstrated reliability rates of components and subsystems. * Conduct independent cost estimate. * Conduct independent program assessment. * Conduct major milestone decision review to enter system demonstration. Knowledge Point 3: Initiation of producing a product to be delivered to customer; Best practice metric: 100 percent of critical manufacturing processes are in control: * Demonstrate manufacturing processes. * Build and test production-representative prototypes to demonstrate product in operational environment. * Test production-representative prototypes to achieve reliability goal. * Collect statistical process control data. * Demonstrate that critical processes are capable and in statistical control. * Independent cost estimate. * Independent program assessment. * Conduct major milestone decision review to begin production. Source: GAO analysis of commercial best practices. [End of table] [End of section] Attachment 2: Technology Readiness Levels: Technology readiness level: 1. Basic principles observed and reported; Description: Lowest level of technology readiness. Scientific research begins to be translated into applied research and development. Examples might include paper studies of a technology's basic properties; Hardware: Software: None (paper studies and analysis); Demonstration Environment: None. Technology readiness level: 2. Technology concept and/or application formulated; Description: Invention begins. Once basic principles are observed, practical applications can be invented. The application is speculative and there is no proof or detailed analysis to support the assumption. Examples are still limited to paper studies; Hardware: Software: None (paper studies and analysis); Demonstration Environment: None. Technology readiness level: 3. Analytical and experimental critical function and/or characteristic proof of concept; Description: Active research and development is initiated. This includes analytical studies and laboratory studies to physically validate analytical predictions of separate elements of the technology. Examples include components that are not yet integrated or representative; Hardware: Software: Analytical studies and demonstration of nonscale individual components (pieces of subsystem); Demonstration Environment: Lab. Technology readiness level: 4. Component and/or breadboard. Validation in laboratory environment; Description: Basic technological components are integrated to establish that the pieces will work together. This is relatively "low fidelity" compared to the eventual system. Examples include integration of "ad hoc" hardware in a laboratory; Hardware: Software: Low fidelity breadboard; Integration of nonscale components to show pieces will work together. Not fully functional or form or fit but representative of technically feasible approach suitable for flight articles; Demonstration Environment: Lab. Technology readiness level: 5. Component and/or breadboard validation in relevant environment; Description: Fidelity of breadboard technology increases significantly. The basic technological components are integrated with reasonably realistic supporting elements so that the technology can be tested in a simulated environment. Examples include "high fidelity" laboratory integration of components; Hardware: Software: High fidelity breadboard; Functionally equivalent but not necessarily form and/or fit (size weight, materials, etc). Should be approaching appropriate scale. May include integration of several components with reasonably realistic support elements/ subsystems to demonstrate functionality; Demonstration Environment: Lab demonstrating functionality but not form and fit. May include flight demonstrating breadboard in surrogate aircraft. Technology ready for detailed design studies. Technology readiness level: 6. System/subsystem model or prototype demonstration in a relevant environment; Description: Representative model or prototype system, which is well beyond the breadboard tested for TRL 5, is tested in a relevant environment. Represents a major step up in a technology's demonstrated readiness. Examples include testing a prototype in a high fidelity laboratory environment or in simulated realistic environment; Hardware: Software: Prototype; Should be very close to form, fit and function. Probably includes the integration of many new components and realistic supporting elements/subsystems if needed to demonstrate full functionality of the subsystem; Demonstration Environment: High-fidelity lab demonstration or limited/ restricted flight demonstration for a relevant environment. Integration of technology is well defined. Technology readiness level: 7. System prototype demonstration in a realistic environment; Description: Prototype near or at planned operational system. Represents a major step up from TRL 6, requiring the demonstration of an actual system prototype in a realistic environment, such as in an aircraft, vehicle or space. Examples include testing the prototype in a test bed aircraft; Hardware: Software: Prototype. Should be form, fit and function integrated with other key supporting elements/subsystems to demonstrate full functionality of subsystem; Demonstration Environment: Flight demonstration in representative realistic environment such as flying test bed or demonstrator aircraft; Technology is well substantiated with test data. Technology readiness level: 8. Actual system completed and "flight qualified" through test and demonstration; Description: Technology has been proven to work in its final form and under expected conditions. In almost all cases, this TRL represents the end of true system development. Examples include developmental test and evaluation of the system in its intended weapon system to determine if it meets design specifications; Hardware: Software: Flight qualified hardware; Demonstration Environment: Developmental Test and Evaluation (DT&E) in the actual system application. Technology readiness level: 9. Actual system "flight proven" through successful mission operations; Description: Actual application of the technology in its final form and under mission conditions, such as those encountered in operational test and evaluation. In almost all cases, this is the end of the last "bug fixing" aspects of true system development. Examples include using the system under operational mission conditions; Hardware: Software: Actual system in final form; Demonstration Environment: Operational Test and Evaluation (OT&E) in operational mission conditions. Source: GAO and its analysis of National Aeronautics and Space Administration data. [End of table] [End of section] Attachment 3: Notional Illustration Showing the Different Paths That a Product's Development Can Take: [Refer to PDF for image: illustration] This illustration is a graph plotting schedule/time against cost/investment. Two paths are shown: 1) Mature product and; 2) Continual changes needed to reach maturity. Mature product: takes less time and has a lower cost due to: * stable design; * Demonstrated; * Built and tested; * Manufacturing processes in control. Continual changes needed to reach maturity: takes more time and has a greater cost due to: * Unstable design; * Late drawings and design changes; * Tooling and design changes (Fully integrated aircraft not built or tested); * Production manufacturing process not in control; * Labor inefficiencies and quality issues. Source: GAO. [End of figure] [End of section] Related GAO Products: Defense Acquisitions: Assessment of Major Weapon Programs. [hyperlink, http://www.gao.gov/products/GAO-09-326SP]. Washington, D.C.: March 30, 2009. Defense Acquisitions: DOD Must Prioritize Its Weapon System Acquisitions and Balance Them with Available Resources. [hyperlink, http://www.gao.gov/products/GAO-09-501T]. Washington, D.C.: March 18, 2009. Defense Acquisitions: DOD Must Balance Its Needs with Available Resources and Follow an Incremental Approach to Acquiring Weapon Systems. [hyperlink, http://www.gao.gov/products/GAO-09-431T]. Washington, D.C.: March 3, 2009. Defense Acquisitions: Perspectives on Potential Changes to DOD's Acquisition Management Framework. [hyperlink, http://www.gao.gov/products/GAO-09-295R]. Washington, D.C.: February 27, 2009. Defense Management: Actions Needed to Overcome Long-standing Challenges with Weapon Systems Acquisition and Service Contract Management. [hyperlink, http://www.gao.gov/products/GAO-09-362T]. Washington, D.C.: February 11, 2009. Defense Acquisitions: Fundamental Changes Are Needed to Improve Weapon Program Outcomes. [hyperlink, http://www.gao.gov/products/GAO-08-1159T]. Washington, D.C.: September 25, 2008. Defense Acquisitions: DOD's Requirements Determination Process Has Not Been Effective in Prioritizing Joint Capabilities. [hyperlink, http://www.gao.gov/products/GAO-08-1060]. Washington, D.C.: September 25, 2008. Defense Acquisitions: A Knowledge-Based Funding Approach Could Improve Major Weapon System Program Outcomes. [hyperlink, http://www.gao.gov/products/GAO-08-619]. Washington, D.C.: July 2, 2008. Defense Acquisitions: Better Weapon Program Outcomes Require Discipline, Accountability, and Fundamental Changes in the Acquisition Environment. [hyperlink, http://www.gao.gov/products/GAO-08-782T]. Washington, D.C.: June 3, 2008. Defense Acquisitions: Assessments of Selected Weapon Programs. [hyperlink, http://www.gao.gov/products/GAO-08-467SP]. Washington, D.C.: March 31, 2008. Best Practices: Increased Focus on Requirements and Oversight Needed to Improve DOD's Acquisition Environment and Weapon System Quality. [hyperlink, http://www.gao.gov/products/GAO-08-294]. Washington, D.C.: February 1, 2008. Cost Assessment Guide: Best Practices for Estimating and Managing Program Costs. [hyperlink, http://www.gao.gov/products/GAO-07-1134SP]. Washington, D.C.: July 2007. Defense Acquisitions: Assessments of Selected Weapon Programs. [hyperlink, http://www.gao.gov/products/GAO-07-406SP.Washington, D.C.: March 30, 2007. Best Practices: An Integrated Portfolio Management Approach to Weapon System Investments Could Improve DOD's Acquisition Outcomes. [hyperlink, http://www.gao.gov/products/GAO-07-388]. Washington, D.C.: March 30, 2007. Defense Acquisitions: Major Weapon Systems Continue to Experience Cost and Schedule Problems under DOD's Revised Policy. [hyperlink, http://www.gao.gov/products/GAO-06-368]. Washington, D.C.: April 13, 2006. DOD Acquisition Outcomes: A Case for Change. [hyperlink, http://www.gao.gov/products/GAO-06-257T]. Washington, D.C.: November 15, 2005. Best Practices: Better Support of Weapon System Program Managers Needed to Improve Outcomes. [hyperlink, http://www.gao.gov/products/GAO-06-110]. Washington, D.C.: November 1, 2005. Defense Acquisitions: Stronger Management Practices Are Needed to Improve DOD's Software-Intensive Weapon Acquisitions. [hyperlink, http://www.gao.gov/products/GAO-04-393]. Washington, D.C.: March 1, 2004. Defense Acquisitions: DOD's Revised Policy Emphasizes Best Practices, but More Controls Are Needed. [hyperlink, http://www.gao.gov/products/GAO-04-53]. Washington, D.C.: November 10, 2003. Best Practices: Setting Requirements Differently Could Reduce Weapon Systems' Total Ownership Costs. [hyperlink, http://www.gao.gov/products/GAO-03-57. Washington, D.C.: February 11, 2003. Defense Acquisitions: Factors Affecting Outcomes of Advanced Concept Technology Demonstration. [hyperlink, http://www.gao.gov/products/GAO-03-52]. Washington, D.C.: December 2, 2002. Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes. [hyperlink, http://www.gao.gov/products/GAO-02-701]. Washington, D.C.: July 15, 2002. Defense Acquisitions: DOD Faces Challenges in Implementing Best Practices. [hyperlink, http://www.gao.gov/products/GAO-02-469T]. Washington, D.C.: February 27, 2002. [End of section] Footnotes: [1] Technology readiness levels, originally developed by the National Aeronautics and Space Administration, are measured on a scale of 1 to 9, beginning with paper studies of a technology's feasibility and culminating with a technology fully integrated into a completed product. See Attachment 2 for the definitions of technology readiness levels. [2] We did this by collecting data directly from program offices using a questionnaire. These programs are primarily in development and, therefore, most relevant to current decisions about which programs should receive substantial investments of research and development funding now and large amounts of procurement funding in the future. Defense Acquisitions: Assessment of Selected Weapons Programs. [hyperlink, http://www.gao.gov/products/GAO-09-326SP]. Washington, D.C.: March 30, 2009. [3] In 2003, DOD revised its primary acquisition policy to state that technologies should be demonstrated in a relevant environment prior to starting an acquisition program. In 2006, this standard became a statutory requirement for all major defense acquisition programs in the National Defense Authorization Act for Fiscal Year 2006, Pub. L. No. 109-163, § 801, codified at 10 U.S.C. § 2366b. [4] GAO, Defense Acquisitions: Assessments of Selected Weapon Programs, [hyperlink, http://www.gao.gov/products/GAO-09-326SP] (Washington, D.C.: Mar. 30, 2009). [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO‘s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO‘s Web site, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548:

The Justia Government Accountability Office site republishes public reports retrieved from the U.S. GAO These reports should not be considered official, and do not necessarily reflect the views of Justia.