Defense Acquisitions
Measuring the Value of DOD's Weapon Programs Requires Starting with Realistic Baselines Gao ID: GAO-09-543T April 1, 2009n 2008, the cumulative cost growth in the Department of Defense's portfolio of 96 major defense acquisition programs was $296 billion and the average delay in delivering promised capabilities to the warfighter was 22 months. These poor outcomes mean that other critical defense and national priorities may go unfunded and that warfighters may go without the equipment they need to counter the changing threats that they face. GAO has examined extensively the issues in DOD's weapon system programs from the perspective of best practices in product development, and believes that significant improvements are possible. Because the ability to measure knowledge, processes, and outcomes is critical to achieving these improvements, GAO has used metrics to review the management and health of these programs from within the framework of best practices. This testimony discusses: 1) "knowledge metrics," used to determine how well programs manage technology, design, and manufacturing risks; 2) outcome metrics--concerning cost, schedule, and capability--that serve as "health indicators" of how well programs are being executed in terms of predicted outcomes; and 3) the prerequisites that GAO believes must be met in order for a program's plans and goals to be realistic.
GAO employs a set of knowledge metrics to determine whether programs have attained the right knowledge at critical points over the course of a weapon system acquisition, and facilitate the identification of potential problems that could lead to cost, schedule, or performance shortfalls. In essence, knowledge supplants risk over time. Key knowledge points and metrics include 1) achieving a high level of technology maturity at the start of program development, 2) reaching design stability at the system-level critical design review, and 3) demonstrating that critical manufacturing processes are in control before starting production. By applying these metrics to selected programs in DOD's 2008 portfolio of major defense acquisitions, GAO found that most programs have started system development without mature technologies and moved into system demonstration with low levels of design stability. GAO has determined that programs with immature technologies and unstable designs have experienced significant cost and schedule growth. Program outcome metrics--quantitative measures of cost, schedule, and performance over time--provide useful indicators of the health of acquisition programs and whether they are meeting their intended goals. When assessed regularly for changes and the reasons that cause changes, these indicators can be valuable tools for improving insight into and oversight of individual programs as well as DOD's total portfolio of major defense acquisitions. The collective performance of the programs in DOD's portfolio is a key indicator of how well the acquisition system generates the return on investment that it promises to the warfighter, Congress and taxpayers. GAO recently reported that outcome metrics for DOD's 2008 major defense acquisition portfolio show worsening performance when compared to the department's 2003 portfolio. For example, total acquisition costs for programs in the 2008 portfolio increased 25 percent from first estimates compared to a 19-percent increase for programs in the 2003 portfolio. DOD is working with GAO and the Office of Management and Budget to develop a comprehensive set of outcome metrics to better assess its portfolio of programs. While knowledge and outcome metrics provide valuable information about the potential problems and health of programs, they are of limited value if DOD does not do a better job ensuring acquisitions begin with realistic plans and baselines prior to development start. GAO believes there is a clear set of prerequisites that must be met by each program's acquisition strategy before a measurement of the program's health will be of real value. These prerequisites include: 1) establishing an evolutionary, knowledge-based business case for each acquisition; 2) separating technology development from product development; 3) limiting time and requirements for product development to manageable levels; 4) employing systems engineering early on in the process to arrive at realistic cost and schedule estimates; 5) committing to fully funding a program once it is approved; and 6) setting priorities from the top to ensure that candidate programs are truly needed and have a solid plan for delivery.
GAO-09-543T, Defense Acquisitions: Measuring the Value of DOD's Weapon Programs Requires Starting with Realistic Baselines
This is the accessible text file for GAO report number GAO-09-543T
entitled 'Defense Acquisitions: Measuring the Value of DOD's Weapon
Programs Requires Starting with Realistic Baselines' which was released
on April 1, 2009.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Testimony:
Before the Panel on Defense Acquisition Reform, Committee on Armed
Services, House of Representatives:
United States Government Accountability Office:
GAO:
For Release on Delivery:
Expected at 7:30 a.m. EDT:
Wednesday April 1, 2009:
Defense Acquisitions:
Measuring the Value of DOD's Weapon Programs Requires Starting with
Realistic Baselines:
Statement of Michael J. Sullivan:
Director Acquisition and Sourcing Management:
GAO-09-543T:
GAO Highlights:
Highlights of GAO-09-543T, a testimony before the House Committee on
Armed Services Panel on Defense Acquisition Reform.
Why GAO Did This Study:
In 2008, the cumulative cost growth in the Department of Defense‘s
portfolio of 96 major defense acquisition programs was $296 billion and
the average delay in delivering promised capabilities to the warfighter
was 22 months. These poor outcomes mean that other critical defense and
national priorities may go unfunded and that warfighters may go without
the equipment they need to counter the changing threats that they face.
GAO has examined extensively the issues in DOD‘s weapon system programs
from the perspective of best practices in product development, and
believes that significant improvements are possible. Because the
ability to measure knowledge, processes, and outcomes is critical to
achieving these improvements, GAO has used metrics to review the
management and health of these programs from within the framework of
best practices.
This testimony discusses: 1) ’knowledge metrics,“ used to determine how
well programs manage technology, design, and manufacturing risks; 2)
outcome metrics--concerning cost, schedule, and capability--that serve
as ’health indicators“ of how well programs are being executed in terms
of predicted outcomes; and 3) the prerequisites that GAO believes must
be met in order for a program‘s plans and goals to be realistic.
What GAO Found:
GAO employs a set of knowledge metrics to determine whether programs
have attained the right knowledge at critical points over the course of
a weapon system acquisition, and facilitate the identification of
potential problems that could lead to cost, schedule, or performance
shortfalls. In essence, knowledge supplants risk over time. Key
knowledge points and metrics include 1) achieving a high level of
technology maturity at the start of program development, 2) reaching
design stability at the system-level critical design review, and 3)
demonstrating that critical manufacturing processes are in control
before starting production. By applying these metrics to selected
programs in DOD‘s 2008 portfolio of major defense acquisitions, GAO
found that most programs have started system development without mature
technologies and moved into system demonstration with low levels of
design stability. GAO has determined that programs with immature
technologies and unstable designs have experienced significant cost and
schedule growth.
Program outcome metrics”quantitative measures of cost, schedule, and
performance over time”provide useful indicators of the health of
acquisition programs and whether they are meeting their intended goals.
When assessed regularly for changes and the reasons that cause changes,
these indicators can be valuable tools for improving insight into and
oversight of individual programs as well as DOD‘s total portfolio of
major defense acquisitions. The collective performance of the programs
in DOD‘s portfolio is a key indicator of how well the acquisition
system generates the return on investment that it promises to the
warfighter, Congress and taxpayers. GAO recently reported that outcome
metrics for DOD‘s 2008 major defense acquisition portfolio show
worsening performance when compared to the department‘s 2003 portfolio.
For example, total acquisition costs for programs in the 2008 portfolio
increased 25 percent from first estimates compared to a 19-percent
increase for programs in the 2003 portfolio. DOD is working with GAO
and the Office of Management and Budget to develop a comprehensive set
of outcome metrics to better assess its portfolio of programs.
While knowledge and outcome metrics provide valuable information about
the potential problems and health of programs, they are of limited
value if DOD does not do a better job ensuring acquisitions begin with
realistic plans and baselines prior to development start. GAO believes
there is a clear set of prerequisites that must be met by each
program‘s acquisition strategy before a measurement of the program‘s
health will be of real value. These prerequisites include: 1)
establishing an evolutionary, knowledge-based business case for each
acquisition; 2) separating technology development from product
development; 3) limiting time and requirements for product development
to manageable levels; 4) employing systems engineering early on in the
process to arrive at realistic cost and schedule estimates; 5)
committing to fully funding a program once it is approved; and 6)
setting priorities from the top to ensure that candidate programs are
truly needed and have a solid plan for delivery.
View [hyperlink, http://www.gao.gov/products/GAO-09-543T] or key
components. For more information, contact Michael J. Sullivan at (202)
512-4841 or sullivanm@gao.gov.
[End of section]
Mr. Chairman and Members of the Committee:
I am pleased to be here this morning to discuss how best to measure and
determine whether DOD's acquisition system is providing value to the
warfighter. Earlier this week, we reported that the cumulative cost
growth in DOD's portfolio of 96 major defense acquisition programs was
$296 billion and the average delay in delivering promised capabilities
to the warfighter was 22 months. These outcomes mean that other
critical defense and national priorities go unfunded and warfighters go
without the equipment they need to counter ever changing threats that
they face. This condition is unacceptable. We believe that significant
improvement in the acquisition of weapon systems is possible and that
the ability to measure knowledge, processes, and outcomes is critical
to achieving that improvement. It is important to note that not one
single metric or set of metrics is enough to monitor acquisitions and
gain efficiencies. Today, we would like to break our discussion about
how to measure the department's acquisitions into 3 basic sections:
* First, we would like to present a set of metrics that we refer to as
"knowledge metrics" and use to determine how well acquisition programs
are managing and retiring predictable technology, design, and
manufacturing risks and gaining knowledge. These metrics are valuable
because they can predict problems and identify causes.
* Second, we would like to discuss a set of outcome measures--
concerning cost, schedule, and capability--that serve as health
indicators. These indicators measure how well programs are being
executed and achieving predicted outcomes in terms of meeting original
baselines for cost, schedule, and performance. These metrics have
intrinsic value as simple measurements, just as a thermometer can warn
a parent that a child has a fever.
* Third, there are certain indicators that we look for--based on the
work we have done examining best practices for product development--
that are, perhaps, more important than these knowledge and health
metrics because they determine from the outset how realistic the
acquisition plans and strategies of programs are. For the sake of
today's discussion, we will refer to them as "prerequisite indicators."
These prerequisites are most important because we question the value of
ANY metric when measuring from an unrealistic baseline.
We know that the knowledge and program health metrics we use to measure
programs' progress and outcomes are valuable when used in realistic,
market-driven product development environments. We also know that ALL
of these metrics are important indicators for decision makers. Our
extensive body of work examining world-class enterprises and the way
they operate has validated their value for programs that must deliver a
new product to market at a certain time and within a certain investment
cost or suffer significant consequences. These metrics work because
they are measuring realistic plans and goals that are supported by
doable requirements, accurate cost and schedule estimates, and stable
funding. The company developing the products suffers dire consequences,
such as loss of market share, if these programs do not succeed.
This statement draws from our extensive body of work on DOD's
acquisition of weapon systems. A list of our key products is provided
at the end of this statement. This work was conducted in accordance
with generally accepted government auditing standards. Those standards
require that we plan and perform the audit to obtain sufficient,
appropriate evidence to provide a reasonable basis for our findings and
conclusions based on our audit objectives. We believe that the evidence
obtained provides a reasonable basis for our findings and conclusions
based on our audit objectives.
"Knowledge" Metrics Identify Potential Problems and Their Likely
Causes:
We have conducted a body of work that examines weapon acquisition
issues from a perspective that draws upon lessons learned from best
practices in product development. Collectively, these practices
comprise a process that is anchored in knowledge. Achieving the right
knowledge at the right time enables leadership to make informed
decisions about when and how best to move into various expensive
acquisition phases. In essence, knowledge supplants risk over time.
This building of knowledge consists of gathering information about
technology, design, and manufacturing at three critical points over the
course of a weapon system program (Figure 1). We have developed
valuable "knowledge metrics" that measure this knowledge build and
allow us to identify potential problems that could lead to cost,
schedule, or performance shortfalls and their likely causes. The
metrics can be described as:
* Knowledge Point 1, evidenced by the balance between a product's
required capabilities and the resources available to meet them. Focus
should be on understanding technological and design implications and
achieving a high level of technology maturity at the start of system
development. This means that the critical technologies needed to meet
essential product requirements must be demonstrated to work in their
intended environment. The technology readiness level for each critical
technology is the metric we use to measure technology maturity.
[Footnote 1]
* Knowledge point 2, evidenced by the development of engineering
prototypes and the completion of engineering drawings for an integrated
product at the system design review. This metric provides tangible
evidence that the product's design is stable, meaning it has a high
probability of meeting customer requirements, as well as cost,
schedule, and reliability targets. A best practice is to achieve design
stability at the system-level critical design review, usually held
midway through development. Completion of at least 90 percent of
engineering drawings is the metric we use to measure design stability.
* Knowledge point 3, evidenced by the demonstration that critical
manufacturing processes are in control--that is, they are repeatable,
sustainable, and capable of consistently producing parts within the
product's quality tolerances and standards--at the start of production.
One hundred percent of critical manufacturing processes in control is
the metric we use to evaluate manufacturing maturity.
Figure 1: Knowledge-Based Acquisition Process:
[Refer to PDF for image: illustration]
Material development start:
Technology Development (A);
PDR: Knowledge point 1: Metric: technology readiness levels at 7 or
higher (B);
Product Development:
- Integration;
- CDR: Knowledge point 2: Metric: 90 percent of engineering drawings
complete;
- Demonstration;
Production: (C): Knowledge point 3: Metric: 100 percent of critical
manufacturing processes in control.
Source: GAO analysis of commercial best practices.
[End of figure]
Each of these metrics gauges the point when the requisite level of
knowledge has been attained for a product in relation to where that
product is in its development. World-class firms we have visited work
hard to establish metrics such as these and their decision makers are
required to consider them before deciding to advance a program to the
next level. Theses types of metrics also help decision makers gauge
progress in meeting cost, schedule, and performance goals and ensure
that managers will (1) conduct activities to capture relevant product
development knowledge, (2) provide evidence that this knowledge has
been captured, and (3) hold decision reviews to determine that
appropriate knowledge has been captured before moving the product to
the next phase. The result is a product development process that
provides critical measurements of knowledge, holds decision makers
accountable, and delivers the expected results in a predictable manner.
Attachment 1 to this statement provides a detailed list of activities
that would provide program managers with the requisite technology,
design, and manufacturing knowledge at key points in time during
development.
We have used these metrics to identify problems on major weapon system
acquisition programs and have found a strong correlation between each
of them and cost and schedule outcomes. For example, for 47 weapon
programs in DOD's 2008 portfolio of major defense acquisitions, we
assessed the knowledge attained at key decision points in the
acquisition process and found the following:[Footnote 2]
* Most programs have started system development without mature
technologies. Only 4 of the 36 programs that provided data on technical
maturity at development start did so with fully mature critical
technologies. Further, only 14 of 39 programs that provided data have
or plan to have demonstrated all of their technologies in a realistic
environment prior to system-level critical design review, at which
point the system's design should be stable. The 5 newer programs--those
initiated since 2003[Footnote 3]--have higher levels of technology
maturity, with all 5 programs demonstrating their technologies in a
relevant environment prior to development start, in accordance with DOD
and statutory criteria. However only 1of these programs met the best
practice standard of demonstrating critical technologies in an
operational environment. Last year, we determined that programs with
immature technologies at the start of system development experienced 44
percent higher cost growth than programs that began with mature
technologies.
* Programs that have held design reviews in recent years reported
higher levels of design knowledge. However, designs, on average, are
still far from stable. For the 24 programs in our assessment that held
a critical design review since 2003, the average percentage of total
expected design drawings releasable at this review was 65 percent,
compared to a best practice standard of 90 percent. We have found that
programs moving forward into system demonstration with low levels of
design stability are more likely than other programs to encounter
costly design changes and parts shortages that, in turn, cause labor
inefficiencies, schedule delays, and quality problems.
Attachment 3 represents our notional depiction of the problems and
outcomes that can typically be expected when these knowledge metrics
are followed versus when they are not. Generally speaking, programs
that move forward without retiring technology, design, and
manufacturing risk at appropriate junctures will encounter a cascade of
problems beginning with design changes and continuing with parts
shortages, changes to manufacturing processes, labor inefficiencies,
and quality problems. All of these problems delay programs and add to
their development costs. We have found, for example, that a significant
portion--about 70 percent--of the total development cost growth in
programs typically occurs after the design review.
Outcome Metrics Provide Insight into the Health and Performance of
Individual Weapon System Programs and DOD's Total Portfolio:
Program outcome metrics--quantitative measures of cost, schedule, and
performance, and changes in these factors over time--provide useful
indicators of the health of acquisition programs and facilitate
analyses of how well programs are meeting cost, schedule, and
performance goals. When assessed regularly for changes and the reasons
that cause changes, such indicators can be valuable tools for improving
insight and oversight of individual programs as well as DOD's total
portfolio of major defense acquisitions. Over the years we have
reported cost, schedule and performance data--good and bad--on numerous
weapon systems. Our work continues to identify systemic and program-
specific causes for cost, schedule, and performance problems and has
led us to designate, since 1990, DOD's management of major weapon
system acquisitions as a high risk area.
To improve acquisition performance and address the factors that keep
weapon acquisitions on the high risk list, DOD is working with us and
the Office of Management and Budget to develop a comprehensive set of
outcome metrics to provide better, comprehensive, and consistent
measures of program cost and schedule performance. Last year, this
cooperative effort resulted in agreement to track trends and changes in
programs from their original baselines, from 5 years ago, and from the
previous year, for the following data points:
* Development cost;
* Procurement cost;
* Total program cost;
* Quantities to be procured;
* Procurement unit costs;
* Total program unit costs;
* Cycle time from Milestone B to Initial Operational Capability:
DOD initiated a pilot study of 7 major defense programs to assess the
adequacy of the proposed metrics and their value in analyzing
performance, and the results proved promising. DOD approved the outcome
metrics and intends to collect and report such data on an annual basis.
Efforts to develop similar metrics on schedule performance continue.
We believe that the metrics DOD plans to use are valuable for providing
insight into the performance of weapon system programs. We have used
similar metrics for many years in assessing programs. For example, we
recently reported that ten of DOD's largest acquisition programs,
commanding about half the overall acquisition dollars in the
department's 2008 portfolio of major programs, have experienced
significant cost growth and have seen quantities reduced by almost a
third (see table 1). The two largest programs--the Joint Strike Fighter
and the Future Combat System--represent significant cost risk moving
forward and will dominate the portfolio for years. Since these programs
consume such a large portion of the funding that DOD spends on research
and development and procurement, their performance also affects other
major weapon acquisitions, smaller acquisition programs, and DOD's
ability to fund and acquire other supplies and equipment as well.
Table 1: Changes in Costs and Quantities for Ten of the Highest Cost
Acquisition Programs:
Fiscal year 2009 dollars in millions:
Program: Joint Strike Fighter;
Total cost: First full estimate: $206,410;
Total cost: Current estimate: $244,772;
Total quantity: First full estimate: 2,866;
Total quantity: Current estimate: 2,456;
Acquisition unit cost: Percentage change: 38.
Program: Future Combat System;
Total cost: First full estimate: $89,776;
Total cost: Current estimate: $129,731;
Total quantity: First full estimate: 15;
Total quantity: Current estimate: 15;
Acquisition unit cost: Percentage change: 45.
Program: Virginia Class Submarine;
Total cost: First full estimate: $58,378;
Total cost: Current estimate: $81,556;
Total quantity: First full estimate: 30;
Total quantity: Current estimate: 30;
Acquisition unit cost: Percentage change: 40.
Program: F-22A Raptor;
Total cost: First full estimate: $88,134;
Total cost: Current estimate: $73,723;
Total quantity: First full estimate: 648;
Total quantity: Current estimate: 184;
Acquisition unit cost: Percentage change: 195.
Program: C-17 Globemaster III;
Total cost: First full estimate: $51,733;
Total cost: Current estimate: $73,571;
Total quantity: First full estimate: 210;
Total quantity: Current estimate: 190;
Acquisition unit cost: Percentage change: 57.
Program: V-22 Joint Services Advanced Vertical Lift Aircraft;
Total cost: First full estimate: $38,726;
Total cost: Current estimate: $55,544;
Total quantity: First full estimate: 913;
Total quantity: Current estimate: 458;
Acquisition unit cost: Percentage change: 186.
Program: F/A-18E/F Super Hornet;
Total cost: First full estimate: $78,925;
Total cost: Current estimate: $51,787;
Total quantity: First full estimate: 1,000;
Total quantity: Current estimate: 493;
Acquisition unit cost: Percentage change: 33.
Program: Trident II Missile;
Total cost: First full estimate: $49,939;
Total cost: Current estimate: $49,614;
Total quantity: First full estimate: 845;
Total quantity: Current estimate: 561;
Acquisition unit cost: Percentage change: 50.
Program: CVN 21 Nuclear Aircraft Class Carrier;
Total cost: First full estimate: $34,360;
Total cost: Current estimate: $29,914;
Total quantity: First full estimate: 3;
Total quantity: Current estimate: 3;
Acquisition unit cost: Percentage change: -13.
Program: P-8A Poseidon Multi-mission Maritime Aircraft;
Total cost: First full estimate: $29,974;
Total cost: Current estimate: $29,622;
Total quantity: First full estimate: 115;
Total quantity: Current estimate: 113;
Acquisition unit cost: Percentage change: 1.
Source: GAO analysis of DOD data.
[End of table]
While program outcome metrics are good measures of individual program
performance, the collective performance of DOD's portfolio of major
defense acquisition programs is a key indicator of how well the
department's acquisition system generates the return on investment it
promises to the warfighter, Congress, and the taxpayer. Portfolio
metrics also provide senior leaders and Congress with a snapshot of the
cumulative impact of current investment decisions and poor program
performance on future budgets. In our annual assessment of selected
weapon programs, we analyzed the performance of DOD programs at the
portfolio level by comparing programs' initial cost, schedule, and
quantity estimates to their current estimates, based on data obtained
from the Selected Acquisition Reports. This year's cumulative results,
reported earlier this week, [Footnote 4] are shown in table 2.
Table 2: Analysis of DOD Major Defense Acquisition Program
Portfolios[A]:
Fiscal year 2009 dollars.
Portfolio size: Number of programs;
Fiscal Year 2003: 77;
Fiscal Year 2007: 95;
Fiscal Year 2008: 96.
Portfolio size: Total planned commitments;
Fiscal Year 2003: $1.2 trillion;
Fiscal Year 2007: $1.6 trillion;
Fiscal Year 2008: $1.6 trillion.
Portfolio size: Commitments outstanding;
Fiscal Year 2003: $724.2 billion;
Fiscal Year 2007: $875.2 billion;
Fiscal Year 2008: $786.3 billion.
Portfolio indicators: Change to total RDT&E costs from first estimate;
Fiscal Year 2003: 37 percent;
Fiscal Year 2007: 40 percent;
Fiscal Year 2008: 42 percent.
Portfolio indicators: Change to total acquisition cost from first
estimate;
Fiscal Year 2003: 19 percent;
Fiscal Year 2007: 26 percent;
Fiscal Year 2008: 25 percent.
Portfolio indicators: Total acquisition cost growth;
Fiscal Year 2003: $183 billion;
Fiscal Year 2007: $301.3 billion[B];
Fiscal Year 2008: $296.4 billion.
Portfolio indicators: Share of programs with 25 percent increase in
program acquisition unit cost growth;
Fiscal Year 2003: 41 percent;
Fiscal Year 2007: 44 percent;
Fiscal Year 2008: 42 percent.
Portfolio indicators: Average schedule delay in delivering initial
capabilities;
Fiscal Year 2003: 18 months;
Fiscal Year 2007: 21 months;
Fiscal Year 2008: 22 months.
Source: GAO analysis of DOD data.
[A] Data were obtained from DOD's Selected Acquisition Reports (dated
December 2002, 2006, and 2007). In a few cases data were obtained
directly from program offices. The number of programs reflects the
programs with Selected Acquisition Reports; however, in our analysis we
have broken a few Selected Acquisition Reports programs into smaller
elements or programs. Not all programs had comparable cost and schedule
data and these programs were excluded from the analysis where
appropriate. Portfolio performance data do not include costs of
developing Missile Defense Agency elements or the DIMHRS program.
[B] The acquisition cost growth for the 2007 portfolio was $295 billion
in 2008 constant dollars.
[End of table]
Analyzing the data and comparing metrics from different time periods
provides unique insights into the relative health of the portfolio and
trends. From 2003 to 2008, the number of programs in DOD's major
defense acquisition portfolio has grown from 77 to 96. Total costs for
these programs now total $1.6 trillion with almost one-half of this
amount still to be spent. Outcome metrics for 2008 show worsening
performance in all categories compared to the 2003 portfolio and mixed
performance--some better, some worse--compared to the 2007 data. While
DOD is committing substantially more investment dollars to developing
and procuring new weapon systems, the total acquisition costs for the
2008 portfolio has grown by $296 billion over initial estimates and the
average schedule delay in delivering capabilities to the warfighter
averages 22 months. Implications for the future are obvious. Continued
cost growth reduces DOD's buying power and results in less funding
being available for other DOD priorities and programs. As program costs
increase, DOD must request more funding to cover overruns, make trade-
offs with existing programs, delay the start of new programs, take
funds from other accounts, or reduce procurement quantities. Continued
failure to deliver weapon systems on time delays providing critical
capabilities to the warfighter and results in operating costly legacy
systems longer than expected, finding alternatives to fill capability
gaps, or going completely without the capability.
Key Prerequisites Need to Be Met to Improve the Value of Metrics and
Achieve Better Acquisition Program Outcomes:
While the metrics discussed above can provide valuable knowledge about
potential problems and additional information on the health of DOD's
acquisition programs, metrics alone may not be sufficient if the
department does not do a better job ensuring that acquisitions begin
with realistic plans and baseline estimates for cost and schedules
prior to development start. We believe there is a clear set of
prerequisites that must be a part of any acquisition strategy before
any measurement of the acquisition's health can be valuable. Otherwise,
metrics measured against unrealistic estimates will do no good. These
key prerequisites for obtaining realistic baselines include:
* Establishing a clear, knowledge-based, evolutionary business case for
the product. This business case must: validate that a need exists;
determine that resources are available to develop a product that will
meet the need; determine that the product developer has a knowledge-
based plan and strategy to deliver the product; establish reasonable
estimates for cost, delivery time and quantities; and ensure available
funding for the product. All of these elements of the business case
should also be agreed upon by major stakeholders across the
requirements, funding, acquisition, and warfighting communities.
* Separating technology development activities from product development
activities. The process of developing technology culminates in
discovery--the gathering of knowledge--and must, by its nature, allow
room for unexpected results and delays. Leading firms do not ask their
product managers to develop technology because they have learned the
hard way that invention cannot be scheduled. When immature technologies
are brought onto the critical path of product development programs too
early, they often cause long delays in an environment where large
workforces must be employed, complex tools, plants, and facilities must
be operated, long and expensive supplier networks must be paid, and the
product itself must sometimes be redesigned once the final form of the
technologies is known. Successful programs give responsibility for
maturing technologies to science and technology organizations, rather
than the program or product development managers, because the science
and technology environment is less costly. We have recommended in the
past that DOD's risks should be taken in the science and technology
arena and that more funding should be made available to this process to
do so.
* Limiting time and requirements for product development to manageable
levels. Product developers should strive to deliver the best available
capabilities within realistic timeframes and should expect to continue
to develop new capabilities when they are technologically feasible. By
limiting product development cycle times to 6 years or less, DOD could
assimilate new technologies into weapon systems more frequently,
accelerate delivery of new technology to the warfighter, hold program
managers accountable, and make more frequent and predictable work in
production, where contractors and the industrial base can profit by
being efficient. Too many major acquisitions currently take the
opposite approach by seeking to deliver a revolutionary "big bang"
capability in one step. This means that programs are more risky,
delivery takes as long as 15 years in some cases, and costs grow at
exponential rates from the original baseline due to the risky nature of
the acquisition strategy. We point to the private sector and some past
defense acquisitions, such as the F-16 program, as models for this
practice.
* Employing early systems engineering discipline in order to develop
realistic cost and schedule estimates prior to development start. Early
systems engineering provides the knowledge a product developer needs to
identify and resolve performance and resource gaps before product
development begins either by reducing requirements, deferring them to
the future, or increasing the estimated cost for the weapon system's
development. Requirements that are too risky given the state of
technology and design should not be allowed into this expensive
environment.
* Making a commitment to fully fund programs once they are approved.
This would require the department to ensure that it does not have too
many programs underway given the amount of available resources.
* Setting priorities from the top to ensure that candidate programs are
truly needed and have a solid plan for delivery. DOD will continue to
experience poor acquisition outcomes until it begins making choices
that reflect the most important needs of the joint warfighter and match
requirements with available resources. The urge to accept all candidate
programs and to go for the "big bang" capability without the knowledge
to achieve it should be resisted. Only the best candidates--defined in
terms of priorities, resource availability, and executability--should
be approved.
There is no doubt that the current state of the department's
acquisition process is too expensive for the taxpayer and not timely
enough for the warfighter. The following illustration reinforces this
point.
Figure 2: Cost Remaining Versus Annual Appropriations for Major Defense
Acquisitions:
[Refer to PDF for image: multiple line graph]
Fiscal year: 1992;
Annual RDTE and Procurement Appropriations: $100 billion;
Costs Remaining for Major Defense Acquisitions: $324 billion.
Fiscal year: 1993;
Annual RDTE and Procurement Appropriations: $91 billion;
Costs Remaining for Major Defense Acquisitions: $271 billion.
Fiscal year: 1994;
Annual RDTE and Procurement Appropriations: $79 billion;
Costs Remaining for Major Defense Acquisitions: $253 billion.
Fiscal year: 1995;
Annual RDTE and Procurement Appropriations: $78 billion;
Costs Remaining for Major Defense Acquisitions: $341 billion.
Fiscal year: 1996;
Annual RDTE and Procurement Appropriations: $78 billion;
Costs Remaining for Major Defense Acquisitions: $345 billion.
Fiscal year: 1997;
Annual RDTE and Procurement Appropriations: $79 billion;
Costs Remaining for Major Defense Acquisitions: $331 billion.
Fiscal year: 1998;
Annual RDTE and Procurement Appropriations: $82 billion;
Costs Remaining for Major Defense Acquisitions: $280 billion.
Fiscal year: 1999;
Annual RDTE and Procurement Appropriations: $89 billion;
Costs Remaining for Major Defense Acquisitions: $233 billion.
Fiscal year: 2000;
Annual RDTE and Procurement Appropriations: $94 billion;
Costs Remaining for Major Defense Acquisitions: $309 billion.
Fiscal year: 2001;
Annual RDTE and Procurement Appropriations: $104 billion;
Costs Remaining for Major Defense Acquisitions: $234 billion.
Fiscal year: 2002;
Annual RDTE and Procurement Appropriations: $111 billion;
Costs Remaining for Major Defense Acquisitions: $586 billion.
Fiscal year: 2003;
Annual RDTE and Procurement Appropriations: $137 billion;
Costs Remaining for Major Defense Acquisitions: $579 billion.
Fiscal year: 2004;
Annual RDTE and Procurement Appropriations: $148 billion;
Costs Remaining for Major Defense Acquisitions: $722 billion.
Fiscal year: 2005;
Annual RDTE and Procurement Appropriations: $165 billion;
Costs Remaining for Major Defense Acquisitions: $745 billion.
Fiscal year: 2006;
Annual RDTE and Procurement Appropriations: $157 billion;
Costs Remaining for Major Defense Acquisitions: $841 billion.
Fiscal year: 2007;
Annual RDTE and Procurement Appropriations: $160 billion;
Costs Remaining for Major Defense Acquisitions: $767 billion.
Source: DOD (data); GAO (analysis and presentation).
[End of figure]
This figure depicts an investment strategy for major weapon systems
that continues to increase the costs to develop our existing weapons
well into the future while the funding available to retire those costs
appears capped at a very low level. While costs continue to rise as the
result of more and more risky programs being added to the portfolio,
our ability to allocate funds for these costs appears to be, at best,
capped at very low percentages of the total cost. We could measure the
risk of these acquisitions much better than we have in the past if we
set the appropriate prerequisites for their initiation, measure the
knowledge that must be in place at various points, and continue to
monitor their health in terms of cost and schedule.
Concluding Remarks:
Measuring the performance of weapon system programs both individually
and collectively is critical for determining whether the warfighter and
the taxpayer are receiving the promised return on investment. No single
metric, however, can capture the whole picture of how well programs are
performing. It is important to look at knowledge and outcome metrics.
Knowledge metrics provide key information for determining whether
programs have the requisite knowledge to move from one phase of
development to the next and are at risk of cost and schedule overruns.
Outcomes metrics are also needed to provide temperature checks on the
health and status of individual programs and the portfolio of programs
as a whole. These metrics are vital for informing program decision
making and helping to manage programs.
Metrics by themselves do not solve problematic acquisitions.
Ultimately, DOD still needs to do a better job planning and executing
programs to achieve better outcomes. Critical to achieving successful
outcomes is establishing knowledge-based, realistic program baselines.
Without realistic baselines, there is no foundation for accurately
measuring the knowledge and health of programs. Over the past several
years, our work has highlighted a number of underlying causes for why
DOD does not effectively manage the acquisition of weapon system
programs. DOD recently revised its acquisition policy to provide a
better foundation for developing weapon systems, however, reform will
not be achieved without fundamental changes to the overall acquisition
culture and environment that exists in DOD. I would be pleased to
discuss these causes and issues with the Committee at a future time.
Mr. Chairman, this concludes my prepared statement. I would be happy to
answer any questions you may have at this time.
Contacts and Acknowledgements:
For further information about this statement, please contact Michael J.
Sullivan (202) 512-4841 or sullivanm@gao.gov. Contact points for our
Office of Congressional Relations and Public Affairs may be found on
the last page of this statement. Individuals who made key contributions
to this statement include Cheryl Andrew, Ridge Bowman, Bruce Fairbairn,
Susan Neill, John Oppenheim, and Ron Schwenn.
[End of section]
Attachment 1: Knowledge-Based Activities:
Knowledge Point 1: Start of product development activities: Best
practice metric: Technology readiness level 7 (indicating technologies
work in an operational environment):
* Demonstrate technologies to high readiness levels.
* Ensure that requirements for the product increment are informed by
preliminary design using systems engineering process (such as
prototyping of preliminary design).
* Establish cost and schedule estimates for product on the basis of
knowledge from preliminary design using system engineering tools (such
as prototyping of preliminary design).
* Constrain development phase (5 to 6 years or less) for incremental
development.
* Ensure development phase fully funded (programmed in anticipation of
milestone).
* Align program manager tenure to complete development phase.
* Contract strategy that separates system integration and system
demonstration activities.
* Conduct independent cost estimate.
* Conduct independent program assessment.
* Conduct major milestone decision review for development start.
Knowledge Point 2: Transition from system integration to system
demonstration activities; Best practice metric: 90 percent of design
drawings are complete by the critical design review:
* Complete system critical design review.
* Complete 90 percent of engineering design drawing packages.
* Complete subsystem and system design reviews.
* Demonstrate with system integration prototype that design meets
requirements.
* Complete the failure modes and effects analysis.
* Identify key system characteristics.
* Identify critical manufacturing processes.
* Establish reliability targets and growth plan on the basis of
demonstrated reliability rates of components and subsystems.
* Conduct independent cost estimate.
* Conduct independent program assessment.
* Conduct major milestone decision review to enter system
demonstration.
Knowledge Point 3: Initiation of producing a product to be delivered to
customer; Best practice metric: 100 percent of critical manufacturing
processes are in control:
* Demonstrate manufacturing processes.
* Build and test production-representative prototypes to demonstrate
product in operational environment.
* Test production-representative prototypes to achieve reliability
goal.
* Collect statistical process control data.
* Demonstrate that critical processes are capable and in statistical
control.
* Independent cost estimate.
* Independent program assessment.
* Conduct major milestone decision review to begin production.
Source: GAO analysis of commercial best practices.
[End of table]
[End of section]
Attachment 2: Technology Readiness Levels:
Technology readiness level: 1. Basic principles observed and reported;
Description: Lowest level of technology readiness. Scientific research
begins to be translated into applied research and development. Examples
might include paper studies of a technology's basic properties;
Hardware: Software: None (paper studies and analysis);
Demonstration Environment: None.
Technology readiness level: 2. Technology concept and/or application
formulated;
Description: Invention begins. Once basic principles are observed,
practical applications can be invented. The application is speculative
and there is no proof or detailed analysis to support the assumption.
Examples are still limited to paper studies;
Hardware: Software: None (paper studies and analysis);
Demonstration Environment: None.
Technology readiness level: 3. Analytical and experimental critical
function and/or characteristic proof of concept;
Description: Active research and development is initiated. This
includes analytical studies and laboratory studies to physically
validate analytical predictions of separate elements of the technology.
Examples include components that are not yet integrated or
representative;
Hardware: Software: Analytical studies and demonstration of nonscale
individual components (pieces of subsystem);
Demonstration Environment: Lab.
Technology readiness level: 4. Component and/or breadboard. Validation
in laboratory environment;
Description: Basic technological components are integrated to establish
that the pieces will work together. This is relatively "low fidelity"
compared to the eventual system. Examples include integration of "ad
hoc" hardware in a laboratory;
Hardware: Software: Low fidelity breadboard; Integration of nonscale
components to show pieces will work together. Not fully functional or
form or fit but representative of technically feasible approach
suitable for flight articles;
Demonstration Environment: Lab.
Technology readiness level: 5. Component and/or breadboard validation
in relevant environment;
Description: Fidelity of breadboard technology increases significantly.
The basic technological components are integrated with reasonably
realistic supporting elements so that the technology can be tested in a
simulated environment. Examples include "high fidelity" laboratory
integration of components;
Hardware: Software: High fidelity breadboard; Functionally equivalent
but not necessarily form and/or fit (size weight, materials, etc).
Should be approaching appropriate scale. May include integration of
several components with reasonably realistic support elements/
subsystems to demonstrate functionality;
Demonstration Environment: Lab demonstrating functionality but not form
and fit. May include flight demonstrating breadboard in surrogate
aircraft. Technology ready for detailed design studies.
Technology readiness level: 6. System/subsystem model or prototype
demonstration in a relevant environment;
Description: Representative model or prototype system, which is well
beyond the breadboard tested for TRL 5, is tested in a relevant
environment. Represents a major step up in a technology's demonstrated
readiness. Examples include testing a prototype in a high fidelity
laboratory environment or in simulated realistic environment;
Hardware: Software: Prototype; Should be very close to form, fit and
function. Probably includes the integration of many new components and
realistic supporting elements/subsystems if needed to demonstrate full
functionality of the subsystem;
Demonstration Environment: High-fidelity lab demonstration or limited/
restricted flight demonstration for a relevant environment. Integration
of technology is well defined.
Technology readiness level: 7. System prototype demonstration in a
realistic environment;
Description: Prototype near or at planned operational system.
Represents a major step up from TRL 6, requiring the demonstration of
an actual system prototype in a realistic environment, such as in an
aircraft, vehicle or space. Examples include testing the prototype in a
test bed aircraft;
Hardware: Software: Prototype. Should be form, fit and function
integrated with other key supporting elements/subsystems to demonstrate
full functionality of subsystem;
Demonstration Environment: Flight demonstration in representative
realistic environment such as flying test bed or demonstrator aircraft;
Technology is well substantiated with test data.
Technology readiness level: 8. Actual system completed and "flight
qualified" through test and demonstration;
Description: Technology has been proven to work in its final form and
under expected conditions. In almost all cases, this TRL represents the
end of true system development. Examples include developmental test and
evaluation of the system in its intended weapon system to determine if
it meets design specifications;
Hardware: Software: Flight qualified hardware;
Demonstration Environment: Developmental Test and Evaluation (DT&E) in
the actual system application.
Technology readiness level: 9. Actual system "flight proven" through
successful mission operations;
Description: Actual application of the technology in its final form and
under mission conditions, such as those encountered in operational test
and evaluation. In almost all cases, this is the end of the last "bug
fixing" aspects of true system development. Examples include using the
system under operational mission conditions;
Hardware: Software: Actual system in final form;
Demonstration Environment: Operational Test and Evaluation (OT&E) in
operational mission conditions.
Source: GAO and its analysis of National Aeronautics and Space
Administration data.
[End of table]
[End of section]
Attachment 3: Notional Illustration Showing the Different Paths That a
Product's Development Can Take:
[Refer to PDF for image: illustration]
This illustration is a graph plotting schedule/time against
cost/investment. Two paths are shown:
1) Mature product and;
2) Continual changes needed to reach maturity.
Mature product: takes less time and has a lower cost due to:
* stable design;
* Demonstrated;
* Built and tested;
* Manufacturing processes in control.
Continual changes needed to reach maturity: takes more time and has a
greater cost due to:
* Unstable design;
* Late drawings and design changes;
* Tooling and design changes (Fully integrated aircraft not built or
tested);
* Production manufacturing process not in control;
* Labor inefficiencies and quality issues.
Source: GAO.
[End of figure]
[End of section]
Related GAO Products:
Defense Acquisitions: Assessment of Major Weapon Programs. [hyperlink,
http://www.gao.gov/products/GAO-09-326SP]. Washington, D.C.: March 30,
2009.
Defense Acquisitions: DOD Must Prioritize Its Weapon System
Acquisitions and Balance Them with Available Resources. [hyperlink,
http://www.gao.gov/products/GAO-09-501T]. Washington, D.C.: March 18,
2009.
Defense Acquisitions: DOD Must Balance Its Needs with Available
Resources and Follow an Incremental Approach to Acquiring Weapon
Systems. [hyperlink, http://www.gao.gov/products/GAO-09-431T].
Washington, D.C.: March 3, 2009.
Defense Acquisitions: Perspectives on Potential Changes to DOD's
Acquisition Management Framework. [hyperlink,
http://www.gao.gov/products/GAO-09-295R]. Washington, D.C.: February
27, 2009.
Defense Management: Actions Needed to Overcome Long-standing Challenges
with Weapon Systems Acquisition and Service Contract Management.
[hyperlink, http://www.gao.gov/products/GAO-09-362T]. Washington, D.C.:
February 11, 2009.
Defense Acquisitions: Fundamental Changes Are Needed to Improve Weapon
Program Outcomes. [hyperlink,
http://www.gao.gov/products/GAO-08-1159T]. Washington, D.C.: September
25, 2008.
Defense Acquisitions: DOD's Requirements Determination Process Has Not
Been Effective in Prioritizing Joint Capabilities. [hyperlink,
http://www.gao.gov/products/GAO-08-1060]. Washington, D.C.: September
25, 2008.
Defense Acquisitions: A Knowledge-Based Funding Approach Could Improve
Major Weapon System Program Outcomes. [hyperlink,
http://www.gao.gov/products/GAO-08-619]. Washington, D.C.: July 2,
2008.
Defense Acquisitions: Better Weapon Program Outcomes Require
Discipline, Accountability, and Fundamental Changes in the Acquisition
Environment. [hyperlink, http://www.gao.gov/products/GAO-08-782T].
Washington, D.C.: June 3, 2008.
Defense Acquisitions: Assessments of Selected Weapon Programs.
[hyperlink, http://www.gao.gov/products/GAO-08-467SP]. Washington,
D.C.: March 31, 2008.
Best Practices: Increased Focus on Requirements and Oversight Needed to
Improve DOD's Acquisition Environment and Weapon System Quality.
[hyperlink, http://www.gao.gov/products/GAO-08-294]. Washington, D.C.:
February 1, 2008.
Cost Assessment Guide: Best Practices for Estimating and Managing
Program Costs. [hyperlink,
http://www.gao.gov/products/GAO-07-1134SP]. Washington, D.C.: July
2007.
Defense Acquisitions: Assessments of Selected Weapon Programs.
[hyperlink, http://www.gao.gov/products/GAO-07-406SP.Washington, D.C.:
March 30, 2007.
Best Practices: An Integrated Portfolio Management Approach to Weapon
System Investments Could Improve DOD's Acquisition Outcomes.
[hyperlink, http://www.gao.gov/products/GAO-07-388]. Washington, D.C.:
March 30, 2007.
Defense Acquisitions: Major Weapon Systems Continue to Experience Cost
and Schedule Problems under DOD's Revised Policy. [hyperlink,
http://www.gao.gov/products/GAO-06-368]. Washington, D.C.: April 13,
2006.
DOD Acquisition Outcomes: A Case for Change. [hyperlink,
http://www.gao.gov/products/GAO-06-257T]. Washington, D.C.: November
15, 2005.
Best Practices: Better Support of Weapon System Program Managers Needed
to Improve Outcomes. [hyperlink,
http://www.gao.gov/products/GAO-06-110]. Washington, D.C.: November 1,
2005.
Defense Acquisitions: Stronger Management Practices Are Needed to
Improve DOD's Software-Intensive Weapon Acquisitions. [hyperlink,
http://www.gao.gov/products/GAO-04-393]. Washington, D.C.: March 1,
2004.
Defense Acquisitions: DOD's Revised Policy Emphasizes Best Practices,
but More Controls Are Needed. [hyperlink,
http://www.gao.gov/products/GAO-04-53]. Washington, D.C.: November 10,
2003.
Best Practices: Setting Requirements Differently Could Reduce Weapon
Systems' Total Ownership Costs. [hyperlink,
http://www.gao.gov/products/GAO-03-57. Washington, D.C.: February 11,
2003.
Defense Acquisitions: Factors Affecting Outcomes of Advanced Concept
Technology Demonstration. [hyperlink,
http://www.gao.gov/products/GAO-03-52]. Washington, D.C.: December 2,
2002.
Best Practices: Capturing Design and Manufacturing Knowledge Early
Improves Acquisition Outcomes. [hyperlink,
http://www.gao.gov/products/GAO-02-701]. Washington, D.C.: July 15,
2002.
Defense Acquisitions: DOD Faces Challenges in Implementing Best
Practices. [hyperlink, http://www.gao.gov/products/GAO-02-469T].
Washington, D.C.: February 27, 2002.
[End of section]
Footnotes:
[1] Technology readiness levels, originally developed by the National
Aeronautics and Space Administration, are measured on a scale of 1 to
9, beginning with paper studies of a technology's feasibility and
culminating with a technology fully integrated into a completed
product. See Attachment 2 for the definitions of technology readiness
levels.
[2] We did this by collecting data directly from program offices using
a questionnaire. These programs are primarily in development and,
therefore, most relevant to current decisions about which programs
should receive substantial investments of research and development
funding now and large amounts of procurement funding in the future.
Defense Acquisitions: Assessment of Selected Weapons Programs.
[hyperlink, http://www.gao.gov/products/GAO-09-326SP]. Washington,
D.C.: March 30, 2009.
[3] In 2003, DOD revised its primary acquisition policy to state that
technologies should be demonstrated in a relevant environment prior to
starting an acquisition program. In 2006, this standard became a
statutory requirement for all major defense acquisition programs in the
National Defense Authorization Act for Fiscal Year 2006, Pub. L. No.
109-163, § 801, codified at 10 U.S.C. § 2366b.
[4] GAO, Defense Acquisitions: Assessments of Selected Weapon Programs,
[hyperlink, http://www.gao.gov/products/GAO-09-326SP] (Washington,
D.C.: Mar. 30, 2009).
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
