Defense Acquisitions
Major Weapon Systems Continue to Experience Cost and Schedule Problems under DOD's Revised Policy Gao ID: GAO-06-368 April 14, 2006The Department of Defense (DOD) is planning to invest $1.3 trillion between 2005 and 2009 in researching, developing, and procuring major weapon systems. How DOD manages this investment has been a matter of congressional concern for years. Numerous programs have been marked by cost overruns, schedule delays, and reduced performance. Over the past 3 decades, DOD's acquisition environment has undergone many changes aimed at curbing cost, schedule, and other problems. In order to determine if the policy DOD put in place is achieving its intended goals, we assessed the outcomes of major weapons development programs initiated under the revised policy. Additionally, we assessed whether the policy's knowledge-based, evolutionary principles are being effectively implemented, and whether effective controls and specific criteria are in place and being used to make sound investment decisions.
Changes made in DOD's acquisition policy over the past 5 years have not eliminated cost and schedule problems for major weapons development programs. Of the 23 major programs we assessed, 10 are already expecting development cost overruns greater than 30 percent or have delayed the delivery of initial operational capability to the warfighter by at least 1 year. The overall impact of these costly conditions is a reduction in the value of DOD's defense dollars and a lower return on investment. Poor execution of the revised acquisition policy is a major cause of DOD's continued problems. DOD frequently bypasses key steps of the knowledge-based process outlined in the policy, falls short of attaining key knowledge, and continues to pursue revolutionary--rather than evolutionary or incremental--advances in capability. Nearly 80 percent of the programs GAO reviewed did not fully follow the knowledge-based process to develop a sound business case before committing to system development. Most of the programs we reviewed started system development with immature technologies, and half of the programs that have held design reviews did so before achieving a high level of design maturity. These practices increase the likelihood that problems will be discovered late in development when they are more costly to address. Furthermore, DOD's continued pursuit of revolutionary leaps in capability also runs counter to the policy's guidance. DOD has not closed all of the gaps in the policy that GAO identified nearly 3 years ago, particularly with regard to adding controls and criteria. Effective controls require decision makers to measure progress against specific criteria and ensure that managers capture key knowledge before moving to the next acquisition phase. However, DOD's policy continues to allow managers to approach major investment decisions with many unknowns. Without effective controls that require program officials to satisfy specific criteria, it is difficult to hold decision makers or program managers accountable to cost and schedule targets. In this environment, decision-making transparency is crucial, but DOD is lacking in this area as well.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-06-368, Defense Acquisitions: Major Weapon Systems Continue to Experience Cost and Schedule Problems under DOD's Revised Policy
This is the accessible text file for GAO report number GAO-06-368
entitled 'Defense Acquisitions: Major Weapon Systems Continue to
Experience Cost and Schedule Problems under DOD's Revised Policy' which
was released on April 14, 2006.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Committees:
United States Government Accountability Office:
GAO:
April 2006:
Defense Acquisitions:
Major Weapon Systems Continue to Experience Cost and Schedule Problems
under DOD's Revised Policy:
GAO-06-368:
GAO Highlights:
Highlights of GAO-06-368, a report to congressional committees.
Why GAO Did This Study:
The Department of Defense (DOD) is planning to invest $1.3 trillion
between 2005 and 2009 in researching, developing, and procuring major
weapon systems. How DOD manages this investment has been a matter of
congressional concern for years. Numerous programs have been marked by
cost overruns, schedule delays, and reduced performance. Over the past
3 decades, DOD‘s acquisition environment has undergone many changes
aimed at curbing cost, schedule, and other problems. In order to
determine if the policy DOD put in place is achieving its intended
goals, we assessed the outcomes of major weapons development programs
initiated under the revised policy. Additionally, we assessed whether
the policy‘s knowledge-based, evolutionary principles are being
effectively implemented, and whether effective controls and specific
criteria are in place and being used to make sound investment decisions.
What GAO Found:
Changes made in DOD‘s acquisition policy over the past 5 years have not
eliminated cost and schedule problems for major weapons development
programs. Of the 23 major programs we assessed, 10 are already
expecting development cost overruns greater than 30 percent or have
delayed the delivery of initial operational capability to the
warfighter by at least 1 year. The overall impact of these costly
conditions is a reduction in the value of DOD‘s defense dollars and a
lower return on investment. The following table illustrates the problem.
Table: Cost and Schedule Outcomes Sorted by Percent of Product
Development Remaining.
Programs: Aerial Common Sensor;
Percent Cost Growth[A]: 45%
Schedule growth in months: 24;
Percent of development remaining: 85%.
Programs: Future Combat System;
Percent Cost Growth[A]: 48%
Schedule growth in months: 48;
Percent of development remaining: 78%.
Programs: Joint Strike Fighter;
Percent Cost Growth[A]: 30%
Schedule growth in months: 23;
Percent of development remaining: 60%.
Programs: Expeditionary Fighting Vehicle;
Percent Cost Growth[A]: 61%
Schedule growth in months: 48;
Percent of development remaining: 49%.
Programs: C-130 Avionics Fighting Vehicle;
Percent Cost Growth[A]: 122%
Schedule growth in months: Delays anticipated;
Percent of development remaining: Undetermined.
Programs: Global Hawk (RQ-4B);
Percent Cost Growth[A]: 166%
Schedule growth in months: Delays Anticipated;
Percent of development remaining: Undetermined.
[A] Cost growth is expressed as the percent change in program
development cost estimates in 2005 base year dollars.
Source: DOD (data); GAO (analysis and presentation)
[End to table]
Poor execution of the revised acquisition policy is a major cause of
DOD‘s continued problems. DOD frequently bypasses key steps of the
knowledge-based process outlined in the policy, falls short of
attaining key knowledge, and continues to pursue revolutionary”rather
than evolutionary or incremental”advances in capability. Nearly 80
percent of the programs GAO reviewed did not fully follow the knowledge-
based process to develop a sound business case before committing to
system development. Most of the programs we reviewed started system
development with immature technologies, and half of the programs that
have held design reviews did so before achieving a high level of design
maturity. These practices increase the likelihood that problems will be
discovered late in development when they are more costly to address.
Furthermore, DOD‘s continued pursuit of revolutionary leaps in
capability also runs counter to the policy‘s guidance.
DOD has not closed all of the gaps in the policy that GAO identified
nearly 3 years ago, particularly with regard to adding controls and
criteria. Effective controls require decision makers to measure
progress against specific criteria and ensure that managers capture key
knowledge before moving to the next acquisition phase. However, DOD‘s
policy continues to allow managers to approach major investment
decisions with many unknowns. Without effective controls that require
program officials to satisfy specific criteria, it is difficult to hold
decision makers or program managers accountable to cost and schedule
targets. In this environment, decision-making transparency is crucial,
but DOD is lacking in this area as well.
What GAO Recommends:
GAO recommends that DOD insert specific criteria into the policy at key
investment points and require programs satisfy those criteria before
allowing them to move forward. In order to insure transparency and
accountability, GAO also recommends that DOD require decision makers to
include the rationale for their decisions in decision documentation.
DOD partially concurred with our recommendations.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Michael J. Sullivan at
(202) 512-4841 or sullivanm@gao.gov.
[End of Section]
Contents:
Letter:
Results in Brief:
Background:
DOD's Revised Policy Has Not Improved Development Program Outcomes:
DOD Is Not Effectively Implementing the Policy's Knowledge-Based,
Evolutionary Approach:
Specific Criteria Are Needed to Ensure Disciplined and Transparent
Investment Decisions:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Comments from the Department of Defense:
Appendix III: Program Data for 23 Programs Initiated under DOD's
Revised Acquisition Policy (as of December 2005):
Related GAO Products:
Tables:
Table 1: Cost and Schedule Outcomes for 23 Programs Initiated under the
Revised Policy (as of December 2005):
Table 2: Cost and Schedule Outcomes for 6 of the 10 Largest Development
Programs Sorted by Percent of System Development Remaining:
Table 3: Assessment of Program Design Maturity:
Table 4: Assessment of Program Acquisition Strategies for GAO's Nine
Case Studies:
Table 5: Types of Controls Considered Best Practices for Successful
Product Development:
Figures:
Figure 1: Development Cost Overruns by Decade (in Fiscal Year 2005
Dollars) and Key Reform Efforts:
Figure 2: Comparison of DOD's Revised Policy and Commercial Best
Practices Model:
Figure 3: Cost and Schedule Growth under DOD's Revised Policy:
Figure 4: Comparison of Programs with Mature versus Immature
Technologies at Start of System Development:
Abbreviations:
DOD: Department of Defense:
GAO: Government Accountability Office:
United States Government Accountability Office:
Washington, DC 20548:
April 13, 2006:
The Honorable John Ensign:
Chairman:
The Honorable Daniel K. Akaka:
Ranking Minority Member:
Subcommittee on Readiness and Management Support Committee on Armed
Services:
United States Senate:
The Honorable Duncan L. Hunter:
Chairman:
Committee on Armed Services:
House of Representatives:
DOD's planned investment in research, development, and procurement of
major weapon systems will total approximately $1.3 trillion between
2005 and 2009, with over $800 billion of that investment yet to be
made. DOD is facing a significant number of problems in managing its
acquisitions. Military operations in Afghanistan and Iraq are consuming
a large share of DOD resources and causing the department to invest
more money sooner than expected to replace or fix existing weapons.
Meanwhile, DOD is intent on transforming military operations while
pursuing multiple megasystems that are expected to be the most
expensive and complex ever. These costly conditions coupled with
increases in spending for other national priorities, such as health
care and social security, make it essential that DOD effectively
leverage its investments, particularly in weapon system acquisitions.
If DOD manages its current portfolio of weapons within traditional
margins of error, the financial consequences could be dire.
DOD's strategy for acquiring major weapon systems has traditionally
been to plan programs that would achieve a big leap forward in
capability within a single development program, a strategy that often
results in major cost and schedule problems. We have assessed weapon
acquisitions as a high-risk area for 15 years, and although U.S.
weapons are among the best in the world, the programs to acquire them
have continued to produce poor cost and schedule outcomes. However, the
current defense acquisition environment continues to be characterized
by cost and schedule growth, a lack of confidence by congressional and
DOD leaders, and no appreciable improvement in the defense acquisition
system. DOD knows what to do to achieve better outcomes. It has written
into policy an approach that advocates that adequate knowledge be
attained at critical junctures before DOD managers agree to invest more
money in the next phase of weapon system development. The policy also
emphasizes evolutionary principles for acquiring weapons rather than
trying to achieve a big leap forward in capability within a single
development program. We have reported in the past that DOD's revised
policy does not incorporate adequate controls to ensure the effective
implementation of a knowledge-based, evolutionary acquisition process.
However, DOD believes that the policy includes the necessary controls
to achieve effective outcomes.
You requested that we evaluate DOD's compliance with and implementation
of its revised acquisition policy intended to produce better cost,
schedule, and performance outcomes for major acquisition programs. In
order to obtain an early assessment of the cost and schedule impact of
the revised policy, and to assess DOD's effectiveness in implementing a
knowledge-based, evolutionary acquisition approach we assessed (1) the
cost and schedule status of major weapons development programs
initiated under the revised policy, (2) whether the policy's knowledge-
based, evolutionary acquisition principles are being effectively
implemented, and (3) whether effective controls and specific criteria
are in place and being used to make sound investment decisions.
In conducting our evaluation, we reviewed pertinent acquisition
statutes, policies, and guidance; analyzed development cost and
schedule data for 23 major acquisition programs approved to start
system development under DOD's revised acquisition policy between
October 2000 and December 2004; conducted case study reviews of nine of
those 23 programs; and interviewed officials from the Office of
Secretary of Defense and each of the military services. We conducted
our review from May 2005 to February 2006 in accordance with generally
accepted government auditing standards. Additional information about
our methodology is contained in appendix I.
Results in Brief:
DOD's revised acquisition policy has not led to improved acquisition
program outcomes. Programs initiated under the revised policy are
already beginning to experience cost and schedule problems similar to
programs managed under prior versions of the policy. Although the
programs we reviewed have been in development for only a short period
of time, nearly half are already estimating development cost growth
greater than 30 percent or are expecting to delay initial delivery to
the warfighter by at least 1 year. Program officials are facing the
familiar predicament of having to add unplanned money or time or to
reduce system capabilities and quantities after initial business cases
have been approved and system development has begun. As a result, DOD
is reducing its buying power and is not achieving the return on
investment it expected when the programs began.
Acquisition officials are not effectively implementing the revised
acquisition policy's knowledge-based process. They regularly bypass key
phases of the early acquisition process, approach key decision points
with limited knowledge about critical technologies and system design,
and do not employ evolutionary acquisition principles. Nearly 80
percent of the programs we assessed were permitted to bypass the
policy's initial major decision review and the associated systems-
engineering process that are intended to ensure that a system's
requirements match available resources and that a sound business case
is developed prior to starting system development. By not consistently
following key processes and strategies, acquisition officials are not
ensuring that a solid foundation of knowledge about cost, schedule, and
performance is established before allowing programs to start system
development, thus resulting in unexecutable business cases. Although
the policy explicitly states that programs shall increase program
knowledge by maturing technologies before beginning system development,
we found that almost three-fourths of the programs started since the
policy was revised began development with immature critical
technologies. Our analysis also indicates that decision makers are
continuing to commit programs to system demonstration and initial
manufacturing before officials have demonstrated high levels of design
knowledge, as emphasized in the policy. In addition, programs like the
Joint Strike Fighter and Future Combat System are still structured to
achieve major leaps in capability within a single development program,
a strategy that has historically proven to be problematic in terms of
cost and schedule outcomes.
Effective implementation of the revised policy is limited by the
absence of effective controls that require compliance and specific
criteria for clearly demonstrating that acceptable levels of knowledge
about technology, design, and manufacturing have been attained at
critical junctures during system development before making further
investments in a program. Without effective controls, the policy cannot
prevent DOD decision makers from starting system development even when
they face significant unknowns about technology, design, and
production. Without specific criteria--or standards against which a
judgment or decision is quantifiably based--decision makers are
permitted to make decisions on the basis of subjective judgment. We
reported this condition in 2003, yet DOD has not closed gaps in the
policy. In the absence of such controls and criteria, DOD faces the
added problems of transparency and accountability because it often does
not sufficiently document the rationale for its decisions to allow
acquisition programs to advance with low levels of technology, design,
and manufacturing knowledge.
This report contains recommendations that DOD require programs to meet
specific knowledge-based criteria at each key decision points in the
acquisition process and require decision makers to provide clear and
specific rationale for their decisions. In addition, we recommend that
before programs enter system development they should be required to
complete disciplined concept and technology development phases that
include specific activities dedicated to capturing knowledge critical
to developing an executable business. DOD partially concurred with our
recommendations. DOD agrees that knowledge-based decision making is
consistent with sound business practice and stated that it would
continue to develop policy that reflects a knowledge-based approach and
improves acquisition outcomes. DOD also agrees that acquisition
decisions should be documented, decision makers should be held
accountable, and that they should provide the rationale for their
decisions.
Background:
Historically, DOD's programs for acquiring major weapon systems have
taken longer, cost more, and often delivered fewer quantities and other
capabilities than planned. GAO has documented these problems for
decades. In 1970, GAO reported that considerable cost growth had been
and was continuing to occur on many current development programs. Since
that report was issued, numerous changes have been made to DOD's
acquisition process and environment to try to improve acquisition
outcomes. Those changes include numerous executive branch initiatives
and legislative actions as well as roughly 11 revisions to DOD's
acquisition policy between 1971 and 2005. Despite these efforts,
defense acquisition programs in the past 3 decades continued to
routinely experience cost overruns, schedule slips, and performance
shortfalls.
Figure 1 illustrates the continued problem of development cost
overruns. The figure depicts the combined cost overruns for large
development programs (programs totaling more than $1 billion for
research, development, testing and evaluation in fiscal year 2005
dollars) in each of the past 3 decades. The figure also identifies some
of the major studies and improvement efforts initiated during this time
frame. As the figure illustrates, efforts to improve acquisition
outcomes have not been successful in curbing acquisition cost problems.
Programs initiated in the 1970s exceeded DOD's initial investment
estimate by 30 percent, or $13 billion (in fiscal year 2005 dollars),
and similar outcomes continued during the subsequent decades despite
numerous reform efforts and policy revisions.
Figure 1: Development Cost Overruns by Decade (in Fiscal Year 2005
Dollars) and Key Reform Efforts:
[See PDF for image]
Source: DOD (data); GAO (analysis and presentation)
[End of figure]
Since the mid-1990s, we have studied the best practices of leading
commercial companies. Taking into account the differences between
commercial product development and weapons acquisitions, we articulated
a best practices product development model that relies on increasing
knowledge when developing new products, separating technology
development from product development, and following an evolutionary or
incremental product development approach. This knowledge-based approach
requires developers to make investment decisions on the basis of
specific, measurable levels of knowledge at critical junctures before
investing more money and before advancing to the next phase of
acquisition. An evolutionary product development process defines the
individual increments on the basis of mature technologies and a
feasible design that are matched with firm requirements. Each increment
should be managed as a separate and distinct acquisition effort with
its own cost, schedule and performance baseline. An increment that
excludes one of these key elements puts an extra burden on decision
makers and provides a weak foundation for making development cost and
schedule estimates. The knowledge-based, evolutionary approach in our
model is intended to help reduce development risks and to achieve
better program outcomes on a more consistent basis.
Hoping to improve acquisition outcomes, DOD leaders initiated
significant revisions to the department's acquisition policy again in
October 2000, by adopting the knowledge-based, evolutionary system
development approach.[Footnote 1] We reported in November 2003, that
much of the revised policy agrees with GAO's extensive body of work and
that of successful commercial firms. DOD's revised policy emphasizes
the importance of and provides a good framework for capturing knowledge
about critical technologies, product design, and manufacturing
processes. If properly implemented and enforced this approach could
help DOD's decision makers gain the confidence they need to make
significant and sound investment decisions for major weapon systems.
Furthermore, the policy's emphasis on evolutionary system development
sets up a more manageable environment for achieving knowledge. We also
noted that DOD's policy strongly suggests the separation of technology
development from system development, a best practice that helps reduce
technological risk at the start of a program and makes cost and
delivery estimates much more predictable.[Footnote 2] Figure 2 depicts
in general how DOD's revised policy adopts key aspects of the best
practices model.
Figure 2: Comparison of DOD's Revised Policy and Commercial Best
Practices Model:
[See PDF for image]
Source: DOD (data); GAO (analysis and presentation)
[End of figure]
Although DOD took significant steps in the right direction, its policy
does not include controls that require program officials to meet the
key criteria that we believe are necessary for ensuring that acceptable
levels of knowledge are actually captured before making additional
significant investments. We previously recommended that DOD design and
implement necessary controls to ensure that appropriate knowledge is
captured and used to make decisions about moving a program forward and
investing more money at critical junctures. DOD officials acknowledged
the advantages of using knowledge-based controls, but stated that they
believed the policy already included enough controls to achieve
effective program results. The officials agreed to monitor the
acquisition process to assess the effectiveness of those controls and
to determine whether additional ones are necessary.
DOD's Revised Policy Has Not Improved Development Program Outcomes:
The cost and schedule outcomes being achieved by development programs
initiated since DOD first issued its revised policy have not improved
over those achieved by programs managed under prior versions of the
policy. Of the 23 major programs we assessed, 10 have already reported
estimated development cost growth greater than 30 percent or expected
delays of at least 1 year in delivery of an initial operational
capability to the warfighter. These programs combined represent a cost
increase of $23 billion (in fiscal year 2005 dollars) and an average
delay in delivery of initial capability of around 2 years. Most of the
other programs were still in the early stages as of December 2005 with
over half of system development remaining and had not yet reported an
adequate amount of cost or schedule data to effectively analyze their
progress. Table 1 contains the cost and schedule increases for the 23
programs we assessed, expressed as a percentage of each program's
development estimate.
Table 1: Cost and Schedule Outcomes for 23 Programs Initiated under the
Revised Policy (as of December 2005):
Program: Expeditionary Fighting Vehicle;
Percent growth in estimated development cost[A]: 61%;
Percent growth in estimated development schedule: 70%.
Program: Active Electronically Scanned Array radar (upgrade for F/A-18
E/F fighter/attack aircraft);
Percent growth in estimated development cost[A]: 14%;
Percent growth in estimated development schedule: 1%.
Program: Global Hawk unmanned aerial vehicle;
Percent growth in estimated development cost[A]: 166%;
Percent growth in estimated development schedule: Undetermined.
Program: Joint Strike Fighter;
Percent growth in estimated development cost[A]: 30%;
Percent growth in estimated development schedule: 23%.
Program: UH-60M helicopter upgrade;
Percent growth in estimated development cost[A]: 151%;
Percent growth in estimated development schedule: 25%.
Program: C-130 Avionics Modernization Program;
Percent growth in estimated development cost[A]: 122%;
Percent growth in estimated development schedule: Undetermined.
Program: C-5 Reliability Enhancement and Re-engining Program;
Percent growth in estimated development cost[A]: 0%;
Percent growth in estimated development schedule: 25%.
Program: Joint Tactical Radio System Cluster 1;
Percent growth in estimated development cost[A]: 31%;
Percent growth in estimated development schedule: 44%.
Program: Joint Tactical Radio System Waveform;
Percent growth in estimated development cost[A]: 44%;
Percent growth in estimated development schedule: Undetermined.
Program: Advanced Anti-radiation Guided Missile;
Percent growth in estimated development cost[A]: 7%;
Percent growth in estimated development schedule: 0%.
Program: Multi-Platform Radar Technology Insertion Program;
Percent growth in estimated development cost[A]: 0%;
Percent growth in estimated development schedule: Undetermined.
Program: Future Combat System;
Percent growth in estimated development cost[A]: 48%;
Percent growth in estimated development schedule: 53%.
Program: E-2 Advanced Hawkeye;
Percent growth in estimated development cost[A]: 5%;
Percent growth in estimated development schedule: 0%.
Program: Warfighter Information Network-Tactical;
Percent growth in estimated development cost[A]: 0%;
Percent growth in estimated development schedule: 0%.
Program: Small Diameter Bomb;
Percent growth in estimated development cost[A]: 0%;
Percent growth in estimated development schedule: 0%.
Program: EA-18G;
Percent growth in estimated development cost[A]: 7%;
Percent growth in estimated development schedule: 0%.
Program: Joint Tactical Radio System Cluster 5;
Percent growth in estimated development cost[A]: 0%;
Percent growth in estimated development schedule: 2%.
Program: Multi-Mission Maritime Aircraft;
Percent growth in estimated development cost[A]: 0%;
Percent growth in estimated development schedule: 0%.
Program: Standard Missile-6 Extended Range Active Missile Block 1;
Percent growth in estimated development cost[A]: 0%;
Percent growth in estimated development schedule: 0%.
Program: Aerial Common Sensor;
Percent growth in estimated development cost[A]: 45%;
Percent growth in estimated development schedule: 36%.
Program: B-2 Radar Modernization Program;
Percent growth in estimated development cost[A]: 0%;
Percent growth in estimated development schedule: 0%.
Program: Patriot/Medium Extended Air Defense System Combined Aggregate
Program (fire unit);
Percent growth in estimated development cost[A]: 0%;
Percent growth in estimated development schedule: 0%.
Program: Mission Planning System;
Percent growth in estimated development cost[A]: 0%;
Percent growth in estimated development schedule: 0%.
Sources: DOD (data); GAO (analysis and presentation).
[A] Cost growth is expressed as the percent change in program
development cost estimates in fiscal year 2005 dollars.
[End of table]
The Army's Future Combat System is a case in point. Less than 3 years
after program initiation and with $4.6 billion invested, the Army has
already increased its development cost estimate $8.9 billion or 48
percent and delayed delivery of initial capability by 4 years over the
original business case. Similarly, just over 1 year after initiating
development of the Aerial Common Sensor aircraft, the Army has reported
that severe weight and design problems discovered during development
have stopped work on the program. As a result, program officials are
anticipating at least a 45 percent cost increase and a delay of 2 years
in delivering an initial capability to the warfighter. These two Army
programs are not the only ones experiencing problems. Table 2 contains
cost and schedule data for 6 of the 10 largest development programs
initiated under the revised policy, including the Future Combat System
and Aerial Common Sensor. As the table illustrates there are several
programs experiencing large cost increases and schedule delays.
Table 2: Cost and Schedule Outcomes for 6 of the 10 Largest Development
Programs Sorted by Percent of System Development Remaining:
Programs: Aerial Common Sensor;
Percent development cost growth: 45%;
Delay in delivery of initial capability in months: 24;
Percent of development remaining: 85%.
Programs: Future Combat System;
Percent development cost growth: 48%;
Delay in delivery of initial capability in months: 48;
Percent of development remaining: 78%.
Programs: Joint Strike Fighter;
Percent development cost growth: 30%;
Delay in delivery of initial capability in months: 23;
Percent of development remaining: 60%.
Programs: Expeditionary Fighting Vehicle;
Percent development cost growth: 61%;
Delay in delivery of initial capability in months: 48;
Percent of development remaining: 49%.
Programs: C-130 Avionics Modernization Program;
Percent development cost growth: 122%;
Delay in delivery of initial capability in months: Delays anticipated
due to program restructure;
Percent of development remaining: Undetermined due to program
restructure.
Programs: Global Hawk (RQ-4B);
Percent development cost growth: 166%;
Delay in delivery of initial capability in months: Delays anticipated
due to program restructure;
Percent of development remaining: Undetermined due to program
restructure.
Sources: DOD (data); GAO (analysis and presentation).
[End of table]
A good measure of acquisition performance is return on investment as
expressed in acquisition program unit cost because unit cost represents
the value DOD is getting for its acquisition dollars invested in a
certain program. The programs listed in table 2 will not achieve the
return on investment that DOD anticipated when they began development.
In the case of Joint Strike Fighter, for example, DOD initially
intended to purchase 2,866 aircraft at an acquisition program unit cost
of about $66 million. The Navy has reduced the number of Joint Strike
Fighter aircraft it plans to buy; technology and design problems
encountered during development have led to the significant cost growth.
As a result, the acquisition program unit cost is now about $84
million, an increase of 27 percent. We recently reported that the risk
of even greater increases is likely because flight testing has not yet
started and the acquisition strategy involves substantial overlap of
development and production. Similar problems have led to increases in
the Future Combat System program. At program initiation, the Army
anticipated that each of 15 units would cost about $5.5 billion to
develop and deliver. Since that time, instability in the program's
technologies and requirements have led to significant cost increases,
leading to a 54 percent increase in acquisition program unit cost, now
estimated to be $8.5 billion.
Regarding all 23 development programs, DOD leaders originally planned
to invest a total of about $83 billion (fiscal year 2005 dollars) for
system development and anticipated delivering an initial operational
capability to the warfighter in 77 months on average. However,
development costs have grown and delivery schedules have been delayed
significantly. DOD now expects to invest over $106 billion in those
same programs, an increase of over $23 billion or 28 percent. The
delivery of initial capability to the warfighter is expected to take an
average of 88 months or nearly 1 year longer than originally planned.
Figure 3 shows changes in these business case elements for these
programs in the short time since their initiation.
Figure 3: Cost and Schedule Growth under DOD's Revised Policy:
[See PDF for image]
Sources: DOD (data); GAO (analysis and presentation).
[End of figure]
DOD Is Not Effectively Implementing the Policy's Knowledge-Based,
Evolutionary Approach:
DOD is not effectively implementing the knowledge-based process and
evolutionary approach emphasized in its acquisition policy. While the
policy outlines a specific knowledge-based process of concept
refinement and technology development[Footnote 3] to help ensure a
sound business case is developed before committing to a new development
program, almost 80 percent of the programs we reviewed were permitted
to bypass this process. Furthermore, the policy emphasizes the need to
mature all critical technologies before starting system development and
to demonstrate that the product's design is mature before beginning
system demonstration. However, nearly three-fourths of the programs
reported having immature critical technologies when they received
approval to start development, and at least half of the programs had
not achieved design maturity before holding their design review and
gaining approval to enter the system demonstration phase of
development. The policy also emphasizes the use of an evolutionary
product development approach, yet program officials continue to
structure major acquisition programs to achieve large advances in
capability within a single step development program. This strategy has
historically resulted in poor cost and schedule outcomes.
Knowledge-Based Process Not Enforced:
DOD decision makers continue to approve programs for system development
that have not followed key elements of the policy's suggested knowledge-
based process. The policy requires program managers to provide senior
decision makers with knowledge about key aspects of a system at
critical investment points in the acquisition process. Our prior
reviews have identified those critical points as the start of system
development or program start (referred to as Milestone B in the DOD
acquisition policy), design readiness review separating system
integration and system demonstration, and production commitment
(Milestone C in the DOD acquisition policy). The most important point
occurs at program start, when system development begins. DOD
acquisition guidance emphasizes the importance of the acquisition
phases preceding program start, noting that the decisions made during
those phases--concept refinement and technology development--generally
define the nature of an entire acquisition program.
Acquisition Officials Are Not Effectively Using Early Processes to
Develop Executable Business Cases:
Acquisition officials continue to begin system development without
following early processes for developing executable business cases. A
business case should provide demonstrated evidence that (1) the
warfighter's needs are real and necessary and that they can best be met
with the chosen concept and (2) the chosen concept can be developed and
produced within existing resources--including technologies, design
knowledge, funding, and the time to deliver the product when it is
needed. Establishing a business case calls for a realistic assessment
of risks and costs; doing otherwise undermines the intent of the
business case and invites failure. This process requires the user and
developer to negotiate whatever trade-offs are needed to achieve a
match between the user's requirements and the developer's resources
before system development begins.
The revised policy and associated guidance emphasize the importance of
following a sound process of systems engineering[Footnote 4] and
decision making prior to initiating a system development program. The
process established in the policy consists of two phases, concept
refinement and technology development, and a major decision review
called Milestone A, which if rigorously followed, would provide
acquisition officials with an opportunity to assess whether program
officials had the knowledge needed to develop an executable business
case. However, almost 80 percent of the programs we reviewed began
system development without holding any prior decision review. Senior
officials with the Office of the Secretary of Defense confirmed that
this is a common practice among defense acquisition programs. This
practice eliminates a key opportunity for decision makers to assess
early product knowledge needed to establish a business case that is
based on realistic cost, schedule, and performance expectations.
Although program officials conduct analysis before starting a
development program, they do not consistently follow a process to
capture the critical knowledge needed to produce executable business
cases, as evidenced by the poor outcomes current programs are
experiencing. Officials with the Office of the Secretary of Defense
recognized this lack of rigor and discipline in acquisition process,
and in February 2004, the Under Secretary of Defense (Acquisition,
Technology and Logistics) issued a department-wide policy memorandum
directing acquisition officials to place greater emphasis on systems
engineering when planning and managing acquisition programs. The policy
requires programs to develop a systems engineering plan that describes
the programs' overall technical approach, including processes,
resources, metrics, and applicable performance incentives. Although
DOD's systems engineering initiative has the potential to improve
program performance, officials have found that the preliminary results
are mixed. Early analysis shows that implementation is inconsistent
while program officials learn to develop and implement systems-
engineering plans.
Programs Continue to Enter System Development with Immature
Technologies:
DOD decision makers continue to permit programs to enter system
development before critical technologies are mature. Our review of
technology readiness assessments and acquisition decision memorandums
for our nine case study programs found that seven of the nine programs
were approved to begin development even though program officials
reported levels of knowledge below the criteria suggested in the policy
and associated guidance, specifically in the area of technology
maturity.[Footnote 5] Those seven programs are not isolated cases. As
illustrated in Figure 4, 13 of the programs (nearly three-fourths) that
received approval to enter system development under the revised policy
did so with less than 100 percent of their critical technologies mature
to the level specified by DOD. Only 2 of those programs had more than
75 percent of their technologies mature when they began (see appendix
III for technology maturity data for each program).
Figure 4: Comparison of Programs with Mature versus Immature
Technologies at Start of System Development:
[See PDF for image]
Sources: DOD (data); GAO (analysis and presentation).
Note: This figure depicts technology maturity status for 18 of the 23
programs in our review. Technology maturity data was not available for
the other 5 programs.
[End of figure]
Even though acquisition policy states that technologies shall be mature
before beginning system development, the practice of accepting high
levels of technology risk at program start continues to be the norm and
not the exception. An official with Office of the Secretary of Defense
responsible for reviewing and validating program assessments of
technology maturity informed us that the office generally views
immature critical technologies at the beginning of development as an
acceptable risk as long as program officials can show that they have a
plan to mature the technologies by the time the program reaches its
design readiness review, which requires additional investments to move
a program from system integration into system demonstration. Therefore,
risk management plans are consistently viewed as acceptable substitutes
for demonstrated knowledge.
Programs Continue Past Design Reviews before Design Maturity is
Demonstrated:
In addition to emphasizing the importance of capturing technology
knowledge before starting system development, DOD's policy also
highlights the importance of demonstrating design maturity before
moving from the integration phase of system development into system
demonstration and initial manufacturing. The policy establishes a
design readiness review between the two phases to determine whether a
product's design is mature and stable and whether the product is ready
to move ahead. While DOD's policy does not require programs to
demonstrate any specific level of design maturity, our past work has
found that a key indicator of design maturity is the completion of 90
percent of the system's engineering drawings. We found that defense
programs that moved forward with lower levels of design maturity, as
indicated by drawing completion, encountered costly design changes and
parts shortages that, in turn, caused labor inefficiencies, schedule
delays, and quality problems. Consequently, those programs required
significant increases in resources--time and money--over what was
estimated at the point each program entered the system demonstration
phase.
We analyzed engineering drawing completion data for 8 programs
initiated under the revised policy that have held a design
review,[Footnote 6] and found that more than half of those programs had
not completed 90 percent of their design drawings before they received
approval to enter the system demonstration phase of development. We
also analyzed drawing-release data for three programs that have not yet
held their design review but have projected the number of drawings
officials anticipate will be completed when their reviews are held.
Based on projections provided by program officials, 2 of those 3
programs are expected to have less than 55 percent of their drawings
complete before they seek approval to begin system demonstration and
initial manufacturing.
Table 3: Assessment of Program Design Maturity:
Program: Joint Tactical Radio System Cluster 1;
Percentage of design drawings complete at design review: 28%.
Program: Global Hawk unmanned aerial vehicle;
Percentage of design drawings complete at design review: 33%.
Program: Active Electronically Scanned Array radar (upgrade for F/A-18
E/F fighter/attack aircraft);
Percentage of design drawings complete at design review: 59%.
Program: Expeditionary Fighting Vehicle;
Percentage of design drawings complete at design review: 81%.
Program: B-2 Radar Modernization Program;
Percentage of design drawings complete at design review: 84%.
Program: E-2 Advanced Hawkeye;
Percentage of design drawings complete at design review: 90%.
Program: EA-18G;
Percentage of design drawings complete at design review: 97%.
Program: C-5 Reliability Enhancement and Re-engining Program;
Percentage of design drawings complete at design review: 98%.
Program: Multi-Platform Radar Technology Insertion Program;
Percentage of design drawings complete at design review: 100%[A].
Program: Joint Strike Fighter;
Percentage of design drawings complete at design review: 52%[A].
Program: Aerial Common Sensor;
Percentage of design drawings complete at design review: 39%[A].
Sources: DOD (data); GAO (analysis and presentation).
[A] Program office projections.
[End of table]
Evolutionary Acquisition Is Not Being Used:
Despite the revised policy's guidance that capabilities should be
developed and delivered in individually defined and separately managed
increments, a majority of major weapon acquisition programs we assessed
continue to be structured to achieve revolutionary increases in
capability within one development program. According to the policy, the
objective of an evolutionary approach is to balance needs and available
capability with resources and put capability into the hands of the user
quickly. The policy states that the success of the strategy depends on
consistent and continuous definition of requirements and the maturation
of technologies that lead to disciplined development and production of
systems that provide increasing capability. In this approach,
requirements that cannot be satisfied within these limits as well as
available financial resources must wait for future generations of the
product and be managed as separate system development programs with
separate milestones, costs, and schedules. In our case studies of nine
acquisition programs initiated under the revised policy, we found only
one program--the Small Diameter Bomb--that satisfied all of the
criteria of an evolutionary approach. In five case studies, we found
that program officials had claimed that their programs were
evolutionary, yet our evidence shows they were not evolutionary in
practice;[Footnote 7] and in three cases, program officials chose not
to use evolutionary acquisition from the outset. Table 3 summarizes our
assessment of the nine case studies.
Table 4: Assessment of Program Acquisition Strategies for GAO's Nine
Case Studies:
Programs in GAO's case study: Future Combat System;
Claim to be evolutionary?: Yes;
Meet evolutionary criteria?: No;
Greater than 30% cost growth or more than 1-year schedule slip: Yes.
Programs in GAO's case study: Global Hawk (RQ-4B);
Claim to be evolutionary?: Yes;
Meet evolutionary criteria?: No;
Greater than 30% cost growth or more than 1-year schedule slip: Yes.
Programs in GAO's case study: Joint Strike Fighter;
Claim to be evolutionary?: Yes;
Meet evolutionary criteria?: No;
Greater than 30% cost growth or more than 1-year schedule slip: Yes.
Programs in GAO's case study: Aerial Common Sensor;
Claim to be evolutionary?: Yes;
Meet evolutionary criteria?: No;
Greater than 30% cost growth or more than 1-year schedule slip: Yes.
Programs in GAO's case study: Multi-Mission Maritime Aircraft;
Claim to be evolutionary?: Yes;
Meet evolutionary criteria?: No;
Greater than 30% cost growth or more than 1-year schedule slip: No.
Programs in GAO's case study: Small Diameter Bomb;
Claim to be evolutionary?: Yes;
Meet evolutionary criteria?: Yes;
Greater than 30% cost growth or more than 1-year schedule slip: No.
Programs in GAO's case study: E-2 Advanced Hawkeye;
Claim to be evolutionary?: No;
Meet evolutionary criteria?: No;
Greater than 30% cost growth or more than 1-year schedule slip: No.
Programs in GAO's case study: Expeditionary Fighting Vehicle;
Claim to be evolutionary?: No;
Meet evolutionary criteria?: No;
Greater than 30% cost growth or more than 1-year schedule slip: Yes.
Programs in GAO's case study: Multiplatform Radar Technology Insertion
Program;
Claim to be evolutionary?: No;
Meet evolutionary criteria?: No;
Greater than 30% cost growth or more than 1-year schedule slip: No.
Sources: DOD (data); GAO (analysis and presentation).
[End of table]
Specific Criteria Are Needed to Ensure Disciplined and Transparent
Investment Decisions:
The revised acquisition policy does not contain effective controls that
require the demonstration of product knowledge measured against
specific criteria to ensure that acquisition officials make
disciplined, transparent, and knowledge-based investment decisions. The
lack of specific required criteria creates an environment in which
unknowns about technology, design, and manufacturing processes are
acceptable. Decision makers and program officials are left with no
objective measures against which to gauge a program's level of
knowledge, making accountability difficult. In the absence of criteria,
transparency in acquisition decisions is essential to ensuring
accountability, but key decision documents do not provide sufficient
information about major decisions. DOD believes that acquisition
decision memorandums, used to document program decisions, provide
adequate transparency. However, the decision memorandums we reviewed
did not contain an explanation of the decision maker's rationale and
rarely identify remaining risks, especially as they relate to the key
knowledge standards emphasized in the policy. Further, the timeliness,
accessibility, and depth, of the data contained in the Selected
Acquisition Reports, DOD's primary means of providing Congress with a
status report of program performance, inhibits the reports' usefulness
as a management and oversight tool.
In November 2003, we reported that the revised acquisition policy
lacked many of the controls that leading commercial companies rely on
to attain an acceptable level of knowledge before making additional
significant investments.[Footnote 8] Controls are considered effective
if they are backed by specific criteria and if decision makers are
required to consider the resulting data before deciding to advance a
program to the next level. Controls used by leading companies help
decision makers gauge progress in meeting cost, schedule, and
performance goals and hold program managers accountable for capturing
relevant product knowledge to inform key investment decisions. The
controls we have articulated as best practices used by successful
commercial product developers are listed below in table 5.
Table 5: Types of Controls Considered Best Practices for Successful
Product Development:
Program start (Milestone B): Start of product development: Demonstrate
technologies to high readiness levels.
Program start (Milestone B): Start of product development: Ensure that
requirements for the product are informed by the systems engineering
process.
Program start (Milestone B): Start of product development: Establish
cost and schedule estimates for product on the basis of knowledge from
preliminary design using system engineering tools.
Program start (Milestone B): Start of product development: Conduct
decision review for program start.
Design Readiness Review: Beginning of System demonstration: Complete 90
percent of design drawings.
Design Readiness Review: Beginning of System demonstration: Complete
subsystem and system design reviews.
Design Readiness Review: Beginning of System demonstration: Demonstrate
with prototype that design meets requirements.
Design Readiness Review: Beginning of System demonstration: Obtain
stakeholders' concurrence that drawings are complete and producible.
Design Readiness Review: Beginning of System demonstration: Complete
the failure modes and effects analysis.
Design Readiness Review: Beginning of System demonstration: Identify
key system characteristics.
Design Readiness Review: Beginning of System demonstration: Identify
critical manufacturing processes.
Design Readiness Review: Beginning of System demonstration: Establish
reliability targets and growth plan on the basis of demonstrated
reliability rates of components and subsystems.
Design Readiness Review: Beginning of System demonstration: Conduct
decision review to enter system demonstration.
Production Commitment (Milestone C): Initiation of low-rate production:
Demonstrate manufacturing processes.
Production Commitment (Milestone C): Initiation of low-rate production:
Build production-representative prototypes.
Production Commitment (Milestone C): Initiation of low-rate production:
Test production-representative prototypes to achieve reliability goal.
Production Commitment (Milestone C): Initiation of low-rate production:
Test production-representative prototypes to demonstrate product in
operational environment.
Production Commitment (Milestone C): Initiation of low-rate production:
Collect statistical process control data.
Production Commitment (Milestone C): Initiation of low-rate production:
Demonstrate that critical processes are capable and in statistical
control.
Production Commitment (Milestone C): Initiation of low-rate production:
Conduct decision review to begin production.
Sources: GAO (analysis and presentation).
[End of table]
Some senior officials with the Office of the Secretary of Defense
believe that the effective use of controls in DOD's policy and the
establishment of more specific criteria for decision making would
improve program outcomes. They note that specific criteria need to be
established and that programs need to be held accountable to those
criteria before being permitted to proceed into the next phase. They
also note that the criteria for moving an acquisition effort from one
phase of the process to the next, primarily documented in acquisition
decision memorandums as exit criteria, are not typically specific and
often do not relate to the key knowledge-based criteria suggested in
the policy.
We found this to be true for our nine case study programs. We reviewed
acquisition decision memorandums in our case studies and determined
that they were not useful in explaining the decision maker's rationale
and in almost all of the cases they did not address the key knowledge
criteria suggested in the acquisition policy. In most instances, the
decision maker simply noted that the program being assessed was ready
to proceed into system development, but did not provide an explanation
of the rationale for the decision. Senior officials with the Office of
the Secretary of Defense told us that they agree that a better
explanation of the decision maker's rationale, specifically in
instances where the knowledge criteria are not fully met, would provide
transparency and ultimately allow for a more accountable decision-
making process. The following two examples illustrate how decision
documentation is lacking:
* The Future Combat System program received approval to enter system
development and demonstration in 2003, with 19 percent of its critical
technologies mature, well below the policy's standard. The acquisition
decision memorandum supporting this decision did not provide the
rationale for approving the system with such a large number of immature
critical technologies. The memo did direct an updated review of the
decision 18 months later and that the program "remain flexible and open
to accommodate trades in the system architecture and in the individual
systems' designs."
* The Joint Strike Fighter program was approved to enter system
development in 2001. The acquisition decision memorandum did not
address the fact that 75 percent of the program's critical technologies
were not mature to the policy's standard. The memorandum did
acknowledge that the program's requirements could be changed or
modified, noting that further refinements in the requirements should be
explored as a potential way to reduce program costs. However, the
memorandum did not explain why the decision maker determined that the
program should enter development without achieving the technology and
requirements knowledge emphasized in the policy.
The acquisition decision memorandums for most of the other programs we
reviewed did not specifically address critical gaps in knowledge, nor
did they effectively explain the decision makers' rationale for deeming
those programs ready to begin system development. In memos where we
found a reference to key knowledge principles, such as technology
maturity, the decision makers acknowledged that more effort was needed
to meet the policy's suggested criteria but considered the risk
acceptable to begin development. These memos did not explain why risks
were considered acceptable. For example, the Navy's Multi-Mission
Maritime Aircraft program had none of its critical technologies mature
at program initiation. The decision maker acknowledged the need to
further mature the critical technologies but approved the program to
enter development. Instead of holding the program to the policy's
criteria for entering development, the decision maker simply directed
the Navy to work with the Office of the Secretary of Defense to
implement risk mitigation and technology maturation plans during the
integration phase of system development.
In addition to the lack of transparency provided through acquisition
decision memoranda, we also found that the data presented to Congress
in DOD's Selected Acquisition Reports (SARs) provided only limited
usefulness as an oversight tool. Since 1969, SARs have been the primary
means by which DOD reports the status of major weapon system
acquisitions to Congress. SARs are reports that are expected to contain
information on the cost, schedule, and performance of major weapon
systems in comparison with baseline values established at program
start, full-scale development, and production decision points. Our
analysis, as well as a previous GAO review,[Footnote 9] of current and
historical SAR data found that the timeliness, accessibility, and depth
of the data contained in the reports limits their usefulness as an
oversight tool. Our prior review noted that a number of opportunities
exist for DOD to give Congress more complete information on the
performance of major defense acquisition programs. DOD agreed that SAR
data could be improved to make it more useful to Congress.
Conclusions:
Failing to consistently implement the knowledge-based process and
evolutionary principles emphasized in the revised acquisition policy--
coupled with a lack of specific criteria for making key investment
decisions--are keeping DOD on its historical path of poor cost and
schedule outcomes. Most programs are incurring the same scope of cost
overruns and schedule delays as programs managed under prior DOD
policies. More consistent use of the early acquisition processes would
improve the quality and viability of program business cases by ensuring
they are founded on knowledge obtained from rigorous and disciplined
analysis. The initiative by Office of the Secretary of Defense to
reinstitute the use of systems engineering is a step in the right
direction. However, in order for this initiative to be effective DOD
must establish and enforce specific criteria at key decision points.
Our past work has identified and recommended criteria and controls that
should be consistently applied at major decision points. The
enforcement of these criteria is critical to ensuring that programs
have the knowledge necessary to successfully move forward through the
acquisition process. DOD officials have acknowledged the advantages of
using knowledge-based criteria and controls, but believe the policy
already includes enough controls to achieve effective program results.
However, without enforceable criteria, defense officials are challenged
to determine whether adequate knowledge has been obtained for investing
taxpayer dollars. The lack of enforceable criteria also makes it
difficult to hold defense officials accountable for their decisions.
Recommendations for Executive Action:
DOD must ensure that appropriate knowledge is captured and used at
critical junctures to make decisions about moving a program forward and
investing more money. We recommend that the Secretary of Defense
require program officials to demonstrate that they have captured
appropriate knowledge at three key points--program start, design review
for transitioning from system integration to system demonstration, and
production commitment--as a condition for investing resources. At a
minimum those controls should require program officials to demonstrate
that they have achieved a level of knowledge that meets or exceeds the
following criteria at each respective decision point:
* Program start (Milestone B): Start of product development:
- Demonstrate technologies to high readiness levels:
- Ensure that requirements for the product are informed by the systems-
engineering process:
- Establish cost and schedule estimates for product on the basis of
knowledge from preliminary design using system engineering tools:
- Conduct decision review for program start:
* Design readiness review: Beginning of system demonstration:
- Complete 90 percent of design drawings:
- Complete subsystem and system design reviews:
- Demonstrate with prototype that design meets requirements:
- Obtain stakeholders' concurrence that drawings are complete and
producible:
- Complete the failure modes and effects analysis:
- Identify key system characteristics:
- Identify critical manufacturing processes:
- Establish reliability targets and growth plan on the basis of
demonstrated reliability rates of components and subsystems:
- Conduct decision review to enter system demonstration:
* Production commitment (Milestone C): Initiation of low-rate
production:
- Demonstrate manufacturing processes:
- Build production-representative prototypes:
- Test production-representative prototypes to achieve reliability goal:
- Test production-representative prototypes to demonstrate product in
operational environment:
- Collect statistical process control data:
- Demonstrate that critical processes are capable and in statistical
control:
- Conduct decision review to begin production:
Furthermore, to ensure that major decisions are transparent and that
program officials and decision makers are held accountable, we
recommend that the Secretary of Defense require decision makers to
include written rationale for each major decision in acquisition
decision documentation. The rationale should address the key knowledge-
based criteria appropriate for milestone decisions, explain why a
program's level of knowledge in each area was deemed acceptable if
criteria have not been met and provide a plan for achieving the
knowledge necessary to meet criteria within a given time frame.
Agency Comments and Our Evaluation:
DOD provided written comments on a draft of this report. The comments
appear in appendix II.
DOD partially concurred with our recommendation that the Secretary of
Defense should establish specific controls to insure that program
officials demonstrate that they have captured a level of knowledge that
meets or exceeds specific criteria at three key points in the
acquisition process: program start, design readiness review, and
production commitment. DOD agreed that knowledge-based decision making
is consistent with sound business practice and stated that it would
continue to develop policy that reflects a knowledge-based approach and
improves acquisition outcomes. DOD noted that it would consider our
recommendations as it reassesses the DOD acquisition business model and
the knowledge required at each decision point. We believe that DOD's
plan to reassess its business model provides a good opportunity to
establish the controls and specific criteria recommended in this
report. Therefore, we are retaining our recommendation that the
Secretary of Defense should establish controls to insure that program
officials demonstrate that they have captured a level of knowledge that
meets or exceeds specific criteria at three key points in the
acquisition process.
DOD also partially concurred with our recommendation that the Secretary
of Defense require decision makers to provide written rationale in
acquisition decision documentation for each major decision. DOD agreed
that acquisition decisions should be documented, decision makers should
be held accountable, and that they should provide the rationale for
their decisions. DOD believes that the implementation of Section 801 of
the National Defense Authorization Act for FY 2006 reinforces these
processes. The act calls for the decision maker to certify that the
program meets certain requirements, such as technology maturity, prior
to starting a new development program at Milestone B. However, the act
is focused on the decision to start a development program and does not
identify specific criteria for programs to be measured against at
design readiness review or production commitment. We believe our
recommendation adds transparency and accountability to the process
because it requires the decision maker to provide the rationale for a
decision to allow a program to move forward, not only at Milestone B
but at other key decision points as well. Therefore, we are retaining
our recommendation that the Secretary of Defense require decision
makers to provide written rationale for each major decision in
acquisition decision documentation.
We are sending copies of this report to the Secretary of Defense; the
Secretaries of the Air Force, Army, and Navy; and the Director of the
Office of Management and Budget. We will provide copies to others on
request. This report will also be available at no charge on GAO's Web
site at [Hyperlink=http://www.gao.gov].
If you have any questions about this report or need additional
information, please call me at (202) 512-4841 (sullivanm@gao.gov).
Contact points for the offices of Congressional Relations and Public
Affairs are located on the last page of this report. Key contributors
to this report were Michael Hazard, Assistant Director; Lily Chin; Ryan
Consaul; Christopher DePerro; Travis Masters; and Adam Vodraska.
Signed by:
Michael J. Sullivan:
Director, Acquisition and Sourcing Management:
[End of section]
Appendix I: Scope and Methodology:
To assess the impact of DOD's revised acquisition policy, we analyzed
cost and schedule data for 23 major defense acquisition programs that
were approved to begin system development under the revised policy. We
did not assess space, missile defense, or ship programs. We collected
our data from Selected Acquisition Reports, presidential budget
documents, ongoing GAO work, and pertinent program officials. We
utilized previous GAO reports related to defense acquisition policies
and worked with knowledgeable GAO staff to ensure the use of current,
accurate data. We also analyzed more than 150 annual Selected
Acquisition Reports covering a 36-year period from 1969 to 2005, to
determine historical trends related to outcomes of acquisition policy
implementation.
We assessed whether the revised policy's knowledge-based, evolutionary
acquisition principles were being effectively implemented by conducting
9 case study reviews and analyzing design maturity data for 11 programs
that have made engineering-drawing data available to GAO. Our case
study programs were the Aerial Common Sensor, Multi-Platform Radar
Technology Insertion Program, Global Hawk Unmanned Aerial Vehicle,
Small Diameter Bomb, Future Combat System, Joint Strike Fighter,
Expeditionary Fighting Vehicle, Multi-Mission Maritime Aircraft, and
the E-2 Advanced Hawkeye. We interacted directly with numerous program
officials to seek input on current developments with their programs. We
studied program documents to assess how well programs understand and
are implementing the revised acquisition policy. We also analyzed
drawing release data for those programs that have either passed their
design review or have provided GAO with estimated drawing release data
for a future design review to assess design maturity. In several cases,
we asked that program offices verify information in these various
documents.
We also reviewed Department of Defense (DOD) Directive 5000.1, DOD
Instruction 5000.2, and the Defense Acquisition Guidebook. In addition
we examined each of the military services' policy directives and
guidance, DOD memorandums to include policy intent and DOD expectations
regarding policy implementation as well as Joint Capabilities
Integration and Development System documents. We interviewed relevant
officials in Washington, D.C., from the Office of the Director, Defense
Research and Engineering, the Joint Staff, the Office of the Secretary
of Defense, and Army, Navy, and Air Force acquisition policy staff in
order to better understand the content of these documents and the
intent of DOD's policy.
We conducted our review from May 2005 to February 2006 in accordance
with generally accepted government auditing standards.
[End of section]
Appendix II: Comments from the Department of Defense:
Acquisition Technology and Logistics:
Office Of The Under Secretary Of Defense:
9000 Defense Pentagon Washington, DC 20301-9000:
April 13 2006:
DPAP/PAIC:
Mr. Michael Sullivan:
Director, Acquisition and Sourcing Management:
U.S. Government Accountability Office:
441 G Street, N.W.
Washington, D.C. 20548:
Dear Mr. Sullivan:
This is the Department of Defense response to the GAO draft report,
"DEFENSE ACQUISITIONS: Major Weapon Systems Continue to Experience Cost
and Schedule Problems under DOD's Revised Policy" (GAO-06-368/GAO Code
120448) DOD Acquisition Policy Implementation. I concur with the
recommendation in the report. Additional comment is attached.
Sincerely,
Signed by:
Shay Assad
Director, Defense Procurement and Acquisition Policy
Attachment: As stated:
GAO DRAFT REPORT - DATED MARCH 7, 2006:
GAO CODE 120448 /GAO-06-368:
"DEFENSE ACQUISITIONS: Major Weapon Systems Continue to Experience Cost
and Schedule Problems under DOD's Revised Policy"
DEPARTMENT OF DEFENSE COMMENT ON THE RECOMMENDATION:
GAO RECOMMENDATION:
DOD must ensure that appropriate knowledge is captured and used at
critical junctures to make decisions about moving a program forward and
investing more money. We recommend that the Secretary of Defense
require program officials to demonstrate that they have captured
appropriate knowledge at three key points - program start, design
review for transitioning from system integration to system
demonstration, and production commitment - as a condition for investing
resources. At a minimum those controls should require program officials
to demonstrate that they have achieved a level of knowledge that meets
or exceeds the following criteria at each respective decision point:
Program start (Milestone B): Start of product development
* Demonstrate technologies to high readiness levels:
* Ensure that requirements for the product are informed by the systems
engineering process:
* Establish cost and schedule estimates for product on the basis of
knowledge from preliminary design using system engineering tools:
* Conduct decision review for program start:
Design readiness review: Beginning of system demonstration
* Complete 90 percent of design drawings:
* Complete subsystem and system design reviews:
* Demonstrate with prototype that design meets requirements:
* Obtain stakeholders' concurrence that drawings are complete and
producible:
* Complete the failure modes and effects analysis:
* Identify key system characteristics:
* Identify critical manufacturing processes:
* Establish reliability targets and growth plan on the basis of
demonstrated reliability rates of components and subsystems:
* Conduct decision review to enter system demonstration:
Production commitment (Milestone C): Initiation of low-rate production
* Demonstrate manufacturing processes:
* Build production-representative prototypes:
* Test production-representative prototypes to achieve reliability goal:
* Test production-representative prototypes to demonstrate product in
operational environment:
* Collect statistical process control data:
* Demonstrate that critical processes are capable and in statistical
control:
* Conduct decision review to begin production:
Furthermore, to ensure that major decisions are transparent and that
program officials and decision makers are held accountable, we
recommend that the Secretary of Defense require decision makers to
include written rationale for each major decision in acquisition
decision documentation. The rationale should address the key knowledge-
based criteria appropriate for milestone decisions, explain why a
program's level of knowledge in each area was deemed acceptable if
criteria have not been met, and provide a plan for achieving the
knowledge necessary to meet criteria within a given time frame.
DOD RESPONSE: Partial Concurrence:
Partially Concur. The Department of Defense agrees that knowledge-based
decision making is consistent with sound business practice.
Consequently, we will continue to develop policy that reflects that
approach and improves acquisition outcomes. In keeping with that
objective and in association with the results of the recently completed
Quadrennial Defense Review, we will soon be reassessing the DOD
acquisition business model and the knowledge required at each decision
point. In that vein, we will consider the GAO proposals during that
process. We also agree that acquisition decisions should be documented,
that decision makers should be held accountable, and that they should
provide the rationale for their decisions-a process reinforced through
implementation of Section 801 of the National Defense Authorization Act
for FY 2006.
[End of section]
Appendix III: Program Data for 23 Programs Initiated under DOD's
Revised Acquisition Policy (as of December 2005):
Program: Expeditionary Fighting Vehicle;
Program start: 12/2000;
Formal Milestone I[A] or Milestone A decision review?: Yes;
Percent technology mature (TRL 6) at program start: 80%;
Percent design drawings complete at design review: 81%;
Percent growth in estimated development cost[C]: 61%;
Percent growth in estimated development schedule: 70%.
Program: Active Electronically Scanned Array radar (upgrade for F/A-18
E/F fighter/attack aircraft);
Program start: 12/2000;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 0%;
Percent design drawings complete at design review: 59%;
Percent growth in estimated development cost[C]: 14%;
Percent growth in estimated development schedule: 1%.
Program: Global Hawk unmanned aerial vehicle;
Program start: 2/2001;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 0%;
Percent design drawings complete at design review: 33%;
Percent growth in estimated development cost[C]: 166%;
Percent growth in estimated development schedule: Undetermined.
Program: UH-60M helicopter upgrade;
Program start: 4/2001;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: Not available;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 151%;
Percent growth in estimated development schedule: 25%.
Program: C-130 Avionics Modernization Program;
Program start: 8/2001;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 100%;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 122%;
Percent growth in estimated development schedule: Undetermined.
Program: Joint Strike Fighter;
Program start: 10/2001;
Formal Milestone I[A] or Milestone A decision review?: Yes;
Percent technology mature (TRL 6) at program start: 25%;
Percent design drawings complete at design review: 52%b;
Percent growth in estimated development cost[C]: 30%;
Percent growth in estimated development schedule: 23%.
Program: C-5 Reliability Enhancement and Re-engining Program;
Program start: 11/2001;
Formal Milestone I[A] or Milestone A decision review?: Yes;
Percent technology mature (TRL 6) at program start: 100%;
Percent design drawings complete at design review: 98%;
Percent growth in estimated development cost[C]: 0%;
Percent growth in estimated development schedule: 25%.
Program: Joint Tactical Radio System Cluster 1;
Program start: 6/2002;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 0%;
Percent design drawings complete at design review: 28%;
Percent growth in estimated development cost[C]: 31%;
Percent growth in estimated development schedule: 44%.
Program: Joint Tactical Radio System Waveform;
Program start: 6/2002;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: Not available;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 44%;
Percent growth in estimated development schedule: Undetermined.
Program: Advanced Anti-radiation Guided Missile;
Program start: 4/2003;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: Not available;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 7%;
Percent growth in estimated development schedule: 0%.
Program: Multi-Platform Radar Technology Insertion Program;
Program start: 4/2003;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 100%;
Percent design drawings complete at design review: 100% b;
Percent growth in estimated development cost[C]: 0%;
Percent growth in estimated development schedule: Undetermined.
Program: Future Combat System;
Program start: 5/2003;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 19%;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 48%;
Percent growth in estimated development schedule: 53%.
Program: E-2 Advanced Hawkeye;
Program start: 6/2003;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 50%;
Percent design drawings complete at design review: 90%;
Percent growth in estimated development cost[C]: 5%;
Percent growth in estimated development schedule: 0%.
Program: Warfighter Information Network-Tactical;
Program start: 7/ 2003;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 25%;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 0%;
Percent growth in estimated development schedule: 0%.
Program: Small Diameter Bomb;
Program start: 10/2003;
Formal Milestone I[A] or Milestone A decision review?: Yes;
Percent technology mature (TRL 6) at program start: 100%;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 0%;
Percent growth in estimated development schedule: 0%.
Program: EA-18G;
Program start: 11/2003;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 60%;
Percent design drawings complete at design review: 97%;
Percent growth in estimated development cost[C]: 7%;
Percent growth in estimated development schedule: 0%.
Program: Joint Tactical Radio System Cluster 5;
Program start: 4/2004;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 50%;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 0%;
Percent growth in estimated development schedule: 2%.
Program: Multi-Mission Maritime Aircraft;
Program start: 5/2004;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 0%;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 0%;
Percent growth in estimated development schedule: 0%.
Program: Standard Missile-6 Extended Range Active Missile Block 1;
Program start: 6/2004;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: Not available;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 0%;
Percent growth in estimated development schedule: 0%.
Program: Aerial Common Sensor;
Program start: 7/2004;
Formal Milestone I[A] or Milestone A decision review?: Yes;
Percent technology mature (TRL 6) at program start: 50%;
Percent design drawings complete at design review: 39% b;
Percent growth in estimated development cost[C]: 45%;
Percent growth in estimated development schedule: 36%.
Program: B-2 Radar Modernization Program;
Program start: 7/2004;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 100%;
Percent design drawings complete at design review: 84%;
Percent growth in estimated development cost[C]: 0%;
Percent growth in estimated development schedule: 0%.
Program: Patriot/Medium Extended Air Defense System Combined Aggregate
Program (fire unit);
Program start: 8/2004;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 83%;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 0%;
Percent growth in estimated development schedule: 0%.
Program: Mission Planning System;
Program start: 12/2004;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: Not available;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 0%;
Percent growth in estimated development schedule: 0%.
Sources: DOD (data); GAO (analysis and presentation).
Note: In this table the term "not available" means that GAO had not
received sufficient data to make an assessment of the given program's
design and/or technology maturity.
[A] Milestone I was a forerunner to Milestone A, the decision review
that currently precedes the start of technology development.
[B] Program office projections.
[C] Cost growth is expressed as the percent change in program
development cost estimates in fiscal year 2005 dollars.
[End of table]
[End of section]
Related GAO Products:
DOD Acquisition Outcomes: A Case for Change. GAO-06-257T. Washington,
D.C.: November 15, 2005.
Defense Acquisitions: Stronger Management Practices Are Needed to
Improve DOD's Software-Intensive Weapon Acquisitions. GAO-04-393.
Washington, D.C.: March 1, 2004.
Best Practices: Setting Requirements Differently Could Reduce Weapon
Systems' Total Ownership Costs. GAO-03-57. Washington, D.C.: February
11, 2003:
Best Practices: Capturing Design and Manufacturing Knowledge Early
Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15,
2002.
Defense Acquisitions: DOD Faces Challenges in Implementing Best
Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.
Best Practices: Better Matching of Needs and Resources Will Lead to
Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8,
2001.
Best Practices: A More Constructive Test Approach Is Key to Better
Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31,
2000.
Defense Acquisition: Employing Best Practices Can Shape Better Weapon
System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000.
Best Practices: DOD Training Can Do More to Help Weapon System Programs
Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August16,
1999.
Best Practices: Better Management of Technology Development Can Improve
Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30,
1999.
Defense Acquisitions: Best Commercial Practices Can Improve Program
Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999.
Defense Acquisition: Improved Program Outcomes Are Possible. GAO/T-
NSIAD-98-123. Washington, D.C.: March 17, 1998.
Best Practices: DOD Can Help Suppliers Contribute More to Weapon System
Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998.
Best Practices: Successful Application to Weapon Acquisition Requires
Changes in DOD's Environment. GAO/NSIAD-98-56. Washington, D.C.:
February 24, 1998.
Best Practices: Commercial Quality Assurance Practices Offer
Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26,
1996.
FOOTNOTES
[1] In addition to the acquisition policy, the process used by DOD to
establish program funding, known as the Planning, Programming,
Budgeting and Execution System (PPBES), and the process used to
determine system requirements, now called the Joint Capabilities
Integration and Development System (JCIDS), also impact program
outcomes. Both processes are currently being studied to determine if
any changes could be made to improve program performance as it relates
to funding and requirements.
[2] GAO. Defense Acquisitions: DOD's Revised Policy Emphasizes Best
Practices, but More Controls Are Needed, GAO-04-53 (Washington, D.C.:
Nov. 10, 2003).
[3] According to DOD Instruction 5000.2, the concept refinement phase
is intended to refine the initial concept and develop a technology
development strategy. Concept refinement ends when the decision
authority approves a preferred solution resulting from the analysis of
alternatives and approves the associated technology development
strategy. After concept refinement, a project enters technology
development at Milestone A, when the decision maker has approved the
technology development strategy. The purpose of this phase is to reduce
technology risk and to determine the appropriate set of technologies to
be integrated into a full system.
[4] Systems engineering is a technical management tool that provides
the knowledge necessary to translate requirements into specific,
achievable capabilities. By using the tools of systems engineering
during these early phases of concept refinement and technology
development acquisition decision makers and developers can work
together to close gaps between requirements and available resources--
well before system development starts.
[5] DOD's revised policy emphasizes the importance of reducing
technology risk and demonstrating technologies in a relevant
environment (technology readiness level 6) prior to program start. A
technology readiness level of 6 means the technology should be very
close to the planned form, fit, and function of its physical
configuration and that it has been tested or proven to work in a
relevant environment such as a laboratory. GAO recommends a higher
level of maturity in its best practice model based on best commercial
practices. This would require a demonstration of the technology in the
environment it is expected to be used.
[6] Some programs did not report having a design readiness review but
did report having a critical design review. Where this was the case, we
assessed those programs' drawing data at their critical design review.
[7] GAO, TACTICAL AIRCRAFT: Opportunity to Reduce Risks in the Joint
Strike Fighter Program with Different Acquisition Strategy, GAO-05-271
(Washington, D.C.: March 15, 2005) and GAO, UNMANNED AERIAL VEHICLES:
Changes in Global Hawk's Acquisition Strategy Are Needed to Reduce
Program Risks, GAO-05-6 (Washington, D.C.: November 5, 2004).
[8] GAO, DEFENSE ACQUISITIONS: DOD's Revised Policy Emphasizes Best
Practices, but More Controls Are Needed, GAO-04-53 (November 10, 2003).
[9] GAO, DEFENSE ACQUISITIONS: Information for Congress on Performance
of Major Programs Can Be More Complete, Timely, and Accessible, GAO-05-
182 (March 28, 2005).
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
