Defense Acquisitions
Application of Lessons Learned and Best Practices in the Presidential Helicopter Program Gao ID: GAO-11-380R March 25, 2011In June 2009, following the expenditure of close to $3 billion and a critical Nunn-McCurdy breach of the cost growth threshold, the Department of Defense (DOD) terminated the Navy's VH-71 presidential helicopter acquisition program and contract because of cost growth, schedule delays, and projected system performance. The Presidential Helicopter VXX program is a successor Navy program to the terminated VH-71 program acquisition and has been initiated to develop aircraft to replace the current, aging presidential helicopter fleet. The Ike Skelton National Defense Authorization Act for Fiscal Year 2011 (the Act) directed GAO to review and report annually to the congressional defense committees on the VXX program through 2013. This is the first of the required GAO reports. It discusses (1) major lessons learned from the terminated VH-71 program that should be applied to the follow-on VXX program and (2) the current acquisition approach of the VXX program and sufficiency of the underlying acquisition plans and related documentation.
Several lessons learned from the acquisition strategy and eventual termination of the VH-71 program apply to the VXX program. For example, had the VH-71 program followed acquisition best practices and conducted early systems engineering, it could have led to a feasible, stable preliminary design ideally before development start. In turn, a stable, early design allows for more accurate program cost estimates and a better foundation for sufficient funding commitments. Instead, it began without completing systems engineering until well after development start. As a consequence, it never achieved design stability and experienced significant cost and schedule problems in development. Its cost estimates doubled--from about $6.5 billion at development start in 2005 to about $13 billion when terminated in 2009. More than good systems engineering is necessary, however. A key to successful development is the ability to make early trade-offs either in the design of a product or the customer's expectations to avoid outstripping the resources available for product development. The VH-71 program was not afforded room needed to pursue these needed trade-offs. Stringent performance requirements (some with no flexibility) were laid out for the system prior to the start of development and did not appear to involve significant consideration of trade-offs of cost, performance, and schedule negotiated between the customer and the developer. The VH-71 program's experience validated the need to execute a knowledge-based acquisition process with discipline, confirming the danger of not replacing risk with knowledge earlier in the acquisition process. VXX program officials seem to understand this lesson learned from the VH-71 program and appear to be establishing a knowledge-based acquisition process emphasizing early systems engineering. One of the primary lessons they learned from the VH-71 program's experience is that there must be an early, solid business case with a rational balance between requirements, cost, and schedule. To accomplish this, they have stated that a rigorous four-phase systems engineering and technical review process will be used. Early VXX program efforts appear to reflect the intent to pursue a knowledge-based acquisition. The VXX program is currently in the materiel solution analysis phase of the acquisition process, and an ICD has been developed to formally document the capabilities required to perform the defined mission, the specific capability gaps that exist, and the need to address them. Our review of the VXX program's ICD indicates that it addresses all three of these areas and also appears to align with acquisition best practices. An AOA has been initiated but not yet completed. According to DOD officials, it will be more robust than the AOA developed for the VH-71 program. The program is in the earliest stages of development, still developing a business case to launch product development. We will, as you have requested, assess and report on a wide range of VXX program activities moving forward. Throughout, we will be looking for the implementation of a knowledge-based acquisition through application of key best practice process controls.
GAO-11-380R, Defense Acquisitions: Application of Lessons Learned and Best Practices in the Presidential Helicopter Program
This is the accessible text file for GAO report number GAO-11-380R
entitled 'Defense Acquisitions: Application of Lessons Learned and
Best Practices in the Presidential Helicopter Program' which was
released on March 25, 2011.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
GAO-11-380R:
United States Government Accountability Office:
Washington, DC 20548:
March 25, 2011:
Congressional Committees:
Subject: Defense Acquisitions: Application of Lessons Learned and Best
Practices in the Presidential Helicopter Program:
In June 2009, following the expenditure of close to $3 billion and a
critical Nunn-McCurdy breach[Footnote 1] of the cost growth threshold,
the Department of Defense (DOD) terminated[Footnote 2] the Navy's VH-
71 presidential helicopter acquisition program and contract because of
cost growth, schedule delays, and projected system performance. The
Presidential Helicopter VXX program is a successor Navy program to the
terminated VH-71 program acquisition and has been initiated to develop
aircraft to replace the current, aging presidential helicopter fleet.
The Ike Skelton National Defense Authorization Act for Fiscal Year
2011 (the Act) directed GAO to review and report annually to the
congressional defense committees on the VXX program through 2013.
[Footnote 3] This is the first of the required GAO reports. It
discusses (1) major lessons learned from the terminated VH-71 program
that should be applied to the follow-on VXX program and (2) the
current acquisition approach of the VXX program and sufficiency of the
underlying acquisition plans and related documentation.
To identify major VH-71 program lessons learned, we examined the cost,
schedule, and performance experience of the terminated VH-71 program
through discussions with agency officials and review of reported
lessons learned and then assessed how those lessons relate to
acquisition best practices. To determine the current status of the VXX
program, we examined available program documents and interviewed
knowledgeable officials. To assess the sufficiency[Footnote 4] of the
program's efforts, we discussed the Navy's plans for the program with
knowledgeable officials and obtained and analyzed the one completed
VXX program baseline acquisition document, the program's Initial
Capabilities Document (ICD),[Footnote 5] to determine whether it
incorporates acquisition best practices and addresses DOD's
acquisition policy and guidance. We also obtained preliminary
information on the ongoing VXX program Analysis of Alternatives
(AOA)[Footnote 6] and made preliminary observations on this effort and
the terminated VH-71 program's AOA conducted in 2003.
We conducted this performance audit from September 2010 to March 2011
in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives.
Background:
The Marine Corps's HMX-1 squadron currently utilizes two types of
helicopters--the VH-3D and the VH-60N--to carryout the presidential
helicopter mission. Over the past several years, numerous
modifications and improvements have been made to both aircraft types
to incorporate emerging technologies and address new requirements.
These improvements have increased the gross weight of the helicopters--
decreasing some aspects of mission capability, for example, range, and
severely limiting the ability to incorporate future improvements
because of the impact of further weight growth.
According to program officials, over time it had become increasingly
difficult to accommodate the demands placed on the HMX-1 aircraft in
support of presidential requirements. The events following the
terrorist attacks on the United States on September 11, 2001,
highlighted the need for improved transportation, communication, and
security capabilities for presidential support aircraft. As a result,
a VXX replacement helicopter program was initiated in April 2002 to
begin fielding a new helicopter in the 2011 time frame. A subsequent
November 2002 White House memo to the Secretary of Defense articulated
a goal of having the new helicopter available earlier, and in
response, the Navy developed an accelerated program plan to develop
and initially field a new helicopter by the end of 2008. The VXX
program was subsequently redesignated the VH-71 program.
In January 2005, the Navy entered into a contract with Lockheed Martin
Systems Integration to develop the replacement helicopter. By January
2009, after schedule slips and significant cost increases, the Navy
reported a Nunn-McCurdy breach of the critical cost growth threshold
and in June 2009 terminated the VH-71 program. However, the need for a
replacement helicopter remains. An ICD was drafted by the Marine Corps
and approved by DOD in August 2009 to start a successor VXX Helicopter
Replacement Program. According to program officials, the current
projected VH-3D and VH-60N service life will support the development
and fielding of the VXX.
In the last few years, Congress and DOD have put in place new defense
acquisition policy provisions that replace risk with knowledge--
placing greater emphasis on front-end planning and establishing sound
business cases for starting programs. For example, both DOD's December
2008 acquisition policy revisions and the Weapon Systems Acquisition
Reform Act of 2009 require programs to invest more time and resources
in the front end of the acquisition process in line with a knowledge-
based acquisition approach.[Footnote 7] As a result, DOD's current
acquisition policy and guidance reflect a knowledge-based acquisition
framework in line with acquisition best practices GAO has been
identifying since the late 1990s. We will use this framework in this
report to analyze both the VH-71 and the VXX programs.
A knowledge-based acquisition framework involves achieving the right
knowledge at the right time--enabling leadership to make informed
decisions about when and how best to move into various acquisition
phases. In essence, knowledge supplants risk over time. Our best
practices work has demonstrated that this building of knowledge
consists of information that should be gathered at three critical
points over the course of a program.
² Knowledge point 1: Resources and requirements match. Achieving a
high level of technology maturity and preliminary system design backed
by robust systems engineering by the start of system development is an
important indicator of whether this match has been made. This means
that the technologies needed to meet essential product requirements
have been demonstrated to work in their intended environment. In
addition, the developer has completed a preliminary design of the
product that shows the design is feasible. If the above conditions are
met, a solid business case is established at this point.
² Knowledge point 2: Product design is stable. This point occurs when
a program determines that a product's design will meet customer
requirements, as well as cost, schedule, and reliability targets. A
best practice is to achieve design stability at the system-level
critical design review, usually held midway through system
development. Completion of at least 90 percent of engineering drawings
at this point provides tangible evidence that the product's design is
stable, and a prototype demonstration shows that the design is capable
of meeting performance requirements.
² Knowledge point 3: Manufacturing processes are mature. This point is
achieved when it has been demonstrated that the developer can
manufacture the product within cost, schedule, and quality targets. A
best practice is to ensure that all critical manufacturing processes
are in statistical control--that is, they are repeatable, sustainable,
and capable of consistently producing parts within the product's
quality tolerances and standards--at the start of production.
Summary:
Several lessons learned from the acquisition strategy and eventual
termination of the VH-71 program apply to the VXX program. For
example, had the VH-71 program followed acquisition best practices and
conducted early systems engineering, it could have led to a feasible,
stable preliminary design ideally before development start. In turn, a
stable, early design allows for more accurate program cost estimates
and a better foundation for sufficient funding commitments. Instead,
it began without completing systems engineering until well after
development start. As a consequence, it never achieved design
stability and experienced significant cost and schedule problems in
development. Its cost estimates doubled--from about $6.5 billion at
development start in 2005 to about $13 billion when terminated in
2009. More than good systems engineering is necessary, however. A key
to successful development is the ability to make early trade-offs
either in the design of a product or the customer's expectations to
avoid outstripping the resources available for product development.
The VH-71 program was not afforded room needed to pursue these needed
trade-offs. Stringent performance requirements (some with no
flexibility) were laid out for the system prior to the start of
development and did not appear to involve significant consideration of
trade-offs of cost, performance, and schedule negotiated between the
customer and the developer.
The VH-71 program's experience validated the need to execute a
knowledge-based acquisition process with discipline, confirming the
danger of not replacing risk with knowledge earlier in the acquisition
process. VXX program officials seem to understand this lesson learned
from the VH-71 program and appear to be establishing a knowledge-based
acquisition process emphasizing early systems engineering. One of the
primary lessons they learned from the VH-71 program's experience is
that there must be an early, solid business case with a rational
balance between requirements, cost, and schedule. To accomplish this,
they have stated that a rigorous four-phase systems engineering and
technical review process will be used. Early VXX program efforts
appear to reflect the intent to pursue a knowledge-based acquisition.
The VXX program is currently in the materiel solution analysis phase
of the acquisition process, and an ICD has been developed to formally
document the capabilities required to perform the defined mission, the
specific capability gaps that exist, and the need to address them. Our
review of the VXX program's ICD indicates that it addresses all three
of these areas and also appears to align with acquisition best
practices. An AOA has been initiated but not yet completed. According
to DOD officials, it will be more robust than the AOA developed for
the VH-71 program. The program is in the earliest stages of
development, still developing a business case to launch product
development. We will, as you have requested, assess and report on a
wide range of VXX program activities moving forward. Throughout, we
will be looking for the implementation of a knowledge-based
acquisition through application of key best practice process controls.
Lesson Learned from the VH-71 Program:
The VH-71 program's failure to follow acquisition best practices was a
critical factor in the program's poor performance that led to its
ultimate termination. It started with a faulty business case, did not
perform appropriate systems engineering analysis to gain knowledge at
the right times, and failed to make necessary trade-offs between
resources and requirements even after years of development. Because of
these failures, the program was unable to achieve stable design and
experienced significant cost growth and schedule delays before being
terminated in 2009.
The Program Began with a High-Risk Business Case:
The VH-71 program was knowingly initiated with a high-risk business
case. To accelerate the program's initial delivery of helicopters from
2011 to 2008, the Navy adopted a two-step acquisition approach and
initiated production at the same time it began development.[Footnote
8] Prior to the start of development, a March 2004 Office of the Under
Secretary of Defense for Acquisition, Technology and Logistics
(OUSD(AT&L)) Defense Acquisition Program Support Assessment of the
program concluded that the program had a high-risk schedule because of
concurrent design and production efforts. However, the Office of the
Secretary of Defense (OSD) recognized and accepted the program's risk
and strategy in its January 2005 Acquisition Decision Memorandum
approving the program's entry into development. Concurrency and lack
of systems engineering knowledge to achieve a match between required
capabilities and resources contributed to the cost overruns and
schedule delays experienced by the program prior to its termination in
2009.
Systems Engineering Knowledge Was Not Available:
The risk of the VH-71 program's business case was even higher than
anticipated because early systems engineering wasn't done when it
should have been to be in accord with acquisition best practices. As
we have previously reported,[Footnote 9] a primary reason for cost and
schedule problems is too many technical unknowns and insufficient
knowledge about performance and production risks. If this early
systems engineering is not performed, significant cost increases can
occur as the system's requirements become better understood by the
government and contractor. The VH-71 program's experience exemplified
this. System requirement reviews are conducted to ensure that system
requirements have been properly identified and that a mutual
understanding between the government and the contractor exists. These
steps should occur prior to the start of development. However, when
the VH-71 program's system requirements review occurred in May 2005--4
months after the start of development--it was found that the
contractor's design was significantly noncompliant with what the
contract required. In June 2006, the VH-71 program's Defense
Acquisition Executive Summary reported that it had taken considerable
effort over the prior 14 months to resolve related issues and that
good progress had been made in aligning the contractor's design with
what the contract required. It also noted, though, that related
activities had resulted in delay of technical reviews and contract
milestones and additional cost.
Having a feasible, stable preliminary design for a weapons program
early in the acquisition process is important for lessening risk, both
by ensuring that there is a match between resources and requirements
and by demonstrating that a product's design can meet customer
requirements, as well as cost, schedule, and reliability targets. The
VH-71 program did not, however, complete needed systems engineering
until well after the start of development for both increments and, in
the case of Increment 1, after approval of production as well, as
Increment 1 was approved to enter development and production at the
same time. OSD approved the Increment 1 entry into development and
production in January 2005. A preliminary design review (PDR), which
best practices indicate should occur prior to a program entering
development, was not completed for the Increment 1 effort until
February 2006--13 months after the start of development. OSD also
approved Increment 2 development in January 2005. A PDR for Increment
2 had not occurred by the time a stop work order was placed on the
Increment 2 effort in December 2007--35 months after the start of
Increment 2 development. The stop work order was placed as a result of
a concern that the required work would exceed available funding.
Similarly, it is a best practice to achieve design stability at a
system-level critical design review (CDR), usually held midway through
system development--that is, halfway between the start of development
and approval to enter production. A CDR for Increment 1 was held in
August 2006 but was not completed as significant elements remained to
be addressed. Subsequently, an airframe CDR did not occur until
February 2007--more than 2 years after the start of Increment 1
development and production. A CDR was scheduled for Increment 2 to
occur in fiscal year 2008, but the stop work order for Increment 2 was
issued prior to the scheduled completion of the CDR. As knowledge from
the successive systems engineering reviews builds, uncertainty and
associated risks in the cost estimate decrease. As noted, this
knowledge was developed late, if at all, in the VH-71 program and the
consequence was cost growth to the degree that at the time the VH-71
program was terminated in June 2009, it had grown from a total
estimated cost of about $6.5 billion at the start of development in
January 2005 to about $13 billion.
The Program Did Not Make Appropriate Trade-offs between Capability
Requirements and Allowable Resources:
While good systems engineering can identify and inform trade-offs, the
customer and developer must be willing to make trade-offs to achieve a
successful business case. We have found that key to successful
developments was the ability to make early trade-offs either in the
design of the product or the customer's expectations to avoid
outstripping the resources available for product development.
Conversely, as we have found with other programs--such as the Armed
Reconnaissance Helicopter program--an unwillingness to make
performance trade-offs can contribute to programs being unexecutable,
ultimately resulting in their termination.
The VH-71 program was not afforded sufficient room needed to pursue
needed trade-offs. Stringent performance requirements (some with no
flexibility) were laid out for the system prior to the start of
development and did not appear to involve significant consideration of
trade-offs of cost, performance, and schedule negotiated between the
customer and the developer. Rather, as a July 2007 Institute for
Defense Analyses[Footnote 10] independent review team reported on the
executability of Increment 2:
"Unfortunately, the requirements for the program are still not well
founded. Initially this reflected a hurried program start, without the
foundation-laying analyses and design and requirements tradeoffs,
necessary to initiate a program with an acceptable level of stability.
Currently, incomplete requirements documentation and poor
configuration management presage further instability in the future.
The [independent review team] also observed that the two directions
given to the program by the Acquisition Decision Memorandum were not
followed:
* Leadership engagement and clear guidance were imperative for program
success.
* Disciplined application of systems engineering practices, early
configuration control decisions, and strict configuration control were
essential."
VXX Program Appears Set to Use a Knowledge-Based Acquisition Approach:
The VH-71 program experience validated the need to execute a knowledge-
based process with discipline and confirmed the danger of not
replacing risk with knowledge earlier in the acquisition process. VXX
program officials seem to understand this and appear to be moving
forward in an attempt to apply these lessons. They have stated that
their aim is to establish an initial knowledge-based acquisition
business case that will emphasize early systems engineering; mature
technologies; an incremental, knowledge-based approach; and the
ability to trade capability for resources. The program is currently in
its earliest stages, nearing completion of the materiel solution
analysis phase, and is likely still years away from establishing its
business case. Therefore, we do not have much acquisition
documentation to review beyond the ICD, information on the preliminary
acquisition planning, and AOA plans.
VXX program management indicated that one of the primary lessons
learned from the VH-71 program experience is that DOD must establish
and maintain a solid business case, where a balance exists between
requirements, cost, and schedule that results in an executable program
with acceptable risk. To accomplish this, the program intends to use a
rigorous four-phase systems engineering and technical review process
constituting a more knowledge-based acquisition approach. This process
differs greatly from the process followed during the VH-71 program.
According to VXX program officials, the VXX program's systems
engineering and technical review process will begin earlier during the
technology development phase (pre-Milestone B). VXX program officials
stated that they plan to enter the systems development phase with a
preliminary technical design and possibly an early prototype of
subsystems.
Early VXX program efforts appear to reflect the intent to pursue a
best practices-aligned knowledge-based acquisition. The VXX program is
currently in the pre-Milestone A[Footnote 11] phase of the acquisition
process. An ICD has been approved and an AOA is currently being
finalized; both are required for Milestone A. After termination of the
VH-71 program contract, DOD conducted an analysis to determine if a
nonmateriel solution existed to fill the identified gap in the
presidential helicopter program's vertical lift capability. DOD
determined that there were no nonmaterial approaches that would
significantly improve or resolve the capability gaps.[Footnote 12] It
was determined that a materiel solution must be pursued, and as a
result, an ICD for the VXX program was developed in August 2009.
According to the Chairman of the Joint Chiefs of Staff instruction
that guides the process that develops ICDs, an ICD is a formal
capability assessment by a military service, defense agency, or other
sponsor and it formally documents (1) the capabilities required to
perform the defined mission, (2) the specific capability gap or set of
gaps that exist, and (3) the need to address the capability gap(s).
[Footnote 13] Our review of the VXX program's ICD indicates that it
addressed all three of these areas. The ICD and its intended use also
appear to align with acquisition best practices. Unlike the
requirements document that led to the VH-71 program, the VXX program's
ICD identifies performance attributes as "preferred values" rather
than key performance parameters. Program officials indicated that the
use of preferred values instead of firm requirements will allow the
program room to do trade-offs needed to achieve a match between the
users' requirements and available resources before development begins--
an acquisition best practice.
According to a DOD official, the Navy expects to issue the results of
its AOA in the second quarter of 2011. According to DOD officials, it
will be more robust than the AOA used to support the VH-71 program.
While we cannot assess its sufficiency until it is complete,
statements by DOD officials on the nature of the AOA effort suggest
the pursuit of an acquisition approach in line with best practices.
Plans are for the AOA to analyze in detail the estimated cost and
effectiveness of a range of potential materiel solutions to support
the need. In the past, we have reported that many AOAs do not
effectively consider a broad range of alternatives for addressing a
need or assess technical and other risks associated with each
alternative.[Footnote 14] Without a sufficient comparison of
alternatives and focus on technical and other risks, AOAs may identify
solutions that are not feasible and decision makers may approve
programs based on limited knowledge. While many factors can affect
cost and schedule outcomes, we have found that programs that had a
limited assessment of alternatives tended to have poorer outcomes than
those that had more robust AOAs. When the AOA is issued, we will
assess it for robustness--the range of alternatives it considers, its
depth of analysis, and its consideration of trade-offs.
As the program progresses, we will continue to assess and report its
performance as directed in the Ike Skelton National Defense
Authorization Act for Fiscal Year 2011 (the Act). Figure 1 illustrates
the acquisition process and where the VXX program currently stands in
that process, and identifies some of the key program documents we will
review in the future and the best practices criteria that the program
should meet as it progresses. We will use this as a template for
future reviews of the program.
Figure 1: Alignment of DOD's Acquisition Process, Key Program
Documents, and Best Practices:
[Refer to PDF for image: illustration]
DOD Acquisition process: Materiel solution analysis:
* Materiel development decision;
* Current VXX status;
* Key program documents:
- ICD.
Milestone A:
Milestone A documents:
AOA;
Component cost estimate;
Systems engineering plan;
Technology development strategy;
Test and evaluation strategy.
DOD Acquisition process: Technology development:
* PDR;
Best practices knowledge-based acquisition model:
Knowledge Point 1: Technologies, time, funding and other resources
match customer needs. Decision to invest in product development.
Key steps:
* PDR completed;
* Technologies demonstrated to high levels;
* Incremental acquisition strategy in place;
* Knowledge-based cost estimate.
Milestone B: Development start;
Milestone B documents:
Acquisition strategy;
Capability development document;
Independent cost estimate;
Technology readiness assessment (TRA);
Test and evaluation master plan.
DOD Acquisition process: Engineering and manufacturing development:
Integrated system design:
CDR;
Knowledge Point 2: Design is stable and performs as expected. Decision
to start building and testing production representative prototypes.
Key steps:
* System-level CDR and subsystem design reviews completed;
* Ninety percent of engineering drawings released;
* Integrated system prototype demonstrated;
* Critical manufacturing processes identified.
DOD Acquisition process: Engineering and manufacturing development:
Capability and manufacturing demo:
Knowledge Point 3: Production meets cost, schedule, and quality
targets. Decision to produce first units for customer.
Key steps:
* Production-representative prototype demonstrated in intended
environment;
* Manufacturing processes in control;
* Product reliability demonstrated via production-representative
prototype testing.
Milestone C: Production start.
Milestone C documents:
Acquisition strategy;
Capability production document;
Independent cost estimate;
TRA;
Test and evaluation master plan.
DOD Acquisition process: Production.
Source: GAO analysis of DOD's acquisition process and GAO's best
practices.
[End of figure]
By using this framework, we will address the issues identified in the
Act. Specifically:
* the extent to which the program is meeting development and
procurement cost, schedule, performance, and risk mitigation goals;
* the progress of developmental and operational testing of aircraft
and plans for correcting deficiencies in aircraft performance,
operational effectiveness, reliability, suitability, and safety;
* the program procurement plans, production results, and efforts to
improve manufacturing efficiency and supplier performance;
* the program's acquisition strategy, including whether it is in
accord with acquisition best practices and DOD's acquisition policy
and regulations;
* risk assessments of its integrated master schedule and test and
evaluation master plan; and:
* our assessment of the sufficiency of the program's ICD (if revised),
AOA, capabilities development document, and a systems requirement
document (if produced).
Overall, we will assess the use of acquisition best practices
throughout the VXX program's acquisition through management's
adherence to key best practice model controls. Those controls are
outlined in the table in enclosure I.
Agency Comments:
DOD provided technical comments on the information in this report,
which GAO incorporated as appropriate, but declined to provide
additional comments.
We are sending copies of this report to interested congressional
committees; the Secretary of Defense; the Under Secretary of Defense
for Acquisition, Technology and Logistics; and the Secretary of the
Navy. This report also is available at no charge on GAO's Web site at
[hyperlink, http://www.gao.gov].
Should you or your staff have any questions on the matters covered in
this report, please contact me at (202) 512-4841 or sullivanm@gao.gov.
Contact points for our Offices of Congressional Relations and Public
Affairs may be found on the last page of this report. Key contributors
to this report are listed in enclosure II.
Signed by:
Michael J. Sullivan, Director:
Acquisition and Sourcing Management:
Enclosures - 2:
List of Committees:
The Honorable Carl Levin:
Chairman:
The Honorable John McCain:
Ranking Member:
Committee on Armed Services:
United States Senate:
The Honorable Daniel K. Inouye:
Chairman:
The Honorable Thad Cochran:
Ranking Member:
Subcommittee on Defense:
Committee on Appropriations:
United States Senate:
The Honorable Howard P. McKeon:
Chairman:
The Honorable Adam Smith:
Ranking Member:
Committee on Armed Services:
House of Representatives:
The Honorable C. W. Bill Young:
Chairman:
The Honorable Norman D. Dicks:
Ranking Member:
Subcommittee on Defense:
Committee on Appropriations:
House of Representatives:
[End of section]
Enclosure I: Best Practices Model Controls at Key Acquisition Process
Points:
Milestone A: Occurs as programs begin the technology development phase;
Criteria:
* Complete robust Analysis of Alternatives.
Milestone B (knowledge point 1): Occurs as programs begin the
engineering and manufacturing development phase (Milestone B). Match
exists between requirements and resources. Completed when technologies
needed to meet essential product requirements have been demonstrated
to work in their intended environments and the producer has completed
a preliminary design of the product;
Criteria:
* Demonstrate high technology readiness levels;
* Ensure that product requirements are informed by the systems
engineering process;
* Establish cost and schedule estimates for the product based on
knowledge from preliminary design using systems engineering tools;
* Complete preliminary design review;
* Conduct decision review for program launch.
Critical design review (knowledge point 2): Occurs at the critical
design review between integration and demonstration. Completed when
design is stable and has been demonstrated through prototype testing
and 90 percent of engineering drawings are releasable to manufacturing
organizations;
Criteria:
* Complete 90 percent of design drawings;
* Complete subsystem and system design reviews;
* Demonstrate with prototype that design meets requirements;
* Obtain stakeholder concurrence that drawings are complete and
producible;
* Complete failure modes and effects analysis;
* Identify key system; characteristics;
* Identify critical manufacturing processes;
* Establish reliability targets and growth plan based on demonstrated
reliability rates of components and subsystems;
* Conduct design review to enter system demonstration.
Milestone C (knowledge point 3): Occurs at low-rate initial production
commitment. Completed when product is ready to be manufactured within
cost, schedule, and quality targets and all key manufacturing
processes are under statistical control and product reliability has
been demonstrated;
Criteria:
* Demonstrate manufacturing processes;
* Build production-representative prototypes;
* Test production-representative prototypes to achieve reliability
goal;
* Test production-representative prototypes to demonstrate the product
in a realistic environment;
* Collect statistical process control data;
* Demonstrate that critical processes are capable and under
statistical control;
* Conduct decision review to begin production.
Source: GAO.
[End of table]
[End of section]
Enclosure II: GAO Contact and Staff Acknowledgments:
GAO Contact:
Michael J. Sullivan, (202) 512-4841 or sullivanm@gao.gov:
Staff Acknowledgments:
Key contributors to this report were Bruce H, Thomas, Assistant
Director; Jerry Clark, Analyst-in-Charge; Marie P. Ahearn; Michael
Aiken; Kenneth Patton; Don Springman; Robert Swierczek; and J. Andrew
Walker.
[End of section]
Related GAO Products:
Defense Acquisitions: Strong Leadership Is Key to Planning and
Executing Stable Weapon Programs. [hyperlink,
http://www.gao.gov/products/GAO-10-522]. Washington, D.C.: May 6, 2010.
Defense Acquisitions: Assessments of Selected Weapon Programs.
[hyperlink, http://www.gao.gov/products/GAO-10-388SP]. Washington,
D.C.: March 30, 2010.
Defense Acquisitions: Many Analyses of Alternatives Have Not Provided
a Robust Assessment of Weapon System Options. [hyperlink,
http://www.gao.gov/products/GAO-09-665]. Washington, D.C.: September
24, 2009.
Defense Acquisitions: Measuring the Value of DOD's Weapon Programs
Requires Starting with Realistic Baselines. [hyperlink,
http://www.gao.gov/products/GAO-09-543T]. Washington, D.C.: April 1,
2009.
Defense Acquisitions: Assessments of Selected Weapon Programs.
[hyperlink, http://www.gao.gov/products/GAO-09-326SP]. Washington,
D.C.: March 30, 2009.
Defense Acquisitions: DOD Must Balance Its Needs with Available
Resources and Follow an Incremental Approach to Acquiring Weapon
Systems. [hyperlink, http://www.gao.gov/products/GAO-09-431T].
Washington, D.C.: March 3, 2009.
GAO Cost Estimating and Assessment Guide: Best Practices for
Developing and Managing Capital Program Costs. [hyperlink,
http://www.gao.gov/products/GAO-09-3SP]. Washington, D.C.: March 2009.
Defense Acquisitions: Perspectives on Potential Changes to Department
of Defense Acquisition Management Framework. [hyperlink,
http://www.gao.gov/products/GAO-09-295R]. Washington, D.C.: February
27, 2009.
Defense Acquisitions: A Knowledge-Based Funding Approach Could Improve
Major Weapon System Program Outcomes. [hyperlink,
http://www.gao.gov/products/GAO-08-619]. Washington, D.C.: July 2,
2008.
Defense Acquisitions: Assessments of Selected Weapon Programs.
[hyperlink, http://www.gao.gov/products/GAO-08-467SP]. Washington,
D.C.: March 31, 2008.
Defense Acquisitions: Assessments of Selected Weapon Programs.
[hyperlink, http://www.gao.gov/products/GAO-07-406SP]. Washington,
D.C.: March 30, 2007.
Defense Acquisitions: Assessments of Selected Major Weapon Programs.
[hyperlink, http://www.gao.gov/products/GAO-06-391]. Washington, D.C.:
March 31, 2006.
Defense Acquisitions: Major Weapon Systems Continue to Experience Cost
and Schedule Problems under DOD's Revised Policy. [hyperlink,
http://www.gao.gov/products/GAO-06-368]. Washington, D.C.: April 13,
2006.
Best Practices: Better Support of Weapon System Program Managers
Needed to Improve Outcomes. [hyperlink,
http://www.gao.gov/products/GAO-06-110]. Washington, D.C.: November
30, 2005.
Best Practices: Capturing Design and Manufacturing Knowledge Early
Improves Acquisition Outcomes. [hyperlink,
http://www.gao.gov/products/GAO-02-701]. Washington, D.C.: July 15,
2002.
Best Practices: Better Matching of Needs and Resources Will Lead to
Better Weapon System Outcomes. [hyperlink,
http://www.gao.gov/products/GAO-01-288]. Washington, D.C.: March 8,
2001.
Best Practices: A More Constructive Test Approach Is Key to Better
Weapon System Outcomes. [hyperlink,
http://www.gao.gov/products/GAO/NSIAD-00-199]. Washington, D.C.: July
31, 2000.
Best Practices: Better Management of Technology Development Can
Improve Weapon System Outcomes. [hyperlink,
http://www.gao.gov/products/GAO/NSIAD-99-162]. Washington, D.C.: July
30, 1999.
[End of section]
Footnotes:
[1] 10 U.S.C. § 2433 establishes the requirement for the Department of
Defense (DOD) to prepare unit cost reports on major defense
acquisition programs or designated major defense subprograms. If a
program exceeds cost growth thresholds specified in the law, this is
commonly referred to as a Nunn-McCurdy breach. DOD is required to
report these breaches to Congress, and in certain circumstances, DOD
must reassess the program and submit a certification to Congress in
order to continue the program, in accordance with 10 U.S.C. § 2433a.
The Nunn-McCurdy breach experienced by the VH-71 program was estimated
to represent program acquisition unit cost (PAUC) growth of more than
50 percent above the original acquisition program baseline--a level
that would have required certification to Congress for continuation of
the program if the program had not been terminated. PAUC represents
the expected acquisition cost for each unit procured, as determined by
dividing the sum of a program's total program development,
procurement, and system-specific military construction costs by the
number of fully configured end items to be procured.
[2] A memorandum canceling the program was issued in May 2009.
Subsequently, a notice terminating the contract was sent to the
contractor in June 2009. In accordance with the termination clause,
the contractor was required to submit a termination settlement
proposal within 1 year. The contractor submitted its termination
proposal in May 2010. The termination proposal is currently being
audited and a final settlement is expected by the end of fiscal year
2011.
[3] Pub. L. No. 111-383 § 233.
[4] We have defined sufficiency as meaning objective and in accord
with acquisition best practices and DOD acquisition policy and
guidance.
[5] A sponsor, usually a military service, submits a capability
proposal called an ICD through DOD's requirements determination
process--the Joint Capabilities Integration and Development System. An
ICD identifies the existence of capability gap(s), the operational
risks associated with the gap(s), and a recommended solution or
preferred set of solutions for filling the gap(s).
[6] An AOA is an evaluation of the performance, effectiveness,
suitability, and estimated costs of alternative systems to meet a
capability.
[7] In December 2008, DOD revised its acquisition instruction--
Department of Defense Instruction 5000.02, Operation of the Defense
Acquisition System. The Weapon Systems Acquisition Reform Act of 2009,
Pub. L. No. 111-23, was enacted on May 22, 2009.
[8] If done correctly, the use of an incremental approach to
development is a best practice that our work has shown enables
organizations to achieve a match between needs and resources. Under
this approach, basic requirements are achieved first, with additional
capabilities planned for future generations of the product. Because
product development is incremental, achieving knowledge is more
manageable. As we have reported, commercial companies have found that
trying to capture the knowledge needed to stabilize the design of a
product with considerable new technical content is an unwieldy task--
especially if the goal is to reduce development cycle times and get
the product to the marketplace as quickly as possible. GAO, Best
Practices: Using a Knowledge-Based Approach to Improve Weapon
Acquisition, [hyperlink, http://www.gao.gov/products/GAO-04-386SP]
(Washington, D.C.: January 2004).
[9] GAO, Defense Acquisitions: Strong Leadership Is Key to Planning
and Executing Stable Weapon Programs, [hyperlink,
http://www.gao.gov/products/GAO-10-522] (Washington, D.C.: May 6,
2010).
[10] Institute for Defense Analyses, VH-71 Presidential Helicopter
Program Assessment, IDA Paper P-4243 (Alexandria, Va., July 2007)
(Distribution authorized to U.S. government agencies and their
contractors; Specific Authority. Other requests must be referred to
OUSD(AT&L).)
[11] The VXX Milestone A is currently expected to occur sometime in
fiscal year 2011.
[12] The analysis typically evaluates military doctrine, organization,
training, materiel, leadership and education, personnel, and
facilities to try to identify a nonmaterial solution to a capability
gap or gaps.
[13] Chairman of the Joint Chiefs of Staff Instruction 3170.01G, Joint
Capabilities Integration and Development System (2009), encl. A, 2(d).
When the Joint Requirements Oversight Council approves an ICD, it
validates the capabilities required to perform the mission as defined,
the gaps in capabilities along with their priorities and operational
risks, and the need to address the capability gaps.
[14] GAO, Defense Acquisitions: Many Analyses of Alternatives Have Not
Provided a Robust Assessment of Weapon System Options, [hyperlink,
http://www.gao.gov/products/GAO-09-665] (Washington, D.C.: Sept. 24,
2009).
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
