Missile Defense
Actions Needed to Improve Information for Supporting Future Key Decisions for Boost and Ascent Phase Elements Gao ID: GAO-07-430 April 17, 2007The Department of Defense (DOD) has spent about $107 billion since the mid-1980s to develop a capability to destroy incoming ballistic missiles. DOD has set key decision points for deciding whether to further invest in capabilities to destroy missiles during the initial phases after launch. In March 2006, DOD issued a report on these capabilities in response to two mandates. To satisfy a direction from the House Appropriations Committee, GAO agreed to review the report. To assist Congress in evaluating DOD's report and preparing for future decisions, GAO studied the extent to which DOD (1) analyzed technical and operational issues and (2) presented complete cost information. To do so, GAO assessed the report's methodology, explanation of assumptions and their effects on results, and whether DOD followed key principles for developing life-cycle costs.
The report DOD's Missile Defense Agency (MDA) submitted to Congress in March 2006 included some useful technical and operational information on boost and ascent phase capabilities by describing these elements, listing upcoming decision points, and discussing geographic areas where boost and ascent elements could intercept missiles shortly after launch. However, the information in the report has several limitations because the analysis did not involve key DOD stakeholders such as the services and combatant commands in preparing the report and did not clearly explain modeling assumptions and their effects on results as required by relevant research standards. MDA's report states that, at this time, some data is limited, and operational concepts that discuss operations from forward locations have not been fully vetted with the services and combatant commands. However, the report did not explain how each element's performance may change if developing technologies do not perform as expected. Also, it did not address the challenges in establishing bases at the locations cited or provide information on the quantity of each element required for various deployment periods. Moving forward, DOD has an opportunity to involve stakeholders in analyzing operational and technical issues so that senior DOD and congressional leaders will have more complete information on which to base upcoming program decisions following key tests in 2008 and 2009 for the Kinetic Energy Interceptor and Airborne Laser boost and ascent phase programs. MDA's report provided some cost estimates for developing and fielding boost and ascent phase capabilities, but these estimates have several limitations and will require refinement before they can serve as a basis for DOD and congressional decision makers to compare life-cycle costs for the elements. MDA's report states that there is uncertainty in estimating life-cycle costs because the elements are early in development. However, based on a comparison of the estimates in the report with key principles for developing life-cycle cost estimates, GAO found that MDA's estimates did not include all cost categories, including costs to establish and sustain operations at U.S. bases and at forward overseas operating locations. Also, MDA's estimates did not calculate costs based on realistic quantities of each element the combatant commanders or services would need to conduct the mission. Finally, MDA did not conduct a sensitivity analysis to assess the effect of key cost drivers on total costs. MDA officials stated that further analysis of the costs for each element along with measures to assess their confidence would help to better inform DOD and congressional decision makers in making investment decisions following key tests in 2008 and 2009.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-07-430, Missile Defense: Actions Needed to Improve Information for Supporting Future Key Decisions for Boost and Ascent Phase Elements
This is the accessible text file for GAO report number GAO-07-430
entitled 'Missile Defense: Actions Needed to Improve Information for
Supporting Future Key Decisions for Boost and Ascent Phase Elements'
which was released on April 17, 2007.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Committees:
United States Government Accountability Office:
GAO:
April 2007:
Missile Defense:
Actions Needed to Improve Information for Supporting Future Key
Decisions for Boost and Ascent Phase Elements:
GAO-07-430:
GAO Highlights:
Highlights of GAO-07-430, a report to congressional committees
Why GAO Did This Study:
The Department of Defense (DOD) has spent about $107 billion since the
mid-1980s to develop a capability to destroy incoming ballistic
missiles. DOD has set key decision points for deciding whether to
further invest in capabilities to destroy missiles during the initial
phases after launch. In March 2006, DOD issued a report on these
capabilities in response to two mandates. To satisfy a direction from
the House Appropriations Committee, GAO agreed to review the report. To
assist Congress in evaluating DOD‘s report and preparing for future
decisions, GAO studied the extent to which DOD (1) analyzed technical
and operational issues and (2) presented complete cost information. To
do so, GAO assessed the report‘s methodology, explanation of
assumptions and their effects on results, and whether DOD followed key
principles for developing life-cycle costs.
What GAO Found:
The report DOD‘s Missile Defense Agency (MDA) submitted to Congress in
March 2006 included some useful technical and operational information
on boost and ascent phase capabilities by describing these elements,
listing upcoming decision points, and discussing geographic areas where
boost and ascent elements could intercept missiles shortly after
launch. However, the information in the report has several limitations
because the analysis did not involve key DOD stakeholders such as the
services and combatant commands in preparing the report and did not
clearly explain modeling assumptions and their effects on results as
required by relevant research standards. MDA‘s report states that, at
this time, some data is limited, and operational concepts that discuss
operations from forward locations have not been fully vetted with the
services and combatant commands. However, the report did not explain
how each element‘s performance may change if developing technologies do
not perform as expected. Also, it did not address the challenges in
establishing bases at the locations cited or provide information on the
quantity of each element required for various deployment periods.
Moving forward, DOD has an opportunity to involve stakeholders in
analyzing operational and technical issues so that senior DOD and
congressional leaders will have more complete information on which to
base upcoming program decisions following key tests in 2008 and 2009
for the Kinetic Energy Interceptor and Airborne Laser boost and ascent
phase programs.
MDA‘s report provided some cost estimates for developing and fielding
boost and ascent phase capabilities, but these estimates have several
limitations and will require refinement before they can serve as a
basis for DOD and congressional decision makers to compare life-cycle
costs for the elements. MDA‘s report states that there is uncertainty
in estimating life-cycle costs because the elements are early in
development. However, based on a comparison of the estimates in the
report with key principles for developing life-cycle cost estimates,
GAO found that MDA‘s estimates did not include all cost categories,
including costs to establish and sustain operations at U.S. bases and
at forward overseas operating locations. Also, MDA‘s estimates did not
calculate costs based on realistic quantities of each element the
combatant commanders or services would need to conduct the mission.
Finally, MDA did not conduct a sensitivity analysis to assess the
effect of key cost drivers on total costs. MDA officials stated that
further analysis of the costs for each element along with measures to
assess their confidence would help to better inform DOD and
congressional decision makers in making investment decisions following
key tests in 2008 and 2009.
What GAO Recommends:
To support future decisions, DOD should include key stakeholders in
assessing operational issues, report on technical progress, and update
and verify life-cycle cost estimates in accordance with key principles
for developing life-cycle costs. In comments on a draft of this report,
DOD agreed to include stakeholders and assess technical progress but
did not agree to prepare or report life-cycle costs in accordance with
key principles.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-430].
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Janet A. St. Laurent at
(202) 512-4402 or stlaurentj@gao.gov; or Paul Francis at (202) 512-2811
or francisp@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
MDA's March 2006 Report Includes Some Useful Information but Has
Several Limitations:
DOD Can Significantly Improve Life-Cycle Cost Estimates to Support
Future Investment and Budget Decisions for Boost and Ascent Phase
Capabilities:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Comments from the Department of Defense:
Appendix III: GAO Contacts and Staff Acknowledgments:
Related GAO Products:
Table:
Table 1: Ballistic Missile Defense Boost and Ascent Phase Elements, Key
Decision Points, and Their Planned Capability Dates:
Abbreviations:
ABL: Airborne Laser:
BMDS: Ballistic Missile Defense System:
DOD: Department of Defense:
FYDP: Future Years Defense Program:
KEI: Kinetic Energy Interceptor:
MDA: Missile Defense Agency:
United States Government Accountability Office:
Washington, DC 20548:
April 17, 2007:
Congressional Committees:
The new security environment includes some states, such as North Korea,
that are aggressively pursuing development of long-range missiles and
weapons of mass destruction. In 2002, President Bush directed that the
Department of Defense (DOD) proceed with plans to develop and put in
place an initial set of ballistic missile defense capabilities
beginning in 2004. DOD has spent about $107 billion since the mid-1980s
to develop a capability to destroy incoming ballistic missiles in all
phases of their flight, including the initial phases after launch,
called the boost and ascent phases.[Footnote 1] The purpose of boost
and ascent phase elements of a ballistic missile defense system (BMDS)
is to engage incoming missiles before release of warheads or
countermeasures. DOD is developing three BMDS boost and ascent phase
capabilities: Airborne Laser (boost phase only), Kinetic Energy
Interceptor (boost and ascent phase), and the Aegis Ballistic Missile
Defense element that will fire the Standard Missile-3 block 2A[Footnote
2] (ascent phase) from BMDS-capable Aegis surface combatants. According
to the Missile Defense Agency (MDA), the agency responsible for
developing an integrated ballistic missile defense system, DOD plans to
spend $1.6 billion for the Airborne Laser; $800 million for the Kinetic
Energy Interceptor; and $200 million for the Aegis Standard Missile-3
Block 2A during fiscal years 2006-08. The long-term costs for these
elements will be much greater if DOD continues their development and
fielding. DOD projects the initial boost and ascent phase capabilities
will not be available until about 2014 through 2020. Congress has
raised questions about the affordability of pursuing both the Airborne
Laser and Kinetic Energy Interceptor programs.
Over the next few years, DOD and Congress will be asked to make
important program and investment decisions on boost and ascent phase
capabilities--at a time when the federal government is facing a large
and growing fiscal imbalance.[Footnote 3] Further, we have identified
the competing demands of investing in transformational capabilities
while continuing to invest in legacy systems as one of the most
significant national defense challenges for the 21st century.[Footnote
4] MDA has established key decision points at which it will decide
whether to continue to invest in certain elements of the integrated
system. For example, one key decision point is the booster flight test
for the Kinetic Energy Interceptor element, scheduled for 2008, at
which time a decision will be made on the program's future. A key
decision point for the Airborne Laser is the lethal shootdown
demonstration scheduled for 2009.[Footnote 5] Finally, in 2009, MDA
plans a motor test for the Standard Missile-3 block 2A.
In March 2006, MDA issued a report on boost and ascent phase
capabilities in response to two mandates--one in the House
Appropriations Committee Report on the Defense Appropriations Bill for
Fiscal Year 2006 and one in the National Defense Authorization Act for
Fiscal Year 2006. The House Appropriations Committee directed the
Secretary of Defense to conduct a study on boost and ascent phase
intercepts including an assessment of operational capabilities,
quantities of operational assets required for various deployment
periods, basing options, and an assessment of life-cycle
costs.[Footnote 6] Life-cycle costs are the total cost to the
government for a program over its full life, consisting of research and
development, production, operations, maintenance, and disposal costs
and are helpful in assessing whether a system's cost is affordable. The
Department of Defense Authorization Act for Fiscal Year 2006 required
the Secretary of Defense to conduct an assessment of U.S. missile
defense programs that are designed to provide capability against threat
ballistic missiles in the boost/ascent phase. The purpose of the
assessment was to compare and contrast capabilities, asset
requirements, and the costs for making the boost and ascent phase
programs operational.[Footnote 7] MDA, on behalf of DOD, prepared one
report to satisfy both requirements and sent the report to all four
defense committees on March 30, 2006. According to MDA officials, that
report was prepared specifically to satisfy the congressional
requirements, not for use at any key decision point.
To satisfy a direction from the House Appropriations Committee report,
we agreed to review the DOD report including assessing the report's
methodologies, assumptions, completeness, and results.[Footnote 8] This
report is our assessment of the March 2006 MDA report and how DOD can
build on this information to support future key decision points.
Accordingly, we assessed the extent to which (1) information reported
by DOD includes analysis of technical and operational issues and
whether any additional information is needed to support future decision
making and (2) DOD presented cost information to Congress that is
complete and transparent. To assess the technical and operational
information, we analyzed the extent to which the MDA report clearly
explained technical maturity, modeling assumptions and their effect on
results, and involved stakeholders in analyses of operational issues--
steps which contribute to a sound and complete quality study and are
embodied in relevant generally accepted research standards[Footnote 9].
To assess the cost information in the March 2006 report, we compared
MDA's cost data with key principles compiled from various DOD and GAO
sources that describe how to develop accurate and reliable life-cycle
cost estimates which are to include development, production, and all
direct and indirect operating and support costs. We discussed the data
and results of this comparison with DOD officials and identified some
limitations, which we discuss in this report. We conducted our work
between June 2006 and February 2007 in accordance with generally
accepted government auditing standards. See appendix I for a more
complete description of our scope and methodology.
This report is one in a series we have issued on ballistic missile
defense (see the list of related GAO products at the end of this
report). Some of these reports have focused on assessments of program
goals and progress in developing each element. For example, our March
2006 report found that MDA made progress in the development and
fielding of some elements but that time pressures caused MDA to stray
from a knowledge-based acquisition strategy.[Footnote 10] This is
significant because straying from a knowledge-based approach opens the
door to greater cost and performance risks by not having the right
information available for decision makers at the right time.[Footnote
11] Another report focused on DOD's planning to operate BMDS. We
reported in May 2006 that DOD has not established criteria that would
have to be met before declaring BMDS operational and that the Future
Years Defense Program (FYDP), a major source of budget information,
does not provide complete and transparent data on ballistic missile
defense operational costs.[Footnote 12]
Results in Brief:
MDA's 2006 report to Congress included some useful technical and
operational information on boost and ascent phase capabilities but the
information in the report has several limitations. Specifically, MDA's
report provided some key information on potential boost and ascent
phase capabilities by describing these elements, listing upcoming
decision points, and discussing geographic areas where these elements
could intercept missiles shortly after launch. However, the analysis
did not involve DOD stakeholders such as the services and combatant
commands, which will have a key role in operating the elements, and did
not clearly explain modeling assumptions and their effects on results
as identified by relevant research standards. For example, the report
assumed that each element would perform as desired and that the
elements could be deployed at optimum locations, but it did not explain
how each element's performance would change if developing technologies
do not perform as expected. In addition, the report did not address the
challenges in establishing bases at the locations cited or provide
information on the quantity of each element required for various
deployment periods. Two key reasons for these limitations, which the
report acknowledges, are that some data on the elements are limited at
this time, and operational concepts that discuss operations from
forward locations have not been fully vetted with the services and
combatant commands. However, moving forward, DOD has an opportunity to
involve stakeholders in analyzing operational and technical issues to
provide more complete data that senior DOD and congressional leaders
can use to make informed program decisions following key tests in 2008
and 2009 for the Kinetic Energy Interceptor and Airborne Laser
programs. To provide decision makers with information to support boost
and ascent phase program decisions at future key decision points, we
are recommending that DOD include all DOD stakeholders in developing
and analyzing operational issues regarding what is needed to support
operations at U.S. bases and potential forward locations, provide
specific information on the technical progress of each element, and use
the results of these analyses at each key decision point.
The Missile Defense Agency provided some cost estimates for developing
and fielding each of the boost and ascent phase capabilities in its
March 2006 report to Congress; however, these estimates have several
limitations and will require refinement before they can serve as a
basis for DOD and congressional decision makers to compare the costs of
boost and ascent phase alternatives. We compared the report's cost
estimates with various DOD and GAO sources that describe key principles
for developing accurate and reliable life-cycle cost estimates. Based
on our analysis, we found that the estimates provided in the report did
not include all cost categories, including costs to establish and
sustain operations at U.S. bases and at forward locations. Also, MDA
did not calculate costs based on quantities the warfighter would need
to conduct the mission, or conduct a sensitivity analysis identifying
the effects of cost drivers. Although the report acknowledges that, at
this time, there is uncertainty in estimating life-cycle costs for
these elements, it did not fully disclose the limitations of the cost
estimates. DOD can significantly improve the completeness of and
confidence in its cost estimates for boost and ascent phase
capabilities as it prepares for future investment and budget decisions.
For example, although MDA did not have the cost estimates in its March
2006 report independently verified because doing so would have required
several months, MDA officials agree that independent verification,
which is another key principle of reliable cost estimates, will be
critical to support major decision points. We are recommending that DOD
prepare, periodically update, and independently verify life-cycle cost
estimates for boost and ascent phase elements in accordance with key
principles for developing accurate and reliable life-cycle cost
estimates and that these verified estimates be used for making
investment decisions and be reported with budget requests.
In commenting on a draft of this report, DOD agreed with our
recommendations regarding the need for analysis of technical progress
and operational issues to support key boost and ascent phase element
decision points. DOD also partially concurred that an independent life-
cycle cost estimate may be needed to inform some key decision points
but said that they may not be needed at others. However, DOD did not
agree to prepare and periodically update full life-cycle cost estimates
for each boost and ascent phase element to support key decision points,
and report independently verified life cycle cost estimates with budget
requests and FYDP funding plans. In its comments, DOD stated that it
will direct an independent evaluation of life-cycle costs if
circumstances warrant or the Director of MDA declares an element mature
enough to provide a militarily useful capability. However, if, as DOD's
comments suggest, such costs are not assessed until circumstances
warrant or MDA's Director makes such a declaration, these costs may not
be available early enough to help shape important program and
investment decisions and consider trade-offs among elements. We
continue to believe our recommendation has merit because the
development of life-cycle cost estimates that include potential
operations and support costs would improve the information available to
decision makers and increase accountability for key decisions that
could involve billions of dollars at a time when DOD will likely face
competing demands for resources. Finally, DOD did not agree to report
independently verified life-cycle cost estimates along with budget
requests and FYDP funding plans since the development of total life-
cycle cost estimates for operationalized BMDS capabilities require
agreement between MDA and the lead military department on the roles and
responsibilities for fielded missile defense capabilities. However,
independent verification allows decision makers to gauge whether the
program is executable. Also, DOD procedures specify that when cost
estimates are presented to the Office of the Secretary of Defense's
Cost Analysis Improvement Group, life-cycle cost estimates should be
compared with the FYDP and differences explained. Therefore, we
continue to believe that our recommendation has merit because, without
an independent cost estimate that can be compared to budget requests
and FYDP funding plans, congressional decision makers may not have all
the necessary information to assess the full extent of future resource
requirements if the boost and ascent phase capabilities go forward, or
assess the completeness of budget requests and FYDP funding plans.
Background:
In 2002, the Secretary of Defense created MDA to develop an integrated
system that would have the ability to intercept incoming missiles in
all phases of their flight.[Footnote 13] In developing BMDS, MDA is
using an incremental approach to field militarily useful capabilities
as they become available. MDA plans to field capabilities in 2-year
blocks. The configuration of a given block is intended to build on the
work completed in previous blocks. For example, Block 2006 is intended
to build on capabilities developed in Block 2004, and is scheduled to
field capabilities during calendar years 2006-07.
The integrated BMDS is comprised of various elements, three of which
are intended to intercept threat missiles in their boost or ascent
phase. Table 1 below describes each of these elements and shows the MDA
projected dates for key decision points, initial capability, and tested
operational capability.
Table 1: Ballistic Missile Defense Boost and Ascent Phase Elements, Key
Decision Points, and Their Planned Capability Dates:
Boost/ascent phase element: Standard Missile-3, block 2A[A];
Description: Aegis Ballistic Missile Defense, which employs the
Standard Missile, is a ship-based system designed to destroy medium,
intermediate, and certain intercontinental ballistic missiles in the
ascent and midcourse phases;
Key decision points: 2009--motor test for the 21-inch diameter missile;
Initial capability: Block 2012;
Tested capability: Block 2014.
Boost/ascent phase element: Kinetic Energy Interceptor (KEI)--land-
based;
Description: KEI is being designed to destroy medium, intermediate, and
intercontinental ballistic missiles during the boost, ascent, and
midcourse phases of flight. A land-based unit may consist of 5
launchers, 10 missiles, and fire control and communications equipment;
Key decision points: 2008--booster flight test;
Initial capability: Block 2014;
Tested capability: Block to be decided.
Boost/ascent phase element: Kinetic Energy Interceptor--sea-based;
Description: KEI is being designed to destroy medium, intermediate, and
intercontinental ballistic missiles during the boost, ascent, and
midcourse phases of flight. DOD is currently studying sea-based
platform alternatives including surface combatants, submarines, or a
new platform;
Key decision points: 2007--platform selection; 2008-- booster flight
test;
Initial capability: Block to be decided;
Tested capability: Block to be decided.
Boost/ascent phase element: Airborne Laser (ABL);
Description: ABL is an air-based missile defense system designed to
destroy all classes of ballistic missiles during the boost phase. ABL
also has onboard sensor capability designed to detect, track, and
transmit targeting data;
Key decision points: 2009--lethal shootdown demonstration;
Initial capability: Block 2016;
Tested capability: Block 2018.
Source: GAO summary of DOD information.
[A] The Standard Missile is intended to be used on BMDS upgraded Aegis
cruisers and destroyers. Other versions of the Standard Missile-3 are
being developed to intercept threat missiles in their midcourse phase.
[End of table]
During the past year, Congress requested additional information and
analyses on the boost and ascent phase elements from DOD. Specifically,
House Report 109-119 on the Department of Defense Appropriations Bill
for Fiscal Year 2006 directed the Secretary of Defense to conduct a
study to review the early engagement of ballistic missiles to include
boost and ascent phase intercepts and submit the report to the
congressional defense committees.[Footnote 14] The report was to
include, but not be limited to:
* an assessment of the operational capabilities of systems against
ballistic missiles launched from North Korea or a location in the
Middle East against the continental United States, Alaska, or Hawaii;
* an assessment of the quantity of operational assets required for
deployment periods of 7 days, 30 days, 90 days, and 1 year;
* basing options; and:
* an assessment of life-cycle costs[Footnote 15] to include research
and development efforts, procurement, deployment, operating, and
infrastructure costs.
In addition, the National Defense Authorization Act for Fiscal Year
2006 required the Secretary of Defense to assess missile defense
programs designed to provide capability against threat ballistic
missiles in the boost/ascent phase of flight. The purpose of this
assessment was to compare and contrast:
* capabilities of those programs (if operational) to defeat ballistic
missiles from North Korea or a location in the Middle East against the
continental United States, Alaska, or Hawaii; and:
* asset requirements and costs for those programs to become operational
with the capabilities referred to above.
MDA, on behalf of DOD, prepared one report to satisfy both of the above
requirements and sent the report to all four defense committees on
March 30, 2006. The report included technical, operational, and cost
information for each of the three boost and ascent phase BMDS elements.
The remainder of this report discusses our assessment of the MDA report
and how DOD can build on this information to support future key
decision points.
MDA's March 2006 Report Includes Some Useful Information but Has
Several Limitations:
MDA's March 2006 report to Congress included some useful technical and
operational information on boost and ascent phase capabilities.
However, the information in the report has several limitations--such as
not including stakeholders in the analysis or explaining how
assumptions affect results. Moving forward, DOD can enhance its ability
to make informed decisions at future key decision points by including
stakeholders DOD-wide in conducting analyses to provide complete
technical and operational information. Otherwise, senior DOD and
congressional decision makers may be limited in their ability to
effectively assess the technical progress and operational effects of
proceeding with one or more boost and ascent phase element.
Report Provided Some Basic Information about Boost and Ascent Phase
Elements:
The March 2006 report to Congress contained some useful technical and
operational information for Congress. For example, the report included
a detailed description of the three boost and ascent phase elements,
which could be useful for those unfamiliar with these elements.
Additionally, the report listed upcoming knowledge points where DOD
will review the progress MDA has made toward developing each of the
boost and ascent phase elements. Further, the report discussed
geographic areas where boost and ascent phase elements could intercept
missiles shortly after launch based on desired technical capabilities.
Also, MDA used a model[Footnote 16] to assess the desired capabilities
of each BMDS element for the March 2006 report to Congress. Further,
the modeling environment was used for several past BMDS analyses and
the results were benchmarked against other models. Finally, MDA
performed a sensitivity analysis that compared how the results in the
modeling changed when different assumptions for targets' propellants,
ascent times, hardness levels, and burn times were used. To provide
context, the report explained that the boost and ascent phase elements
are in the early stages of development and that the operational
concepts are not yet mature.
MDA's Analyses Have Limitations but DOD Can Improve Information to
Support Future Decisions:
The information in the March 2006 report has several limitations
because the analyses did not involve stakeholders and did not clearly
explain modeling assumptions and their effects on results as identified
by relevant research standards. The relevant research standards and our
prior work have shown that coordination with stakeholders from study
design through reporting, and clearly explained assumptions and their
effects on results, can enable DOD officials to make fully informed
program decisions. As a result, the March 2006 report presents an
incomplete picture of technical capabilities, such as development
challenges to be overcome in order to achieve desired performance, and
it does not clearly explain the effects of operational assumptions,
such as basing locations, asset quantities, and base support
requirements. As a step in the right direction, MDA stated that it
plans to develop criteria to assess the boost/ascent phase elements at
major decision points in a process involving the combatant commands.
Although MDA officials told us that they consult stakeholders in a
variety of forums other than the March 2006 report, they did not
clearly state whether or how the services or other DOD stakeholders
would be involved in developing criteria for key decision points or the
extent to which their analyses would include information on technical
and operational issues.
Data Collection and Analysis for the March 2006 Report Did Not Involve
Key DOD Stakeholders:
MDA's analyses did not involve soliciting or using information from key
DOD stakeholders such as the services, combatant commands, and joint
staff from study design through reporting. For example, officials from
the Office of the Secretary of Defense for Program Analysis and
Evaluation and the Defense Intelligence Agency stated there were areas
where additional information would have improved the fidelity of the
results. First, the officials stated that there is uncertainty that the
boost and ascent phase elements would achieve their desired
capabilities within the timeframe stated in the report. Second,
officials from both organizations stated that the report could have
been enhanced by presenting different views of the type and capability
of threats the United States could face and when these threats could
realistically be expected to be used by adversaries. Third, officials
from the Office of the Secretary of Defense for Program Analysis and
Evaluation said that the MDA report did not distinguish between
countermeasures that could be used in the near term and countermeasures
that may be more difficult to implement. MDA officials said that they
worked with the Office of the Secretary of Defense for Program Analysis
and Evaluation in conducting analyses before they began work on the
March 2006 report. MDA also stated that it discussed the draft March
2006 report with Office of the Secretary of Defense for Program
Analysis and Evaluation officials and included some of their comments
in the report's final version. However, without communication with
stakeholders from study design through reporting, MDA may not have had
all potential inputs that could have affected how the type, capability,
and likelihood of countermeasures to the boost and ascent phase
elements were presented in its report.
Additionally, MDA did not solicit information from the services,
combatant commands, or Joint Staff regarding operational issues that
could have affected information about basing and the quantities of
elements that could be required to support operations. Although the
elements have to be located in close proximity to their intended
targets, and the report discusses placing the elements at specific
forward overseas locations, the report does not include a basing
analysis explaining what would need to be done to support operations at
these locations. Specifically, the report did not include any
discussion of the infrastructure or security/force protection that will
be needed for the BMDS elements. Although the report mentions some
support requirements--such as the Airborne Laser's need for unique
maintenance and support equipment and skilled personnel to maintain the
laser--the report did not fully explain how these support requirements
would be determined, who would provide or fund them, or explain the
operational effect if this support is not provided. For instance,
without an adequate forward operating location, the boost and ascent
phase elements would have to operate from much further away which would
significantly limit the time an element is in close proximity to
potential targets. Developing such information with the services, Joint
Staff, and combatant commands could provide a much more complete
explanation of operational issues and challenges. The services
typically perform site analyses to ascertain what support is needed for
a new weapon system at either a U.S. or overseas location. This
comprehensive analysis examines a range of issues from fire protection
to security, to infrastructure, to roads and airfields. In addition,
U.S. Strategic Command and service officials told us that this type of
support must be planned for in advance when adding a new system to any
base, either in the United States or a forward location.
MDA also did not involve stakeholders in assessing the quantities of
each element for deployment periods of 7 days, 30 days, 90 days, and 1
year. The report stated that limited data exist at this time for a full
assessment of this issue, and service, Joint Staff, and MDA officials
acknowledged that the quantities of each element used in the report are
MDA assumed quantities. Service, Joint Staff, and U.S. Strategic
Command officials stated that they have not completed analyses to
assess quantities the warfighters may require. We understand that
operational concepts will continue to evolve and could affect required
quantities. However, stakeholders such as the services, Joint Staff, or
combatant commands could have assisted MDA in assessing potential
quantities required for various deployment periods. In addition, MDA
did not solicit information from the services, Joint Staff, or
combatant commands to determine if those organizations were conducting
force structure analyses for the boost and ascent phase elements. We
learned that the Navy had done a preliminary analysis in July 2005 and
that the Joint Staff has begun a capabilities mix study and both
include, in part, an analysis of quantities. Thus, in preparing for
future decision points, MDA's analysis could be strengthened by
including stakeholders to leverage other analyses. For example, MDA
could have presented a range of scenarios to show how the quantities
required to intercept adversary missiles could vary depending upon the
number of sites covered and whether continuous, near-continuous, or
sporadic coverage is provided.
Assumptions and Their Effects on Data Presented in the MDA Report Were
Not Clearly Explained:
The March 2006 report to Congress did not clearly explain the
assumptions used in the modeling of the BMDS elements' capabilities and
did not explain the effects those assumptions may have had on the
results. First, the model inputs for the technical analysis assumed
desired rather than demonstrated performance, and the report does not
fully explain challenges in maturing technologies or how these
performance predictions could change if the technologies are not
developed as desired or assumed. For example, although the model MDA
used is capable of showing different results based on different
performance assumptions, the report did not explain how the number of
successful intercepts may change if less than 100 percent of the
desired technical capabilities are developed as envisioned. Thus the
results represent the best expected outcome. Second, the report does
not explain the current status of technical development or the
challenges in maturing each element's critical technologies as desired
or assumed in the report. DOD best practices define Technology
Readiness Levels on a scale of 1-9, and state which level should be
reached to progress past specific program decision points.[Footnote 17]
However, the March 2006 report does not explain the current Technology
Readiness Level for any of the boost and ascent phase elements'
critical technologies or the extent to which the technology has to
mature to attain the performance assumed in the report. For example,
the report does not explain that some of the technologies for the
Airborne Laser have to improve between 60 percent and 80 percent and
the report does not discuss any of the challenges MDA faces in doing
so.
DOD Can Significantly Improve Life-Cycle Cost Estimates to Support
Future Investment and Budget Decisions for Boost and Ascent Phase
Capabilities:
The March 2006 report to Congress provides cost estimates for each of
the boost and ascent phase capabilities; however, the cost estimates in
the report have several limitations that raise questions about their
usefulness. We compared the report's cost estimates with various DOD
and GAO sources that describe key principles for developing accurate
and reliable life-cycle cost estimates.[Footnote 18] Based on our
analysis, we found that MDA did not include all cost categories,
calculate costs based on warfighter quantities, and did not conduct a
sensitivity analysis to assess the effects of cost drivers. Moreover,
although MDA's report acknowledges uncertainty in the cost estimates,
the report does not fully disclose the limitations of the cost
estimates. DOD can significantly improve the completeness of and
confidence in cost estimates for boost and ascent phase capabilities as
it prepares for future investment and budget decisions. For example,
although DOD did not have its cost estimate for its March 2006 report
independently verified because doing so would have taken several
months, MDA officials agreed that independently verified cost estimates
will be critical to support major decision points for boost and ascent
phase capabilities.[Footnote 19] In addition, as these capabilities
mature, MDA officials agreed that showing cost estimates over time and
conducting uncertainty analyses will be needed to support key program
and investment decisions.
Cost Estimates Did Not Include All Cost Categories:
The cost estimates provided in the MDA report included some
development, production, and operations/support costs for each boost
and ascent phase element but were not fully developed or verified
according to key principles for developing life-cycle cost estimates.
Life-cycle costs are the total cost to the government for a program
over its full life, including the costs of research and development,
investment, operating and support, and disposal.
Based on our comparison of the life-cycle cost estimates in the report
with key principles for developing life-cycle cost estimates, we found
that the estimates were incomplete in several ways. First, the cost
estimates did not include all cost categories, such as costs to
establish and sustain operations at U.S. bases. Instead, MDA assumed
that the elements would be placed at existing bases with sufficient
base support, infrastructure and security; however, some of these costs
such as infrastructure could be significant. For example, an MDA
planning document cited about $87 million for infrastructure costs to
support a ground-based BMDS element (Terminal High Altitude Area
Defense). Army officials confirmed that training facilities, missile
storage buildings, and a motor pool were built at a U.S. base
specifically to support this element and it is likely that similar
infrastructure would be needed to support the land-based Kinetic Energy
Interceptor. Additionally, MDA's cost estimates did not include costs
to establish and sustain operations at forward overseas locations, even
though the report states that the elements will have to be located in
close proximity to their targets, and the operational concepts for
Kinetic Energy Interceptor and Airborne Laser, although in early
development, state that these elements will be operated from forward
locations. Again, these are important factors to consider--the Airborne
Laser operational concept and the MDA report acknowledge that unique
support will be required to support operations at any forward location
for the Airborne Laser such as chemical facilities, unique ground
support equipment, and maintenance. Service, Joint Staff, and U.S.
Strategic Command officials also said that these elements would have to
be located forward and could be used as a strategic deterrent in
peacetime.
Second, the production and operating cost estimates were not based on
warfighter quantities, that is, quantities of each element that the
services and combatant commands may require to provide needed coverage
of potential targets. MDA assumed a certain quantity of each element.
For example, MDA officials told us that they assumed 96 Standard
Missile-3 block 2A missiles because, at the time MDA prepared the
report, they planned to buy 96 block 1A missiles developed to intercept
short-range ballistic missiles. However, MDA did not solicit input from
the services, Joint Staff, or combatant commands on whether they had
done or begun analyses to determine element quantities.
Third, MDA did not conduct a sensitivity analysis to identify the
effects of cost drivers. A sensitivity analysis is a way to identify
risk by demonstrating how the cost estimates would change in response
to different values for specific cost drivers. Therefore, a sensitivity
analysis should be performed when developing cost estimates, and the
results should be documented and reported to decision makers. This
means, for example, that MDA could have computed costs with and without
significant categories of costs such as forward bases to identify the
effect that adding forward bases would have on operating costs. The
House Armed Services Committee report on the National Defense
Authorization Bill for Fiscal Year 2006 recognized that operational
capabilities and costs must be taken into account when making decisions
on future funding support. Finally, the cost estimates did not estimate
costs over time--a process known as time phasing--which can assist
decision makers with budgetary decisions. The MDA report showed an
annual cost estimate but did not state for how many years the
development, production, and operating costs may be incurred. Although
MDA officials stated they did not prepare time-phased cost estimates in
order to prepare the report to Congress in a timely manner, they agreed
that showing cost estimates over time would be important information to
support investment decisions at key decision points.
Cost Estimates Were Not Assessed or Independently Verified for MDA's
2006 Report but MDA Officials See Need to Do So for Key Decision
Points:
Key principles for developing life-cycle cost estimates also include
two steps for assessing the confidence of cost estimates. However, MDA
did not take these steps to assess the confidence of the estimates
reported in March 2006. First, the Missile Defense Agency did not
conduct a risk analysis to assess the level of uncertainty for most of
the cost estimates in the MDA report. Risk and uncertainty refer to the
fact that, because a cost estimate is a prediction of the future, it is
likely that the estimated cost will differ from the actual cost. It is
useful to perform a risk analysis to quantify the degree of uncertainty
in the estimates. By using standard computer simulation techniques, an
overall level of uncertainty can be developed for cost estimates. In
contrast, MDA officials told us that they could only provide a
judgmental confidence level for the most of the cost estimates. Second,
MDA did not have the cost estimates in the report verified by an
independent organization such as DOD's Cost Analysis Improvement Group
because doing so would have taken several months. However, MDA
officials agreed that independent verification of cost estimates would
be important information to support investment decisions at key
decision points. According to the key principles that we have
identified, all life-cycle cost estimates should be independently
verified to assure accuracy, completeness, and reliability.[Footnote
20] MDA has recognized the value in independently developed cost
estimates. In 2003, MDA and the Cost Analysis Improvement Group
developed a memorandum of understanding that said, in part, the Cost
Analysis Improvement Group would develop independent cost estimates for
the approved BMDS and its elements as appropriate during development in
anticipation of transition to production, but MDA officials said that
little work was completed under this agreement, which has expired.
Developing complete cost estimates in which decision makers can have
confidence is important since life-cycle cost estimates usually form
the basis for investment decisions and annual budget requests.
Specifically, life-cycle cost estimates that include all cost
categories, show costs over time, include warfighter quantities,
include an assessment of cost drivers, and are independently verified
are important because accurate life-cycle cost estimates can be used in
formulating funding requests contained in the President's Budget and
DOD's future funding plan, the Future Years Defense Program (FYDP)
submitted to Congress. Therefore, there is a need for DOD to provide
transparent budget and cost planning information to Congress. In May
2006, GAO reported that the FYDP, a major source of budget and future
funding plans, does not provide complete and transparent data on
ballistic missile defense operational costs because the FYDP's
structure does not provide a way to identify and aggregate these costs.
It is important that Congress has confidence in boost and ascent phase
estimates because Congress has indicated that it is concerned with the
affordability of pursuing both the Airborne Laser and Kinetic Energy
Interceptor programs in parallel through 2008.
As we reported in 2003, DOD assumes increased investment risk by not
having information available for decision makers at the right time, and
the level of anticipated spending magnifies this risk.[Footnote 21]
Otherwise, senior DOD and congressional decision makers may be limited
in their ability to assess the relative cost of the elements if all
cost categories are not included and cost drivers are not identified.
Considering competing demands, this could also limit Congress's ability
to consider investment decisions or evaluate whether continued
expenditures are warranted. MDA officials stated that, in developing
the cost estimates for the March 2006 report, they decided not to
follow some of the key principles for developing life-cycle cost
estimates such as time phasing and independent verification of the cost
estimates in order to complete the report in a timely manner. However,
the officials also agreed that these key principles are important in
developing complete, accurate, and reliable life-cycle cost estimates
for supporting investment decisions at key decision points. Therefore,
in the future, when preparing cost estimates to be used in support of
key decision points, MDA could provide decision makers with more
complete, accurate, and reliable cost estimates by better adhering to
key principles for developing life-cycle cost estimates.
Conclusions:
Our review of MDA's March 2006 report on boost and ascent phase
elements identified a number limitations but helps to illuminate the
kind of information that DOD and congressional decision makers will
need following upcoming tests for boost and ascent phase elements. We
recognize that the March 2006 report was prepared in response to
congressional direction rather than to support program decisions. We
also recognize that, at the time of MDA's report, these elements were
early in their development and information was incomplete and changing.
Thus, the focus of our analysis was to identify additional information
that could enhance future program and investment decisions.
In particular, the House Armed Services Committee has raised questions
about the affordability of pursuing both the Kinetic Energy Interceptor
and the Airborne Laser in parallel through the projected knowledge
point demonstrations,[Footnote 22] which are now scheduled for 2008 and
2009 respectively. It is important that these decisions be both well-
informed and transparent because of the long-term funding consequences.
DOD and congressional decision makers' ability to assess which elements
can be fully developed, integrated, and operated relative to the others
will be enhanced if they have the benefit of information based on more
rigorous analysis than that contained in MDA's March 2006 report.
Looking forward, as DOD strengthens its analyses to support future key
decisions, DOD and congressional decision makers will be able to use
more complete information to assess force structure, basing, support,
and infrastructure requirements, as well as technical maturity, budget
requests, and FYDP spending plans, in deciding whether or not to
continue developing one, two, or all three boost and ascent phase
elements and in what quantities.
Recommendations for Executive Action:
To provide decision makers with information that enables them to
clearly understand the technical progress and operational implications
of each boost and ascent phase element and make fully informed, fact-
based, program decisions at future key decision points, we recommend
that the Secretary of Defense direct the Under Secretary of Defense for
Acquisition, Technology, and Logistics to take the following actions to
support key decision points for the BMDS boost and ascent phase
elements:
* Include all DOD stakeholders (including services, combatant commands,
Joint Staff) in developing and analyzing operational issues regarding
what is needed to support operations at U.S. bases and potential
forward locations, including basing assessments, force structure and
quantity requirements, infrastructure, security/force protection,
maintenance, and personnel.
* Provide specific information on the technical progress of each
element. Specifically, the analysis should explain current technical
maturity versus desired technical maturity and capabilities of all
major components and subsystems, reasonable model inputs on element
performance, and provide a clear explanation of assumptions and their
effect on results.
* Use the results of these analyses at each key decision point.
To provide decision makers with complete and reliable data on the costs
of each boost/ascent phase BMDS element to enhance investment and
budget decisions, we recommend that the Secretary of Defense take the
following actions:
* Direct the Under Secretary of Defense for Acquisition, Technology,
and Logistics to require MDA to prepare and--to support key decision
points--periodically update a full life-cycle cost estimate for each
boost/ascent phase element, in accordance with key principles for
developing accurate and reliable life-cycle cost estimates, that
includes all operational costs, including costs to establish and
sustain operations at U.S. bases and forward locations, and that is
based on warfighter quantities, includes sensitivity analyses, and
reflects time phasing.
* Direct an independent group, such as the Cost Analysis Improvement
Group, to prepare an independent life-cycle cost estimate for each
capability at each key decision point.
* Direct MDA and services to report independently verified life-cycle
cost estimates along with budget requests and FYDP funding plans for
each boost/ascent phase element.
Agency Comments and Our Evaluation:
In written comments on a draft of this report, DOD agreed with our
recommendations regarding the need for analysis of technical progress
and operational issues to support key boost and ascent phase element
decision points. DOD also agreed that an independent life-cycle cost
estimate may be needed to inform some key decision points while they
may not be needed at other decision points. However, DOD did not agree
to prepare and periodically update full life-cycle cost estimates for
each boost and ascent phase element to support key decision points, and
report independently verified life cycle cost estimates with budget
requests and FYDP funding plans. As discussed below, we continue to
believe our recommendations have merit and that DOD should take the
additional actions we have recommended to provide a rigorous analytical
basis for future decisions, enhance the transparency of its analyses,
and increase accountability for key decisions that could involve
billions of dollars. The department's comments are reprinted in their
entirety in appendix II.
DOD agreed with our recommendations that all DOD stakeholders be
included in developing and analyzing operational issues, that specific
information on technical progress be provided to explain current versus
desired capabilities, and that the results of both analyses be used at
key decision points. DOD stated in its comments that officials from
MDA, the military departments, the combatant commanders, and other
organizations are collaborating to develop an operational BMDS.
Moreover, the annual BMDS Transition and Transfer Plan is coordinated
with the service secretaries and other stakeholders and serves as a
repository for plans, agreements, responsibilities, authorities, and
issues. DOD also stated that key program decisions are and will
continue to be informed by detailed technical analysis, including
assessment of element technical maturity. However, DOD did not clearly
explain how future decision making will be enhanced or how analyses of
operational issues will be conducted if, as in the case of the Kinetic
Energy Interceptor, DOD has not assigned a service responsibility for
operating the element once it is developed. We continue to believe that
DOD and congressional decision makers will need more complete
information on support requirements at upcoming decision points as well
as a clear comparison of current versus desired technical capabilities
in deciding whether or not to continue developing one, two, or all
three boost and ascent phase elements.
Regarding our recommendations to improve cost estimates used to support
key investment decisions, DOD partially concurred that independent life-
cycle cost estimates may be required to inform some key decision points
but stated that other key decision points may not. However, DOD did not
agree that it should routinely prepare and periodically update a full
life-cycle cost estimate for each boost and ascent phase element. DOD
said that it continuously assesses all aspects of its development
efforts and will direct an independent evaluation of life- cycle costs
for boost and ascent phase elements if circumstances warrant or if
MDA's Director declares an element mature enough to provide a
militarily useful capability. However, if, as DOD's comments suggest,
such costs are not assessed until circumstances warrant or MDA's
Director declares an element mature enough to provide a militarily
useful capability, these costs may not be available early enough to
help shape important program and investment decisions and consider
trade-offs among elements. Moreover, DOD's Operating and Support Cost
Estimating Guide, published by the Cost Analysis Improvement Group,
states that when the Cost Analysis Improvement Group assists the Office
of the Secretary of Defense components in their review of program
costs, one purpose is to determine whether a new system will be
affordable to operate and support.[Footnote 23] Therefore, such
analysis must be done early enough to provide cost data that will be
considered in making a decision to field, produce, or transition an
element. We continue to believe our recommendation has merit because
the development of life-cycle cost estimates that include potential
operations and support costs would improve the information available to
decision makers and increase accountability for key decisions that
could involve billions of dollars at a time when DOD will likely face
competing demands for resources.
Finally, DOD did not agree to report independently verified life-cycle
cost estimates along with budget requests and FYDP funding plans for
each boost and ascent phase element. DOD stated that operations and
support segments of the budget are organized by functional area rather
than by weapon system and are dependent on operations and support
concepts of the employing military department. DOD further stated that
development of total life-cycle cost estimates for operational BMDS
capabilities requires agreement between MDA and the lead military
department on roles and responsibilities for fielded BMDS capabilities
that transcend the annual transition planning cycle but serve as a
basis for budget submittals. We recently reported that MDA enjoys
flexibility in developing BMDS but this flexibility comes at the cost
of transparency and accountability.[Footnote 24] One purpose of cost
estimates is to support the budget process by providing estimates of
the funding required to efficiently execute a program. Also,
independent verification of cost estimates allows decision makers to
gauge whether the program is executable. Thus, cost estimating is the
basis for establishing and defending budgets and is at the heart of the
affordability issue. This principle is stated in DOD procedures which
specify that when cost results are presented to the Office of the
Secretary of Defense Cost Analysis Improvement Group, the program
office-developed life-cycle cost estimate should be compared with the
FYDP and differences explained.[Footnote 25] Therefore, we continue to
believe that our recommendation has merit because, without an
independent cost estimate that can be compared to budget requests and
FYDP funding plans, congressional decision makers may not have all the
necessary information to assess the full extent of future resource
requirements if the boost and ascent phase capabilities go forward, or
assess the completeness of the cost estimates that are in the budget
request and FYDP funding plans.
We are sending copies of this report to the Secretary of Defense; the
Commander, U.S. Strategic Command; the Director, Missile Defense
Agency; Chairman, the Joint Chiefs of Staff; and the Chiefs of Staff of
the Army, Navy, and Air Force. We will make copies available to others
upon request. In addition, the report will be available at no charge on
the GAO Web site at http://www.gao.gov.
If you or your staff have any questions, please call either Janet St.
Laurent on (202) 512-4402 or Paul Francis on (202) 512-2811. Contact
points for our Offices of Congressional Relations and Public Affairs
may be found on the last page of this report. Staff members who made
key contributions to this report are listed in appendix III.
Signed by:
Janet A. St. Laurent:
Director, Defense Capabilities and Management:
Signed by:
Paul L. Francis:
Director, Acquisition, Sourcing, and Management:
List of Committees:
The Honorable Carl Levin:
Chairman:
The Honorable John McCain:
Ranking Minority Member:
Committee on Armed Services:
United States Senate:
The Honorable Daniel K. Inouye:
Chairman:
The Honorable Ted Stevens:
Ranking Minority Member:
Subcommittee on Defense:
Committee on Appropriations:
United States Senate:
The Honorable Ike Skelton:
Chairman:
The Honorable Duncan L. Hunter:
Ranking Minority Member:
Committee on Armed Services:
House of Representatives:
The Honorable John P. Murtha:
Chairman:
The Honorable C. W. Bill Young:
Ranking Minority Member:
Subcommittee on Defense:
Committee on Appropriations:
House of Representatives:
[End of section]
Appendix I: Scope and Methodology:
During this review, we focused on assessing the analytical approach the
Missile Defense Agency (MDA) used to develop its March 2006 report to
Congress, as well as the methodology for developing the cost estimates
for each of the three Ballistic Missile Defense System (BMDS) boost and
ascent phase elements.
To assess the extent to which the Department of Defense (DOD) is
developing technical and operational information useful for oversight
and that will support decision making at key points, we compared the
analytical approach DOD used to develop its March 2006 report with
generally accepted research standards that are relevant for defense
studies such as this, that define a sound and complete study, and that
cover all phases of a study--design, execution, and presentation of
results. The following were our sources for these standards:
* GAO, Government Auditing Standards: 2003 Revision, GAO-03-673G
(Washington, D.C.: June 2003);
* GAO, Designing Evaluations, GAO/PEMD-10.1.4 (Washington, D.C.: March
1991);
* GAO, Dimensions of Quality, GAO/QTM-94-1 (Washington, D.C.: February
2004);
* RAND Corporation, RAND Standards for High-Quality Research and
Analysis (Santa Monica, Calif.: June 2004);
* Air Force, Office of Aerospace Studies, Analysts Handbook: On
Understanding the Nature of Analysis (January 2000);
* Air Force, Office of Aerospace Studies, Air Force Analysis Handbook,
A Guide for Performing Analysis Studies: For Analysis of Alternatives
or Functional Solution Analysis (July 2004);
* Department of Defense, DOD Modeling and Simulation (M&S)
Verification, Validation, Accreditation (VV&A), Instruction 5000.61
(Washington, D.C.: May 2003);
* Department of Defense, Data Collection, Development, and Management
in Support of Strategic Analysis, Directive 8260.1 (Washington, D.C.:
Dec. 2, 2003); and:
* Department of Defense, Implementation of Data Collection,
Development, and Management for Strategic Analyses, Instruction 8260.2
(Washington, D.C.: Jan. 21, 2003).
For a more complete description of these standards and how we
identified them, see GAO-06-938, appendix I.[Footnote 26] In applying
these standards, we focused on the extent to which stakeholders were
involved in study design and analysis as well as the extent to which
assumptions were reasonable and their effects on results were clearly
explained. We assessed MDA briefings that explained the modeling used
for the technical analysis projecting the elements' capabilities. To
assess the basis for the assumed performance parameters used to model
each element's performance, we traced and verified a nonprobability
sample of these parameters to their source documentation and concluded
that they were generally supported. To evaluate the DOD report's
characterization of threats, we reviewed Defense Intelligence Agency
documents and discussed the type and capability of threats and expected
BMDS capabilities with officials from the Office of the Secretary of
Defense for Program Analysis and Evaluation and the Defense
Intelligence Agency. In addition, to gain an understanding of the
extent to which DOD has assessed warfighter quantities for the boost
and ascent phase elements, the development of operational concepts, and
operational implications of employing the boost and ascent phase
elements at forward locations, we evaluated DOD and service guidance on
assessing sites and support for new weapon systems and discussed these
issues with officials from the Joint Staff; U.S. Army Headquarters and
Space and Missile Defense Command; U.S. Strategic Command; the office
of the Chief of Naval Operations Surface Warfare Directorate, Ballistic
Missile Defense Division; Air Combat Command; and the office of the
Secretary of the Air Force for Acquisition, Global Power Directorate.
Finally, we discussed the results of all our analyses with officials in
the Joint Staff; U.S. Strategic Command; the Army's Space and Missile
Defense Command; Office of the Secretary of Defense for Acquisition,
Technology, and Logistics; Missile Defense Agency; the office of the
Chief of Naval Operations Surface Warfare Directorate, Ballistic
Missile Defense Division; the office of the Secretary of the Air Force
for Acquisition Global Power Directorate; and Air Combat Command.
To assess the extent to which DOD presented cost information to
Congress that is complete and transparent, we first assessed how MDA
developed its estimates and then compared the method by which those
estimates were prepared to key principles compiled from various DOD and
GAO sources that describe how to develop accurate and reliable life-
cycle cost estimates to determine their completeness and the extent to
which DOD took steps to assess confidence in the estimates. The
following were our sources for compiling the cost criteria:
* Department of Defense, Assistant Secretary of Defense (Program
Analysis and Evaluation), Cost Analysis Guidance and Procedures, DOD
Manual 5000.4-M (December 1992);
* Department of Defense, Office of the Secretary of Defense Cost
Analysis Improvement Group, Operating and Support Cost Estimating Guide
(May 1992);
* Department of Defense, Defense Acquisition University, Defense
Acquisition Guidebook (online at http://akss.dau.mil/dag);
* Department of Defense, Defense Acquisition University, Introduction
to Cost Analysis (April 2006);
* Air Force, Office of Aerospace Studies, Air Force Analysis Handbook:
A Guide for Performing Analysis Studies for Analysis of Alternatives or
Functional Solution Analysis (July 2004);
* Air Force, Base Support and Expeditionary Site Planning, Air Force
Instruction 10-404 (March 2004); and:
* GAO, GAO Cost Assessment Guide (currently under development).
In addition, we met with DOD officials from MDA, U.S. Strategic
Command, the Joint Staff, Army, Navy and Air Force to determine the
extent to which they were involved in developing the cost estimates for
the DOD report. Finally, we corroborated our methodology and results
with officials from the Office of the Under Secretary of Defense,
Program, Analysis and Evaluation (Cost Analysis Improvement Group) and
the Office of the Under Secretary of Defense (Comptroller) and they
agreed that our methodology for examining the report's cost estimates
was reasonable and consistent with key principles for developing
accurate and reliable life-cycle cost estimates. We identified some
data limitations with the cost estimates which we discuss in this
report.
We provided a draft of this report to DOD for its review and
incorporated its comments where appropriate. Our review was conducted
between June 2006 and February 2007 in accordance with generally
accepted government auditing standards.
[End of section]
Appendix II: Comments from the Department of Defense:
Office Of The Under Secretary Of Defense:
3000 Defense Pentagon Washington, DC 20301-3000:
Acquisition, Technology And Logistics:
Mar 28 8 2007:
Ms. Janet St. Laurent:
Director, Defense Capabilities and Management:
U. S. Government Accountability Office:
441 G Street, N.W.
Washington, DC 20548:
Dear Ms. St. Laurent:
This is the Department of Defense (DoD) response to the GAO Draft
Report GAO-07-430, "Missile Defense: Actions Needed to Improve
Information for Supporting Future Key Decisions for Boost and Ascent
Phase Elements," dated February 28, 2007 (GAO Code 350861).
The DoD concurs with three, partially concurs with one, and non-concurs
with two of the draft report's recommendations. The rationale for our
position is included in the enclosure.
We appreciate the opportunity to comment on the draft report. My point
of contact for this effort is Mr. David Crim, (703) 697-5385,
David.Crim@osd.mil:
Sincerely,
Signed by:
Dave G. Ahern:
Director:
Portfolio Systems Acquisition:
Enclosure:
As stated:
GAO Draft Report - Dated February 28, 2007 GAO Code 350861/GAO-07-430:
"Missile Defense: Actions Needed to Improve Information for Supporting
Future Key Decisions for Boost and Ascent Phase Elements"
Department Of Defense Comments To The Recommendation:
Recommendation 1: The GAO recommended that the Secretary of Defense
direct the Under Secretary of Defense for Acquisition, Technology, and
Logistics to include all DoD stakeholders (including services,
combatant commands, Joint Staff) in developing and analyzing
operational issues regarding what is needed to support operations at
U.S. bases and potential forward locations, including basing
assessments, force structure and quantity requirements, infrastructure,
security/force protection; maintenance, and personnel, for the
Ballistic Missile Defense System (BMDS) boost and ascent phase
elements. (Page 21/GAO Draft Report):
DoD Response: Concur. As reported in the DoD Report to Congress on
Transition of Ballistic Missile Defense Programs to the Military
Departments for 2007, the DoD has made significant progress in planning
for the transition of operating and support responsibilities and
authorities for elements of the BMDS to the Military Departments.
Members of the Missile Defense Agency (MDA), the Military Departments,
the Combatant Commands, the Joint Staff, and the Office of the
Secretary of Defense are collaborating by way of an Integrated Product
Team (IPT) to operationalize the BMDS. This Transition and Transfer IPT
produces an annual DoD document referred to as the BMDS Transition and
Transfer Plan (T&TP) that serves as a repository for plans, agreements,
responsibilities, authorities, and issues. The T&TP is submitted by MDA
to USD(AT&L) annually on March l, and then coordinated with Service
Secretaries and other stakeholders, and, with regards to
operationalizing the BMDS, serves as the basis for MDA and Service
budget formulation.
Recommendation 2: The GAO recommended that the Secretary of Defense
direct the Under Secretary of Defense for Acquisition, Technology, and
Logistics to provide specific information on the technical progress of
each BMDS boost and ascent phase element to support key decision
points. Specifically, the analysis should explain current technical
maturity of all major components and subsystems in comparison to
desired technical maturity, reasonable model inputs on element
performance, and a clear explanation of assumptions and their effect on
results. (Page 21/GAO Draft Report):
DoD Response: Concur. Key program decisions are and will continue to be
informed by detailed technical analysis, including assessments of
element technical maturity, and any sensitivity of element performance
estimates to the models and simulations used.
Recommendation 3: The GAO recommended that the Secretary of Defense
direct the Under Secretary of Defense for Acquisition, Technology, and
Logistics to use the results of these analyses at each key decision
point for the BMDS boost and ascent phase elements. (Page 21 /GAO Draft
Report):
DoD Response: Concur. Key program decisions are and will continue to be
informed by detailed technical analysis, including assessments of
element technical maturity, and any sensitivity of element performance
estimates to the models and simulations used.
Recommendation 4: The GAO recommended that the Secretary of Defense
direct the Under Secretary of Defense for Acquisition, Technology, and
Logistics to require the Missile Defense Agency (MDA) to prepare and
periodically update a full life cycle cost estimate for each boost/
ascent phase element, in accordance with key principles for developing
accurate and reliable life cycle cost estimates, that includes all
operational costs, including costs to establish and sustain operations
at U.S. bases and forward locations, is based on warfighter quantities,
includes sensitivity analyses, and reflects time phasing. (Pages 21-22/
GAO Draft Report):
DoD Response: Non-concur. As noted in the DoD Report to Congress on
Transition of Ballistic Missile Defense Programs to the Military
Departments for 2007, DoD guidance directs that BMDS elements be
managed in three phases: development; transition, and procurement and
operations. The Director, MDA is to manage the BMDS through the
development and transition phases, baseline the capability and
configuration of each element and capability block, and define
interoperability standards. The Military Departments provide forces, as
needed, to support fielding of early and or contingency capability.
During early fielding and the transition phase, responsibilities are
shared by MDA and the Military Departments; specific responsibilities
and authorities are documented in the T&TP, agreed to by MDA and the
Military Departments, and subsequently programmed for by the applicable
Department or Agency. Beyond early fielding and or transition, and
after an appropriate milestone decision, the Military Departments will
provide resources to procure and support the transitioned elements as
programs in their own right and as reflected by the planned force
structure. DoD continuously assesses all aspects of its developments
and will direct an independent evaluation of life cycle costs for boost
and ascent phase elements if circumstances warrant or the Director, MDA
declares an element mature enough to provide a militarily useful
capability. In making this declaration, the Director, MDA will rely on
all data available, including available cost estimates. These estimates
will be refined in conjunction with designated lead Military
Departments as part of the Transition and Transfer IPT and documented
in the T&TP plan.
Recommendation 5: The GAO recommended that the Secretary of Defense
direct an independent group, such as the Cost Analysis Improvement
Group, to prepare an independent Life Cycle Cost Estimate for each
capability at each key decision point. Pages 21-22/GAO Draft Report):
DoD Response: Partially concur. While some of the key decision points
may require independent life cycle cost estimates, others may not. As
noted in the DoD Response to Recommendation 4 above, the Director, MDA
may declare an element mature enough to provide a militarily useful
capability. Based upon this recommendation, DoD will determine if that
capability is ready to begin early fielding or proceed to a milestone
review. DoD stakeholders will rely on independent life cycle cost
estimates to inform these and other key decisions as applicable.
Recommendation 6: The GAO recommended that the Secretary of Defense
direct MDA and the Military Services to report independently verified
life cycle cost estimates along with budget requests and Future Years
Defense Plan (FYDP) funding plans for each boost/ascent phase element.
(Pages 21-22/GAO Draft Report):
DoD Response: Non-concur. Operations and support segments of the budget
are organized by functional area rather than by weapon system. The
functional areas are characterized by personnel, unit-level
consumption, various levels of maintenance, sustaining support, and
indirect support costs. Although the costs of certain elements (such as
personnel) can often be identified explicitly, other elements may be
common to more than one weapon system and are dependent on operations
and support concepts of the employing Military Department. The cost of
these latter elements may be allocated among respective systems and
support organizations. Development of total life cycle cost estimates
for operationalized BMDS capabilities requires agreement between MDA
and lead Military Departments on roles and responsibilities for fielded
missile defense capabilities. The T&TP, updated annually or as
significant changes warrant, constitutes a "snapshot" of a work-in-
progress intended to capture current BMDS capability transition status,
roles, responsibilities, and future transition plans for the components
and elements of the BMDS. The annual snapshot is taken with the
intention of clarifying roles and responsibilities and documenting
agreements between the Military Departments and MDA. These agreements
cover responsibilities, deliverables, funding, and schedules that
transcend the annual BMDS transition plan cycle. The T&TP is submitted
early enough in the Programming Planning and Budgeting Cycle so that it
can be used by MDA and the Services as a basis for their BMDS budget
submittals and to provide needed insights into estimated life cycle
costs of BMDS capabilities.
[End of section]
Appendix III: GAO Contacts and Staff Acknowledgments:
GAO Contacts:
Janet A. St. Laurent, (202) 512-4402, stlaurentj@gao.gov:
Paul L. Francis, (202) 512-2811, francisp@gao.gov:
Acknowledgments:
In addition to the individuals named above, Barbara H. Haynes and
Gwendolyn R. Jaffe, Assistant Directors; Brenda M. Waterfield; Todd
Dice; Jeffrey R. Hubbard; Nabajyoti Barkakati; Hai V. Tran; Ron La Due
Lake; and Susan C. Ditto made key contributions to this report.
[End of section]
Related GAO Products:
Defense Transportation: Study Limitations Raise Questions about the
Adequacy and Completeness of the Mobility Capabilities Study and
Report. GAO-06-938. Washington, D.C.: September 20, 2006.
Defense Management: Actions Needed to Improve Operational Planning and
Visibility of Costs for Ballistic Missile Defense. GAO-06-473.
Washington, D.C.: May 31, 2006.
Defense Acquisitions: Missile Defense Agency Fields Initial Capability
but Falls Short of Original Goal. GAO-06-327. Washington, D.C.: March
15, 2006.
Defense Acquisitions: Actions Needed to Ensure Adequate Funding for
Operation and Sustainment of the Ballistic Missile Defense System. GAO-
05-817. Washington, D.C.: September 6, 2005.
Military Transformation: Actions Needed by DOD to More Clearly Identify
New Triad Spending and Develop a Long-term Investment Approach. GAO-05-
962R. Washington, D.C.: August 4, 2005.
Military Transformation: Actions Needed by DOD to More Clearly Identify
New Triad Spending and Develop a Long-term Investment Approach. GAO-05-
540. Washington, D.C.: June 30, 2005.
Defense Acquisitions: Status of Ballistic Missile Defense Program in
2004. GAO-05-243. Washington, D.C.: March 31, 2005.
Future Years Defense Program: Actions Needed to Improve Transparency of
DOD's Projected Resource Needs. GAO-04-514. Washington, D.C.: May 7,
2004.
Missile Defense: Actions Are Needed to Enhance Testing and
Accountability. GAO-04-409. Washington, D.C.: April 23, 2004.
Missile Defense: Actions Being Taken to Address Testing
Recommendations, but Updated Assessment Needed. GAO-04-254. Washington,
D.C.: February 26, 2004.
Missile Defense: Additional Knowledge Needed in Developing System for
Intercepting Long-Range Missiles. GAO-03-600. Washington, D.C.: August
21, 2003.
Missile Defense: Alternate Approaches to Space Tracking and
Surveillance System Need to Be Considered. GAO-03-597. Washington,
D.C.: May 23, 2003.
Missile Defense: Knowledge-Based Practices Are Being Adopted, but Risks
Remain. GAO-03-441. Washington, D.C.: April 30, 2003.
Missile Defense: Knowledge-Based Decision Making Needed to Reduce Risks
in Developing Airborne Laser. GAO-02-631. Washington, D.C.: July 12,
2002.
Missile Defense: Review of Results and Limitations of an Early National
Missile Defense Flight Test. GAO-02-124. Washington, D.C.: February 28,
2002.
Missile Defense: Cost Increases Call for Analysis of How Many New
Patriot Missiles to Buy. GAO/NSIAD-00-153. Washington, D.C.: June 29,
2000.
Missile Defense: Schedule for Navy Theater Wide Program Should Be
Revised to Reduce Risk. GAO/NSIAD-00-121. Washington, D.C.: May 31,
2000.
FOOTNOTES
[1] The boost phase is the period immediately after launch when the
missile's booster stages are still thrusting. Next is the ascent phase
which ends when the ballistic missile completes deployment of reentry
vehicles and possible decoys.
[2] Other versions of the Standard Missile-3 are being developed to
intercept threat missiles in their midcourse phase.
[3] GAO, Budget Process: Long-term Focus Is Critical, GAO-04-585T
(Washington, D.C.: Mar. 23, 2004).
[4] GAO, 21st Century Challenges: Reexamining the Base of the Federal
Government, GAO-05-325SP (Washington, D.C.: February 2005).
[5] Although there are other interim decision points leading up to the
lethal shootdown demonstration, MDA officials stated that the 2009 key
decision point is one where decisions may be made about the future of
the Airborne Laser program.
[6] H.R. Rep. No. 109-119 (2005).
[7] Pub.L. No. 109-163, § 231 (2005).
[8] While the House Appropriations Committee report originally directed
us to conduct a joint study with DOD, we agreed with the Defense
Subcommittee, House Appropriations Committee that, in lieu of a joint
study, we would assess the Missile Defense Agency's March 2006 report
on boost and ascent phase capabilities.
[9] In a September 2006 report, GAO identified frequently occurring,
generally accepted research standards that are relevant for defense
studies and define a sound and complete study. GAO, Defense
Transportation: Study Limitations Raise Questions about the Adequacy
and Completeness of the Mobility Capabilities Study and Report, GAO-06-
938 (Washington, D.C.: Sept. 20, 2006). Also, see app. I for a more
complete description of our scope and methodology.
[10] GAO, Defense Acquisitions: Missile Defense Agency Fields Initial
Capability but Falls Short of Original Goals, GAO-06-327 (Washington,
D.C.: Mar. 15, 2006).
[11] GAO, Missile Defense: Knowledge-Based Practices Are Being Adopted,
but Risks Remain, GAO-03-441 (Washington, D.C.: Apr. 30, 2003).
[12] GAO, Defense Management: Actions Needed to Improve Operational
Planning and Visibility of Costs for Ballistic Missile Defense, GAO-06-
473 (Washington, D.C.: May 31, 2006).
[13] The boost phase is the period immediately after launch when the
missile's booster stages are still thrusting and typically lasts 3-5
minutes for intercontinental ballistic missiles. The ascent phase is
when the booster stages have stopped thrusting and dropped away leaving
a warhead and possible decoys. The midcourse phase, lasting for about
20 minutes, begins after the missile has stopped accelerating and the
warhead travels through space on a predictable path. The final or
terminal phase begins when the warhead reenters the atmosphere and
lasts approximately a minute or less.
[14] H.R. Rep. No. 109-119 (2005).
[15] For most major acquisition programs, DOD prepares a life-cycle
cost estimate that is independently verified for major program reviews.
A life-cycle cost includes all costs associated with a weapon system's
research and development, investment, which includes military
construction, operations and support, and disposal.
[16] The model, called WILMA, supports quantitative evaluation of
ballistic missile defense system performance in terms of probabilities
of success.
[17] Technology Readiness Levels are measured on a scale of 1-9,
beginning with paper studies of a technology's feasibility (level 1)
and culminating with a technology fully integrated into a completed
product (level 9).
[18] The criteria for developing accurate and reliable life-cycle cost
estimates were compiled from several sources including the DOD Manual
5000.4-M Cost Analysis Guidance and Procedures, the Defense Acquisition
Guidebook, and an exposure draft of GAO's Cost Assessment Guide. See
app. I for a more complete description of our scope and methodology.
[19] As we reported in GAO-05-817, independent cost estimates are
normally first required for Milestone B, the decision to enter system
development and demonstration. GAO, Defense Acquisitions: Actions
needed to Ensure Adequate Funding for Operation and Sustainment of the
Ballistic Missile Defense System, GAO-05-817 (Washington, D.C.: Sept.
6, 2005). However, because BMDS elements do not enter DOD's acquisition
cycle until Milestone C, the requirements under Section 2434 of Title
10 may not be applied to the BMDS until the transition phase that leads
to the transfer to a military service. However, developing cost
estimates using the key principles that we describe for developing
accurate and reliable life-cycle cost estimates would provide useful
information to DOD decision makers and Congress.
[20] As we reported in GAO-05-817, independent cost estimates are
normally first required for Milestone B, the decision to enter system
development and demonstration. However, because BMDS elements do not
enter DOD's acquisition cycle until Milestone C, the requirements under
Section 2434 of Title 10 may not be applied to the BMDS until the
transition phase that leads to the transfer to a military service.
However, developing cost estimates using the key principles that we
describe for developing accurate and reliable life-cycle cost estimates
would provide useful information to DOD decision makers and Congress.
[21] GAO-03-441.
[22] H.R. Rep. No. 109-452 (2006).
[23] DOD, Office of the Secretary of Defense Cost Analysis Improvement
Group, Operating and Support Cost Estimating Guide (May 1992).
[24] GAO, Defense Acquisitions: Missile Defense Acquisition Strategy
Generates Results but Delivers Less at a Higher Cost, GAO-07-387
(Washington, D.C.: Mar. 15, 2007).
[25] DOD, Assistant Secretary of Defense (Program Analysis and
Evaluation), Cost Analysis Guidance and Procedures, DOD 5000.4-M
(December 1992).
[26] GAO, Defense Transportation: Study Limitations Raise Questions
about the Adequacy and Completeness of the Mobility Capabilities Study
and Report, GAO-06-938 (Washington, D.C.: Sept. 20, 2006).
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site (www.gao.gov). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site.
To have GAO e-mail you a list of newly posted products every afternoon,
go to www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125
Washington, D.C. 20548:
Public Affairs:
Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548:
