DOD Weapon Systems
Missed Trade-off Opportunities During Requirements Reviews Gao ID: GAO-11-502 June 16, 2011The Weapon Systems Acquisition Reform Act of 2009 (WSARA) directed the Joint Requirements Oversight Council (JROC) to ensure trade-offs among cost, schedule, and performance objectives are considered as part of its requirements review process. WSARA also directed GAO to assess the implementation of these requirements. This report addresses (1) the extent to which the JROC has considered trade-offs within programs, (2) the quality of resource estimates presented to the JROC, and (3) the extent to which the JROC is prioritizing requirements and capability gaps. To do so, GAO analyzed requirement documents reviewed by the JROC in fiscal year 2010, which identified capability gaps or performance requirements for new major defense acquisition programs. GAO also assessed resource estimates presented to the JROC against best practices criteria in the GAO Cost Estimating and Assessment Guide.
The JROC considered trade-offs made by the military services before validating requirements for four of the seven proposed programs it reviewed in fiscal year 2010. According to DOD officials, the most significant trade-offs are made by the military services during the analysis of alternatives (AOA), which occurs between the JROC's review of an Initial Capabilities Document (ICD) and its review of a Capability Development Document (CDD). The AOA is intended to compare the operational effectiveness, cost, and risks of a number of alternative potential solutions. The JROC does not formally review the trade-off decisions made as a result of an AOA until it reviews a proposed program's CDD. As a result, the JROC does not have an opportunity to provide military advice on trade-offs and the proposed solution before it is selected, and a significant amount of time and resources can be expended in technology development before the JROC gets to formally weigh in. The military services did not consistently provide high-quality resource estimates to the JROC for proposed programs in fiscal year 2010. GAO found the estimates presented to the JROC were often unreliable when assessed against best practices criteria. In most cases, the military services had not effectively conducted uncertainty and sensitivity analyses or examined the effects of changing assumptions and ground rules, all of which could further the JROC's efforts to ensure that programs are fully funded and provide a sound basis for making cost, schedule, and performance trade-offs. The JROC does not currently prioritize requirements, consider redundancies across proposed programs, or prioritize and analyze capability gaps in a consistent manner. As a result, the Joint Staff is missing an opportunity to improve the management of DOD's joint portfolio of weapon programs. According to Army, Air Force, and Navy officials, having a better understanding of warfighter priorities from the JROC would be useful to inform both portfolio management efforts and service budgets. A DOD review team examining the JROC's requirements review process is considering changes that would address the prioritization of requirements on a departmentwide basis. GAO recommends that the JROC establish a mechanism to review AOA results earlier in the acquisition process, require higher quality resource estimates from requirements sponsors, prioritize requirements across proposed programs, and address potential redundancies during requirements reviews. The Joint Staff partially concurred with GAO's recommendations and generally agreed with their intent, but differed with GAO on how to implement them.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Michael J. Sullivan Team: Government Accountability Office: Acquisition and Sourcing Management Phone: (937) 258-7915GAO-11-502, DOD Weapon Systems: Missed Trade-off Opportunities During Requirements Reviews
This is the accessible text file for GAO report number GAO-11-502
entitled 'DOD Weapon Systems: Missed Trade-off Opportunities During
Requirements Reviews' which was released on June 16, 2011.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
United States Government Accountability Office:
GAO:
Report to Congressional Committees:
June 2011:
DOD Weapon Systems:
Missed Trade-off Opportunities During Requirements Reviews:
GAO-11-502:
GAO Highlights:
Highlights of GAO-11-502, a report to congressional committees.
Why GAO Did This Study:
The Weapon Systems Acquisition Reform Act of 2009 (WSARA) directed the
Joint Requirements Oversight Council (JROC) to ensure trade-offs among
cost, schedule, and performance objectives are considered as part of
its requirements review process. WSARA also directed GAO to assess the
implementation of these requirements. This report addresses (1) the
extent to which the JROC has considered trade-offs within programs,
(2) the quality of resource estimates presented to the JROC, and (3)
the extent to which the JROC is prioritizing requirements and
capability gaps. To do so, GAO analyzed requirement documents reviewed
by the JROC in fiscal year 2010, which identified capability gaps or
performance requirements for new major defense acquisition programs.
GAO also assessed resource estimates presented to the JROC against
best practices criteria in the GAO Cost Estimating and Assessment
Guide.
What GAO Found:
The JROC considered trade-offs made by the military services before
validating requirements for four of the seven proposed programs it
reviewed in fiscal year 2010. According to DOD officials, the most
significant trade-offs are made by the military services during the
AOA, which occurs between the JROC‘s review of an Initial Capabilities
Document (ICD) and its review of a Capability Development Document
(CDD). The AOA is intended to compare the operational effectiveness,
cost, and risks of a number of alternative potential solutions. The
JROC does not formally review the trade-off decisions made as a result
of an AOA until it reviews a proposed program‘s CDD. As a result, the
JROC does not have an opportunity to provide military advice on trade-
offs and the proposed solution before it is selected, and a
significant amount of time and resources can be expended in technology
development before the JROC gets to formally weigh in.
Figure: AOA‘s Relationship to JROC Requirements Reviews:
[Refer to PDF for image: illustration]
ICD developed: No cost, schedule, performance objectives.
(Requirements development)
JROC review: Too early for trade-offs.
Analysis of alternatives: Most significant trade-offs made.
(Acquisition activity)
Milestone A. (Acquisition milestone)
Technology development: Requirements refined and vetted. (Requirements
development)
CDD developed: Cost, schedule, performance objectives identified.
(Acquisition activity)
JROC review: Requirements rarely change.
Milestone B. (Acquisition milestone)
Source: GAO analysis of DOD policy.
[End of figure]
The military services did not consistently provide high-quality
resource estimates to the JROC for proposed programs in fiscal year
2010. GAO found the estimates presented to the JROC were often
unreliable when assessed against best practices criteria. In most
cases, the military services had not effectively conducted uncertainty
and sensitivity analyses or examined the effects of changing
assumptions and ground rules, all of which could further the JROC‘s
efforts to ensure that programs are fully funded and provide a sound
basis for making cost, schedule, and performance trade-offs.
The JROC does not currently prioritize requirements, consider
redundancies across proposed programs, or prioritize and analyze
capability gaps in a consistent manner. As a result, the Joint Staff
is missing an opportunity to improve the management of DOD‘s joint
portfolio of weapon programs. According to Army, Air Force, and Navy
officials, having a better understanding of warfighter priorities from
the JROC would be useful to inform both portfolio management efforts
and service budgets. A DOD review team examining the JROC‘s
requirements review process is considering changes that would address
the prioritization of requirements on a departmentwide basis.
What GAO Recommends:
GAO recommends that the JROC establish a mechanism to review analysis
of alternatives (AOA) results earlier in the acquisition process,
require higher quality resource estimates from requirements sponsors,
prioritize requirements across proposed programs, and address
potential redundancies during requirements reviews. The Joint Staff
partially concurred with GAO‘s recommendations and generally agreed
with their intent, but differed with GAO on how to implement them.
View [hyperlink, http://www.gao.gov/products/GAO-11-502] or key
components. For more information, contact Michael J. Sullivan at (202)
512-4841 or sullivanm@gao.gov.
[End of section]
Contents:
Letter:
Background:
JROC Did Not Always Consider Trade-offs or Influence Trade-off
Decisions:
Military Services Did Not Consistently Provide High-Quality Resource
Estimates to JROC:
JROC Did Not Consistently Prioritize Requirements and Capability Gaps:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Requirements Documents Reviewed by the JCB or JROC in
Fiscal Year 2010:
Appendix III: JROC Cost Breach Reviews in Fiscal Year 2010:
Appendix IV: Best Practice Criteria for Resource Estimates:
Appendix V: Comments from the Department of Defense:
Appendix VI: GAO Contact and Acknowledgments:
Tables:
Table 1: JROC Consideration of Trade-offs Among Cost, Schedule, and
Performance Objectives for Seven Proposed Programs in Fiscal Year 2010:
Table 2: JROC Cost Breach Reviews in Fiscal Year 2010:
Table 3: Best Practice Criteria for Resource Estimates:
Figures:
Figure 1: JCIDS Reviews and the DOD Acquisition Process:
Figure 2: AOA's Relationship to JROC Reviews:
Figure 3: GAO Assessment of Resource Estimates Presented to the JROC
During CDD Reviews in Fiscal Year 2010:
Figure 4: Requirements Documents Reviewed by the JCB or JROC in Fiscal
Year 2010:
Abbreviations:
AIAMD SOS: Army Integrated Air and Missile Defense System of Systems:
AOA: Analysis of Alternatives:
CAPE: Cost Assessment and Program Evaluation:
CDD: Capability Development Document:
CPD: Capability Production Document:
CSAR-X: Combat Search and Rescue Replacement Vehicle:
CVLSP: Common Vertical Lift Support Platform:
DOD: Department of Defense:
FCB: Functional Capabilities Board:
ICD: Initial Capabilities Document:
JCB: Joint Capabilities Board:
JCIDS: Joint Capabilities Integration and Development System:
JPALS: Joint Precision Approach and Landing System:
JROC: Joint Requirements Oversight Council:
KPP: Key Performance Parameter:
KSA: Key System Attribute:
MDAP: Major Defense Acquisition Program:
USD AT&L: Under Secretary of Defense for Acquisition, Technology and
Logistics:
WSARA: The Weapon Systems Acquisition Reform Act of 2009:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
June 16, 2011:
The Honorable Carl Levin:
Chairman:
The Honorable John McCain:
Ranking Member:
Committee on Armed Services:
United States Senate:
The Honorable Howard P. McKeon:
Chairman:
The Honorable Adam Smith:
Ranking Member:
Committee on Armed Services:
House of Representatives:
With the prospect of slowly growing or flat defense budgets for years
to come, the Department of Defense (DOD) must get better returns on
its weapon system investments and find ways to deliver more capability
to the warfighter for less than it has in the past. In this
environment, DOD's capacity to make effective trade-offs among cost,
schedule, and performance objectives when developing and validating
weapon system requirements and acquiring those systems will be
important to achieving a balance between DOD's weapon system
investments and resources available to it.
The Weapon Systems Acquisition Reform Act of 2009 (WSARA) took several
steps to encourage DOD to engage in a more robust discussion of trade-
offs among cost, schedule, and performance objectives before beginning
a new weapon system program. First, WSARA directed DOD's Joint
Requirements Oversight Council (JROC), which validates joint military
requirements, to ensure trade-offs among cost, schedule, and
performance objectives are considered as part of its process for
assessing and prioritizing requirements.[Footnote 1] Additionally,
WSARA stated that the requirements development process must be
structured to enable incremental, evolutionary, or spiral acquisition
approaches,[Footnote 2] and that acquisition, budget, and cost
estimating officials should be provided an opportunity to develop
resource estimates and raise cost and schedule issues before
performance objectives are established.[Footnote 3] Finally, WSARA
amended the U.S. Code to require the JROC to seek and consider, among
other things, the input of combatant commanders when assisting the
Chairman of the Joint Chiefs of Staff in identifying, assessing, and
approving joint military requirements; considering trade-offs; and
establishing and assigning priority levels, among other areas.
[Footnote 4]
WSARA also directed GAO to assess the implementation of these
requirements.[Footnote 5] This report addresses (1) the extent to
which the JROC has considered trade-offs among cost, schedule, and
performance objectives within programs; (2) the quality and
effectiveness of efforts to estimate the level of resources needed to
fulfill joint military requirements; and (3) the extent to which the
JROC is prioritizing requirements and capability gaps. In addition,
the Ike Skelton National Defense Authorization Act for Fiscal Year
2011 requires GAO to conduct a comprehensive review of the JROC's
requirements validation process and report its results in 2012.
[Footnote 6]
To conduct our work, we focused on JROC activities in fiscal year
2010. We chose this time period to allow for any changes the JROC
would implement as a result of the enactment of WSARA in May 2009. To
determine the extent to which the JROC has considered cost, schedule,
and performance trade-offs within programs, we reviewed the seven
Capability Development Documents (CDD) submitted to the JROC in fiscal
year 2010, JROC decision memos related to the CDDs, and analyses of
alternatives (AOA) conducted by the military services prior to JROC
reviews. We focused on CDDs because they are the first requirements
documents that contain cost, schedule, and performance objectives.
Additionally, we reviewed documentation from 15 JROC reviews of
programs that incurred substantial cost growth after program start to
determine if cost, schedule, and performance trade-offs were made. To
determine the quality and effectiveness of efforts to estimate the
level of resources needed to fulfill joint military requirements, we
assessed the resource estimates used to support the 7 CDDs presented
to the JROC for approval. To do so, we applied GAO's best practice
criteria for cost estimates and reviewed supporting documentation.
[Footnote 7] Each program was also provided with a copy of our
assessment of their resource estimates for review and comment. To
determine the extent to which the JROC prioritized requirements and
capability gaps, we reviewed the 13 Initial Capabilities Documents
(ICD) and 7 CDDs submitted to the JROC in fiscal year 2010, and any
discussions of priorities and redundancies contained in each document.
We also interviewed officials from DOD, the Joint Staff, and military
service headquarters about the extent to which the JROC and its
supporting bodies have addressed prioritization issues.
We conducted this performance audit from June 2010 to June 2011 in
accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings based on our audit objectives. Appendix I contains detailed
information on our scope and methodology.
Background:
DOD uses three interrelated processes to deliver capabilities to the
U.S. military: the Joint Capabilities Integration and Development
System (JCIDS), which validates gaps in joint warfighting capabilities
and requirements that resolve those gaps; the Defense Acquisition
System, which develops and fields weapon systems to meet these
requirements; and the Planning, Programming, Budgeting and Execution
process, which allocates the funding needed to develop, acquire, and
field these weapon systems. The JCIDS process is overseen by the JROC,
which supports the Chairman of the Joint Chiefs of Staff in advising
the Secretary of Defense on joint military capability needs. The JROC
is chaired by the Vice Chairman of the Joint Chiefs of Staff, and
includes one senior leader from each of the military services, such as
the Vice Chief of Staff of the Army or the Vice Chief of Naval
Operations.
The JROC has a number of statutory responsibilities related to the
identification, validation, and prioritization of joint military
requirements. The JROC assists the Chairman of the Joint Chiefs of
Staff with a number of tasks, including (1) identifying, assessing,
and approving joint military requirements; (2) establishing and
assigning priority levels for joint military requirements; and (3)
reviewing the estimated level of resources required to fulfill each
requirement and ensuring that the resource level is consistent with
the requirement's priority. The JROC also assists acquisition
officials in identifying alternatives to any acquisition programs that
experience significant cost growth.
Since 2008, Congress has added to the JROC's statutory
responsibilities and increased the number of JROC members and advisors
who provide input to it. The National Defense Authorization Act for
Fiscal Year 2008 amended the U.S. Code to require that the Under
Secretary of Defense for Acquisition, Technology and Logistics (USD
AT&L), the Under Secretary of Defense (Comptroller), and the Director
of the Office of Program Analysis and Evaluation serve as advisors to
the JROC on matters within their authority and expertise.[Footnote 8]
In 2009, WSARA expanded the role of the JROC by directing it to assist
the Chairman of the Joint Chiefs of Staff in (1) ensuring that trade-
offs among cost, schedule, and performance objectives are considered
for joint military requirements; and (2) establishing an objective
period of time within which an initial operational capability should
be delivered. WSARA also stated that the newly constituted Director of
Cost Assessment and Program Evaluation (CAPE) would advise the JROC.
[Footnote 9] The Ike Skelton National Defense Authorization Act for
Fiscal Year 2011 allowed the Vice Chairman of the Joint Chiefs of
Staff to direct senior leaders from combatant commands to serve as
members of the JROC when matters related to the area of responsibility
or functions of that command are under consideration. It also added
the Under Secretary of Defense for Policy, the Director of Operational
Test and Evaluation, and other civilian officials designated by the
Secretary of Defense as advisors to the JROC on issues within their
authority and expertise.[Footnote 10]
The JROC is supported in the JCIDS process by two Joint Capabilities
Boards (JCB) and seven Functional Capabilities Boards (FCB), each of
which is chaired by a general/flag officer or civilian equivalent.
JCBs and FCBs are responsible for specific Joint Capability Areas,
such as Force Protection, Logistics, or Battlespace Awareness. The
JCBs, FCBs, and associated FCB Working Groups review requirements
documents prior to JROC reviews. The JCB also serves as the validation
authority for requirements documents that are not associated with
major defense acquisition programs (MDAP).[Footnote 11] In some
instances, the JROC will not meet in person to approve requirements
documents if there are no outstanding issues to discuss.
The JROC and its supporting organizations review requirements
documents related to capability gaps and the MDAPs intended to fill
those gaps prior to key acquisition milestones. These requirements
documents--the Initial Capabilities Documents (ICD), Capability
Development Documents (CDD), and Capability Production Documents
(CPD)--are submitted by capability sponsors, which are generally the
military services, but can also be other DOD agencies or combatant
commands. Figure 1 depicts how JCIDS reviews align with the
acquisition process.
Figure 1: JCIDS Reviews and the DOD Acquisition Process:
[Refer to PDF for image: process illustration]
Capabilities assessment:
JCIDS reviews: Initial capabilities document (focus of this report).
Analysis of alternatives:
Acquisition milestone reviews:
Materiel Development Decision.
Milestone A: Validate selected alternatives.
Technology development:
JCIDS reviews: Capability development document (focus of this report).
Acquisition milestone reviews: Milestone B: Program start.
Engineering and manufacturing development:
JCIDS reviews: Capability production document.
Acquisition milestone reviews: Milestone C: Production start.
Production:
Source: GAO analysis of DOD policy.
Note: Our audit primarily focused on JROC efforts to ensure programs
were initiated on a sound basis, i.e., requirements validation prior
to program start.
[End of figure]
The ICD is the first requirements document reviewed in JCIDS. It is
intended to identify a specific capability gap, or set of gaps, in
joint military capabilities that are determined to require a materiel
solution as a result of a capabilities-based assessment. DOD policy
requires that the JROC validate the ICD prior to a Materiel
Development Decision, which is the formal entry point into the
acquisition process. The ICD does not contain specific cost, schedule,
or performance objectives. Once the JROC validates an ICD, the
Milestone Decision Authority, working with appropriate stakeholders,
shall determine whether to proceed to a Materiel Development
Decision.[Footnote 12] After the Materiel Development Decision, the
capability sponsor initiates an AOA to consider alternative solutions
to fulfilling the capability need described in an ICD, and possible
trade-offs among cost, schedule, and performance for each alternative
are considered.
The CDD is the second requirements document reviewed in JCIDS. It can
address capability gaps presented in one or more ICDs. The CDD is
intended to define a proposed program's Key Performance Parameters
(KPP), Key System Attributes (KSA), and other performance attributes.
KPPs are the system characteristics that the CDD sponsor considers
most critical to delivering that military capability, while KSAs are
system attributes the CDD sponsor considers essential for an effective
military capability, but a lower priority than the KPPs. DOD policy
calls for the JROC to validate the CDD to inform the Milestone B
decision, which marks the official start of an acquisition program and
entry into the engineering and manufacturing development phase. The
CDD is the first requirements document that contains cost, schedule,
and performance objectives.
The CPD is the third and final requirements document reviewed in
JCIDS. It is intended to refine the KPPs, KSAs, and performance
attributes validated in the CDD. DOD policy calls for the JROC to
validate the CPD to inform the Milestone C decision, which marks a
program's entry into production. Appendix II identifies the ICDs,
CDDs, and CPDs reviewed by the JCB or JROC in fiscal year 2010.
In addition to JCIDS reviews, the JROC reviews MDAP requirements after
a program experiences cost growth beyond JROC-specified amounts,
triggering what is called a tripwire review. The JROC's tripwire
policy directs military services to brief the JROC when unit cost
increases by 10 percent over the current baseline or 25 percent over
the original baseline. DOD officials explained this allows the JROC to
consider relaxing, deferring, or deleting a program's KPPs if they are
found to be unachievable or of lesser priority than reducing program
cost. The JROC also participates in what are commonly known as
critical Nunn-McCurdy[Footnote 13] reviews by providing an assessment
of whether the program is essential to national security. A critical
Nunn-McCurdy breach occurs when the program acquisition unit cost or
the average procurement unit cost increases by at least 25 percent
over the current baseline estimate or at least 50 percent over the
original baseline estimate. Appendix III lists the JROC's 15 fiscal
year 2010 Nunn-McCurdy and tripwire reviews.
JROC Did Not Always Consider Trade-offs or Influence Trade-off
Decisions:
The JROC considered trade-offs made by the military services before
validating requirements for four of the seven proposed programs it
reviewed in fiscal year 2010, and provided input to the military
services on the cost, schedule, and performance objectives for two of
the seven programs. The JROC's requirements review was the final step
in a long requirements vetting process, with most trade-offs being
made by the military services earlier in the process. Key stakeholders
from the offices of the Under Secretary of Defense (Comptroller), USD
AT&L, Director of CAPE, and the combatant commands were all satisfied
with their opportunities to provide input to the JROC; but they
provided limited input on trade-offs among cost, schedule, and
performance objectives, and used other means to influence trade-offs.
Perhaps most importantly, none of the JROC's requirements reviews
align with the AOA, which is where the military services reported
making the most significant trade-offs. As a result, a program can
spend significant time in technology development before the JROC gets
to formally weigh in on these trade-offs through the JCIDS process.
The JROC also reviews MDAP requirements after a program enters
development and experiences substantial cost growth. DOD and the JROC
stated that requirements were not the primary causes of cost growth
for the 15 programs reviewed for this purpose in fiscal year 2010 and
the JROC did not change any KPPs to mitigate the reported cost growth.
JROC Did Not Always Consider Trade-offs when Validating Requirements
for Proposed Programs:
The JROC considered trade-offs made by the military services before
validating requirements for four of the seven proposed programs it
reviewed in fiscal year 2010. On three programs, the JROC did not
receive information on the potential cost and schedule implications of
each of the alternatives considered. Table 1 summarizes the JROC's
consideration of cost, schedule, and performance objectives for the
seven proposed MDAPs it reviewed in fiscal year 2010.
Table 1: JROC Consideration of Trade-offs Among Cost, Schedule, and
Performance Objectives for Seven Proposed Programs in Fiscal Year 2010:
Proposed program: Common Vertical Lift Support Platform (CVLSP);
Capability description: The CVLSP program is expected to provide a
helicopter that will support nuclear security and passenger transport
missions, and improve carrying capacity, range, speed, survivability,
and battlespace awareness among other capabilities;
Did JROC consider cost, schedule, and performance trade-offs? Yes. The
JROC established rapid fielding as the priority and instructed the Air
Force to discuss trade-offs with the acquisition and testing
communities if the system's requirements were adversely affecting its
rapid fielding.
Proposed program: HH-60 Recapitalization;
Capability description: The HH-60 Recapitalization program is expected
to provide a helicopter that will support personnel recovery missions,
and be capable of operating day or night, in adverse weather, and
amongst a variety of threats;
Did JROC consider cost, schedule, and performance trade-offs? Yes. The
JROC approved the decision to change combat radius, survivability,
cabin space, payload, and airspeed requirements to decrease cost.
Proposed program: Army Integrated Air and Missile Defense System of
Systems (AIAMD SOS), Increment 2;
Capability description: The AIAMD SOS, Increment 2 program is expected
to provide an air and missile defense system that will integrate
sensors and weapons across an integrated fire control network;
Did JROC consider cost, schedule, and performance trade-offs? Yes. The
JROC approved the decision to accept greater cost and schedule risk in
order to meet performance objectives.
Proposed program: Ground Soldier System, Increment 1[A];
Capability description: The Ground Soldier System, Increment 1 program
is expected to provide a communications system that will improve
ground combat leaders' battle command and situational awareness
capabilities;
Did JROC consider cost, schedule, and performance trade-offs? Yes. The
JROC emphasized the importance of weight and mobility and directed the
Army to seek JROC approval if the operating weight exceeded a specific
threshold.
Proposed program: Joint Precision Approach and Landing System (JPALS),
Increment 2;
Capability description: The JPALS, Increment 2 program is expected to
provide avionics systems and mobile and fixed ground stations that
will increase the efficiency of approach and landing operations;
Did JROC consider cost, schedule, and performance trade-offs? No. The
JROC received brief descriptions of seven potential solutions, but the
cost and schedule implications of each alternative were not identified.
Proposed program: P-8A, Increment 3[B];
Capability description: The P-8A, Increment 3 program is expected to
provide an antisurface warfare weapon and improved communications
capabilities;
Did JROC consider cost, schedule, and performance trade-offs? No. The
JROC received brief descriptions of seven analyses supporting the P-8A
program, but they did not explicitly discuss Increment 3 cost,
schedule, and performance objectives.
Proposed program: Ship to Shore Connector;
Capability description: The Ship to Shore Connector program is
expected to provide an air cushion vehicle for transporting vehicles,
cargo, and personnel from ship to shore, and will replace the current
Landing Craft Air Cushion;
Did JROC consider cost, schedule, and performance trade-offs? No. The
JROC received brief descriptions of three potential solutions, but the
cost and schedule implications of each alternative were not identified.
Source: GAO analysis of JCIDS and acquisition documents.
[A] The Ground Soldier System, Increment 1 program has been renamed
Nett Warrior.
[B] The proposed P-8A, Increment 3 program has not yet reached
Milestone A, but at the direction of the Joint Staff, its requirements
were approved in the same document as the proposed P-8A, Increment 2
program.
[End of table]
The JROC's review of the CDD for a proposed program is the final step
in a long requirements vetting process, and DOD officials reported
that trade-offs typically occur earlier in the process. Each military
service conducts its own internal requirements reviews for its
proposed programs, which are used to refine requirements documents
before they are submitted into JCIDS. Military service officials
reported that they make significant trade-offs during these internal
reviews, and that KPPs and technical requirements rarely change after
requirements documents are submitted into JCIDS because extensive
analysis has already been conducted. For the seven proposed MDAPs we
reviewed, the military services generally submitted requirements to
the JROC that would be fully funded, provide initial capability within
6 years, utilize critical technologies that were nearing maturity, and
be acquired using an incremental approach. These characteristics are
consistent with provisions in the Weapon Systems Acquisition Reform
Act (WSARA) related to how the requirements process should be
structured[Footnote 14] and aspects of GAO's best practices for weapon
system acquisitions.[Footnote 15]
Two of the proposed program requirements presented to the JROC
included major trade-offs among cost, schedule, and performance
objectives and revisions to their acquisition approaches that had been
made after predecessor programs were canceled over affordability
concerns. The Air Force initiated the HH-60 Recapitalization program
after the Combat Search and Rescue Replacement Vehicle (CSAR-X)
program was canceled, and the HH-60 Recapitalization program is
expected to decrease cost by changing cabin space, velocity, and range
from the CSAR-X requirements. In 2007, the Army, with input from a
Functional Capabilities Board, decided to use an incremental
acquisition approach for the Ground Soldier System in order to reduce
costs, meet schedule demands, and avoid some of the mistakes made
during the Land Warrior program, which was canceled because of funding
and cost issues.
Key Stakeholders Provided Limited Input into JCIDS, but Use Other
Means to Influence Trade-offs:
The JROC received limited input on trade-offs among cost, schedule,
and performance objectives from key stakeholders when validating
requirements for the seven proposed MDAPs we reviewed from fiscal year
2010. Both WSARA and the National Defense Authorization Act for Fiscal
Year 2008 directed the JROC to consult with the Under Secretary of
Defense (Comptroller), the USD AT&L, and the Director of CAPE.
[Footnote 16] Additionally, WSARA instructed the JROC to consult with
the combatant commands. Officials from these organizations reported
that they had ample opportunity to participate in JROC requirements
reviews, and Joint Staff officials said efforts to involve these
stakeholders preceded WSARA. However, officials from the offices of
the Under Secretary of Defense (Comptroller), USD AT&L, and the
Director of CAPE also reported that the acquisition and
budgeting/funding processes are the primary mechanisms through which
they influence programs, rather than JCIDS. For example, CAPE oversees
AOAs for MDAPs and has an opportunity to provide input and guidance on
AOA considerations. Further, the combatant commands reported that they
most often submit prioritized lists of capability gaps directly to the
Chairman of the Joint Chiefs of Staff as part of the resource
allocation process, which is separate from JCIDS.[Footnote 17]
Nonetheless, joint stakeholders did provide some significant input
during the JROC's reviews of the seven proposed programs in fiscal
year 2010. For example, the Army more fully defined a Ground Soldier
System, Increment 1 KPP in response to input from DOD's Joint
Interoperability Test Command, and in another instance, the Army added
a KSA to the AIAMD SOS, Increment 2 CDD due to input from the office
of the USD AT&L, the Defense Information Systems Agency, and the Joint
Staff. Neither of these changes involved trade-offs among cost,
schedule, and performance objectives.
JCIDS Reviews Are Not Aligned with the Most Significant Trade-off
Decisions:
The JROC does not formally review the trade-off decisions made as a
result of an AOA until a proposed program's CDD enters the JCIDS
process. According to DOD officials, the most significant trade-offs
are made by the military services between ICD and CDD reviews during
the AOA, which is intended to compare the operational effectiveness,
cost, and risks of a number of alternative potential solutions. For
example, during the CVLSP AOA, the Air Force decided to decrease troop
transport capacity in order to reduce cost. Alternatively, during the
AIAMD SOS AOA, the Army decided to pursue the most costly option
reviewed because it provided greater capability. A significant amount
of time and resources can be expended before the JROC gets to weigh in
on these trade-offs during CDD reviews. For example, the JROC reviewed
the AOA summary for JPALS, Increment 2, 4 years after the conclusion
of the AOA. During the time between the AOA and the CDD review, the
technology intended to enable the chosen alternative is developed.
Figure 2 shows the AOA's relationship to both the requirements and
acquisition processes.
Figure 2: AOA's Relationship to JROC Reviews:
[Refer to PDF for image: illustration]
ICD developed: No cost, schedule, performance objectives.
(Requirements development)
JROC review: Too early for trade-offs.
Analysis of alternatives: Most significant trade-offs made.
(Acquisition activity)
Milestone A. (Acquisition milestone)
Technology development: Requirements refined and vetted. (Requirements
development)
CDD developed: Cost, schedule, performance objectives identified.
(Acquisition activity)
JROC review: Requirements rarely change.
Milestone B. (Acquisition milestone)
Source: GAO analysis of DOD policy.
[End of figure]
Joint Staff officials have stated that establishing a JROC review of
the AOA would allow it to provide military advice on trade-offs and
the proposed materiel solution before Milestone A, and an ongoing
Joint Staff review of JCIDS is considering an increased role for the
JROC at this point.[Footnote 18] According to the Joint Staff,
increased JROC engagement at these early stages of the acquisition
process is warranted to align it with other elements of recent
acquisition reforms. For example, WSARA emphasized that the AOA should
fully consider possible trade-offs among cost, schedule, and
performance objectives for each alternative considered,[Footnote 19]
and in September 2010, USD AT&L issued a memorandum that emphasized
the need for trade-offs from a program's inception.[Footnote 20] The
memorandum also dictated that affordability targets shall be
established at the conclusion of the AOA and that these targets will
be treated like KPPs, even though they will be set and managed by the
acquisition, not requirements, community.
JROC Did Not Change KPPs when Programs Incurred Substantial Cost
Growth:
The JROC did not change any KPPs during 15 reviews of programs that
reported substantial cost growth in fiscal year 2010. According to the
Joint Staff, by holding requirements firm and accepting increased cost
and schedule delays, the JROC essentially traded cost and possibly
schedule for performance. In fiscal year 2010, the JROC reviewed six
programs after they experienced a critical Nunn-McCurdy breach and
nine programs as part of the tripwire process. During all 15 reviews,
DOD and the JROC stated that requirements were not the primary causes
of cost growth. For all six programs that experienced a critical Nunn-
McCurdy cost breach, the JROC validated the system's capabilities as
being essential to national security and did not make any changes to
their KPPs. For all nine programs that were approaching Nunn-McCurdy
thresholds, the JROC did not identify opportunities to mitigate cost
growth by modifying requirements. Most of these programs were in
production in fiscal year 2010, and changing requirements at this late
stage might not have mitigated the reported cost growth. When the JROC
reviewed the Family of Advanced Beyond Line-of-Sight Terminals
program, which was still in development, it concluded that the
program's requirements could not be met in an affordable manner. The
JROC did not immediately defer any of the program's requirements, but
instead requested that USD AT&L identify potential alternatives for
the program, including reviewing whether adjustments to performance
requirements would be appropriate.
Military Services Did Not Consistently Provide High-Quality Resource
Estimates to JROC:
The military services did not consistently provide high-quality
resource estimates to the JROC to support its review of requirements
for 7 proposed programs in fiscal year 2010. We found the estimates
presented to the JROC were often unreliable when assessed against best
practices criteria. The type of resource estimates the military
services presented to the JROC varied from ones that had been
validated by the military services' cost analysis agencies to less
rigorous rough-orders-of-magnitude estimates. In most cases, the
military services had not effectively conducted uncertainty and
sensitivity analyses, which establish confidence levels for resource
estimates, based on the knowledge available, and examine the effects
of changing assumptions and ground rules. Lacking risk and uncertainty
analysis, the JROC cannot evaluate the range of resources that might
be necessary to cover increased costs resulting from unexpected design
complexity, technology uncertainty, and other issues. The lack of this
information affects the JROC's efforts to ensure that programs are
fully funded and its ability to consider the resource implications of
cost, schedule, and performance trade-offs.
JROC Received Resource Estimates That Did Not Meet Best Practices:
The JROC first receives resource estimates for proposed programs when
it reviews CDDs, and when we reviewed the CDD resource estimates
presented to the JROC in fiscal year 2010, we found that they were
generally unreliable when assessed against our best practices
criteria.[Footnote 21] While most of the resource estimates
substantially met our criteria for a comprehensive resource estimate,
they generally were not very accurate, credible, or well-documented.
Appendix IV includes a list of the best practices against which we
assessed these resource estimates.
The type of resource estimates the military services presented to the
JROC varied from ones that had been validated by the military
services' cost analysis agencies to less rigorous rough-orders-of-
magnitude estimates.[Footnote 22] According to Joint Staff officials,
military services can initiate CDD reviews at any point in the
acquisition process prior to program start, even if good resource
estimates are not available. For example, the JROC validated the P-8A,
Increment 3 CDD more than 2 years before the program was expected to
start, before an AOA had been completed, and with a rough-order-of-
magnitude estimate. Joint Staff officials reported that they depend on
CAPE to review the quality of resource estimates during the JCIDS
process, but CAPE cost assessment officials told us that they rarely
participate in JCIDS reviews.
Regardless of the type of resource estimate, uncertainty and
sensitivity analysis can establish confidence levels for resource
estimates, based on the knowledge available at the time, and examine
the effects of changing assumptions and ground rules, including those
related to trade-offs among cost, schedule, and performance
objectives. The military services sponsoring the requirements
generally did not effectively meet best practices for uncertainty and
sensitivity analyses using the knowledge they had available to them
for any of the seven resource estimates we reviewed. Figure 3
summarizes our assessment of the resource estimates presented to the
JROC against our best practices criteria.
Figure 3: GAO Assessment of Resource Estimates Presented to the JROC
During CDD Reviews in Fiscal Year 2010:
[Refer to PDF for image: illustrated table]
CDDs: Ship to Shore Connector;
Type of resource estimate: Naval Center for Cost Analysis validated
estimate;
Characteristics of a reliable resource estimate:
Comprehensive: Substantially met – evidence satisfied a large portion
of the criterion;
Accurate: Substantially met – evidence satisfied a large portion of
the criterion;
Credible: Substantially met – evidence satisfied a large portion of
the criterion;
Well-documented: Partially met – evidence satisfied about half of the
criterion.
CDDs: JPALS, Increment 2;
Type of resource estimate: Air Force cost estimate;
Characteristics of a reliable resource estimate:
Comprehensive: Substantially met – evidence satisfied a large portion
of the criterion;
Accurate: Substantially met – evidence satisfied a large portion of
the criterion;
Credible: Minimally met – evidence satisfied a small portion of the
criterion;
Well-documented: Substantially met – evidence satisfied a large
portion of the criterion.
CDDs: CVLSP;
Type of resource estimate: Air Force Cost Analysis Agency validated
estimate[A];
Characteristics of a reliable resource estimate:
Comprehensive: Substantially met – evidence satisfied a large portion
of the criterion;
Accurate: Partially met – evidence satisfied about half of the
criterion;
Credible: Partially met – evidence satisfied about half of the
criterion;
Well-documented: Partially met – evidence satisfied about half of the
criterion.
CDDs: AIAMD SOS, Increment 2;
Type of resource estimate: Army cost position;
Characteristics of a reliable resource estimate:
Comprehensive: Substantially met – evidence satisfied a large portion
of the criterion;
Accurate: Partially met – evidence satisfied about half of the
criterion;
Credible: Partially met – evidence satisfied about half of the
criterion;
Well-documented: Partially met – evidence satisfied about half of the
criterion.
CDDs: Ground Soldier System, Increment 1;
Type of resource estimate: Program office estimate;
Characteristics of a reliable resource estimate:
Comprehensive: Substantially met – evidence satisfied a large portion
of the criterion;
Accurate: Partially met – evidence satisfied about half of the
criterion;
Credible: Minimally met – evidence satisfied a small portion of the
criterion;
Well-documented: Partially met – evidence satisfied about half of the
criterion.
CDDs: HH-60 Recapitalization;
Type of resource estimate: Rough-order-of-magnitude estimate;
Characteristics of a reliable resource estimate:
Comprehensive: Minimally met – evidence satisfied a small portion of
the criterion;
Accurate: Minimally met – evidence satisfied a small portion of the
criterion;
Credible: Not met – no evidence satisfied any of the criterion;
Well-documented: Minimally met – evidence satisfied a small portion of
the criterion.
CDDs: P-8, Increment 3;
Type of resource estimate: Rough-order-of-magnitude estimate;
Characteristics of a reliable resource estimate:
Comprehensive: Minimally met – evidence satisfied a small portion of
the criterion;
Accurate: Not met – no evidence satisfied any of the criterion;
Credible: Not met – no evidence satisfied any of the criterion;
Well-documented: Not met – no evidence satisfied any of the criterion.
Source: GAO analysis of DOD data.
[A] Air Force Cost Analysis Agency found the CVLSP estimate adequate
for an AOA, but not for budgeting purposes.
[End of figure]
Most Resource Estimates Substantially Met Comprehensiveness Criteria:
Five of the seven CDD resource estimates substantially met our
criteria for a comprehensive resource estimate. The resource estimates
generally completely defined their respective programs, and included
most, if not all, life-cycle costs. The Ship to Shore Connector,
CVLSP, and JPALS, Increment 2 resource estimates also effectively
documented all cost-influencing ground rules and assumptions, although
the other resource estimates did not. Additionally, only the Ship to
Shore Connector's work breakdown structure effectively met our
criteria, which require that work breakdown structures are product-
oriented and at an appropriate level of detail. If a resource estimate
does not specifically break out common costs, such as government-
furnished equipment costs, or does not include an associated work
breakdown structure dictionary, cost estimators cannot ensure that the
estimate includes all relevant costs.
The HH-60 Recapitalization and P-8A, Increment 3 resource estimates
did not effectively meet any of our best practices for a comprehensive
resource estimate. Unless resource estimates account for all costs,
they cannot enhance decision making by allowing for design trade-off
studies to be evaluated on a total cost, technical, and performance
basis. Additionally, unless ground rules and assumptions are clearly
documented, the resource estimate will not have a basis for resolving
areas of potential risk.
Most Resource Estimates Did Not Substantially Meet Accuracy Criteria:
Only two of the seven CDD resource estimates substantially met our
criteria for an accurate resource estimate, while three partially met
the criteria, and two did not meet or minimally met the criteria. We
found that the Ship to Shore Connector, CVLSP, AIAMD SOS, Increment 2,
and the Ground Soldier System, Increment 1 resource estimates
contained few, if any, minor mistakes, and that the Ship to Shore
Connector, CVLSP, and JPALS, Increment 2 resource estimates were
appropriately adjusted for inflation. Additionally, we found that the
Ship to Shore Connector and JPALS, Increment 2 resource estimates were
based on historical records of actual experiences from other
comparable programs.
However, we generally found that the resource estimates were not
consistent with our best practices. Accurate resource estimates are
rooted in historical data, which provide cost estimators with insight
into actual costs of similar programs, and can be used to challenge
optimistic assumptions and bring more realism to a resource estimate.
Unless an estimate is based on an assessment of the most likely costs,
and reflects the degree of uncertainty given all of the risks
considered, management will not be able to make well-informed
decisions.
Most Resource Estimates Were Not Credible:
Four of the seven CDD resource estimates did not meet or minimally met
our criteria for a credible resource estimate, and only the Ship to
Shore Connector resource estimate substantially met the criteria. The
Ship to Shore Connector and AIAMD SOS, Increment 2 resource estimates
included sensitivity analyses that identified a range of possible
costs based on varying assumptions, parameters, and data inputs, but
none of the other resource estimates included this analysis. As a best
practice, sensitivity analysis should be included in all resource
estimates because it examines the effects of changing assumptions and
ground rules. Since uncertainty cannot be avoided, it is necessary to
identify the cost elements that represent the most risk and, if
possible, cost estimators should quantify that risk. When an agency
fails to conduct sensitivity analysis to identify the effect of
uncertainties associated with different assumptions, this increases
the chance that decisions will be made without a clear understanding
of the impact on cost.
Additionally, only the Ship to Shore Connector resource estimate
effectively met our best practices for risk and uncertainty analysis.
For management to make good decisions, the program estimate must
reflect the degree of uncertainty so that a level of confidence can be
given about the estimate. An estimate without risk and uncertainty
analysis is unrealistic because it does not assess the variability in
the resource estimate from effects such as schedules slipping,
missions changing, and proposed solutions not meeting users' needs.
Lacking risk and uncertainty analysis, management cannot determine a
defensible level of contingency reserves that is necessary to cover
increased costs resulting from unexpected design complexity,
technology uncertainty, and other issues.
Further, none of the planned programs effectively met our criteria for
an independent cost estimate when they were reviewed by the JROC. An
independent cost estimate is considered one of the best and most
reliable resource estimate validation methods because it provides an
independent view of expected program costs that tests the program
office and service estimates for reasonableness. Without an
independent cost estimate, decision makers lack insight into a
program's potential costs because these estimates frequently use
different methods and are less burdened with organizational bias.
Moreover, independent cost estimates tend to incorporate adequate
risk, and therefore tend to be more conservative by forecasting higher
costs than the program office. A program estimate that has not been
reconciled with an independent cost estimate has an increased risk of
proceeding underfunded because an independent cost estimate provides
an objective and unbiased assessment of whether the program estimate
can be achieved. Alternatively, programs can reinforce the credibility
of their resource estimates through cross-checking, which determines
whether alternative cost estimating methods produce similar results.
However, only the Ship to Shore Connector resource estimate
effectively met our best practices for cross-checking.
Most Resource Estimates Were Not Well-documented:
Only the JPALS, Increment 2 resource estimate substantially met our
criteria for a well-documented resource estimate, while four of the
seven CDD resource estimates partially met our criteria, and two of
the resource estimates did not meet or minimally met the criteria. The
JPALS, Increment 2 and CVLSP resource estimates sufficiently described
the calculations performed and estimating methodologies used to derive
each program element's cost. Additionally, the JPALS, Increment 2,
Ship to Shore Connector, and AIAMD SOS, Increment 2 documentation
clearly discusses the technical baseline description, and the data in
the technical baseline are consistent with the resource estimate.
However, none of the documents effectively described how the resource
estimates were developed in a manner that a cost analyst unfamiliar
with the program could understand what was done and replicate it.
We generally found that the resource estimates were not consistent
with our best practices for a well-documented resource estimate.
Documentation is essential for validating and defending a resource
estimate. Without a well-documented resource estimate, a convincing
argument of an estimate's validity cannot be presented, and decision
makers' questions cannot be effectively answered. Poorly documented
resource estimates cannot explain the rationale of the methodology or
the calculations underlying the cost elements. Further, a well-
documented resource estimate is essential for an effective independent
review to ensure that the resource estimate is valid and credible.
Unless the estimate is fully documented, it will not support
reconciliation with an independent cost estimate, hindering
understanding of cost elements and their differences.
Military Services Generally Presented Resource Estimates That Were
Fully Funded to JROC:
The JROC required the military services to show that the proposed
programs were fully funded to the resource estimates presented by the
military services before it validated requirements for five of the
seven proposed MDAPs we reviewed from fiscal year 2010; the two other
proposed MDAPs were funded at more than 97 and 99 percent
respectively.[Footnote 23] However, we found that these resource
estimates were generally unreliable, which undermined the JROC's
efforts. In 2007, the JROC issued guidance instructing the military
services to commit to funding the requirements that the JROC
validates. The guidance emphasized the need for full funding in an
effort to facilitate sound fiscal and risk decisions.[Footnote 24]
However, the JROC does not explicitly consider a requirement's
affordability in a broader context during JCIDS reviews. DOD funding
plans are captured in the future-years defense program, which presents
resource information for the current year and the following 4 years.
The future-years defense program is updated twice per year to reflect
the military services' input and the budget the President submits to
Congress. Statute and DOD acquisition policy also require programs to
be fully funded through the period covered by the future-years defense
program.[Footnote 25]
One of the seven proposed MDAPs we reviewed from fiscal year 2010
included a funding shortfall when its requirements were being reviewed
through JCIDS, but its CDD was not approved until the shortfall had
been addressed. Specifically, when the JCB reviewed the CVLSP CDD, the
funding plan included a $1.3 billion shortfall through fiscal year
2015. The JCB chairman directed the Air Force to modify the program's
funding plan before proceeding to the JROC review. When the Air Force
briefed the JROC on the CVLSP CDD approximately 8 months later, it
presented a funding plan that fully funded the program through the
future-years defense program time frame. The revised funding plan also
included more money for the program beyond the future-years defense
program time frame, and the total program cost increased from $14.2
billion to $15.2 billion.
Despite JROC efforts to ensure programs are fully funded, the military
services retain primary control over their budgets, and ultimately,
JROC decisions are influential but not binding. When the JCB reviewed
the JPALS, Increment 2 CDD, it requested clarification on the Air
Force's funding plan, and emphasized the need for full funding prior
to program start. The funding plan presented to the JCB included a
$77.7 million shortfall through fiscal year 2015, and the Air Force
had cut JPALS funding in the past. Following the JCB review, the JROC
issued a decision memorandum that documented the Air Force's
commitment to fully funding JPALS, Increment 2. However, in fiscal
years 2011 and 2012, the Air Force only funded approximately 30
percent of the resource estimate presented to the JCB.
JROC Did Not Consistently Prioritize Requirements and Capability Gaps:
The JROC does not currently prioritize requirements, consider
redundancies across proposed programs, or prioritize and analyze
capability gaps in a consistent manner. As a result, the Joint Staff
is missing an opportunity to improve military service and
departmentwide portfolio management efforts. A portfolio management
approach to weapon system investments would involve taking a
disciplined, integrated approach to prioritizing needs and allocating
resources in order to eliminate redundancies, gain efficiencies, and
achieve a balanced mix of executable programs. According to Army, Air
Force, and Navy officials, having a better understanding of warfighter
priorities from the JROC would be useful to inform both portfolio
management efforts and service budgets. A DOD review team examining
the JCIDS process is considering changes that would address the
prioritization of requirements. During its review of the capability
gaps presented in 12 ICDs in fiscal year 2010, the JROC did receive
some information on priorities and potential redundancies; however,
the sponsors presented this information in an inconsistent manner,
making it difficult for the JROC to assess the relative priority of
capability gaps across different ICDs.
JROC Does Not Prioritize Requirements or Consider Redundancies Across
Proposed Programs:
Under the current JCIDS process, the JROC does not prioritize
requirements or consider redundancies across proposed programs during
CDD reviews. In the National Defense Authorization Act for Fiscal Year
2008, Congress amended the U.S. Code to direct the JROC to help assign
priority levels for joint military requirements and ensure that
resource levels associated with those requirements are consistent with
the level of priority.[Footnote 26] The House Armed Services Committee
report accompanying the authorization act stated that clear JROC
priorities and budget guidance would allow for joint decision making,
as opposed to service-centric budget considerations.[Footnote 27] In
addition, we have previously recommended that DOD develop an analytic
approach within JCIDS to better prioritize and balance the capability
needs of the military services, combatant commands, and other defense
components.[Footnote 28] According to the Joint Staff and military
service officials, prioritization across programs still primarily
occurs through the Planning, Programming, Budgeting and Execution
process, which is the responsibility of the military services and the
Office of the Under Secretary of Defense (Comptroller).
The JCIDS manual does not currently require an analysis of potential
redundancies during CDD reviews. In our recently issued report on
government duplication, we noted that service-driven requirements and
funding processes continue to hinder integration and efficiency and
contribute to unnecessary duplication in addressing warfighter needs.
[Footnote 29] We have also previously reported that ineffective
collaboration precluded opportunities for commonality in unmanned
aircraft systems.[Footnote 30] In fiscal year 2010, the JROC met to
consider joint efficiencies between two such systems: the Navy's Broad
Area Maritime Surveillance system and the Air Force's Global Hawk
system. The JROC requested that the Navy and Air Force ensure that a
common component was interoperable between the two systems, and that
the Air Force consider an all-weather capability developed by the
Navy. The JROC has also supported joint development efforts for these
programs and requested annual status updates. According to Broad Area
Maritime Surveillance program officials, the Air Force and Navy
programs are investigating commonality opportunities, including sense-
and-avoid capabilities, a consolidated maintenance hub, and basing
options for both systems. The JROC did not meet to consider any other
joint efficiencies across military services in fiscal year 2010.
The Joint Staff has acknowledged that the JROC should play a larger
role in prioritizing needs and addressing redundancies. In July 2010,
the Vice Chairman of the Joint Chiefs of Staff initiated a review of
the JCIDS process. One of the goals of the review team was to develop
metrics and criteria to ensure the JCIDS process has the ability to
rank or prioritize needs. The review team's charter states that these
metrics must enable more structured reviews of portfolio gaps and
redundancies. According to the Joint Staff, the review team is
considering a number of recommendations including asking the JROC to
prioritize requirements based on the urgency and significance of the
need. This list of priorities could be used to inform military service
budgets. Joint Staff officials have also stated that redundancies may
be addressed more directly in the future as part of an enhanced
portfolio management effort.
Lack of JROC Prioritization Results in Missed Opportunities to Manage
Portfolios Better:
We have previously reported that DOD has not taken a portfolio
management approach to weapon system investments, which would involve
taking a disciplined, integrated approach to prioritizing needs and
allocating resources in order to eliminate redundancies, gain
efficiencies, and achieve a balanced mix of executable programs.
[Footnote 31] In September 2010, USD AT&L issued guidance intended to
increase efficiencies and eliminate redundancies, and it presented the
Army's portfolio management activities as an example to emulate.
[Footnote 32] The Army uses capability portfolio reviews of capability
gaps and proposed and existing programs to revalidate, modify, or
terminate requirements and ensure the proper allocation of funds
between them. The Army has established 17 portfolios, including
aviation, air and missile defense, and combat vehicle modernization.
An Army official involved in the portfolio reviews said that he has
requested on several occasions for the Joint Staff to prioritize
warfighter needs; however, the JROC has not done so. Instead, the Army
relies on its own prioritization information during the portfolio
reviews to help determine the capability areas where it is willing to
assume risk. Air Force and Navy officials have also stated that they
could benefit from JROC prioritization of requirements, and that this
information would be useful in order to better allocate resources
during their budget formulation activities.
Sponsors Do Not Prioritize Capability Gaps or Analyze Potential
Redundancies in a Consistent Manner:
The JROC has required that capability sponsors prioritize capability
gaps and identify redundancies when developing ICDs,[Footnote 33] and
capability sponsors generally complied with these requirements in the
12 validated ICDs we reviewed from fiscal year 2010.[Footnote 34]
However, the sponsors presented this information in an inconsistent
manner, making it difficult for the JROC and the military services to
assess priorities and redundancies across ICDs or use this information
to inform resource allocation decisions. For example, the Electronic
Health Record ICD prioritized its gaps in numerical order from 1 to
10,[Footnote 35] but the Command and Control On-The-Move ICD labeled
half its gaps medium priority and the other half high priority.
[Footnote 36] The JCIDS operation manual provides limited guidance on
how capability sponsors should prioritize the gaps, stating only that
the prioritization should be based on the potential for operational
risk associated with the shortfalls. The JCIDS manual also directs
capability sponsors to identify redundancies and assess whether the
overlap is operationally acceptable or whether it should be evaluated
as part of the trade-offs to satisfy capability gaps. Three of the 12
validated ICDs we reviewed from fiscal year 2010 did not address
redundancies. Furthermore, only one of these ICDs presented to the
JROC in fiscal year 2010 included an evaluation of the overlaps. The
JROC did not address these omissions when it validated the documents.
Conclusions:
In the last several years, Congress has passed legislation to give the
JROC a greater role in prioritizing military requirements and shaping
sound acquisition programs by encouraging cost, schedule, and
performance trade-offs. Taken together, these steps have the potential
to improve the affordability and execution of DOD's portfolio of major
defense acquisition programs. However, the JROC has largely left
prioritization and trade-off decisions to the military services,
despite having a unique, joint perspective, which would allow it to
look across the entire department to identify efficiencies and
potential redundancies. To more effectively leverage its unique
perspective, the JROC would have to change the way it views its role,
more regularly engage the acquisition community in trade-off
discussions at early acquisition milestones, and more effectively
scrutinize the quality of the resource estimates presented by the
military services. Until it does so, the JROC will only be a marginal
player in DOD's efforts to align the department's available resources
with its warfighting requirements.
Recommendations for Executive Action:
To enhance the JROC's role in DOD-wide efforts to deliver better value
to the taxpayer and warfighter, we recommend that the Vice Chairman of
the Joint Chiefs of Staff, as chairman of the JROC, take the following
five actions:
* Establish a mechanism to review the final AOA report prior to
Milestone A to ensure that trade-offs have been considered and to
provide military advice on these trade-offs and the proposed materiel
solution to the Milestone Decision Authority.
* Require that capability sponsors present resource estimates that
have been reviewed by a military service's cost analysis organization
to ensure best practices are being followed.
* Require that capability sponsors present key results from
sensitivity and uncertainty analyses, including the confidence levels
associated with resource estimates, based on the program's current
level of knowledge.
* Assign priority levels to the CDDs based on joint force capability
gaps and redundancies against current and anticipated threats, and
provide these prioritization levels to the Under Secretary of Defense
(Comptroller) and the military services to be used for resource
allocation purposes.
* Modify the JCIDS operations manual to require that CDDs discuss
potential redundancies across proposed and existing programs, and
address these redundancies when validating requirements.
Agency Comments and Our Evaluation:
The Joint Staff provided us written comments on a draft of this
report. The comments are reprinted in appendix V. The Joint Staff also
provided technical comments, which we addressed in the report, as
appropriate.
In its comments, the Joint Staff partially concurred with all five of
our recommendations, generally agreeing that there is a need to take
action to address the issues we raised, but differing in terms of the
specific actions that should be taken.
The Joint Staff partially concurred with our recommendation that the
Vice Chairman of the Joint Chiefs of Staff, as chairman of the JROC,
establish a mechanism to review the final AOA report prior to
Milestone A to ensure that trade-offs have been considered and to
provide military advice on these trade-offs and the proposed materiel
solution to the Milestone Decision Authority. The Joint Staff noted
that its ongoing review of JCIDS will include a recommendation that
AOA results be briefed to FCBs. However, the FCB will only elevate
these briefings to the JCB or JROC on an exception basis. The Joint
Staff explained that this approach would allow the JROC to provide
more informed advice to a Milestone Decision Authority without adding
another round of staffing, an additional JCIDS document, or an
official validation of AOA results. We agree that the Joint Staff
should seek to implement this recommendation in the most efficient and
effective way possible; however, given our finding that the most
significant trade-off decisions are made as a result of an AOA, we
continue to believe that the results should be reviewed by the JROC.
The Joint Staff partially concurred with our recommendation that the
Vice Chairman of the Joint Chiefs of Staff require that capability
sponsors present resource estimates that have been reviewed by a
military service's cost analysis organization to ensure best practices
are being followed. The Joint Staff stated that program office cost
estimates are compared to independent cost estimates during CDD
reviews. However, none of the seven CDD cost estimates we reviewed
effectively met our criteria for an independent cost estimate. As a
result, we believe that the Joint Staff needs to take additional
action to ensure that resource estimates presented by capability
sponsors have been reviewed by a military service's cost analysis
organization. The Joint Staff also stated that its ongoing review of
JCIDS will examine how to highlight this area during CDD reviews.
The Joint Staff partially concurred with our recommendation that the
Vice Chairman of the Joint Chiefs of Staff require that capability
sponsors present key results from sensitivity and uncertainty
analyses, including the confidence levels associated with resource
estimates, based on the program's current level of knowledge. The
Joint Staff stated that our recommendation needs further study to
understand the expected outcomes and the required authorities for the
JROC, and its ongoing review of JCIDS will examine how to highlight
this area. We believe that the JROC cannot fully consider trade-offs
or the affordability of a proposed program unless it receives
information on the risk and uncertainty associated with resource
estimates; it does not need additional authority to require capability
sponsors to present the results of this type of analysis before it
approves proposed requirements. The Joint Staff also noted that the
Director, CAPE, has cost analysis responsibilities for resource
estimates. CAPE cost assessment officials reported that they rarely
participated in JCIDS reviews. As a result, the JROC may have to be
more proactive in reaching out to CAPE to help it understand the risk
and uncertainty associated with the resource estimates it receives.
The Joint Staff partially concurred with our recommendation that the
Vice Chairman of the Joint Chiefs of Staff assign priority levels to
CDDs based on joint force capability gaps and redundancies against
current and anticipated threats, and provide these prioritization
levels to the Under Secretary of Defense (Comptroller) and the
military services to be used for resource allocation purposes. The
Joint Staff agreed that the identification of joint priorities could
enhance a number of processes, including program and budget reviews.
It noted that its ongoing review of JCIDS will recommend a
prioritization framework through which CDDs will inherit priority
levels based on the requirements and capability gaps identified in
ICDs or Joint Urgent Operational Needs. However, the Joint Staff
argued against prioritizing based on CDDs directly because it would
provide less flexibility. We believe that the proposed approach could
be effective if the Joint Staff addresses the inconsistencies we found
in the way ICDs prioritize gaps. In addition, we continue to believe
that the prioritization framework should facilitate an examination of
priorities across CDDs.
The Joint Staff partially concurred with our recommendation that the
Vice Chairman of the Joint Chiefs of Staff modify the JCIDS operations
manual to require that CDDs discuss potential redundancies across
proposed and existing programs, and address these redundancies when
validating requirements. The Joint Staff stated that its ongoing
review of JCIDS will address this issue by establishing unique
requirements as a higher priority than unnecessarily redundant
requirements, and by establishing a post-AOA review, which could also
be used to identify unnecessary redundancies. The Joint Staff did not
address whether it would update the JCIDS operations manual as
recommended and stated that reviewing assessments of redundancies in
CDDs would be late in the JCIDS process. We believe that potential
redundancies should be discussed at multiple points, including during
CDD reviews, because we found that several years can pass between the
conclusion of an AOA and this review. During that time, new redundancy
issues could emerge.
We are sending copies of this report to the Secretary of Defense; the
Chairman and Vice Chairman of the Joint Chiefs of Staff; the
Secretaries of the Army, Navy, and Air Force; and the Director of the
Office of Management and Budget. In addition, the report will be made
available at no charge on the GAO Web site at [hyperlink,
http://www.gao.gov].
If you or your staff have any questions concerning this report, please
contact me at (202) 512-4841. Contact points for our offices of
Congressional Relations and Public Affairs may be found on the last
page of this report. Staff members making key contributions to this
report are listed in Appendix VI.
Signed by:
Michael J. Sullivan:
Director, Acquisition and Sourcing Management:
[End of section]
Appendix I: Scope and Methodology:
To conduct our work, we reviewed relevant sections of Title 10 of the
U.S. Code, the Weapon Systems Acquisition Reform Act of 2009 (WSARA),
and the National Defense Authorization Act for Fiscal Year 2008 to
establish the role of the Joint Requirements Oversight Council (JROC)
in considering trade-offs among cost, schedule, and performance
objectives; reviewing the estimated level of resources needed to
fulfill these requirements; and prioritizing requirements. We also
reviewed Department of Defense (DOD), Joint Staff, and military
service guidance documents, as well as those for the Joint
Capabilities Integration and Development System (JCIDS) for developing
and validating military requirements, to determine how these roles
have been implemented in policy. To determine how these policies have
been implemented in practice, we analyzed information and capability
documents contained in the Joint Staff's Knowledge Management/Decision
Support tool.[Footnote 37] To do so, we first established how many
requirements documents--Initial Capabilities Documents (ICD),
Capability Development Documents (CDD), and Capability Production
Documents (CPD)--were reviewed by the JROC and Joint Capabilities
Board (JCB) during fiscal year 2010. We selected fiscal year 2010 as
our time frame because WSARA was enacted in May 2009, and this would
allow for any changes the JROC would implement as a result of this
legislation. We then focused our analysis on the unclassified
requirement documents reviewed by the JROC and JCB in fiscal year 2010
which identified capability gaps or defined performance requirements
for new major defense acquisition programs: 13 ICDs and 7 CDDs. We
assessed these documents, as well as briefings presented to the JROC
or the JCB, associated meeting minutes, and JROC decision memos. We
also examined 15 JROC reviews of programs that incurred substantial
cost growth after program start in fiscal year 2010 to determine if
cost, schedule, and performance trade-offs were made. We chose this
time period to allow for any changes the JROC would implement as
result of the enactment of WSARA in May 2009.
To determine the extent to which the JROC has considered trade-offs
among cost, schedule, and performance objectives within programs, we
reviewed the seven CDDs submitted to the JROC and analyzed the
information presented on trade-offs. We focused on CDDs because they
are the first requirements documents that contain cost, schedule, and
performance objectives. We also examined JROC decision memos to
identify whether the JROC provided input on cost, schedule, and
performance objectives for the seven proposed programs and analyses of
alternatives (AOA) conducted by the military services prior to JROC
reviews. We also met with officials from the Joint Staff; Department
of the Air Force; Department of the Army; Department of the Navy;
Office of the Director of Cost Assessment and Program Evaluation
(CAPE); Office of the Under Secretary of Defense (Comptroller); Office
of the Assistant Secretary of Defense for Research and Engineering;
and respective program offices about these issues. To obtain combatant
command views on their participation in the joint requirements process
since the implementation of WSARA, we developed a survey administered
to DOD's 10 combatant commands.[Footnote 38] The survey addressed a
range of topics related to the joint requirements process, including
the means for combatant commands to provide information on their
capability needs. To understand the Joint Staff's ongoing internal
JCIDS review, we assessed the review charter and met with the Joint
Staff officials managing the review to discuss the recommendations
from the review and how they might affect the JROC's consideration of
trade-offs. We also observed joint requirements meetings and reviewed
prior GAO reports.
To determine the quality and effectiveness of efforts to estimate the
level of resources needed to fulfill joint military requirements, we
assessed the resource estimates used to support the seven unclassified
proposed major defense acquisition programs reviewed by the JROC in
fiscal year 2010 against the best practices in our cost estimating
guide.[Footnote 39] We used these criteria to determine the extent to
which these resource estimates were credible, well documented,
accurate, and comprehensive. We scored each best practice as either
being Not Met--DOD provided no evidence that satisfies any of the
criterion, Minimally Met--DOD provided evidence that satisfies a small
portion of the criterion, Partially Met--DOD provided evidence that
satisfies about half of the criterion, Substantially Met--DOD provided
evidence that satisfies a large portion of the criterion, and Met--DOD
provided complete evidence that satisfies the entire criterion. We
determined the overall assessment rating by assigning each individual
rating a number: Not Met = 1, Minimally Met = 2, Partially Met = 3,
Substantially Met = 4, and Met = 5. Then, we took the average of the
individual assessment ratings to determine the overall rating for each
of the four characteristics. To perform this analysis, we obtained and
analyzed program resource estimate supporting documentation, including
service cost positions, technical descriptions, work breakdown
structures, technology readiness assessments, program schedules, and
AOA reports. We also interviewed program and cost estimating
officials, when necessary, to gather additional information on these
resource estimates and the cost models used to produce them. Each
program was also provided with a copy of our assessment of their
resource estimates for review and comment.
To determine the extent to which the JROC prioritized requirements and
capability gaps, we reviewed the 13 ICDs and 7 CDDs submitted to the
JROC and any discussions of priorities and redundancies contained in
each document. We also met with officials from the Joint Staff;
Department of the Air Force; Department of the Army; Department of the
Navy; and Office of the Under Secretary of Defense (Comptroller) to
discuss the extent to which the JROC and its supporting bodies have
addressed prioritization issues. To understand the Joint Staff's
ongoing internal JCIDS review, we assessed the review charter and met
with the Joint Staff officials managing the review to discuss the
recommendations from the review and how they might affect the JROC's
prioritization of requirements. We also observed joint requirements
meetings and reviewed prior GAO reports.
We conducted this performance audit from June 2010 to June 2011 in
accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives.
[End of section]
Appendix II: Requirements Documents Reviewed by the JCB or JROC in
Fiscal Year 2010:
In fiscal year 2010, the Joint Capabilities Boards (JCB) and Joint
Requirements Oversight Council (JROC) combined to review a total of 45
new requirements documents, including 11 that were classified, 2 that
were information technology programs, and 8 documents that were not
associated with major defense acquisition programs. The remaining 24
requirements documents are identified in figure 4.[Footnote 40]
Figure 4: Requirements Documents Reviewed by the JCB or JROC in Fiscal
Year 2010:
[Refer to PDF for image: table]
ICD:
Sponsor: Air Force;
Capability Gap: Advanced Pilot Trainer.
Sponsor: Air Force;
Capability Gap: Joint Future Theater Lift.
Sponsor: Army;
Capability Gap: Ground Combat Vehicle.
Sponsor: Army;
Capability Gap: Joint Direct-Support Airborne Intelligence,
Surveillance, and Reconnaissance.
Sponsor: Army;
Capability Gap: Positioning, Navigation, and Timing Assurance.
Sponsor: Defense Intelligence Agency;
Capability Gap: Biometric Enabled Intelligence.
Sponsor: Defense Intelligence Agency;
Capability Gap: Joint Intelligence Operations Center Information
Technology Enterprise.
Sponsor: Deputy Assistant Secretary of Defense (Force Health
Protection and Readiness) and Joint Staff Surgeon/J4 Health Service
Support Division;
Capability Gap: Joint Force Health Protection.
Sponsor: TRICARE Management Agency;
Capability Gap: Electronic Health Record.
Sponsor: US Joint Forces Command;
Capability Gap: Command and Control On-The-Move.
Sponsor: US Joint Forces Command;
Capability Gap: Global Force Management.
Sponsor: US Northern Command;
Capability Gap: Maritime Domain Awareness Joint Integrating Concept.
Sponsor: US Strategic Command and US Joint Forces Command;
Capability Gap: Joint Aerial Layer Network.
CDD:
Sponsor: Air Force;
Capability Gap: Common Vertical Lift Support Platform.
Sponsor: Air Force;
Capability Gap: HH-60 Recapitalization.
Sponsor: Army;
Capability Gap: Integrated Air and Missile Defense System of Systems,
Increment 2.
Sponsor: Army;
Capability Gap: Ground Soldier System, Increment 1.
Sponsor: Navy;
Capability Gap: Joint Precision Approach and Landing System, Increment
2.
Sponsor: Navy;
Capability Gap: P-8A Poseidon Multi-mission Maritime Aircraft,
Increment 2 and 3.
Sponsor: Navy;
Capability Gap: Ship to Shore Connector.
CPD:
Sponsor: Air Force;
Capability Gap: Joint Air-to-Surface Standoff Missile - Extended Range
System, Increment 1.
Sponsor: Army;
Capability Gap: Apache Block III.
Sponsor: Marine Corps;
Capability Gap: Global Combat Support System - Marine Corps/Logistics
Chain Management.
Sponsor: Marine Corps;
Capability Gap: V-22 Block C/20.
Source: DOD's Knowledge Management and Decision Support Tool.
[End of figure]
[End of section]
Appendix III: JROC Cost Breach Reviews in Fiscal Year 2010:
The Joint Requirements Oversight Council (JROC) conducted 15 reviews
following cost breaches in fiscal year 2010--6 Nunn-McCurdy reviews
and 9 tripwire reviews. Table 2 identifies these reviews.
Table 2: JROC Cost Breach Reviews in Fiscal Year 2010:
Nunn-McCurdy reviews:
Apache Block III.
Advanced Threat Infrared Countermeasure/Common Missile Warning System.
DDG 1000.
Joint Strike Fighter.
Remote Minehunting System.
Wideband Global Satellite Communication.
Tripwire reviews:
Apache Block III.
Apache Longbow Block II.
C-130 Avionics Modernization Program.
CH-47F.
Family of Advanced Beyond Line-of-Sight Terminals, Increment 1.
Global Hawk.
Patriot Advance Capability 3.
Tactical Tomahawk.
Vertical Take-off and Landing Tactical Unmanned Air Vehicle.
Source: GAO analysis of DOD information.
[End of table]
[End of section]
Appendix IV: Best Practice Criteria for Resource Estimates:
Table 3 below presents the best practice criteria against which we
assessed the resource estimates presented to the Joint Requirements
Oversight Council during fiscal year 2010 Capability Development
Document (CDD) reviews.
Table 3: Best Practice Criteria for Resource Estimates:
Characteristic: Comprehensive;
Best practices:
* The cost estimate includes all life cycle costs.
* The cost estimate completely defines the program, reflects the
current schedule, and is technically reasonable.
* The cost estimate work breakdown structure is product-oriented,
traceable to the statement of work/objective, and at an appropriate
level of detail to ensure that cost elements are neither omitted nor
double-counted.
* The estimate documents all cost-influencing ground rules and
assumptions.
Characteristic: Well-documented;
Best practices:
* The documentation should capture the source data used, the
reliability of the data, and how the data were normalized.
* The documentation describes in sufficient detail the calculations
performed and the estimating methodology used to derive each element's
cost.
* The documentation describes step by step how the estimate was
developed so that a cost analyst unfamiliar with the program could
understand what was done and replicate it.
* The documentation discusses the technical baseline description and
the data in the baseline is consistent with the estimate.
The documentation provides evidence that the cost estimate was
reviewed and accepted by management.
Characteristic: Accurate;
Best practices:
* The cost estimate results are unbiased, not overly conservative or
optimistic and based on an assessment of most likely costs.
* The estimate has been adjusted properly for inflation.
* The estimate contains few, if any, minor mistakes.
* The estimate is based on a historical record of cost estimating and
actual experiences from other comparable programs.
Characteristic: Credible;
Best practices:
* The cost estimate includes a sensitivity analysis that identifies a
range of possible costs based on varying major assumptions,
parameters, and data inputs.
* A risk and uncertainty analysis was conducted that quantified the
imperfectly understood risks and identified the effects of changing
key cost driver assumptions and factors.
* Major cost elements were crossed-checked to see whether results were
similar.
* An independent cost estimate was conducted by a group outside the
acquiring organization to determine whether other estimating methods
produce similar results.
Source: GAO-09-3SP.
[End of table]
[End of section]
Appendix V: Comments from the Department of Defense:
The Joint Staff:
Washington, DC:
June 7, 2011:
Reply ZIP Code: 20318-0300.
Mr. Michael J. Sullivan:
Director, Acquisition and Sourcing Management:
U.S. Government Accountability Office:
441 G Street, N.W.
Washington, D.C. 20548:
Dear Mr. Sullivan,
Thank you for the opportunity to review GAO Draft Report GA0-11-502,
"DOD Requirements Reviews: Missed Trade-off Opportunities," dated 4 May
2011 (GAO Code 120916). Please find enclosed the DOD response to the
report's recommendations.
The Joint Staff point of contact is Mr. Randolph Wood; J-8; 703-614-
9628, randolph.wood@js.pentagon.mil.
Sincerely,
Signed by:
Craig A. Franklin:
Major General, USAF:
Vice Director, Joint Staff:
[End of letter]
Enclosure:
GAO Draft Report Dated 4 May 2011:
GAO-11-502 (GAO Code 120916):
"DOD Requirements Reviews: Missed Trade-Off Opportunities"
Department Of Defense Comments On The GAO Recommendations:
Recommendation 1: The GAO recommends that the Vice Chairman of the
Joint Chiefs of Staff, as Chairman of the Joint Requirements Oversight
Council (JROC), establish a mechanism to review the final analysis of
alternatives (AoA) report prior to Milestone A to ensure that trades-
offs have been considered and to provide military advice on these
trades-offs and the proposed material solution to the milestone
decision authority. (See page 27/GAO Draft Report.)
DOD Response: Partially concur. One of the Joint Capability Development
Process Review (JCDPR) recommendations is to have AoA results and
recommended path forward briefed to the appropriate Functional
Capabilities Board (FCB). At the discretion of the FCB Chair, by
exception, the briefing could be elevated to the Joint Capabilities
Board (JCB)/JROC for further review. This recommendation does not add
another round of staffing, an additional Joint Capabilities
Integration and Development System document, or an official validation
of the AoA results. It does allow the JROC to provide more informed
advice to the milestone decision authority. Concurrently, continued
participation by Joint Staff representatives in the AoA Senior
Advisory Group, Executive Steering Committee, and Integrated Product
Team will ensure that timely military advice on trade-offs will be
considered during the execution of the AoA.
Recommendation 2: The GAO recommends that the Vice Chairman of the
Joint Chiefs of Staff, as Chairman of the JROC, require that
capability sponsors present resource estimates that have been reviewed
by a military Service's cost analysis organization to ensure best
practices are being followed. (See page 27/GAO Draft Report.)
DOD Response: Partially concur. When capability development documents
(CDDs) are entered into staffing for Joint Staff review and approval,
a comparison of an independent cost estimate to the program office's
cost estimate is included. A method to highlight and examine this area
will be incorporated in the staffing and briefing process leading to a
JROC validation, as part of the ongoing JCDPR.
Recommendation 3: The GAO recommends that the Vice Chairman of the
Joint Chiefs of Staff, as Chairman of the JROC, require that
capability sponsors present key results from sensitivity and
uncertainty analyses, including the confidence levels associated with
resource estimates, based on the program's current level of knowledge.
(See page 27/GAO Draft Report.
DOD Response: Partially concur. The Director, Cost Assessment and
Program Evaluation has oversight responsibility for analyses of
alternatives and cost analyst responsibilities for resource estimates.
This recommendation needs further study to understand the expected
outcomes and the required authorities for the JROC. The JROC approves
joint military requirements to inform the decision points of the
acquisition process. However, a method to highlight and examine this
area will be incorporated in the staffing and briefing process leading
to a JROC validation as part of the ongoing JCDPR.
Recommendation 4: The GAO recommends that the Vice Chairman of the
Joint Chiefs of Staff, as Chairman of the JROC, assign priority levels
to the CDDs based on joint force capability gaps and redundancies
against current and anticipated threats, and provide these
prioritization levels to the Under Secretary of Defense
(Comptroller/Chief Financial Officer) and the military Services to be
used for resource allocation purposes. (See page 27 through 28/GAO
Draft Report.)
DOD Response: Partially concur on the prioritization means. One of the
JCDPR recommendations is a prioritization framework. Details are still
being finalized, but the current recommendation includes prioritizing
capability requirements to provide traceability from the capability
requirement to capability gap to the solution being pursued (CDD).
Therefore, the CDDs will "inherit" priority from the requirements and
gaps being satisfied, and these requirements and gaps may come from
one or more Initial Capability Documents or Joint Urgent Operational
Needs. Priorities based upon CDDs directly would be a less flexible
construct for prioritization.
DOD Response: Partially concur on the prioritization usage. The
prioritization framework developed under the JCDPR has been designed
to provide more informed decision making to a wide variety of
"consumers" across the Department. Processes that could be enhanced or
streamlined by the availability of joint prioritization include the
Program and Budget Review, combatant command Integrated Priority Lists
and Capability Gap Assessments, and Service Program Objective
Memorandum generation. In addition, prioritization could inform Joint
Experimentation and Science and Technology efforts.
Recommendation 5: The GAO recommends that the Vice Chairman of the
Joint Chiefs of Staff, as Chairman of the JROC, modify the Joint
Capabilities Integration and Development System operations manual to
require that a CDD discuss potential redundancies when validating
requirements. (See page 28/GAO Draft Report.)
DOD Response: Partially concur. Reviewing assessment of redundancies
in the CDD is late in the process for this activity; however, a method
to examine this area during CDD staffing and briefing will be
addressed as part of the ongoing JCDPR. Two of the recommendations
from the JCDPR effort address the issue of unnecessary redundancy
prior to the generation of a CDD. One of the Joint assessment criteria
in the prioritization framework looks specifically at the notion of
requirement uniqueness or redundancy. Unique requirements without
suitable alternatives gain a higher priority than requirements that
would be unnecessarily redundant with other existing requirements or
capability solutions. This helps encourage the drive to commonality
and collaboration where multiple organizations have similar or
identical requirements. The recommendation for a post-AoA review
provides a second opportunity for the FCB (or JCB/JROC) to review the
alternatives studied and the recommended path forward. If the
recommended approach introduces unnecessary redundancy, the JROC will
provide appropriate recommendations to the MDA for adjustments to the
path forward.
[End of section]
Appendix VI: GAO Contact and Acknowledgments:
GAO Contact:
Michael J. Sullivan, (202) 512-4841 or sullivanm@gao.gov:
Acknowledgments:
In addition to the contact named above, Ronald E. Schwenn, Assistant
Director; Noah B. Bleicher; Stephen V. Marchesani; Kenneth E. Patton;
Karen A. Richey; Anna K. Russell; and Nathan A. Tranquilli made key
contributions to this report.
[End of section]
Footnotes:
[1] Pub. L. No. 111-23, § 201(b), (codified at 10 U.S.C. § 181
(b)(1)(C)).
[2] Pub. L. No. 111-23, § 201(a)(2).
[3] Pub. L. No. 111-23, § 201(a).
[4] Pub. L. No. 111-23, § 105(b), (codified at 10 U.S.C. § 181(d)(2)).
[5] Pub. L. No. 111-23, § 105(c). GAO, Defense Management:
Perspectives on the Involvement of the Combatant Commands in the
Development of Joint Requirements, [hyperlink,
http://www.gao.gov/products/GAO-11-527R] (Washington, D.C.: May 2011)
reports on the extent to which the JROC solicits and considers input
from combatant commanders.
[6] Pub L. No. 111-383, § 862.
[7] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for
Developing and Managing Capital Program Costs, [hyperlink,
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009).
[8] Pub. L. No. 110-181, § 942(d).
[9] Pub. L. No. 111-23, § 201 (b) and (c).
[10] Pub. L. No. 111-383 § 841.
[11] Major defense acquisition programs are those programs identified
by DOD that require eventual total research, development, test, and
evaluation expenditures, including all planned increments, of more
than $365 million or procurement expenditures, including all planned
increments, of more than $2.19 billion in fiscal year 2000 constant
dollars.
[12] The Milestone Decision Authority is an acquisition official with
the authority to approve a program's entry into the next phase of the
acquisition process. For MDAPs, the Milestone Decision Authority is
the USD AT&L, DOD component head, or a component acquisition executive.
[13] 10 U.S.C. § 2433a.
[14] Pub. L. No. 111-23 § 201(a)(2)(B).
[15] GAO, Defense Acquisitions: A Knowledge-Based Funding Approach
Could Improve Major Weapon System Program Outcomes, [hyperlink,
http://www.gao.gov/products/GAO-08-619] (Washington, D.C.: July 2008).
[16] Prior to WSARA, the Director of the Office of Program Analysis
and Evaluation, which is named as a JROC advisor in the National
Defense Authorization Act of Fiscal Year 2008, performed some of the
cost estimating and program evaluation roles now performed by CAPE.
The Deputy Director of Cost Assessment performs independent cost
estimates for MDAPs, among other roles. The Deputy Director of Program
Evaluation performs evaluations of the net costs and benefits of a
proposed system compared with alternatives, among other roles.
[17] [hyperlink, http://www.gao.gov/products/GAO-11-527R].
[18] The Joint Capabilities Development Process Review Integrated
Process Team is providing recommendations to improve JCIDS'
responsiveness and decision support to the JROC, combatant commands,
military services, and defense agencies. The target completion date
for implementing the review's recommendations is June 30, 2011.
[19] Pub. L. No. 111-23 § 201(d)(1).
[20] Under Secretary of Defense for Acquisition, Technology and
Logistics, Better Buying Power: Guidance for Obtaining Greater
Efficiency and Productivity in Defense Spending (Washington, D.C.:
Sept. 14, 2010).
[21] [hyperlink, http://www.gao.gov/products/GAO-09-3SP].
[22] A rough-order-of-magnitude estimate is a quick, high-level
estimate that generally involves less time and effort than a budget-
quality estimate.
[23] Full funding refers to a budgetary allocation in the future-years
defense program.
[24] JROC, Funding Guidance for Joint Requirements Oversight Council
Directed Action, JROC Memorandum 067-07 (Washington, D.C.: Mar. 23,
2007).
[25] An MDAP may also not receive Milestone B approval until the
Milestone Decision Authority certifies that funding is available to
execute the program through the period covered by the future-years
defense program. 10 U.S.C. § 2366b(a)(1)(D). DOD Instruction 5000.02,
Operation of the Defense Acquisition System (Dec. 8, 2008) also
requires programs to be fully funded through the future-years defense
program.
[26] Pub. L. No. 110-181, § 942, (codified at 10 U.S.C. § 181(b)).
[27] H.R. Rep. No. 110-146 at 381 (2007).
[28] GAO, Defense Acquisitions: DOD's Requirements Determination
Process Has Not Been Effective in Prioritizing Joint Capabilities,
[hyperlink, http://www.gao.gov/products/GAO-08-1060] (Washington,
D.C.: September 2008).
[29] GAO, Opportunities to Reduce Potential Duplication in Government
Programs, Save Tax Dollars, and Enhance Revenue, [hyperlink,
http://www.gao.gov/products/GAO-11-318SP] (Washington, D.C.: March
2011).
[30] GAO, Defense Acquisitions: Opportunities Exist to Achieve Greater
Commonality and Efficiencies among Unmanned Aircraft Systems,
[hyperlink, http://www.gao.gov/products/GAO-09-520] (Washington, D.C.:
July 2009).
[31] [hyperlink, http://www.gao.gov/products/GAO-07-388].
[32] Under Secretary of Defense for Acquisition, Technology and
Logistics, Better Buying Power: Guidance for Obtaining Greater
Efficiency and Productivity in Defense Spending (Washington, D.C.:
Sept. 14, 2010).
[33] Manual for the Operation of the Joint Capabilities Integration
and Development System (July 31, 2009).
[34] While the JROC validated 12 ICDs during fiscal year 2010,
according to Joint Staff officials, an additional ICD received during
this time period was pending final certification as of January 2011.
[35] Electronic Health Record ICD is designed to address 10 gaps in
the Joint Force Health Protection concept of operations, including
comprehensive medical and dental documentation, inpatient and
outpatient order entry and management, and consult and referral
management.
[36] Command and Control On-The-Move ICD is the capability to maintain
situational awareness and make timely and informed decisions while
nonstationary. It is designed to address six gaps, including the
limited capability to share information with mission partners and an
inability to plan collaboratively while on the move.
[37] This system is designed to track the status of requirements
documents--ICDs, CDDs, CPDs---submitted to the JCIDS process for
review. It is the key system that the Joint Staff and other DOD
components use to review and comment on proposals. To assess the
reliability of the data system, we interviewed Joint Staff officials
knowledgeable about the system, and cross-checked information in the
system. We determined that the data were sufficiently reliable for the
purposes of this report.
[38] Combatant commands are DOD's operational commanders. There are
six combatant commands with geographic responsibilities and four with
functional responsibilities. However, in August 2010, the Secretary of
Defense recommended the closure of the U.S. Joint Forces Command.
[39] [hyperlink, http://www.gao.gov/products/GAO-09-3SP].
[40] Joint Staff officials reported that in fiscal year 2010, two key
requirements documents were reviewed by a Functional Capabilities
Board, but not a JCB or JROC.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
