Joint Strike Fighter
DOD Plans to Enter Production before Testing Demonstrates Acceptable Performance Gao ID: GAO-06-356 March 15, 2006The Joint Strike Fighter (JSF) is DOD's most expensive aircraft program. The program represents 90 percent of the remaining planned investment for recapitalizing DOD's aging tactical aircraft fleet. GAO is required by law to review the program annually for 5 years, beginning in fiscal year 2005. This is our second report and GAO assessed the program's acquisition approach--in terms of capturing knowledge for key investment decisions--and identified an alternative to improve outcomes.
DOD is investing heavily in procuring JSF aircraft before flight testing proves it will perform as expected. For example, the JSF program plans to produce 424 low-rate initial production aircraft, at a total estimated cost of more than $49 billion, by 2013--the same time at which the program plans to complete initial operational testing. Producing aircraft before testing demonstrates the design is mature increases the likelihood of design changes that will lead to cost growth, schedule delays, and performance problems. Because the program will lack key design and testing knowledge, DOD plans to use cost reimbursement contracts to procure early production aircraft. This type of contract places a substantially greater cost risk on DOD and the taxpayers. Confidence that investment decisions will deliver expected capability within cost and schedule goals increases as testing proves the JSF will work as expected. At the same time, the JSF program has not adopted an evolutionary approach to acquiring the aircraft--despite DOD policy that prefers such an approach. Instead, the JSF program has contracted to develop and deliver the aircraft's full capability in a single-step, 12-year development program--a daunting task given the need to incorporate the technological advances that, according to DOD, represent a quantum leap in capability. DOD's buying power has already been reduced. Since initial estimates, program acquisition unit costs have increased by 28 percent, or $23 million. Development costs have increased 84 percent, planned purchases have decreased by 535 aircraft, and the completion of development has slipped 5 years, delaying delivery of capabilities to the warfighter. With more than 90 percent of the JSF investment remaining, DOD officials have the opportunity to adopt a knowledge-based and evolutionary acquisition strategy that would maximize DOD's return on its investment. The acquisition approach used for the F-16 fighter, the Air Force's JSF predecessor, could provide a model for delivering JSF capabilities to the warfighter sooner and recapitalizing tactical aircraft forces more quickly while lowering risk. The F-16 program successfully evolved capabilities over the span of about 30 years, with an initial capability delivered to the warfighter about 4 years after development started.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-06-356, Joint Strike Fighter: DOD Plans to Enter Production before Testing Demonstrates Acceptable Performance
This is the accessible text file for GAO report number GAO-06-356
entitled 'Joint Strike Fighter: DOD Plans to Enter Production before
Testing Demonstrates Acceptable Performance' which was released on
March 15, 2006.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Committees:
United States Government Accountability Office:
GAO:
March 2006:
Joint Strike Fighter:
DOD Plans to Enter Production before Testing Demonstrates Acceptable
Performance:
GAO-06-356:
GAO Highlights:
Highlights of GAO-06-356, a report to congressional committees:
Why GAO Did This Study:
The Joint Strike Fighter (JSF) is DOD‘s most expensive aircraft
program. The program represents 90 percent of the remaining planned
investment for recapitalizing DOD‘s aging tactical aircraft fleet.
GAO is required by law to review the program annually for 5 years,
beginning in fiscal year 2005. This is our second report and GAO
assessed the program‘s acquisition approach”in terms of capturing
knowledge for key investment decisions”and identified an alternative to
improve outcomes.
What GAO Found:
DOD is investing heavily in procuring JSF aircraft before flight
testing proves it will perform as expected. For example, the JSF
program plans to produce 424 low-rate initial production aircraft, at a
total estimated cost of more than $49 billion, by 2013”the same time at
which the program plans to complete initial operational testing.
Producing aircraft before testing demonstrates the design is mature
increases the likelihood of design changes that will lead to cost
growth, schedule delays, and performance problems. Because the program
will lack key design and testing knowledge, DOD plans to use cost
reimbursement contracts to procure early production aircraft. This type
of contract places a substantially greater cost risk on DOD and the
taxpayers. Confidence that investment decisions will deliver expected
capability within cost and schedule goals increases as testing proves
the JSF will work as expected.
Overlap of Production Investments and Testing:
[See PDF for image]
[End of figure]
At the same time, the JSF program has not adopted an evolutionary
approach to acquiring the aircraft”despite DOD policy that prefers such
an approach. Instead, the JSF program has contracted to develop and
deliver the aircraft‘s full capability in a single-step, 12-year
development program”a daunting task given the need to incorporate the
technological advances that, according to DOD, represent a quantum leap
in capability. DOD‘s buying power has already been reduced. Since
initial estimates, program acquisition unit costs have increased by 28
percent, or $23 million. Development costs have increased 84 percent,
planned purchases have decreased by 535 aircraft, and the completion of
development has slipped 5 years, delaying delivery of capabilities to
the warfighter. With more than 90 percent of the JSF investment
remaining, DOD officials have the opportunity to adopt a knowledge-
based and evolutionary acquisition strategy that would maximize DOD‘s
return on its investment. The acquisition approach used for the F-16
fighter, the Air Force‘s JSF predecessor, could provide a model for
delivering JSF capabilities to the warfighter sooner and recapitalizing
tactical aircraft forces more quickly while lowering risk. The F-16
program successfully evolved capabilities over the span of about 30
years, with an initial capability delivered to the warfighter about 4
years after development started.
What GAO Recommends:
The Congress should consider delaying authorizations and appropriations
for JSF procurement until a new business case is developed and flight
testing demonstrates the design and integrated mission systems work.
GAO included this matter for consideration because DOD did not plan to
make changes as a result of recommendations.
GAO is recommending that DOD delay investing in production until flight
testing shows that the JSF performs as expected, and that the program
develop a plan, consistent with DOD‘s preferred policy, to adopt an
evolutionary approach that limits new content for each increment to
proven technologies and design. DOD partially concurred, but believes
that its current practices achieve our recommendations‘ objectives.
www.gao.gov/cgi-bin/getrpt?GAO-06-356.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Michael J. Sullivan at
(202) 512-4841 or sullivanm@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
JSF Acquisition Strategy Will Begin Procurement before Testing
Demonstrates the Aircraft's Performance:
Evolutionary Acquisition Strategy Provides a Less Risky Alternative for
the JSF Program:
Conclusions:
Matters for Congressional Consideration:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Changes in JSF Program Purchase Quantities and Costs:
Appendix III: Comments from the Department of Defense:
Table:
Table 1: Military Services' Planned Use for the Joint Strike Fighter:
Figures:
Figure 1: Overlap of Low-Rate Initial Production Investments and
Testing Demonstrations of the JSF Variants:
Figure 2: JSF Program Acquisition:
Figure 3: JSF's Annual Funding Requirements, as of December 2004:
Figure 4: F-16 Incremental Development Approach:
Figure 5: Planned JSF Blocks:
Figure 6: Potential Application of an Incremental Development Approach
to the JSF Program:
Abbreviations:
DOD: Department of Defense:
JSF: Joint Strike Fighter:
OSD: Office of the Secretary of Defense:
United States Government Accountability Office:
Washington, DC 20548:
March 15, 2006:
Congressional Committees:
Since the Joint Strike Fighter (JSF) program began in 1996, Congress
has appropriated nearly $25 billion for its development. It is the
Department of Defense's (DOD) most expensive aircraft acquisition
program. As the program is currently planned, DOD estimates it will
spend $257 billion to develop and procure about 2,443 aircraft and
related support equipment by 2027 and an additional $347 billion to
operate and support these aircraft once they have been fielded.
JSF program goals are to develop and field a family of stealthy, strike
fighter aircraft for the Navy, Air Force, and Marine Corps, and United
States allies. Design plans call for three variants of the aircraft,
but the program aims to provide maximum commonality to minimize life
cycle costs. JSF is a central part of DOD's overall recapitalization
strategy for its tactical aircraft fleet, representing 90 percent of
the remaining planned investment for its major tactical aircraft
programs. How DOD manages the JSF program in the future will be
critical for getting the most out of this remaining large investment
and for success in replacing the aging fleet of tactical aircraft.
The Ronald W. Reagan National Defense Authorization Act for Fiscal Year
2005 (P.L. 108-375) requires GAO to review the JSF program annually for
5 years.[Footnote 1] In March 2005, we reported the JSF's original
business case was unexecutable and recommended that DOD establish an
executable program consistent with best practices and DOD policy
regarding evolutionary acquisitions.[Footnote 2] We also recommended
that this new business case be accompanied by a knowledge-based
acquisition approach[Footnote 3]--an approach that ensures attainment
and use of demonstrated product knowledge before making future
investments. In commenting on our report, DOD stated that JSF's
restructured acquisition plan would incorporate a knowledge-based,
evolutionary approach consistent with DOD policy. This is our second
report. Specifically, we (1) determined whether the JSF program
acquisition strategy captures critical knowledge in time to make
production investment decisions and (2) identified an alternative to
the current acquisition strategy to improve JSF program outcomes.
The act also requires us to certify whether we had access to sufficient
information to make informed judgments on the matters contained in our
report. We were provided sufficient information to assess the plans to
capture technology, design, and manufacturing knowledge for the three
JSF variants, the evolutionary nature of the acquisition strategy, and
the opportunities to improve outcomes. At the time of our review, the
Office of the Secretary of Defense (OSD) was still preparing its fiscal
year 2007 budget request as well as conducting its independent
assessment of the program cost estimate, which is not expected to be
completed until sometime in 2006. Additionally, the Quadrennial Defense
Review was ongoing during our review, and we did not have insights into
the potential outcomes or how the results of the study might affect the
JSF program. We performed our work from June 2005 through March 2006 in
accordance with generally accepted government auditing standards. For
more on our scope and methodology, see appendix I.
Results in Brief:
Despite recent program restructuring, the JSF program continues to base
its acquisition strategy on a highly concurrent approach that makes
significant investments in manufacturing capabilities and production
aircraft before flight testing demonstrates the JSF's performance. As a
result, significant development risk remains, and it is likely that
current cost and schedule goals will not be met. The program plans to
proceed into low-rate initial production in 2007 with inadequate
testing to prove a mature design for any of the three basic JSF variant
airframes, without developing critical software, or without a fully
integrated aircraft with advanced mission systems and prognostic
maintenance capabilities. All three production representative
variants[Footnote 4] will not be in flight testing until 2009, and a
fully configured, integrated development aircraft will not begin flight
testing until 2011--4 years after production begins. By this time, DOD
plans to have ordered 190 aircraft at a cost of about $26 billion. By
2013, when initial operational testing is expected to be complete, the
program plans to have procured 424 aircraft at an expected cost of $49
billion. Because it will lack necessary technology, design, and
performance knowledge, DOD plans to procure early production aircraft
using cost-reimbursable-type contracts. This type of contract places a
significantly greater cost risk on DOD.
DOD has revised its acquisition policy to embrace evolutionary
acquisition, allowing managers to develop and evolve a product through
small, time-phased development increments--an alternative still
available to the JSF program. Instead of establishing time-phased
requirements for aircraft to be delivered in sequence that could first
meet DOD's need to recapitalize its aging fleet of aircraft and then
evolve the aircraft to eventually achieve improved capabilities in
future increments, DOD chose a single-step approach to develop and
deliver the JSF with ultimate capabilities in a single 12-year system
development program. The large amount of uncertainty in this approach
has already led to poor cost and schedule estimates for the JSF program
and a reduction in DOD's buying power. The length and scope of work
remaining continue to make it difficult to predict the cost and time
needed to complete the program. The Air Force's JSF predecessor, the F-
16 Fighter program, provides a model for a less risky alternative JSF
acquisition strategy that delivers weapons to the warfighter more
quickly and that recapitalizes tactical aircraft forces sooner. The F-
16 program successfully delivered 2,200 aircraft with incremental
improvements as technology became available over the span of about 30
years. An initial F-16 capability was delivered to the warfighter
within about 4 years after development began and substantial quantities
were purchased with each increment to meet warfighter needs, including
the replacement of aging legacy tactical fighter aircraft.
To improve the chances for a successful outcome, we are recommending
the JSF program delay production and investments in production
capability until the aircraft design qualities and integrated mission
capabilities of the fully configured and integrated JSF aircraft
variants have been proven to work in flight testing. We are also
recommending the Secretary of Defense report to the Congress by July
2006 on the feasibility of an incremental acquisition approach that
follows the intent of DOD evolutionary acquisition policy and delivers
a first increment that limits new content to proven technologies and
design.
DOD partially concurred with our recommendations stating that the
current JSF acquisition strategy and management practices will be able
to achieve the objectives of our recommendations, and therefore,
further limits on production are unnecessary. DOD also stated that the
program's strategy balances technical risk, financial resources, and
operational needs to reduce program risk and is based on a knowledge-
based, incremental approach. We continue to believe DOD's planned
strategy to make significant investments in production concurrent with
testing is very risky and is similar to strategies of past programs
with poor outcomes. We believe that DOD has an opportunity to reduce
risk by adopting an acquisition strategy based on capturing technology,
design, and manufacturing knowledge. We also believe that smaller, more
manageable commitments in capabilities would make JSF program cost and
schedule more predictable and deliver needed capabilities to the
warfighter sooner. Because DOD does not plan to make changes as a
result of our recommendations, we believe Congress should consider
delaying authorizations and appropriations for JSF procurement until
DOD develops a new business case and demonstrates the aircraft design
qualities and integrated mission capabilities of the fully configured
and integrated JSF variants work as designed based on actual flight
testing.
Background:
JSF is a joint, multinational acquisition program for the Air Force,
Navy, Marine Corps, and eight cooperative international partners. The
program began in November 1996 with a 5-year competition between
Lockheed Martin and Boeing to determine the most capable and affordable
preliminary aircraft design. Lockheed Martin won the competition, and
the program entered system development and demonstration in October
2001.
DOD has set major expectations for the program. The program's objective
is to develop and deploy a technically superior and affordable fleet of
aircraft that support the warfighter in performing a wide range of
missions in a variety of theaters. The single-seat, single-engine
aircraft is being designed to be self-sufficient or part of a
multisystem and multiservice operation, and to rapidly transition
between air-to-surface and air-to-air missions while still airborne. To
achieve its mission, JSF will incorporate low observable technologies,
defensive avionics, advanced onboard and offboard sensor
fusion,[Footnote 5] internal and external weapons, and advanced
prognostic maintenance capability. According to DOD, these technologies
represent a quantum leap over legacy tactical aircraft capabilities. At
the same time, the JSF aircraft design includes three variants: a
conventional takeoff and landing variant for the Air Force; an aircraft
carrier-suitable variant for the Navy; and a short takeoff and vertical
landing variant for the Marine Corps, the Air Force, and the United
Kingdom. JSF is intended to replace a substantial number of aging
fighter and attack aircraft in DOD's current inventory (see table 1).
Table 1: Military Services' Planned Use for the Joint Strike Fighter:
Service: Air Force;
Planned use: Replacement for the F-16 and A-10; complement the F-22A.
Service: Marine Corps;
Planned use: Replacement for the AV-8B and F/A- 18 A/C/D.
Service: Navy;
Planned use: Complement the F/A-18 E/F.
Source: DOD data.
[End of table]
In recent years, DOD has revised its acquisition policy to embrace an
evolutionary, or incremental, approach to improve program outcomes. The
acquisition policy states evolutionary acquisition is the preferred DOD
strategy for rapid acquisition of mature technology for the user by
delivering capability in increments, recognizing up front the need for
future capability improvements. The objective is to balance needs with
resources in order to put capability into the hands of the warfighter
more quickly. The policy states that a product enters system
development and demonstration when an affordable increment of
militarily useful capability has been identified, technology has been
demonstrated in a relevant environment, and a system can be developed
for production in a short time frame (normally less than 5 years). Each
increment of an evolutionary acquisition will have its own decision
milestones and baseline--cost, schedule, and performance requirements.
In commenting on our March 2005 JSF report, DOD stated that JSF's
restructured acquisition plan would balance technical, cost, and
schedule risk and incorporate a knowledge-based, evolutionary approach
to system acquisition consistent with DOD policy.
JSF Acquisition Strategy Will Begin Procurement before Testing
Demonstrates the Aircraft's Performance:
DOD's acquisition strategy for JSF has not substantially changed as a
result of the program's restructuring last year. In May 2005, DOD
leadership approved the program's plan, which intends to start
production in 2007--more than 2 years before all three JSF variants
have completed some flight testing of the aircraft's basic design, 4
years before a fully configured and integrated aircraft is expected to
be flight tested, and 6 years before development and initial
operational testing are scheduled to be completed.[Footnote 6] Before
development has ended, DOD will have potentially signed procurement
contracts for 424 JSF aircraft at an estimated cost of $49 billion.
Starting production before ensuring the design is mature through flight
testing significantly increases the risk of costly design changes that
will push the program over budget and behind schedule. Evidence of the
risk associated with concurrently developing, testing, and producing
the JSF aircraft can be seen in the program office strategy to place
initial production orders on a cost reimbursement contract, placing an
unusually high risk burden on the government during the early
production phase.
Key Testing Events Will Not Be Completed before Significant Procurement
Investments Are Made:
The JSF program plans to begin low-rate initial production of the
aircraft before many of JSF's technology advances and basic flying
qualities are flight-tested and to produce substantial quantities of
aircraft before initial operational testing is completed. According to
JSF program plans, DOD's low-rate initial production quantities will
increase from 5 aircraft a year in 2007 to 133 a year, before
development and initial operational testing are completed in
2013.[Footnote 7] This production rate will require DOD to invest
significantly in tooling, facilities, and personnel. Initial contractor
cost estimates indicate that close to $1 billion of new tooling will be
needed to support low-rate initial production rates. Early contractor
data also show that the program will need to double its manufacturing
workforce by 2008 and will need six times the number of manufacturing
personnel before low-rate initial production ends. Total monthly
spending for production activities in 2013 is expected to approach $1
billion, a significant increase from $100 million a month when
production is scheduled to begin in 2007.
The cost of discovering design problems during production could be
significant if testing shows that large, structural components of the
aircraft require modifications. Design changes needed in one variant
could also ripple through the other two variants, reducing efficiencies
necessary to lower production and operational costs with common parts
and manufacturing processes for the three variants. Some industry
officials have indicated that the cost of design changes such as these
could be 10 to 1,000 times greater, depending on how far the product
has progressed into production.
When the JSF program is expected to begin low-rate initial production
in 2007,[Footnote 8] the program will have completed less than 1
percent of the flight test program and none of the three JSF variants
will have a production representative prototype built and in flight
testing. Features critical to JSF's operational success, such as the
low-observable and highly common airframe and the advanced mission
systems, will not have been demonstrated in a flight test environment.
The program plans to proceed into low-rate initial production without
demonstrating that (1) the aircraft's flying qualities function within
the parameters of the flight envelope--that is, the set limits for
altitude, speed, and angles of attack; (2) the aircraft design is
reliable; or (3) a fully integrated and capable aircraft system can
perform as intended. These are key indicators of a program's readiness
for entering production and making significant investments in tooling,
facilities, and materials. When the first fully integrated and capable
development JSF is expected to fly, in 2011, DOD will have committed to
buy 190 aircraft at an estimated cost of $26 billion under current
plans. By the time testing is planned to be completed, in 2013, DOD
will have procured more than double that amount--424 aircraft at an
estimated cost of about $49 billion. Figure 1 shows the significant
overlap in development and testing and the major investments in
production capability and aircraft that inject additional cost and
delivery risks that could delay delivery of a proven capability to the
warfighter.
Figure 1: Overlap of Low-Rate Initial Production Investments and
Testing Demonstrations of the JSF Variants:
[See PDF for image]
[A] Flight testing data reflect the percentage of the total flight
tests completed at the time of the planned investment decision, which
is currently planned for January of each year.
[End of figure]
Under the current schedule, the JSF program plans to manufacture and
deliver 15 flight test aircraft and 7 ground test (nonflying) articles
in 5 years--an aggressive schedule when compared with schedules of
other programs with fewer variables. Current JSF schedules are already
showing delivery of early test aircraft could be later than the planned
delivery date. Unplanned manufacturing and technical problems can delay
the completion of a flight test program, increase the number of flight
test hours needed to verify that the system will work as intended, and
affect when key knowledge will be available for making investment
decisions. For example, when the B-2 program began flight testing in
July 1989, it estimated that the flight test program would last
approximately 4.5 years and require about 3,600 flight test hours. When
the test program ended in 1997, the flight test hours had grown to
5,000 hours, or by 40 percent, over an 8-year period. Program officials
cited several causes, including difficulties in manufacturing test
aircraft and correcting deficiencies from unanticipated problems
discovered during testing. The F-22A encountered similar delays,
increasing a planned 4-year flight test program to about 8 years and
affecting the program's ability to conduct operational testing and move
into production on schedule.
Initial Procurement Decisions Will Be Made before the Basic Aircraft
Design Is Demonstrated:
While each JSF variant is similar--all are being designed to have
stealth airframes, fly at supersonic speeds, shoot air-to-air missiles,
and drop bombs on a target--there are subtle airframe design
differences to support the services' different operational concepts and
environments. Test officials acknowledge that each airframe variant
will require flight testing to demonstrate that each will fly as
intended. Yet at the time the JSF program expects to begin low-rate
initial production, only 1 of 15 flight test aircraft is expected to
have started testing. Further, the aircraft's test flights are not
scheduled to begin until August 2006, less than 6 months before the
planned decision to begin low-rate initial production,[Footnote 9] and
will not assess the aircraft in more demanding operational
environments, such as high angles of attack. Moreover, the first flight
test aircraft will not include a large number of design changes that
resulted from an effort to significantly reduce aircraft weight over
the past 2 years and, therefore, will not represent the planned
production configuration. The first demonstration of a production
representative airframe that includes the latest design changes is
scheduled for late 2007--after production has been initiated--with
first flight of the short takeoff and vertical landing variant.
All three variants will not be in flight testing until 2009, when the
carrier variant is expected to have its first flight. Several key test
events identified by the contractor to demonstrate the flight
characteristics of these aircraft in the intended operational
environment are not scheduled until 2009 or later. These include
shipboard operations for the carrier and short takeoff and vertical
landing variants. According to the contractor, the full-flight envelope
for all three variants will not be fully demonstrated until 2011 or
2012--4 to 5 years after low-rate initial production is scheduled to
begin.
According to DOD, modeling and simulation will be used to evaluate the
performance of the aircraft's flying qualities and to support decisions
to invest in production. However, the Rand Corporation recently
reported in a study on testing and evaluation that modeling is not a
substitute for flight testing.[Footnote 10] Rand found that even in
performance areas that are well understood, it is not unusual for
flight testing to uncover problems that were not apparent in
simulations. Examples include flight effects on the wing of the F/A-18
EF and buffeting of stores externally carried on various aircraft when
flown in certain conditions. Additionally, OSD testing officials
indicated that flight testing of each variant is necessary to
demonstrate designed capabilities. Our past work has found that flying
quality problems were identified during actual flight testing on
programs like the F-22A, B-2A, and V-22.
Finally, two major ground tests of the airframe's structural integrity-
-fatigue and static testing--will be in their very early stages or not
have started when production begins. Fatigue testing, which measures
the aircraft's durability over its expected life, is slated to begin in
2008 with testing to show structural fatigue over one lifetime of
flying to be completed in 2009 and a second lifetime completed by 2010.
Fatigue testing uses actual JSF airframe structures that are subjected
to the long-term stresses expected over the aircraft's life. Standards
require this testing to be done to represent the stresses expected over
two lifetimes. Static testing, which measures the aircraft's ability to
withstand the stresses expected to be encountered throughout the
aircraft's flight regime, is slated to begin in 2007 and last through
2009. These tests are important to reduce the risk of structural
problems emerging during production or after aircraft are fielded.
Mission Capabilities Will Not Be Tested until 4 Years after Procurement
Has Begun:
In addition to late design testing, many of the mission systems planned
for JSF will not be available for initial flight testing either.
Although laboratory tests are under way, DOD does not plan to flight-
test several of the new technologies needed for the JSF to perform its
intended missions until 2009 at the earliest. Defense Operational Test
and Evaluation officials have stated that flight-testing capabilities
in a production representative test aircraft and in the operational
environment planned for the new system are important to reducing risk.
This actual environment differs from what can be demonstrated in a
laboratory and has historically identified unexpected problems. For
example, the F-22A software worked as expected in the laboratory, but
when tested in the aircraft, significant problems were identified.
These problems delayed testing and the delivery of a proven capability.
The different levels of mission capability will be tested in JSF
aircraft as follows.
* Block 1, an initial air interdiction capability, is scheduled to
begin testing in 2009, with initial operational testing scheduled for
2011--4 years after DOD plans to begin production.
* Block 2, an improved air interdiction and close air support
capability, is scheduled to begin testing in 2010, with initial
operational testing scheduled for 2012. This block will include several
critical technologies that are not fully mature, such as the advanced
missions systems and prognostics and health maintenance, but are
critical to meeting requirements like sortie generation and mission
capabilities.
* Block 3, the fully integrated and capable JSF, is scheduled to begin
testing in 2011. At this time less than 50 percent of the planned
mission capability testing will have been completed. This is close to
the same point that the F-22A and other past programs experienced
difficulties integrating all the complex software and hardware
components into the aircraft. Flight testing to evaluate the
effectiveness and suitability of the fully integrated system is
expected to continue until the full-rate decision in late 2013.
Aggressive Software Development Schedule May Cause Delays:
JSF's expected performance is largely dependent on demonstrating
software that supports vehicle, mission system, and other capabilities.
The program plans to develop over 19 million lines of code--
substantially more than the lines of code needed for the F-22A. The
software is planned to be developed in five blocks. The first block is
scheduled for completion in 2006, and the last block is scheduled for
completion in 2011. At the time the program enters low-rate initial
production, the program will have completed less than 35 percent of the
software needed for the system's functionality. Past programs have
encountered difficulties in developing software, delaying flight test
schedules. Data provided by the program office indicate that the
program is already showing early signs of falling behind its software
delivery schedule. JSF program officials recognize the risk associated
with this large software effort.
The Current Acquisition Strategy Requires Prolonged Reliance on Cost
Reimbursement Contracts That Place an Increased Risk on DOD:
According to program officials, the uncertainties inherent in
concurrently developing, testing, and producing the JSF aircraft
prevent the pricing of initial production orders on a fixed price
basis. Consequently, the program office plans to place initial
production orders on a cost reimbursement basis. Cost reimbursement
contracts provide for payment of allowable incurred costs, to the
extent prescribed in the contract. Such contracts are generally used
only when the uncertainties involved in contract performance do not
permit costs to be estimated with sufficient accuracy to use any type
of fixed price contract. Cost reimbursement contracts place substantial
risk on the buyer--in this case DOD--because the contractor's
responsibility for performance costs is minimized or reduced. In
contrast, a fixed price contract provides for a pre-established price,
and places more risk and responsibility for costs and resulting profit
or loss on the contractor and provides more incentive for efficient and
economical performance.
The program plans to transition to fixed-price-type contracts once the
air vehicle has a mature design, has been demonstrated in flight test,
and is producible at established cost targets. According to program
officials, this transition will occur sometime before full-rate
production begins in 2013. The program office believes the combination
of the early concept development work, the block development approach,
and the relatively small numbers of aircraft in the initial production
buys allow decisions to be made earlier than normal with an acceptable
level of risk.
Evolutionary Acquisition Strategy Provides a Less Risky Alternative for
the JSF Program:
The JSF acquisition strategy currently plans a single-step approach to
deliver a quantum leap in tactical fighter capability by 2013 and has
already felt the negative cost and schedule impacts from the executing
this approach. The length and scope of the remaining effort in the JSF
program make it even more difficult to accurately estimate cost and
delivery schedules. The JSF funding profile--which requires an average
of $11 billion annually for the next two decades--is also at risk to
increase if costs continue to grow or schedules are further delayed to
develop the ultimate JSF capabilities. An alternative acquisition
strategy, such as used by the F-16 program, that sequences capabilities
over time based on proven technologies and design would reduce risk and
deliver aircraft sooner. This evolutionary approach is actually the
preferred approach in DOD's acquisition policy for acquiring new
systems for more rapid delivery of incremental capabilities to the
warfighter.
JSF Program Is Designed to Deliver Full Required Capability in a Single-
Step Development Program:
Instead of establishing time-phased requirements for aircraft to be
delivered in sequence that could first meet DOD's need to recapitalize
its aging fleet of aircraft and then evolve the aircraft to eventually
achieve improved capabilities in future system development increments,
DOD chose a single-step development approach to deliver the full
required capability by the end of system development in 2013. That
approach is now planned to last 17 years (5 years in concept
development and 12 years in system development)--including the
development of immature, undemonstrated technologies and will take at
least two to three times longer than the development time of leading
commercial firms or that suggested by defense acquisition policy. The
JSF acquisition strategy is shown in figure 2.
Figure 2: JSF Program Acquisition:
[See PDF for image]
[A] The bar for production is not drawn to scale. JSF production lasts
for 20 years.
[End of figure]
While JSF's acquisition strategy calls for delivering a small number of
aircraft with limited capabilities, the program has committed to
deliver the full required capability by the end of system development
and demonstration in 2013. For JSF, this includes a set of objectives
that exceeds those of aircraft development programs of the past. JSF
will use cutting-edge technology to fuse data from the aircraft itself
and other sources, from all aspects of the aircraft, and display the
information to the pilot. The aircraft must be able to rapidly
transition from ground attack to air-to-air missions while airborne.
JSF also expects advances in technologies for mission systems,
prognostics, and autonomic logistics support requiring hardware
development as well as extensive and complex software development and
integration to have lower cost of ownership than the legacy aircraft
while being able to deploy rapidly, sustain high mission reliability,
and maintain a high sortie generation rate.
Past single-step development programs have been unsuccessful in
predicting acquisition costs and delivery schedules. For example,
development costs for the F-22A fighter, B-2 bomber, Crusader artillery
vehicle, and Comanche helicopter skyrocketed, and production quantities
were either substantially reduced or the program was canceled. Such
outcomes are a poor return on investment and a failure to recapitalize
aging legacy systems. So far the JSF program is experiencing similar
results. Since the program's start, development cost has increased 84
percent, the development schedule has increased by almost 5 years, and
planned delivery of capabilities to the warfighter has been delayed.
DOD now plans to buy 535 fewer aircraft than originally planned. As a
result, DOD's buying power has been significantly reduced--program
acquisition unit costs have increased by 28 percent, or $23 million,
since first estimates. See appendix II for more details on JSF
outcomes.
Funding Needs Could Increase Given the Level of Risk to Complete the
JSF Program:
Despite a lengthy restructuring effort that added over $7 billion and
18 months to the development program, JSF officials have stated that
the restructured program has little or no flexibility for future
changes or unanticipated risks. Furthermore, the length and scope of
the remaining effort make it even more difficult to accurately estimate
cost and completion schedules. While it has been over 9 years since the
program started, the first flight test aircraft still has not been
delivered. The program has planned almost 8 years to complete the
remaining activities of the system development and demonstration phase.
These remaining activities include:
* fully maturing 7 of the 8 critical technologies;
* completing the designs and releasing the engineering drawings for all
three variants;
* manufacturing and delivering 15 flight test aircraft and 7 ground
test articles;
* developing 19 million lines of software code; and:
* completing a 7-year, over 12,000-hour flight test program.
The JSF program's latest planned funding profile for development and
procurement, produced in December 2004, expects annual funding
requirements to hover close to $13 billion in between 2012 and 2022,
peaking at $13.8 billion in 2013. If the program fails to achieve its
current estimated costs, funding challenges could be even greater than
they are today. Thus, even a modest cost increase would have dramatic
impacts on funding. For example, a 10 percent increase in production
costs would amount to over $21 billion (see fig. 3).
Figure 3: JSF's Annual Funding Requirements, as of December 2004:
[See PDF for image]
[End of figure]
The current cost estimate reflects the position of the JSF program
office. The Office of Secretary of Defense Cost Analysis Improvement
Group was to update its formal independent cost estimate in the spring
of 2005, and the Navy and the Air Force were expected to fully fund the
program consistent with the estimate. The group now does not expect to
formally complete its estimate until the spring of 2006, but its
preliminary estimate was substantially higher than the program
office's. According to Cost Analysis Improvement Group officials, an
assessment of the software and mission systems requirements based on
more recent information could further increase the estimate of JSF
costs.
An Incremental Development Approach Provides an Alternative Model for
Reducing JSF Risks:
An incremental development approach consistent with DOD's policy on
evolutionary acquisition and best practices has potential to deliver
warfighter capabilities in planned product increments that would
increase JSF capabilities over time. The F-16 fighter program, the Air
Force's JSF predecessor, successfully evolved capabilities over the
span of about 30 years, delivering increases of capabilities quickly
and often, as technologies became available. That program may provide a
model for a possible alternative acquisition strategy for the JSF
program. Structuring the program into separate and manageable
increments based on what is achievable now and in the future would
allow more predictable cost and delivery estimates.
Over the past three decades, the Air Force successfully procured more
than 2,200 F-16s. The F-16 acquisition approach allowed the timely and
affordable delivery of aircraft and capability to meet the warfighter's
needs, including the recapitalization of aging aircraft. By using an
evolutionary approach to develop the aircraft, the program was able to
quickly deliver new and improved capabilities to the warfighter and
increase the aircraft's capability as new technologies were matured and
added to the aircraft. The first increment, developed during the 1970s,
provided a "day fighter" aircraft with basic air-to-air and air-to-
ground capabilities. This allowed the developer to deliver new and
useful military capability to the warfighter in less than 4 years. With
each subsequent increment, new technology was used to improve the
engine, radar, structure, avionics, and other systems that allow the
aircraft today to perform close air support, ground attack, air
defense, and suppression of enemy defense missions (see fig. 4). The
evolutionary approach also enriched the industrial base capabilities by
extending the life of the production over the length of this
incremental approach.
Figure 4: F-16 Incremental Development Approach:
[See PDF for image]
Note: This gives the number of aircraft delivered to the U.S. Air Force
only, no foreign military sales or other allied government sales
included.
[End of figure]
In contrast, JSF's fully configured design represents a quantum leap in
capability that far exceeds the capability of legacy systems that JSF
is intended to replace. While the program is using a block structure--
where each block adds capabilities over the proceeding block--the
blocks are part of a single development effort, and DOD is on contract
with the developer to deliver the warfighter the full capability (see
fig. 5). The program's block structure provides for an escalating
capability, but DOD already plans to buy 95 percent of JSF aircraft
with the ultimate capability (block 3). Unlike the approach used with
the F-16, this risky approach will likely be like past programs that
have encountered significant increases in cost and time, not allowing
DOD to quickly recapitalize the aging legacy aircraft.
Figure 5: Planned JSF Blocks:
[See PDF for image]
[End of figure]
An evolutionary approach would entail delivering a first product
increment with at least as much capability as currently operating
legacy aircraft with sufficient quantities to allow DOD to retire older
planes sooner and reduce operating inefficiencies. DOD has repeatedly
raised concerns about the age of its fighter aircraft fleet, which was
bought largely in the 1970s and 1980s and will need to be replaced
around 2010. Delays in fielding JSF aircraft may increase costs to
maintain legacy aircraft to meet force structure requirements. Limiting
development to 5-year increments or less, as suggested in the DOD
acquisition policy, would force smaller, more manageable commitments in
capabilities and make costs and schedules more predictable. Some of the
more challenging JSF capabilities, such as the advanced mission systems
or prognostics technologies, would be deferred, kept in the technology
base (off the critical path of the JSF program), and added to follow-on
increments once they are demonstrated in the more conducive technology
development environment.[Footnote 11] In addition to considering
available resources--including technology and design knowledge, budget,
and time--each increment should be based on the warfighter's most
immediate needs and the number of aircraft needed to maintain a viable
fleet. Figure 6 shows what an incremental approach might look like for
JSF.
Figure 6: Potential Application of an Incremental Development Approach
to the JSF Program:
[See PDF for image]
[A] The bar for production is not drawn to scale. JSF production lasts
for 20 years.
[End of figure]
Because an incremental approach would reduce the amount of risk in each
development phase, it would make the program manager and contractor
accountable for a lower-risk system development phase with precise cost
and schedule targets. A shorter system development phase also makes it
more feasible to align a program manager's tenure to the completion of
the phase, holding the program manager accountable for decisions made.
It also would enable the use of fixed-price-type contracts for
production that contains a pricing structure that reduces government
risk.
Conclusions:
While the JSF program plays a critical role in DOD plans to
recapitalize the services' aging tactical aircraft fleet, DOD's current
acquisition approach for the JSF program continues to be risky and
could further jeopardize meeting this important objective. The JSF
program has already encountered increases to estimated development
costs, delays to planned deliveries, and reductions in the planned
number of JSF to be procured that have eroded DOD's buying power. We
believe the current acquisition strategy to begin production in 2007 is
too risky. By committing to procure large quantities of the aircraft
before completing testing that demonstrates that the design is mature
and reliable, DOD has significantly increased the risk of further
compromising its return on investment--as well as adding more cost and
delaying the delivery of critical capabilities to the warfighter. Also,
making sizable investments in tooling and other manufacturing
capability needed to produce JSFs at higher rates before a fully
integrated aircraft is demonstrated in testing places the program at
risk for expensive design changes as testing uncovers problems.
Deferring production decisions until all three variants' performance
has been demonstrated throughout their flight envelopes and with the
full integrated capability would allow additional time to capture
knowledge and help to significantly minimize these risks.
To continue with the current plan to deliver the ultimate JSF
capability will require (1) execution of a 7-year test program without
further schedule delays, (2) development of 5 major blocks of software
(19 million lines of code), and (3) new and yet undemonstrated
technologies for advanced mission systems that must collect, analyze,
and synthesize information from other platforms or sensors not all a
part of the JSF aircraft. These are necessary to ensure the JSF has
increased situational awareness to destroy enemy air defenses and
perform deep strike missions with advanced weapons. The JSF program
will also need, annually, an average of $11 billion of development and
procurement funding for the next two decades.
All of these factors add to the challenges faced by DOD in trying to
manage this highly concurrent program, increasing the risk it will have
the same poor outcomes experienced by similar major acquisition
programs with significant cost and schedule growth and delayed
modernization and recapitalization of aging systems. DOD has an
alternative. With 90 percent of DOD's remaining planned investment in
JSF, it can adopt a knowledge-driven and evolutionary acquisition
approach to reduce JSF program risks, recapitalize its aging tactical
air force sooner, and deliver needed capabilities to the warfighter
more quickly. The experiences of the F-16 program--evolving and
improving capabilities over time while providing the warfighter combat
capability--provide a precedent for this. Requirements for the first
increment of JSF under this evolutionary approach would match a level
supported by current knowledge of technologies and design. This would
allow the testing of those reduced requirements to support a knowledge-
based low-rate production decision sooner and allow delivery of a
useful product and in sufficient quantities to start replacing the
aging legacy fighter force. Capabilities that demand as yet
undemonstrated technologies, for example, advanced mission systems and
software, would be included as requirements in subsequent future JSF
aircraft increments--managed as separate development programs--as
technology is demonstrated in the technology base and becomes
available.
Matters for Congressional Consideration:
DOD does not plan to change its business case or acquisition plan for
developing and buying the JSF. Without changes, the acquisition plan
will put at risk $50 billion for procuring JSF aircraft at the same
time the program develops and tests the aircraft's expected performance
capabilities over a 7-year, 12,000 hour flight test program. The JSF
program has continually missed its cost and schedule targets over the 5
years it has been in development. If DOD were to make smaller, more
manageable commitments in capabilities, it would make cost and schedule
more predictable and deliver needed capabilities to the war fighter
sooner. For these reasons, Congress may want to consider delaying
authorizations and appropriations for procuring JSF aircraft until:
1. DOD develops a knowledge-based business case that matches
requirements with proven technologies and design knowledge and
available funding. Capabilities that demand technological advances not
yet demonstrated should be part of future increments that are funded
and managed separately once demonstrated.
2. DOD demonstrates the aircraft design qualities and integrated
mission capabilities of the fully configured and integrated JSF
variants work as designed based on actual flight testing.
Recommendations for Executive Action:
We are making the following recommendations to reduce program risk and
increase the likelihood of successful program outcomes by delivering
capabilities to the warfighter when needed and within available
resources. Specifically, we recommend that the Secretary of Defense:
3. Delay the investment in production aircraft until sufficient testing
has at least demonstrated the basic airframe design of each JSF variant
in important parts of the flight envelope, including combinations of
critical altitudes, speeds, and angles of attack.
4. Once the basic airframe design has been demonstrated, limit
production quantities to the number of aircraft needed for operational
testing and that can be produced using the current production
capability (based on existing tooling, materials, and personnel) until
a fully integrated aircraft (with advanced mission systems and
predictive maintenance systems) demonstrates through flight testing the
required capabilities.
5. Plan an incremental acquisition strategy that follows the intent of
DOD evolutionary acquisition policy and delivers a first increment that
limits new content to proven technologies and design. The plan should
be completed and reported to the Congress by July 2006, and include
adjustments in associated programs as necessary to maintain needed
capabilities.
Agency Comments and Our Evaluation:
DOD provided us with written comments on a draft of this report. The
comments appear in appendix III.
DOD partially concurred with our recommendation that the Secretary of
Defense delay the investment in production aircraft until sufficient
testing has at least demonstrated the basic airframe design of each JSF
variant in important parts of the flight envelope, including
combinations of critical altitudes, speeds, and angles of attack. DOD
agreed that a knowledge-based approach is critical to making prudent
decisions and that this type of approach is being used by JSF. However,
DOD's response did not provide a plan of action to show how it will
ensure flight testing demonstrates acceptable performance before
significant production investments are made. In the past, Congress has
raised concerns about the costly outcomes of highly concurrent
development and production efforts that are not "flying before buying."
DOD points out that the JSF program is in its fifth year of a 12-year
development, but starting production with nearly 7 years of development
and testing remaining leaves a high probability that, because it is not
flying before buying, it will incur costly design changes and delay
getting capability to the warfighter. Until there has been testing to
demonstrate that all three variants will work as intended, we believe
entering production in 2007 places financial investments at significant
risk. The risk increases as spending for production goes from about
$100 million a month in 2007 to over $500 million a month just 2 years
later.
DOD partially concurred with our recommendation to the Secretary of
Defense to limit production quantities to the number of aircraft needed
for operational testing and that can be produced using the current
production capability until a fully integrated aircraft demonstrates
the required capabilities through flight testing. DOD stated it is
limiting production until aircraft capabilities are fully demonstrated
and that further limits are not necessary. It stated that various
program reviews to assess performance, including test objectives, are
conducted to ensure associated program risks are understood and
mitigated. We believe DOD's plan to invest almost $50 billion to buy
over 400 aircraft concurrent with testing is very risky. Significant
efforts remain in the JSF program to demonstrate the aircraft will
perform as expected. A number of major DOD acquisition programs have
employed highly concurrent acquisition strategies in the past and
experienced significant cost increases and schedule delays. DOD stated
its use of modeling and simulation and laboratory testing reduces risk.
However, DOD operational test officials have stated that demonstrations
need to occur in the actual aircraft in an operational environment to
verify that the system works as intended. For example, the F-22A
avionics software performed successfully in the laboratory but
experienced significant problems that delayed the program and increased
costs once it entered actual flight testing in the F-22A aircraft. JSF
software and advanced electronics are more complex than the F-22A's.
While DOD believes it can manage program risk by holding regular
program reviews, DOD's own experience has shown this approach is not
effective. Accepting and managing risk instead of capturing technology,
design, and manufacturing knowledge, as suggested in DOD policy, has
made it difficult for DOD to make informed decisions at key points.
DOD partially concurred with our recommendation to the Secretary of
Defense to plan an incremental acquisition strategy that follows the
intent of DOD evolutionary acquisition policy and delivers a first
increment that limits new content to proven technologies and design and
to report this plan to Congress by July 2006. It stated the JSF
Acquisition Strategy fully complies with policy and is a knowledge-
based, incremental approach that includes three blocks of increasing
degrees of warfighter capability. We believe DOD's strategy does not
provide the benefits of an evolutionary approach, as suggested by DOD's
policy and best practices. In commenting on our report, DOD did not
address the salient points concerning the true significance of adopting
an incremental acquisition approach--reducing program risk, delivering
needed capabilities to the warfighter quicker, and recapitalizing the
aging tactical aircraft fleet sooner. The JSF strategy resembles other
past major acquisition programs that have attempted to achieve the
ultimate capability in a single development increment, producing nearly
all of the fleet with the fully required capability from the outset.
DOD has allowed technology development to spill over into product
development, weakening the foundation for program cost or schedule
estimates. This has led to disastrous outcomes for other major
acquisition programs in the past. We continue to believe the successful
F-16 program can serve as a model for the JSF program. The F-16 program
evolved capabilities over a 30-year period, buying substantial
quantities of each increment in order to recapitalize aging tactical
aircraft and provide new capabilities to the warfighter more quickly.
If DOD were to make smaller, more manageable commitments in
capabilities, it would make cost and schedule more predictable and
deliver needed capabilities to the warfighter sooner.
We are sending copies of this report to the Secretary of Defense; the
Secretaries of the Air Force, Army, and Navy; and the Director of the
Office of Management and Budget. We will also provide copies to others
on request. In addition, the report will be available at no charge on
the GAO Web site at http://www.gao.gov.
If you or your staff have any questions concerning this report, please
contact me at (202) 512-4841. Contact points for our offices of
Congressional Relations and Public Affairs may be found on the last
page of this report. Other staff making key contributions to this
report were Michael Hazard, Assistant Director; Lily Chin; Matthew
Drerup; Matthew Lea; Gary Middleton; Karen Sloan; and Adam Vodraska.
Signed by:
Michael J. Sullivan:
Director:
Acquisition and Sourcing Management:
List of Congressional Committees:
The Honorable John Warner:
Chairman:
The Honorable Carl Levin:
Ranking Minority Member:
Committee on Armed Services:
United States Senate:
The Honorable Ted Stevens:
Chairman:
The Honorable Daniel K. Inouye:
Ranking Minority Member:
Subcommittee on Defense:
Committee on Appropriations:
United States Senate:
The Honorable Duncan L. Hunter:
Chairman:
The Honorable Ike Skelton:
Ranking Minority Member:
Committee on Armed Services:
House of Representatives:
The Honorable C.W. Bill Young:
Chairman:
The Honorable John P. Murtha:
Ranking Minority Member:
Subcommittee on Defense:
Committee on Appropriations:
House of Representatives:
[End of section]
Appendix I: Scope and Methodology:
To determine if the Joint Strike Fighter (JSF) program will capture
critical product knowledge before making production investments, we
compared the program's plans and results to date against best practice
standards for applying knowledge to support major program investment
decisions. The best practice standards are based on a GAO body of work
that encompasses nearly 10 years and visits to over 25 major commercial
companies. Our work has shown that valuable lessons can be learned from
the commercial sector, as well as successful Department of Defense
(DOD) cases, and can be applied to the development of weapon systems.
We reviewed JSF management reports, acquisition plans, test plans, risk
assessments, cost reports, independent program assessments, and program
status briefings. We identified gaps in product knowledge at the
production decision points, reasons for these gaps, and the risks to
program cost and schedule associated with moving forward. We reviewed
DOD's acquisition policy to determine whether JSF's approach met its
framework and intent. We interviewed officials from the DOD acquisition
program management office and prime contractor to gain their
perspectives on program risks and their approaches to managing risks.
To evaluate whether the current acquisition plan follows an
evolutionary or incremental approach, a key best practice for meeting
business case goals, we examined the JSF program's acquisition
framework and the actions taken by DOD to address the recommendations
made in our 2005 report. We compared the JSF program's approach with
the approaches used by leading companies and successful DOD programs to
evolve products to their ultimate capabilities. We also reviewed DOD
guidance on structuring evolutionary acquisition programs. To determine
the JSF program outcomes to date, we used the program estimates that
justified the program when it started in 1996. This was the point JSF
transitioned from a technology development environment to an
acquisition program environment, with the commitment to deliver a
family of strike aircraft that meets the Air Force, Navy, and Marine
Corps needs. At that time, total production, acquisition, and ownership
costs had not been estimated. The total production, acquisition, and
ownership cost estimates were first established to support the decision
to enter the system development and demonstration phase in 2001. We
used these estimates as the baseline for these costs. We also obtained
information on past DOD programs from Selected Acquisition Reports and
prior work conducted by GAO over the past 25 years.
To identify opportunities to improve JSF program outcomes, we assessed
the potential of applying an incremental development approach to the
JSF acquisition program based on the commercial model and DOD guidance.
We also examined the program history of the F-16 fighter, the JSF
predecessor, which applied an incremental approach over its 30-year
acquisition. The F-16 acquisition has been touted by DOD and others as
a successful program and a model for others to follow. To examine the
program history of the F-16, we met with F-16 program officials and
analyzed acquisition plans, management reports, and program outcomes.
In performing our work, we obtained information and interviewed
officials from the JSF Joint Program Office, Arlington, Virginia;
Lockheed Martin Aeronautical Systems, Fort Worth, Texas; Defense
Contract Management Agency, Fort Worth, Texas; and offices of the
Director, Operational Test and Evaluation, and Acquisition, Technology
and Logistics, Program Analysis and Evaluation-Cost Analysis
Improvement Group, which are part of the Office of Secretary of Defense
in Washington, D.C. We also obtained information from the F-16 program
office and Rand Corporation, Santa Monica, California.
[End of section]
Appendix II: Changes in JSF Program Purchase Quantities and Costs:
Expected quantities:
Development quantities;
November 1996 (program start): 10;
October 2001 (system development start): 14;
As of December 2005: 15;
Changes since initial estimate: 50.
Procurement quantities (U.S. only);
November 1996 (program start): 2,978;
October 2001 (system development start): 2,852;
As of December 2005: 2,443;
Changes since initial estimate: (18).
Total quantities;
November 1996 (program start): 2,988;
October 2001 (system development start): 2,866;
As of December 2005: 2,458;
Changes since initial estimate: (18).
Cost estimates (then year dollars in billions):
Development;
November 1996 (program start): $24.8;
October 2001 (system development start): $34.4;
As of December 2005: $45.7;
Changes since initial estimate: 84.
Procurement;
November 1996 (program start): Not available;
October 2001 (system development start): $196.6;
As of December 2005: $210.7;
Changes since initial estimate: 7.
Other;
November 1996 (program start): Not available;
October 2001 (system development start): $2.0;
As of December 2005: $.2;
Changes since initial estimate: (90).
Total program acquisition;
November 1996 (program start): Not available;
October 2001 (system development start): $233.0;
As of December 2005: $256.6;
Changes since initial estimate: 10.
Unit cost estimates (then year dollars in millions):
Average procurement;
November 1996 (program start): Not available;
October 2001 (system development start): $69;
As of December 2005: $86;
Changes since initial estimate: 25.
Program acquisition;
November 1996 (program start): Not available;
October 2001 (system development start): $81;
As of December 2005: $104;
Changes since initial estimate: 28.
Unit recurring flyaway costs (fiscal year 2002 dollars in millions):
Conventional takeoff and landing;
November 1996 (program start): $31.5;
October 2001 (system development start): $37.0;
As of December 2005: $44.5;
Changes since initial estimate: 41.
Short takeoff and vertical landing;
November 1996 (program start): $33.7-39.3;
October 2001 (system development start): $45.8;
As of December 2005: $58.7;
Changes since initial estimate: 49-74.
Carrier;
November 1996 (program start): $34.9-42.7;
October 2001 (system development start): $47.8;
As of December 2005: $61.7;
Changes since initial estimate: 44-77.
Source: GAO analysis of DOD data:
[End of table]
[End of section]
Appendix III: Comments from the Department of Defense:
OFFICE OF THE UNDER SECRETARY OF DEFENSE:
3000 DEFENSE PENTAGON:
ACQUISITION TECHNOLOGY AND LOGISTICS:
WASHINGTON. DC 20301-3000:
FEB 28 2006:
Mr. Michael J. Sullivan:
Director, Acquisition and Sourcing Management:
U.S. Government Accountability Office:
441 G Street, N. W.:
Washington, D.C. 20548:
Dear Mr. Sullivan:
This is the Department of Defense (DoD) response to the Government
Accountability Office (GAO) draft report 06-356, "JOINT STRIKE FIGHTER:
DOD Plans to Enter Production before Testing Demonstrates Acceptable
Performance," dated January 26, 2006 (GAO Code 120450). The Department
of Defense partially concurs with the three recommendations in the
draft report. Details of the partial concurrences are contained in the
enclosure.
The Department appreciates the opportunity to comment on the draft
report.
Sincerely,
Signed by:
Mark D. Schaeffer:
Acting Director:
Defense Systems:
Enclosure: As stated:
GAO DRAFT REPORT - DATED JANUARY 26, 2006:
GAO CODE 120450/GAO-06-356:
"JOINT STRIKE FIGHTER: DOD Plans to Enter Production before Testing
Demonstrates Acceptable Performance"
DEPARTMENT OF DEFENSE COMMENTS TO THE RECOMMENDATIONS:
RECOMMENDATION 1: The GAO recommended that the Secretary of Defense
delay the investment in production aircraft until sufficient testing
has at least demonstrated the basic airframe design of each JSF variant
in important parts of the flight envelope, including combinations of
critical altitudes, speeds, and angles of attack. (p. 22/GAO Draft
Report):
DOD RESPONSE: Partially Concur. The Department agrees that a knowledge-
based approach is critical to making prudent acquisition decisions. The
JSF Acquisition Strategy incorporates this type of approach.
Established decision criteria to support each production decision
include specific test deliverables.
The JSF is in the 5th year of a 12-year development effort. In 2004,
the program was restructured to allow additional development time to
meet performance requirements associated with the Short Takeoff and
Vertical Landing (STOVL) variant. Due to the high degree of commonality
throughout the three JSF variants, the design changes from the STOVL
weight reduction effort benefit the Conventional Takeoff and Landing
(CTOL) and Carrier Variant (CV). The restructured program extended the
System Development and Demonstration (SDD) phase by 18 months. The
Department approved an updated Acquisition Strategy in May 2005 that
reflects changes since the start of SDD, including the restructuring. A
significant aspect of the restructuring was a comprehensive review and
update to integrated test and evaluation planning.
The Department is confident that the Acquisition Strategy and
management practices currently in place achieve the objectives of GAO's
recommendation.
RECOMMENDATION 2: The GAO recommended that the Secretary of Defense
limit production quantities to the number of aircraft needed for
operational testing and that can be produced using the current
production capability (based on existing tooling, materials, and
personnel) until a fully integrated aircraft (with advanced mission
systems and predictive maintenance systems) demonstrates through flight
testing the required capabilities. (p. 22/GAO Draft Report):
DOD RESPONSE: Partially Concur. The Department plans limited production
until aircraft capabilities are fully demonstrated. Further limitations
on production are not necessary. To reduce both risk and cost, the JSF
Acquisition Strategy takes advantage of the benefits of flight test and
significant advances in modeling, simulation, and lab testing that have
been achieved in the commercial and military sectors. The JSF
Acquisition Strategy provides the most effective balance of technical
risk, financial resources, and the Services' operational needs.
The JSF Acquisition Strategy includes clear entry and exit criteria for
critical milestones to ensure that technologies are mature, and
required incremental test objectives are achieved before obligating
funds for SDD or production. The Department conducts acquisition
reviews via Integrating Integrated Product Teams and Overarching
Integrated Product Teams, which support Defense Acquisition Board
reviews. Configuration Steering Board and Service Acquisition Executive
reviews are conducted quarterly to assess program performance,
including test objectives, ensuring associated program risks are
understood and appropriately mitigated.
RECOMMENDATION 3: The GAO recommended that the Secretary of Defense
plan an incremental acquisition strategy that follows the intent of DOD
acquisition policy and delivers a first increment that limits new
content to proven technologies and design. The plan should be completed
and reported to the Congress by July 2006, and include adjustments in
associated programs as necessary to maintain needed capabilities. (p.
22/GAO Draft Report):
DOD RESPONSE: Partially Concur. The Department's approved JSF
Acquisition Strategy fully complies with acquisition policy. It is
premised on a knowledge-based, incremental approach that provides
increasing degrees of warfighting capability with each block. Technical
maturity, test accomplishments and operational effectiveness are key
considerations at each acquisition decision point.
Early production aircraft will have Block 1 capability for standup of
initial training squadrons. Block 2 upgrades will support Marine Corps
Initial Operational Capability. Block 3 will provide full avionics and
key weapons capability. Previously delivered Block 1 and 2 aircraft
will be upgraded to Block 3 capability. This approach provides initial,
incremental capability to meet the most immediate, achievable
warfighting requirements while development on more complex capabilities
continues.
The Department structured the JSF program from the beginning to
emphasize jointness, technology maturation and concept demonstrations,
as well as early cost and performance trades integral to the weapons
system requirements definition process. The Department approved the
updated JSF Acquisition Strategy in May 2005. Department and JSF
Program leadership have provided program updates to Congress since then
and will continue to do so.
[End of section]
FOOTNOTES
[1] Section 213 of the act requires us to assess the extent to which
the system development and demonstration program is currently meeting
key cost, schedule, and performance goals; the likelihood that the
program will be completed within estimated costs; and the program's
current acquisition plan leading to production.
[2] GAO, Tactical Aircraft: Opportunity to Reduce Risks in Joint Strike
Fighter Program with Different Acquisition Strategy, GAO-05-271
(Washington, D.C.: Mar. 15, 2005).
[3] A business case provides demonstrated evidence that (1) the
warfighter need exists and that it can best be met with the chosen
concept and (2) the concept can be developed and produced within
existing resources--including proven technologies, design knowledge,
adequate funding, and adequate time to deliver the product when needed.
Establishing a business case calls for a realistic assessment of risks
and costs; doing otherwise undermines the intent of the business case
and invites failure.
[4] A production representative aircraft is one that is built using
production-like tooling and methods and using production workers and is
of the design expected be built to satisfy the requirements of the
warfighter. Flight-testing a production representative prototype that
incorporates all critical software is the primary method to capture key
design knowledge that indicates the system will work as planned, is
reliable, and is ready for production. The higher the percentage of
testing completed, the lower the risk of investing in production.
[5] Sensor fusion is the ability to take information from both multiple
onboard and offboard aircraft sensors and display the information in an
easy-to-use format for the pilot. It is vitally important, because the
JSF is a single-seat aircraft, and the pilot needs support to carry out
multiple types of missions.
[6] Initial operational testing consists of field tests intended to
demonstrate a system's effectiveness and suitability for military use.
[7] These figures do not include the potential for orders for
international partners during low-rate initial production. Preliminary
data indicate that these orders could significantly increase this rate.
[8] DOD policy contains three critical milestone decision points in its
acquisition process. One of these, referred to as milestone C,
authorizes entry into low-rate initial production. However, the JSF
program is deviating from that milestone framework and has delayed this
critical milestone decision point to the full-rate production decision
point in 2013--more than 6 years after entering low-rate initial
production. This delay compromises the controls expected in the DOD
acquisition policy for the start of production.
[9] During this approximately 6-month period of initial tests, which
includes about 30 planned flights, the program intends to demonstrate
the aircraft's flight qualities in limited flight conditions, takeoff
and landing, and initial air refueling. However, recent program data
indicate that the first flight may be delayed until October 2006,
reducing the number of flights that may be completed before the
production decision.
[10] Rand Corporation, Test and Evaluation Trends and Costs for
Aircraft and Guided Weapons (Santa Monica, California, 2004).
[11] These technologies are not expected to be fully mature until 2011.
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
