NASA's James Webb Space Telescope
Knowledge-Based Acquisition Approach Key to Addressing Program Challenges Gao ID: GAO-06-634 July 14, 2006The National Aeronautics and Space Administration's (NASA) James Webb Space Telescope (JWST) is being designed to explore the origins and nature of the universe. It should allow scientists to look deeper into space--and thus farther back in time--than ever before. The program, however, has experienced cost growth of more than $1 billion and its schedule has slipped nearly 2 years. NASA recently restructured the program and now anticipates a launch no sooner than June 2013. Because of the cost and schedule problems, under the Comptroller General's authority, we reviewed the JWST program to determine the extent to which this procurement follows NASA acquisition policy and GAO best practices for ensuring that adequate product knowledge is used to make informed investment decisions
Although the JWST program recently revised its acquisition strategy to conform to NASA's acquisition policies, the program still faces considerable challenges because it has not fully implemented a knowledge-based approach, which our past work has shown is often a key factor in program success. In a recent report, we made recommendations that NASA take steps to ensure that projects follow a knowledge-based approach for product development. NASA concurred and revised its acquisition policy. When we initiated our work and before the JWST program's recently revised acquisition strategy, program officials intended to have NASA commit to program start, which is the end of the formulation phase and the beginning of the implementation phase, with immature technologies, according to best practices, and without a preliminary design. During our review, we discussed these shortfalls with NASA officials, and they revised their acquisition strategy to conform to NASA policy. However, the current strategy still does not fully incorporate a knowledge-based approach which ensures that resources match requirements in terms of knowledge, time, and money before program start. If program officials follow the current plan, the maturity of key technologies may not be adequately tested prior to program start. In addition, it appears the program will not have sufficient funding resources to ensure the program's success. In light of the fiscally constrained environment the federal government and NASA will face in the years ahead, adopting a knowledge-based approach will not only increase the JWST program's chances for success but also lay the foundation for comparison between competing programs.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-06-634, NASA's James Webb Space Telescope: Knowledge-Based Acquisition Approach Key to Addressing Program Challenges
This is the accessible text file for GAO report number GAO-06-634
entitled 'NASA's James Webb Space Telescope: Knowledge-Based
Acquisition Approach Key to Addressing Program Challenges' which was
released on July 13, 2006.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Committees:
United States Government Accountability Office:
GAO:
July 2006:
NASA'S James Webb Space Telescope:
Knowledge-Based Acquisition Approach Key to Addressing Program
Challenges:
James Webb Space Telescope:
GAO-06-634:
GAO Highlights:
Highlights of GAO-06-634, a report to congressional committees
Why GAO Did This Study:
The National Aeronautics and Space Administration‘s (NASA) James Webb
Space Telescope (JWST) is being designed to explore the origins and
nature of the universe. It should allow scientists to look deeper into
space”and thus farther back in time”than ever before. The program,
however, has experienced cost growth of more than $1 billion and its
schedule has slipped nearly 2 years. NASA recently restructured the
program and now anticipates a launch no sooner than June 2013. Because
of the cost and schedule problems, under the Comptroller General‘s
authority, we reviewed the JWST program to determine the extent to
which this procurement follows NASA acquisition policy and GAO best
practices for ensuring that adequate product knowledge is used to make
informed investment decisions
What GAO Found:
Although the JWST program recently revised its acquisition strategy to
conform to NASA‘s acquisition policies, the program still faces
considerable challenges because it has not fully implemented a
knowledge-based approach, which our past work has shown is often a key
factor in program success. In a recent report, we made recommendations
that NASA take steps to ensure that projects follow a knowledge-based
approach for product development. NASA concurred and revised its
acquisition policy. When we initiated our work and before the JWST
program‘s recently revised acquisition strategy, program officials
intended to have NASA commit to program start, which is the end of the
formulation phase and the beginning of the implementation phase, with
immature technologies, according to best practices, and without a
preliminary design. During our review, we discussed these shortfalls
with NASA officials, and they revised their acquisition strategy to
conform to NASA policy. However, the current strategy still does not
fully incorporate a knowledge-based approach which ensures that
resources match requirements in terms of knowledge, time, and money
before program start. If program officials follow the current plan, the
maturity of key technologies may not be adequately tested prior to
program start. In addition, it appears the program will not have
sufficient funding resources to ensure the program‘s success. In light
of the fiscally constrained environment the federal government and NASA
will face in the years ahead, adopting a knowledge-based approach will
not only increase the JWST program‘s chances for success but also lay
the foundation for comparison between competing programs.
Figure: Conceptual Drawing of NASA's JWST.
[See PDF for Image]
[End of Figure]
What GAO Recommends:
GAO recommends that the NASA administrator: (1) direct the JWST program
to fully apply a knowledge-based acquisition approach to ensure that
adequate knowledge is attained at key decision points and also to hold
the program accountable and
(2) instruct the JWST program to continue to adhere to NASA acquisition
policy and go forward only after demonstrating that it is meeting
incremental knowledge markers and has sufficient funds to execute the
program. NASA concurred with GAO‘s recommendations.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-634].
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Allen Li at (202) 512-
4841 or lia@gao.gov.
[End of Section]
Contents:
Letter:
Results in Brief:
Background:
JWST's Revised Strategy Does Not Fully Incorporate a Knowledge-Based
Approach That Could Reduce Risks and Better Inform Decision Making:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Technology Readiness Levels:
Appendix III: Comments from the National Aeronautics and Space
Administration:
Appendix IV: GAO Contact and Staff Acknowledgments:
Related GAO Products:
Figures:
Figure 1: Conceptual Drawing of NASA's JWST:
Figure 2: Comparison of NASA's Life Cycle with a Knowledge-Based
Acquisition Life Cycle:
Figure 3: Technology Maturity Levels for Product Development:
Abbreviations:
CSA: Canadian Space Agency:
ESA: European Space Agency:
JWST: James Webb Space Telescope:
KP1: Knowledge point 1:
NARN: on-advocate Review:
NASA: National Aeronautics and Space Administration:
PDR: Preliminary Design Review:
TRL: Technology Readiness Levels:
United States Government Accountability Office:
Washington, DC 20548:
July 14, 2006:
Congressional Committees:
As the expected follow-on to the tremendously successful Hubble Space
Telescope, the National Aeronautics and Space Administration's (NASA)
James Webb Space Telescope (JWST) is being designed to explore the
early universe and allow scientists to shed light on the origins and
nature of the universe by allowing them to look deeper into space--and
thus farther back in time--than ever before. Recently, however, NASA
acknowledged that the program[Footnote 1] has experienced cost growth
exceeding $1 billion--which increased its life-cycle cost estimate from
$3.5 billion to $4.5 billion--and its schedule has slipped nearly 2
years. The agency restructured the program and is now anticipating a
launch no sooner than June 2013.
Because of the restructuring and past cost and schedule problems, we
reviewed the program to determine the extent to which the JWST
program's acquisition strategy follows NASA acquisition policy and
Government Accountability Office (GAO) best practices for ensuring that
adequate product knowledge is used to make informed investments. We
conducted our work under the Comptroller General's authority and are
addressing this report to you because of your committee's or
subcommittee's interest in NASA activities.
To assess the extent to which the JWST acquisition strategy follows
NASA policy and GAO best practices for ensuring readiness to proceed
into implementation, we reviewed NASA policy guidance and compared it
with the JWST program's acquisition strategy. We also benchmarked the
JWST acquisition strategy to best practices. We interviewed NASA and
contractor officials to clarify our understanding of the program's
management approach and technology development plan. We analyzed cost
and schedule information and discussed the impact of the investment in
the JWST on other NASA programs with NASA officials. We attended two
design reviews, including one at the prime contractor's facility. We
performed our review from August 2005 through May 2006 in accordance
with generally accepted government auditing standards.
Results in Brief:
Although the JWST program recently revised its acquisition strategy to
conform to NASA's acquisition policies, the program still faces
considerable challenges because it has not fully implemented a
knowledge-based approach. Our past work on the best practices of
product developers in government and industry has found that using a
knowledge-based approach is often a key factor in program success. We
recently made recommendations that NASA take steps to ensure that
projects follow a knowledge-based approach for product
development.[Footnote 2] NASA concurred and revised its acquisition
policy. When we initiated our work and before the JWST program's
recently revised acquisition strategy, program officials intended to
have NASA commit to the program and start implementation with immature
technologies, according to best practices, and without a preliminary
design. During our review, we discussed these shortfalls with NASA
officials, and they revised their acquisition strategy to conform to
NASA policy. However, the current strategy still does not fully
incorporate a knowledge-based approach that ensures that resources
match requirements in terms of knowledge, time, and money before
program start, which is the end of the formulation phase and the
beginning of the implementation phase. If program officials follow the
current plan, the maturity of key technologies may not be adequately
tested prior to program start. For example, a test to demonstrate
critical performance parameters is scheduled to occur after the program
start decision and some planned test items may not provide the validity
needed to adequately verify technology maturity. In addition, it
appears the program will not have sufficient funding resources to
ensure the program's success. According to a review conducted by NASA's
Independent Program Assessment Office, the program's contingency
funding is too low and phased in too late in the program to support the
planned launch date and provide the necessary resources to address as
yet unforeseen problems. In light of the fiscally constrained
environment the federal government and NASA will face in the years
ahead, adopting a knowledge-based approach will not only increase the
JWST program's chances for success but also lay the foundation for
comparison between competing programs. As more programs, such as the
JWST, move into implementation, using a knowledge-based approach will
allow NASA to assess these development efforts in a consistent format
to confirm the continued viability of the investment.
To increase the JWST program's chances of successful product
development and to better inform NASA's decision-making process, we are
recommending that the NASA Administrator (1) direct the JWST program to
apply a knowledge-based acquisition approach, including incremental
markers, to ensure that adequate knowledge is attained at key decision
points and to hold the program accountable and (2) instruct the JWST
program to continue to adhere to NASA acquisition policy and base the
program's go/no-go decision not only on adherence to that policy, but
also on demonstrating that it is meeting incremental knowledge markers
and that adequate funds are available to execute the program.
In written comments on a draft of this report, NASA concurred with our
recommendations. NASA's comments are included in their entirety in
appendix III.
Background:
The JWST--identified by the National Research Council as the top
priority new initiative for astronomy and physics for the current
decade--is a large deployable space-based observatory being developed
to study and answer fundamental questions ranging from the formation
and structure of the universe to the origin of planetary systems and
the origins of life. Often referred to as the replacement to Hubble,
the JWST is more of a next generation telescope--one that scientists
believe will be capable of seeing back to the origins of the universe
(Big Bang). The JWST will have a large, segmented primary mirror--6.5
meters (about 21 feet) in diameter--which is a leap ahead in technology
over the last generation of mirrors. The observatory requires a
sunshield approximately the size of a tennis court to allow it to cool
to the extremely cold temperature (around 40 degrees Kelvin, or minus
388 degrees Fahrenheit) necessary for the telescope and science
instruments to work. The mirror and the sunshield--both critical
components--must fold up to fit inside the launch vehicle and open to
their operational configuration once the JWST is in orbit. In addition,
the observatory will house science instruments--such as a near-
infrared[Footnote 3] camera, a near-infrared spectrograph,[Footnote 4]
a mid-infrared instrument, and a fine guidance sensor--to enable
scientists to conduct various research activities.
Figure 1: Conceptual Drawing of NASA's JWST:
[See PDF for image]
Source: Northrup Grumman Corporation.
[End of figure]
The JWST is an international collaboration among the United States, the
European Space Agency (ESA), and the Canadian Space Agency (CSA). ESA
will provide the near-infrared spectrograph science instrument, the
optical bench assembly of the mid-infrared instrument, and the launch
of the JWST by means of an Ariane 5 expendable launch vehicle. CSA's
contribution will be the fine guidance sensor to enable stable
pointing.
Recently, the JWST program recognized significant cost growth and
schedule slippage. In March 2005, NASA identified about $1 billion cost
growth, which increased the JWST's life-cycle cost estimate from $3.5
billion to $4.5 billion. In addition, the program's schedule slipped
nearly 2 years. As a result, the program began a series of re-
baselining efforts to revise its acquisition strategy. In summer 2005,
NASA Headquarters chartered two independent review teams--an
Independent Review Team from NASA's Independent Program Assessment
Office and a Science Assessment Team--to evaluate the program. The
Independent Review Team was charged with examining the program's new
cost/schedule/ technical baseline and reported in mid-April 2006 that
(1) the JWST's scientific performance met the expectations of the
science community, (2) the technical content was complete and sound,
and (3) the Goddard Space Flight Center and contractor teams were
effective. However, the team was concerned about the program's early
year funding constraints.
The Science Assessment Team, an international team of outside experts,
was established to evaluate scientific capabilities of the JWST in the
2015 time frame in light of other astronomical facilities that would be
available. The team concluded that the financial savings gained from
the reduction in the size of the primary mirror area would not be worth
the resultant loss of scientific capabilities. The team recommended
relaxing some science requirements and simplifying other aspects of the
mission, such as integration and testing, to reduce the program's cost
risk. For example, the team recommended relaxing the contamination
requirements, allowing the project to test the mirrors using an
innovative approach that will reduce costs. The team also recommended
that the JWST de-emphasize the shorter wavelengths, since other
astronomical facilities would be available to cover that range.
JWST's Revised Strategy Does Not Fully Incorporate a Knowledge-Based
Approach That Could Reduce Risks and Better Inform Decision Making:
The JWST program recently revised its acquisition strategy to conform
to NASA's acquisition policies; however, the program still faces
considerable challenges. GAO best practices work has found that using a
knowledge-based approach is a key factor in program success. When we
initiated our work and before the program's recently revised
acquisition strategy, program officials intended to have NASA commit to
the program and start implementation with immature technologies,
according to best practices, and without a preliminary design. During
our review, we discussed these shortfalls with NASA officials, and they
revised their acquisition strategy to align their decision milestones
in accordance with NASA acquisition policy. While this is a good step,
the current strategy does not fully incorporate a knowledge-based
approach that could reduce the program's risks by ensuring that
resources match requirements at program start. By closely following a
knowledge-based approach, the JWST program will increase its chances
for success and better inform NASA's decision making.
Immature Technologies, Design Challenges, and Testing Restrictions
Still Pose Risks:
The JWST contains several innovations, including lightweight optics, a
deployable sunshield, and a folding segmented mirror. Although the
program began risk reduction activities early to develop and mature
some technologies, such as the lightweight segmented folding mirror,
the program is challenged with maturing some of its other critical
technologies. For example, the sunshield, which consists of five layers
of membranes, must be folded for launch but then unfurled to its
operational configuration--with enough tension to prevent wrinkle
patterns that could interfere with the telescope's mirrors, but not so
much tension to cause tears in the fabric. The sunshield must also be
aligned with the rest of the observatory so that only the top layer of
the sunshield is visible to the primary mirror and a correct angle
between the observatory and the sun and other heat-radiating bodies is
maintained to enable the telescope and science instruments to preserve
the very cold temperature--about 40 degrees Kelvin--critical for
achieving the JWST's mission. In addition, using passive cooling
devices, such as heat switches, to allow specific areas of the
telescope to cool down, represent additional challenges since these
items will be used in new configurations. NASA also recently
substituted the cryo-cooler used for the mid-infrared instrument for a
lower technology component to save mass. According to JWST officials,
the program recently awarded the development contract for the cryo-
cooler. In addition, the micro shutter array, which will allow the JWST
to program specific patterns of the electromagnetic spectrum for
viewing, is a new technology being developed by the Goddard Space
Flight Center and is still at a relatively low level of maturity. JWST
officials acknowledge that they are concerned about maturing the cryo-
cooler and the micro shutter array.
In addition, the program also faces design challenges related to the
launch vehicle and the observatory's stability. For example, program
officials told us that they may need to request a waiver because the
telescope will not fit within the criteria limits of the launch
vehicle's envelop without making design modifications. Furthermore, due
to the late selection of the launch vehicle, the project office and
prime contractor are just beginning to discuss interfaces,
transportation at the launch site, and the additional space issue with
Ariane 5 officials. Also, the project faces the unresolved problem of
finding the best way to keep the observatory stable. The large
sunshield, observatory attitude changes, and other effects conspire to
produce unbalanced torques, which can make the observatory unstable.
The project continues to look at ways to resolve this problem,
including thrusters to rebalance the observatory, but project officials
say this will continue to be a challenge.
Another overriding concern is NASA's inability to test the entire
observatory in its operational environment, since there is no test
facility in the United States large enough to perform this test. The
plan is to incrementally test components and subsystems on the ground
in laboratories simulating the observatory's operational environment
and to make extensive use of modeling and simulation. According to the
memorandum summarizing the January 2006 System Definition Review, a key
concern is that the JWST is pushing the limits of ground test
facilities and cannot be tested at the observatory level; therefore,
requiring complicated integration and testing with a series of
subsystem tests and analyses. In its April 2006 assessment of the JWST
program, the Independent Review Team reported that there are several
exceptions to the "test as you fly"[Footnote 5] guideline and that
mitigation strategies need to be developed before the end of the
preliminary design phase.
Containing Further Cost Growth and Schedule Slippage:
In March 2005, the JWST program recognized that its cost had grown by
about $1 billion, increasing the JWST's life-cycle cost estimate from
$3.5 billion to $4.5 billion. About half of the cost growth was due to
schedule slippage--a 1-year schedule slip because of a delay in the
decision to use an ESA-supplied Ariane 5 launch vehicle and an
additional 10-month slip caused by budget profile limitations in fiscal
years 2006 and 2007. More than a third of the cost increase was caused
by requirements and other changes. An increase in the program's
contingency funding accounted for the remainder--about 12 percent--of
the growth.
Despite an increase in the program's contingency funding, the
Independent Review Team found that the contingency funding is still
inadequate. In its April 2006 assessment of the JWST program's re-
baselining, the Independent Review Team expressed concern over the
program's contingency funding, stating that it is too low and phased in
too late. According to the team, the program's contingency from 2006
through 2010 of only $29 million, or about 1.5 percent,[Footnote 6]
after "liens" and "threats"[Footnote 7] is inadequate.[Footnote 8] The
team also stated that a 25 percent to 30 percent total contingency is
appropriate for a program of this complexity. The program's total
contingency is only about 19 percent. The team warned that because of
the inadequate contingency, the program's ability to resolve issues,
address program risk areas, and accommodate unknown problems is very
limited. Therefore, the team concluded that from a budget perspective,
the re-baselined program is not viable for a 2013 launch. The team
recommended that before the Non-Advocate Review (NAR)[Footnote 9]
leading to program start, steps should be taken by the Science Mission
Directorate to assure that the JWST program contains an adequate time-
phased funding contingency to secure a stable launch date.
The JWST program remains at risk of incurring additional cost growth
and schedule slippage because of the technical challenges that must be
resolved--immature technologies, design challenges, and testing
restrictions. Our best practices work indicates that immature
technology increases the risk of cost increases and schedule slips.
Unresolved technology challenges can cascade through a product
development cycle often resulting in an unstable design that will
require more testing and thus more time and money to fix the problems.
Subsequently, it will be difficult to prepare a reliable cost estimate
until these challenges are resolved.
Knowledge-Based Approach Key to Overcoming Challenges:
Our past work on the best practices of product developers in government
and industry has found that the use of a knowledge-based approach is a
key factor in successfully addressing challenges such as those faced by
the JWST program. Over the last several years, we have undertaken a
body of work on how leading developers in industry and government use a
knowledge-based approach to deliver high quality products on time and
within budget.[Footnote 10] A knowledge-based approach to product
development efforts enables developers to be reasonably certain that,
at critical junctures or "knowledge points" in the acquisition life
cycle, their products are more likely to meet established cost,
schedule, and performance baselines and therefore provides them with
information needed to make sound investment decisions. The marker for
the first juncture--knowledge point 1 (KP1)--occurs just prior to
program start. At KP1, the customer's requirements match the product
developer's resources in terms of knowledge, time, and money. At KP 2,
the product design is stable, and production processes are mature at KP
3. Product development efforts that have not followed a knowledge-based
approach can frequently be characterized by poor cost, schedule, and
performance outcomes.
We recently reported that NASA's revised acquisition policy for
developing flight systems and ground support projects incorporates some
aspects of the best practices used by successful developers.[Footnote
11] For example, NASA policy requires projects to conduct a major
decision review--NAR--before moving from formulation to implementation.
Further, before moving from formulation to implementation, projects
must validate requirements and develop realistic cost and schedule
estimates, human capital plans, a preliminary design, and a technology
plan--all key elements for matching needs to resources before
commitment to a major investment is made at project start. Figure 2
compares NASA's life cycle with a knowledge-based acquisition life
cycle.
Figure 2: Comparison of NASA's Life Cycle with a Knowledge-Based
Acquisition Life Cycle:
[See PDF for image]
Sources: NASA and GAO analysis.
[End of figure]
While the policy incorporates elements of a knowledge-based approach,
we also reported that NASA's acquisition policies lack the necessary
requirements to ensure that programs proceed and are funded only after
an adequate level of knowledge at key junctures. For example, NASA
policy does not require that programs demonstrate technologies at high
levels of maturity at program start. Further, although NASA policy does
require project managers to establish a continuum of technical and
management reviews, the policy does not specify what these reviews
should be nor does it require major decision reviews at other key
points in a product's development. These best practices could be used
to further reduce program risks.
In order to close the gaps between NASA's current acquisition
environment and best practices on knowledge-based acquisition, we
recommended that NASA take steps to ensure that NASA projects follow a
knowledge-based approach for product development. Specifically, we
recommended that NASA (1) in drafting its systems engineering policy,
incorporate requirements for flight systems and ground support projects
to capture specific product knowledge by key junctures in project
development and use demonstration of this knowledge as exit criteria
for decision making at key milestones and (2) revise NASA Procedural
Requirements 7120.5C to institute additional major decision reviews
following the NAR for flight systems and ground support projects, which
result in recommendations to the appropriate decision authority at key
milestones. NASA concurred with our recommendations and agreed to
revise its policies.
One of the resources needed at program start is mature technology. Our
best practices work has shown that technology readiness levels
(TRL)[Footnote 12]--a concept developed by NASA--can be used to gauge
the maturity of individual technologies. Specifically, TRL 6--
demonstrating a technology as a fully integrated prototype in a
realistic environment--is the level of maturity needed to minimize
risks for space systems entering product development. To achieve TRL 6,
technology maturity must be demonstrated in a relevant environment
using a prototype or model. (See app. II for a detailed description and
definition of TRLs and test environments.)
Figure 3: Technology Maturity Levels for Product Development:
[See PDF for image]
Source: HAO>
[End of figure]
A knowledge-based approach also involves the use of incremental markers
to ensure that the required knowledge has been attained at each
critical juncture. For example, exit criteria at KP1 should include
demonstrated maturity of critical technologies, completed trade-offs
and finalized requirements, and initial cost and schedule estimates
using results from the preliminary design review. The approach ensures
that managers will (1) conduct activities to capture relevant product
development knowledge, (2) provide evidence that knowledge was
captured, and (3) hold decision reviews to determine that appropriate
knowledge was captured to allow a move to the next phase. If the
knowledge attained at each juncture does not justify the initial
investment, the project should not go forward and additional resources
should not be committed.
Risks Not Fully Addressed by Recently Revised Acquisition Strategy:
Prior to the program's recent acquisition strategy revision, program
officials were not following NASA acquisition policy[Footnote 13] and
were set to commit to the program and start implementation with
immature technologies, according to best practices, and without a
preliminary design. For instance, the schedule called for convening the
NAR before the end of preliminary design. NASA policy indicates that
the NAR and Preliminary Design Review (PDR) should be aligned. Even at
the pre-NAR[Footnote 14] in July 2003, the plan had been to have the
NAR before the PDR,[Footnote 15] although the two reviews were closer
together than the more recent plan.
During our review, we discussed these shortfalls with NASA officials.
To their credit, they revised their acquisition strategy to conform to
NASA policy. Currently, the mission NAR--upon which the program start
decision will be based--will be aligned with the mission PDR (scheduled
for March 2008). We believe this is a positive step, since it will
ensure that a preliminary design--a key element for matching needs to
resources--is established before program start. The revised strategy
also splits the NAR into two parts--a technical NAR and a mission NAR.
The purpose of the technical NAR (scheduled for January 2007) will be
to determine whether the project has successfully retired its invention
risk, i.e., critical technologies have achieved TRL 6, according to a
NASA official. Technology issues will not be revisited after the
technical NAR unless problems arise. However, it is unclear if the
critical technologies will be demonstrated to a level of fidelity
required by best practices at the technical NAR. Furthermore, the
strategy does not fully incorporate a knowledge-based approach that
could address the program's risks by ensuring--through the use of exit
criteria--that resources match requirements in terms of knowledge,
time, and money before program start. For example:
* Under a knowledge-based approach, adequate testing is required to
demonstrate that key technologies are mature--at TRL 6--prior to
program start. This is particularly important for the JWST, given the
program's challenges with testing restrictions and the fact that the
observatory cannot be serviced in space. In some cases, such as the
sunshield, backup technologies do not exist, thus increasing the
importance of adequately maturing and testing critical technologies. If
key components--like the sunshield--fail, then the entire observatory
will be lost. This requires greater fidelity in the testing, even as
early as demonstrating the maturity of key technologies prior to
program start.
To achieve TRL 6 (the maturity level required by best practices for
program start), technology maturity must be demonstrated as a
representative model or prototype--which is very close to the actual
system in form, fit, and function--in a relevant environment. However,
there is risk that the current JWST technology development plan will
not result in the appropriate demonstration of technology maturity. For
example, the half-scale thermal vacuum test of the entire
observatory[Footnote 16] at Johnson Space Center is currently planned
for September 2008, and so the knowledge gained regarding the maturity
of the sunshield's thermal and dynamic performance[Footnote 17] is
pushed out 6 months beyond the PDR/NAR/program start date of March
2008. When JWST program officials briefed us in August 2005, the TRL
levels for thermal and dynamic performance of the sunshield were both
assessed to be at TRL 4, and the plan to get to TRL 6 was to test these
subsystems during this half-scale thermal vacuum test. However, in fall
2005 program officials reviewed the technology development plan and
concluded that only the materials for the sunshield's membrane are
technology development items, while other items affecting the
configuration and deployment of the sunshield--such as thermal and
dynamic performance--are considered engineering challenges. JWST
officials stated that earlier testing of sample materials demonstrated
the sunshield's thermal performance and a demonstration using a 1/10th
scale model demonstrated dynamic performance[Footnote 18] and satisfied
TRL 6 requirements. However, we have found in our best practices work
that demonstrating a technology to a TRL 6 typically involves
demonstrating that a prototype--close to the form, fit, and
functionality intended for the product--has been demonstrated in an
environment that closely represents the anticipated operational
environment. In our past review of development programs, we have found
that if this level of maturity is not demonstrated before a product
development effort is launched, a program increases the likelihood of
cost growth and schedule delays as it tries to close the knowledge gap
between the technologies' maturity level and the product's design
requirements.
* The JWST program's inadequate contingency runs contrary to another
premise of a knowledge-based approach--having sufficient resources in
terms of funding available to ensure a program's success. As discussed
in an earlier section, the Independent Review Team stated that the
program's contingency from 2006 through 2010 of only about 1.5 percent
after "liens" and "threats" is inadequate. The team warned that,
because of the inadequate contingency, the program's ability to resolve
issues, address program risk areas, and accommodate unknown problems is
very limited. The team concluded that, from a budget perspective, the
re-baselined program is not viable for a 2013 launch.
Knowledge-Based Approach Would Allow the JWST Program to Better Inform
NASA's Decision-Making Process:
A good basis for making informed investment decisions is essential in
the fiscally constrained environment that now exists across the federal
government. Our nation faces large, growing, and structural long-term
fiscal imbalances. Given the severity of those fiscal challenges and
the wide range of federal programs, hard choices need to be considered
across the government, and NASA is no exception. NASA must compete with
other departments and agencies for part of a constricted discretionary
spending budget.
In the near future, NASA will need to determine the resources necessary
to develop the systems and supporting technologies to achieve the
President's Vision for Space Exploration--while simultaneously
financing its other priority programs--and structure its investment
strategy accordingly. Initial implementation of the Vision as explained
in NASA's Exploration Systems Architecture Study calls for completing
the International Space Station, developing a new crew exploration
vehicle, and returning to the moon no later than 2020. NASA estimates
that it will cost approximately $104 billion over the next 13 years to
accomplish these initial goals. These priorities, along with NASA's
other missions, will be competing within NASA for funding. It will
likely be difficult for decision makers to agree on which projects to
invest in and which projects, if any, to terminate. The NASA
Administrator has acknowledged that NASA faces difficult choices about
its missions in the future--for example, between human space flight,
science, and aeronautics missions.
In the President's fiscal year 2007 budget request for NASA, the JWST
has the largest budget allocation of all programs in the Science
Mission Directorate's Astrophysics Division for the 5-year budget
horizon from fiscal year 2007 through fiscal year 2011--nearly $2
billion of the division's $6.9 billion total budget, or about 29
percent. An inadequately informed decision to commit to the estimated
$4.5 billion total funding for the JWST would significantly impact
NASA's science portfolio, since funding given to the JWST will not
available for other programs. Early in the planning for how to handle
the JWST program's cost growth, NASA officials recognized the impact
that the JWST's cost growth could have on other programs. In a July
2005 briefing to the Agency Program Management Council[Footnote 19]
soon after the cost growth was identified, NASA officials stated that
"something must give if JWST stays in the portfolio." The choices
discussed were (1) relaxing requirements or (2) adding budget and
schedule, which would mean that other missions would be deferred or
deleted from the portfolio.
In addition, committing to the JWST program obligates the government
contractually, since it allows the prime contractor to begin
implementation tasks on the very long prime contract extending from
October 2002 through launch--currently planned for June 2013--plus one
year. The contract states that until the project achieves the
implementation milestone, contract spending is limited to formulation
activities, except for long-lead items and other activities approved in
writing. After the implementation milestone is achieved at program
start, the contracting officer will notify the contractor by letter to
proceed to implementation. According to the contracting officer, the
assumption is that this is the go-ahead for the whole program.
To make well-informed decisions, NASA needs the knowledge to assess the
value of its programs--like the JWST program--in relationship to each
other. In May 2004, we reported that, of 27 NASA programs we examined,
17 had cost increases averaging about 31percent.[Footnote 20] One of
the programs in our sample was another infrared telescope program--the
Spitzer Space Telescope--and it was plagued by schedule slippages
caused by delays in the delivery of components, flight software, the
mission operation system, and launch delays, all contributing to a 29.3
percent increase in program costs. In general, we found the programs in
the sample lacked sufficient knowledge needed to make informed
acquisition decisions. Insufficient knowledge to make informed
investment decisions can further complicate the already-difficult
choices that NASA faces. Conversely, sufficient knowledge at key
junctures can facilitate well-informed investment decisions and protect
the government from incurring contractual liabilities before it is
appropriate. A knowledge-based approach ensures that comprehensive and
comparable programmatic data are obtained.
Conclusions:
Within the JWST program, NASA officials have accomplished a great deal,
such as the development of the large, segmented mirror that is a leap
ahead in technology. Moreover, the program has support from the larger
scientific community. To enhance the program's chances for success,
program officials have chosen a path forward which follows NASA's
policies for ensuring readiness to proceed into implementation/product
development. However, the JWST program's revised strategy does not
fully address the risks associated with the many challenges that the
program still faces--including maturing technology, mitigating testing
restrictions, and ensuring that adequate funding is available for
contingencies. This puts the program at risk of further cost growth and
schedule slippage. The program needs to have sufficient knowledge at
key junctures to successfully address its challenges and use
incremental markers to make certain that resources in terms of
knowledge, time, workforce, and money match the requirements. Given the
severity of the fiscal challenges our nation faces and the wide range
of competing federal programs, hard choices need to be considered
across the government, and NASA is no exception. Using a knowledge-
based approach for NASA's new development programs such as the JWST
could help the agency make the difficult choices about how to allocate
its limited budget resources among competing priorities by utilizing
common and consistent criteria in program evaluations.
Recommendations for Executive Action:
To increase the JWST program's chances of successful product
development, we recommend that the NASA Administrator take the
following actions:
² Direct the JWST program to fully apply a knowledge-based acquisition
approach--to include incremental markers--that will not only ensure
that adequate knowledge is attained at key decision points, but also
hold the program accountable. These markers should include, but not be
limited to:
* schedules that demonstrate the maturity of all critical technologies
prior to program start;
* criteria to ensure the validity of test articles;
* criteria to demonstrate that mature component designs being used in
new configurations meet form, fit, and function standards; and:
* criteria to ensure that sufficient contingency funding can be
provided and phased appropriately.
² Instruct the JWST program to continue to adhere to NASA acquisition
policy and base the program's go/no-go review (NAR) decision not only
on adherence to that policy, but also on (1) the program's ability to
demonstrate whether it is meeting the knowledge markers outlined
earlier and (2) whether adequate funds are available to execute the
program.
Agency Comments and Our Evaluation:
In written comments on a draft of this report, NASA concurred with our
two recommendations and outlined actions that the agency plans to take
to implement such recommendations. NASA said that it endorses the
knowledge-based approach recommended and that it believes the current
JWST program plan is consistent with that approach. NASA's recognition
of the value of obtaining knowledge prior to moving to subsequent
acquisition phases and acknowledgment that it plans to use exit
criteria as knowledge markers for other JWST mission-level reviews are
welcome steps toward establishing an agency-wide risk reduction
culture. Now, it will be critical for NASA decision makers to enforce
adherence to the discipline of the knowledge-based approach and ensure
that critical product knowledge is indeed demonstrated before allowing
the JWST program to proceed. In the years ahead, NASA decision makers
will likely face pressures to grant waivers for going forward with
immature technologies, allow programs to be restructured, and thus
marginalize accountability. For a program such as the JWST, whose
investment is already substantial and successful outcome eagerly
anticipated by the science community, adherence to such knowledge-based
principles will need to be strictly enforced. As identified in this
report, NASA would be well served by applying its own technology
readiness standards (reprinted in appendix II) as part of its exit
criteria, and demonstrating that critical technologies are at the TRL 6
level prior to program start using a representative model or prototype-
-which is very close to the actual system in form, fit, and function--
in a relevant environment. Emphasis by decision makers on the
application of "form, fit, and function standards" and "validity of
test articles" as exit criteria for the JWST program start and entry
into Phase C will help address our concern that the current JWST
technology development plan may not result in the appropriate
demonstration of technology maturity prior to program start. NASA's
comments are reprinted in appendix III.
We are sending copies of this report to interested congressional
committees and to the NASA Administrator. We will make copies available
to others upon request. In addition, the report will be available at no
charge on the GAO Web site at [Hyperlink, http://www.gao.gov].
If you or your staff have any questions concerning this report, please
contact me at (202) 512-4841 or lia@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. Key contributors to this report are
acknowledged in appendix IV.
Signed by:
Allen Li:
Director Acquisition and Sourcing Management:
List of Congressional Committees:
The Honorable Kay Bailey Hutchison:
Chairman:
The Honorable Bill Nelson:
Ranking Minority Member:
Subcommittee on Science and Space:
Committee on Commerce, Science, and Transportation:
United State Senate:
The Honorable Richard C. Shelby:
Chairman:
The Honorable Barbara A. Mikulski:
Ranking Minority Member:
Subcommittee on Commerce, Justice, Science, and Related Agencies:
Committee on Appropriations:
United States Senate:
The Honorable Sherwood L. Boehlert:
Chairman:
The Honorable Bart Gordon:
Ranking Minority Member:
Committee on Science:
House of Representatives:
The Honorable Ken Calvert:
Chairman:
The Honorable Mark Udall:
Ranking Minority Member:
Subcommittee on Space and Aeronautics:
Committee on Science:
House of Representatives:
The Honorable Frank R. Wolf:
Chairman:
The Honorable Alan B. Mollohan:
Ranking Minority Member:
Subcommittee on Science, the Departments of State, Justice, and
Commerce, and, Related Agencies:
Committee on Appropriations:
House of Representatives:
[End of section]
Appendix I: Scope and Methodology:
To assess the extent to which the JWST acquisition strategy follows
NASA policy and GAO best practices for ensuring readiness to proceed
into implementation, we reviewed NASA policy on program management and
compared the JWST project office's management approach to NASA policy.
Additionally, we analyzed the JWST acquisition strategy and benchmarked
it to best practices. We interviewed NASA and contractor officials to
clarify our understanding of the JWST management approach and
technology development plan in relation to NASA policy and guidelines
and best practices. To deepen our understanding of JWST technical
issues, we attended the 3-day Sunshield Subsystem Concept Design Review
as well as the 4-day JWST System Definition Review.
To evaluate the impact of the JWST acquisition strategy on NASA's
ability to assess the program and make informed investment decisions in
the context of its other priorities, we analyzed available JWST cost
and schedule data and conducted interviews with program officials to
clarify our understanding of the information. Furthermore, we requested
and reviewed documentary support breaking out the components of the
cost increases and schedule slippage. We also interviewed program
officials to clarify our understanding of the potential impact that
investment in the JWST will have on other NASA programs. In addition,
we reviewed statements of the NASA Administrator, budget documents,
GAO's High-Risk Series, and GAO's 21st Century Challenges to better
evaluate the JWST's significance in the larger NASA and federal
government context.
To accomplish our work, we visited NASA Headquarters, Washington, D.C;
Goddard Space Flight Center, Greenbelt, Maryland; Marshall Space Flight
Center, Huntsville, Alabama; Northrop Grumman Space Technology, Redondo
Beach, California; and Ball Aerospace and Technologies Corporation,
Boulder, Colorado.
We performed our review from August 2005 through May 2006 in accordance
with generally accepted government auditing standards.
[End of section]
Appendix II: Technology Readiness Levels:
Technology Readiness Level: TRL 1: Basic principles observed and
reported;
Description: Lowest level of technology readiness. Scientific research
begins to be translated into applied research and development. Examples
might include paper studies of a technology's basic properties;
Hardware Software: None. (Paper studies and analysis.);
Demonstration Environment: None.
Technology Readiness Level: TRL 2: Technology concept and/or
application formulated;
Description: Invention begins. Once basic principles are observed,
practical applications can be invented. The application is speculative
and there is no proof or detailed analysis to support the assumption.
Examples are still limited to paper studies;
Hardware Software: None. (Paper studies and analysis.);
Demonstration Environment: None.
Technology Readiness Level: TRL 3: Analytical and experimental critical
function and/or characteristic proof of concept;
Description: Active research and development is initiated. This
includes analytical studies and laboratory studies to physically
validate analytical predictions of separate elements of the technology.
Examples include components that are not yet integrated or
representative;
Hardware Software: Analytical studies and demonstration of nonscale
individual components (pieces of subsystem);
Demonstration Environment: Lab.
Technology Readiness Level: TRL 4: Component and/or breadboard.
Validation in laboratory environment;
Description: Basic technological components are integrated to establish
that the pieces will work together. This is relatively "low fidelity"
compared to the eventual system. Examples include integration of "ad
hoc" hardware in a laboratory;
Hardware Software: Low fidelity breadboard. Integration of nonscale
components to show pieces will work together. Not fully functional or
form or fit but representative of technically feasible approach
suitable for flight articles;
Demonstration Environment: Lab.
Technology Readiness Level: TRL 5: Component and/or breadboard
validation in relevant environment;
Description: Fidelity of breadboard technology increases significantly.
The basic technological components are integrated with reasonably
realistic supporting elements so that the technology can be tested in a
simulated environment. Examples include "high fidelity" laboratory
Integration of components;
Hardware Software: High fidelity breadboard. Functionally equivalent
but not necessarily form and/or fit (size weight, materials, etc.)
Should be approaching appropriate scale. May include integration of
several components with reasonably realistic support elements/
subsystems to demonstrate functionality;
Demonstration Environment: Lab demonstrating functionality but not form
and fit. May include flight demonstrating breadboard in surrogate
aircraft. Technology ready for detailed design studies.
Technology Readiness Level: TRL 6: System/subsystem model or prototype
demonstration in a relevant environment;
Description: Representative model or prototype system, which is well
beyond the breadboard tested for TRL 5, is tested in a relevant
environment. Represents a major step up in a technology's demonstrated
readiness. Examples include testing a prototype in a high fidelity
laboratory environment or in simulated operational environment;
Hardware Software: Prototype--Should be very close to form, fit and
function. Probably includes the integration of many new components and
realistic supporting elements/subsystems if needed to demonstrate full
functionality of the subsystem;
Demonstration Environment: High-fidelity lab demonstration or limited/
restricted flight demonstration for a relevant environment. Integration
of technology is well defined.
Technology Readiness Level: TRL 7: System prototype demonstration in an
operational environment;
Description: Prototype near or at planned operational system.
Represents a major step up from TRL 6, requiring the demonstration of
an actual system prototype in an operational environment, such as in an
aircraft, vehicle or space. Examples include testing the prototype in a
test bed aircraft;
Hardware Software: Prototype. Should be form, fit and function
integrated with other key supporting elements/subsystems to demonstrate
full functionality of subsystem;
Demonstration Environment: Flight demonstration in representative
operational environment such as flying test bed or demonstrator
aircraft. Technology is well substantiated with test data.
Technology Readiness Level: TRL 8: Actual system completed and "flight
qualified" through test and demonstration;
Description: Technology has been proven to work in its final form and
under expected conditions. In almost all cases, this TRL represents the
end of true system development. Examples include developmental test and
evaluation of the system in its intended weapon system to determine if
it meets design specifications;
Hardware Software: Flight qualified hardware;
Demonstration Environment: Developmental test and evaluation in the
actual system application.
Technology Readiness Level: TRL 9: Actual system "flight proven"
through successful mission operations;
Description: Actual application of the technology in its final form and
under mission conditions, such as those encountered in operational test
and evaluation. In almost all cases, this is the end of the last "bug
fixing" aspects of true system development. Examples include using the
system under operational mission conditions;
Hardware Software: Actual system in final form;
Demonstration Environment: Operational test and evaluation in
operational mission conditions.
Source: GAO and its analysis of NASA data.
[End of table]
[End of section]
Appendix III: Comments from the National Aeronautics and Space
Administration:
National Aeronautics and Space Administration:
Office of the Administrator:
Washington, DC 20546-0001:
June 26, 2006:
Mr. Allen Li:
Director, Acquisition and Sourcing Management:
United States Government Accountability Office:
Washington, DC 20548:
Dear Mr. Li:
NASA appreciates the opportunity to comment on your draft report
General Accountability Office (GAO) GAO-06-634 entitled "Knowledge-
Based Acquisition Approach Key to Addressing Program Challenges," which
pertains to the James Webb Space Telescope (JWST) program. NASA
endorses the knowledge-based acquisition approach recommended by the
GAO. In part due to earlier GAO recommendations and NASA management
changes, NASA believes the current JWST program plan is consistent with
a knowledge-based approach and that the appropriate maturity of JWST
technologies will be demonstrated well in advance of an Agency decision
to proceed into Phase C.
The draft report references an earlier GAO report (GAO-06-218),
entitled "Implementing a Knowledge-Based Acquisition Framework Could
Lead to Better Investment Decisions and Project Outcomes," in which the
GAO recommended that NASA take steps to ensure that NASA projects
follow a knowledge-based approach for product development. In a letter
to GAO dated December 15, 2005, NASA agreed with the GAO
recommendations, noting that while NASA was already employing many of
the recommended practices, some of those practices were not apparent in
existing NASA acquisition policy documents. NASA is currently in the
process of revising relevant NASA acquisition policy documents in
accordance with the commitments made in the letter to the GAO.
In the current draft report, GAO recommends that the NASA Administrator
take the following actions:
Recommendation 1* Direct the JWST program to apply a knowledge-based
acquisition approach to include incremental markers-that will not only
ensure that adequate knowledge is attained at key decision points, but
also hold the program accountable. These markers should include, but
not be limited to:
* schedules that demonstrate the maturity of all critical technologies
prior to program start;
* criteria to ensure the validity of test articles;
* criteria to demonstrate that mature component designs being used in
new configurations meet form, fit, and function standards; and:
* criteria to ensure that sufficient contingency funding can be
provided and phased appropriately.
Concur - NASA concurs with this recommendation. The recently replanned
JWST program includes a set of mission-level reviews that exceed the
minimum set of reviews required by NASA Procedural Requirements 7123.
Explicit exit criteria (including the criteria listed in this GAO
report recommendation), are developed for each mission-level review to
serve as incremental knowledge markers to ensure that adequate
knowledge has been attained before proceeding to the next mission
phase. Major JWST mission-level reviews include:
* Technology Non-Advocate Review (T-NAR) planned for January 2007:
* Preliminary Design Review (PDR) planned for March 2008
* Non-Advocate Review (NAR) planned for March 2008 Critical Design
Review (CDR) planned for July 2009
* Test Readiness Review:
A successful PDR/NAR will be required for Agency approval to proceed
into Phase C, and a successful CDR will be required for Agency approval
to proceed to Phase D.
Recommendation 2: Instruct the JWST program to continue to adhere to
NASA acquisition policy and base the program's go/no-go review (NAR)
decision not only on adherence to that policy, but also on (1) the
program's ability to demonstrate whether it is meeting the knowledge
markers outlined earlier and (2) whether adequate funds are available
to execute the program.
Concur - NASA concurs with this recommendation. NASA will employ the
monthly JWST program status reporting processes, the annual budget
planning processes, and the mission-level reviews (listed above) to
confirm that the JWST program continues to adhere to NASA acquisition
policy. NASA will also ensure that Agency approval to proceed to Phase
C will be based on the program's ability to demonstrate that it is
meeting the appropriate knowledge markers, as well as on whether
adequate funds are available to execute the program.
Thank you for the opportunity to respond to this draft report.
Signed by:
Shana Dale:
Deputy Administrator:
[End of section]
Appendix IV: GAO Contact and Staff Acknowledgments:
GAO Contact:
Allen Li (202) 512-4841:
Staff Acknowledgments:
In addition to the individual named above, Jim Morrison, Assistant
Director; Greg Campbell; Keith Rhodes; Sylvia Schatz; Erin Schoening;
Hai Tran; and Ruthie Williamson made key contributions to this report.
[End of section]
Related GAO Products:
NASA Reports:
NASA: Implementing a Knowledge-Based Acquisition Framework Could Lead
to Better Investment Decisions and Project Outcomes. GAO-06-218.
Washington, D.C.: December 21, 2005.
NASA's Space Vision: Business Case for Prometheus 1 Needed to Ensure
Requirements Match Available Resources. GAO-05-242. Washington, D.C.:
February 28, 2005.
Space Reports:
Space Acquisitions: Stronger Development Practices and Investment
Planning Need to Address Continuing Problems. GAO-05-891T. Washington,
D.C.: July 12, 2005.
Defense Acquisitions: Incentives and Pressures That Drive Problems
Affecting Satellite and Related Acquisitions. GAO-05-570R. Washington,
D.C.: June 23, 2005.
Defense Acquisitions: Space-Based Radar Effort Needs Additional
Knowledge before Starting Development. GAO-04-759. Washington, D.C.:
July 23, 2004.
Defense Acquisitions: Risks Posed by DOD's New Space Systems
Acquisition Policy. GAO-04-379R. Washington, D.C.: January 29, 2004.
Space Acquisitions: Committing Prematurely to the Transformational
Satellite Program Elevates Risks for Poor Cost, Schedule, and
Performance Outcomes. GAO-04-71R. Washington, D.C.: December 4, 2003.
Defense Acquisitions: Improvements Needed in Space Systems Acquisition
Policy to Optimize Growing Investment in Space. GAO-04-253T.
Washington, D.C.: November 18, 2003.
Defense Acquisitions: Despite Restructuring, SBIRS High Program Remains
at Risk of Cost and Schedule Overruns. GAO-04-48. Washington, D.C.:
October 31, 2003.
Defense Acquisitions: Improvements Needed in Space Systems Acquisition
Management Policy. GAO-03-1073. Washington, D.C.: September 15, 2003.
Military Space Operations: Common Problems and Their Effects on
Satellite and Related Acquisitions. GAO-03-825R. Washington, D.C.: June
2, 2003.
Military Space Operations: Planning, Funding, and Acquisition
Challenges Facing Efforts to Strengthen Space Control. GAO-02-738.
Washington, D.C.: September 23, 2002.
Polar-Orbiting Environmental Satellites: Status, Plans, and Future Data
Management Challenges. GAO-02-684T. Washington, D.C.: July 24, 2002.
Defense Acquisitions: Space-Based Infrared System-Low at Risk of
Missing Initial Deployment Date. GAO-01-6. Washington, D.C.: February
28, 2001.
Best Practices Reports:
Defense Acquisitions: Assessments of Selected Major Weapon Programs.
GAO-05-301. Washington, D.C.: March 31, 2005.
Defense Acquisitions: Stronger Management Practices Are Needed to
Improve DOD's Software-Intensive Weapon Acquisitions. GAO-04-393.
Washington, D.C.: March 1, 2004.
Defense Acquisitions: Assessments of Selected Major Weapon Programs.
GAO-04-248. Washington, D.C.: March 31, 2004.
Defense Acquisitions: DOD's Revised Policy Emphasizes Best Practices,
but More Controls Are Needed. GAO-04-53. Washington, D.C.: November 10,
2003.
Defense Acquisitions: Assessments of Selected Major Weapon Programs.
GAO-03-476. Washington, D.C.: May 15, 2003.
Best Practices: Setting Requirements Differently Could Reduce Weapon
Systems' Total Ownership Costs. GAO-03-57. Washington, D.C.: February
11, 2003.
Best Practices: Capturing Design and Manufacturing Knowledge Early
Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15,
2002.
Defense Acquisitions: DOD Faces Challenges in Implementing Best
Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.
Best Practices: Better Matching of Needs and Resources Will Lead to
Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8,
2001.
Best Practices: A More Constructive Test Approach Is Key to Better
Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31,
2000.
Defense Acquisition: Employing Best Practices Can Shape Better Weapon
System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000.
Best Practices: DOD Training Can Do More to Help Weapon System Program
Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August
16, 1999.
Best Practices: Better Management of Technology Development Can Improve
Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30,
1999.
Defense Acquisitions: Best Commercial Practices Can Improve Program
Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999.
Defense Acquisition: Improved Program Outcomes Are Possible. GAO/T-
NSIAD-98-123. Washington, D.C.: March 18, 1998.
Best Practices: Successful Application to Weapon Acquisition Requires
Changes in DOD's Environment. GAO/NSIAD-98-56. Washington, D.C.:
February 24, 1998.
Major Acquisitions: Significant Changes Underway in DOD's Earned Value
Management Process. GAO/NSIAD-97-108. Washington, D.C.: May 5, 1997.
Best Practices: Commercial Quality Assurance Practices Offer
Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26,
1996.
FOOTNOTES
[1] The JWST is a one-project program, according to a NASA official.
The terms "program" and "project" are used interchangeably throughout
this report.
[2] GAO, NASA: Implementing a Knowledge-Based Acquisition Framework
Could Lead to Better Investment Decisions and Project Outcomes, GAO-06-
218 (Washington, D.C.: Dec. 21, 2005).
[3] Infrared radiation is one of the many types of "light" that
comprise the electromagnetic spectrum. Infrared light is situated
outside of the visible spectrum and has wavelengths longer than visible
light. Astronomers generally divide the infrared portion of the
electromagnetic spectrum into three regions: near infrared, mid
infrared, and far infrared. The JWST will be sensitive to near-infrared
and mid-infrared radiation.
[4] A spectrograph is an instrument for dispersing radiation (as
electromagnetic radiation or sound waves) into a spectrum and
photographing or mapping the spectrum.
[5] "Test as you fly" means performing the final performance and
environmental test with the spacecraft fully integrated in the same
configuration that it will be in when it launches, according an agency
official.
[6] Some budget cuts were restored after the Independent Review Team's
assessment, increasing this amount to about 3 percent.
[7] A "lien" is a potential cost to a project, direct or indirect,
which may or may not come to fruition, for which a portion of funding
reserves is set aside. According to a JWST project official, "threats"
are things that concern a project or engineer, which may or may not
come true, but which bear watching to see if they have validity;
however, they do not require the same rigor as "liens."
[8] According to a member of the Independent Review Team, "threats"
were included in the analysis because after examining the project
office's "threat" list, the team concluded that the "threats" had a
high probability of occurring and were therefore more like "liens."
[9] The NAR--a program/project milestone review prescribed by NASA
Procedural Requirements 7120.5C--is intended to provide NASA management
with an independent assessment of a program's readiness to move into
implementation and the final design phase.
[10] Our best practice reviews are identified in the "Related GAO
Products" section at the end of this report.
[11] GAO-06-218.
[12] TRLs characterize the readiness of technologies for hand-off to
project implementers. Nine levels are defined representing concepts
from fundamental research level through technologies fully qualified
and demonstrated in flight.
[13] NASA Procedural Requirements 7120.5C, which states that its
requirements are applicable to all programs and projects currently in
formulation as of the effective date of March 22, 2005.
[14] The pre-NAR is an independent review of programs/projects
conducted at the end of the concept development phase to assess
readiness to proceed into the preliminary design phase.
[15] The PDR is the project milestone review which establishes the
basis for proceeding with a detailed design. The purpose of the PDR is
to demonstrate that the preliminary design meets all system
requirements with an acceptable level of risk within the planned cost
and schedule.
[16] According to the mission systems engineer, the half-scale thermal
vacuum test will be done using a half-scale model of the entire
observatory. Deployments, including the sunshield, will be tested, and
the sunshield membrane will be vibrated during the test.
[17] The purpose of dynamic testing is to determine how the sunshield
behaves structurally when shaken at different frequencies in order to
predict the influence of disturbances on the pointing control of the
JWST's optics.
[18] The main components of the 1/10th scale model test article were a
central mounting block, four support tubes, and four Kapton film
layers. Therefore, the 1/10th scale model was not a scale version of
the current JWST sunshield, which consists of five layers of Kapton
membranes with special coatings, booms, hinges, deployment motors, edge
cables, stowed boom restraints, stowed membrane containment structure,
and other mechanisms.
[19] The Agency Program Management Council is one of a system of
Governing Program Management Councils responsible for assessing program
and project formulation and implementation as well as providing
oversight and direction.
[20] GAO, NASA: Lack of Disciplined Cost-Estimating Processes Hinders
Effective Program Management, GAO-04-642 (Washington, D.C.: May 28,
2004).
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
