NASA
Assessments of Selected Large-Scale Projects Gao ID: GAO-09-306SP March 2, 2009The National Aeronautics and Space Administration (NASA) plans to invest billions in the coming years in science and exploration space fl ight initiatives. The scientifi c and technical complexities inherent in NASA's mission create great challenges in managing its projects and controlling costs. In the past, NASA has had diffi culty meeting cost, schedule, and performance objectives for some of its projects. The need to effectively manage projects will gain even more importance as NASA seeks to manage its wide-ranging portfolio in an increasingly constrained fiscal environment. Per congressional direction, this report provides an independent assessment of selected NASA projects. In conducting this work, GAO compared projects against best practice criteria for system development including attainment of knowledge on technologies and design as well as various aspects of program management. The projects assessed are considered major acquisitions by NASA--each with a life-cycle cost of over $250 million. No recommendations are provided.
GAO assessed 18 NASA projects with a combined life-cycle cost of more than $50 billion. Of those, 10 out of 13 projects that had entered the implementation phase experienced signifi cant cost and/or schedule growth. For these 10 projects, development costs increased by an average of 13 percent from baseline cost estimates that were established just 2 or 3 years ago and they had an average launch delay of 11-months. In some cases, cost growth was considerably higher than what is reported because it had occurred prior to the most recent baseline. Many of the projects we reviewed experienced challenges in developing new technologies or retrofi tting older technologies as well as in managing their contractors, and more generally, understanding the risks and challenges they were up against when they started their efforts. GAO's previous work has consistently shown that reducing the kinds of problems this assessment identifi es in acquisition programs hinges on developing a sound business case for a project. In essence, this means establishing fi rm requirements, maturing technologies, and assuring other vital resources, such as time and funding, are suffi cient before making long-term commitments to acquisitions. NASA has acted to adopt practices that would ensure programs proceed based on a sound business case and undertaken an array of initiatives aimed at improving program management, cost estimating, and contractor oversight. Continued attention to these efforts should help maximize NASA's acquisition investments.
GAO-09-306SP, NASA: Assessments of Selected Large-Scale Projects
This is the accessible text file for GAO report number GAO-09-306SP
entitled 'NASA: Assessments of Selected Large-Scale Projects' which was
released on March 2, 2009.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Committees:
United States Government Accountability Office:
GAO:
March 2009:
NASA:
Assessments of Selected Large-Scale Projects:
GAO-09-306SP:
GAO Highlights:
Highlights of GAO-09-306, a report to congressional committees.
Why GAO Did This Study:
The National Aeronautics and Space Administration (NASA) plans to
invest billions in the coming years in science and exploration space
flight initiatives. The scientific and technical challenges inherent in
NASA‘s mission create great challenges in managing its projects and
controlling costs. In the past, NASA has had difficulty meeting cost,
schedule, and performance objectives for some of its projects. The need
to effectively manage projects will gain even more importance as NASA
seeks to manage its wide-ranging portfolio in an increasingly
constrained fiscal environment.
In response to the explanatory statement of the House Committee on
Appropriations accompanying the Consolidated Appropriations Act of
2008, this report provides an independent assessment of selected NASA
projects. In conducting this work, GAO compared projects against best
practice criteria for system development including attainment of
knowledge on technologies and design as well as various aspects of
program management. The projects assessed are considered major
acquisitions by NASA”each with a lifecycle cost of over $250 million.
No recommendations are provided.
What GAO Found:
GAO assessed 18 NASA projects in various phases of project development
and found that 10 out of 13 projects that had entered the
implementation phase experienced significant cost and/or schedule
growth. For those projects, development costs increased by an average
of 13 percent from baseline cost estimates that were established just 2
or 3 years ago and they had an average launch delay of 11-months. In
some cases, cost growth was considerably higher than what is reported
because it had occurred prior to the most recent baseline. Many of the
projects we reviewed indicated that they had experienced challenges in
developing new technologies or retrofitting older technologies as well
as in managing their contractors, and more generally, understanding the
risks and challenges they were up against when they started their
efforts.
GAO‘s previous work has consistently shown that reducing the kinds of
problems this assessment identifies in acquisition programs hinges on
developing a sound business case for a project. In essence, this means
establishing firm requirements, maturing technologies, and assuring
other vital resources, such as time and funding are sufficient before
making long-term commitments to acquisitions. NASA has acted to adopt
practices that would ensure programs proceed based on a sound business
case and undertaken an array of initiatives aimed at improving program
management, cost estimating, and contractor oversight. Continued
attention to these efforts should help maximize NASA‘s acquisition
investments.
Figure: Photographs:
[Refer to PDF for image]
Photographs of:
Gamma-ray Large Area Space Telescope; James Webb Space Telescope;
Dawn;
Mars Science Laboratory.
Source: Kennedy Space Center, Cape Canaveral. JWST Project Office;
NASA/JPL/McREL; Background: William K. Hartmann, UCLA: Mars Science
Laboratory.
[End of figure]
To view the full product, including the scope and methodology, click on
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-09-306]. For more
information, contact Cristina Chaplain at (202) 512-4841 or
ChaplainC@gao.gov.
[End of section]
Contents:
Foreword:
Letter:
A Sound Business Case Underpins Successful Acquisition Outcomes:
NASA Has Made Efforts To Improve Its Acquisitions:
Project Assessments:
Our Observations:
Assessments of Individual Projects:
Aquarius:
Ares I Crew Launch Vehicle (CLV):
Dawn:
Gamma-ray Large Area Space Telescope (GLAST):
Glory:
Global Precipitation Measurement (GPM) Mission:
Herschel:
James Webb Space Telescope (JWST):
Kepler:
Landsat Data Continuity Mission (LDCM):
Lunar Reconnaissance Orbiter (LRO):
Mars Science Laboratory (MSL):
NPOESS Preparatory Project (NPP):
Orbiting Carbon Observatory (OCO):
Orion Crew Exploration Vehicle (CEV):
Solar Dynamics Observatory (SDO):
Stratospheric Observatory for Infrared Astronomy (SOFIA):
Wide-field Infrared Survey Explorer (WISE):
Agency Comments and Our Evaluation:
Appendixes:
Appendix I: Comments from the National Aeronautics and Space
Administration:
Appendix II: NASA Life-Cycle For Flight Systems Compared To A Knowledge-
Based Approach:
Appendix III: Objectives, Scope, and Methodology:
Appendix IV: Technology Readiness Levels:
Appendix V: GAO Contact and Staff Acknowledgments:
Table:
Table 1: Assessment of Challenges for NASA Projects in the
Implementation Phase:
Figure:
Figure 1: NASA‘s Life Cycle for Flight Systems Compared to a Knowledge-
Based Approach:
Abbreviations:
AFB: Air Force Base:
AFS: Air Force Station:
AIA: Atmospheric Imaging Assembly:
APS: aerosol polarimetry sensor:
ASI: Argenzia Spaciale Italiana (Italian Space Agency):
CCD: charged-coupled device:
CDR: critical design review:
CERES: Clouds and Earth‘s Radiant Energy System:
CEV: Crew Exploration Vehicle:
CONAE: Comision Nacional de Actividades Espaciales (Space Agency of
Argentina):
CrIS: Cross-track Infrared Sounder:
CSA: Canadian Space Agency:
DCI: data collection instrument:
DLR: German Aerospace Center:
DPR: dual-frequency precipitation radar:
EO-1: Earth Observatory Satellite:
ESA: European Space Agency:
EVE: Extreme Ultraviolet Variability Experiment:
GBM: gamma-ray burst monitor:
GLAST: Gamma-ray Large Area Space Telescope:
GMI: GPM microwave imager:
GPM: Global Precipitation Measurement (mission):
HIFI: Heterodyne Instrument for the Far Infrared:
HMI: Helioseismic and Magnetic Imager:
IPO: Integrated Program Office:
JAXA: Japan Aerospace Exploration Agency:
JPL: Jet Propulsion Laboratory:
KDP: key decision point:
LAT: large area telescope:
LCROSS: Lunar Crater Observation and Sensing Satellite:
LDCM: Landsat Data Continuity Mission:
LRO: Lunar Reconnaisance Orbiter:
MEP: Mars Exploration Program:
MSL: Mars Science Laboratory:
NAR: nonadvocate review:
NASA: National Aeronautics and Space Administration:
NIR: near infrared (bands):
NPR: NASA Procedural Requirements:
NPOESS: National Polar-Orbiting Operational Environmental Satellite
System:
NPP: NPOESS Preparatory Project:
OCO: Orbiting Carbon Observatory:
OLI: Operational Land Imager:
PA&E: Office of Program Analysis and Evaluation (NASA):
PDR: preliminary design review:
PICA: phenolic impregnated carbon ablator:
SDO: Solar Dynamics Observatory:
SDR: system definition review:
SOFIA: Stratospheric Observatory for Infrared Astronomy:
SSS: sea surface salinity:
TIM: total irradiance monitor:
TIRS: Thermal Infrared Sensor:
TRL: technology readiness level:
TSIS: Total Solar Irradiance Sensor:
USGS: U.S. Geological Service:
VIIRS: Visible Infrared Imaging Radiometer Suite:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
March 2, 2009:
Congressional Committees:
In response to the explanatory statement of the House Committee on
Appropriations accompanying the Consolidated Appropriations Act of
2008, this report provides our assessment of large-scale NASA projects.
NASA is at a critical juncture. The agency is in the midst of phasing
out the Space Shuttle program and beginning another major undertaking,
the Constellation program--which will create the next generation of
spacecraft for human spaceflight and is expected to cost upward of $230
billion. This massive effort, unparalleled since the transition from
the Apollo program to the Shuttle program, presents the agency with
myriad complex and interdependent challenges.
NASA is taking on this endeavor against a backdrop of growing national
government fiscal imbalance and budget deficits that continue to strain
all our federal agencies' resources. While NASA's budget represents
less than 2 percent of the nation's fiscal discretionary budget, the
agency is increasingly being asked to expand its portfolio to support
important scientific missions including the study of climate change.
Therefore, it is exceedingly important that these resources be managed
as effectively and efficiently as possible.
In the past, this has not always been the case. NASA has had difficulty
meeting cost, schedule, and performance objectives for some of its
projects, and in fact, it had to cancel prior attempts to replace the
space shuttle, after billions had already been spent, in the face of
cost overruns and program management problems. However, to its credit,
NASA has developed a comprehensive plan to address systemic acquisition
management weaknesses and it is in the initial stages of implementing
the plan. Moreover, as we have urged it to do, NASA recently
incorporated best practice criteria for system development in its
acquisition policy, though our review shows more needs to be done to
ensure the policy is followed. To maximize NASA's ability to invest in
science and space exploration, senior leaders should focus attention to
adopting best practices and demonstrate a willingness to fix and/or
terminate projects that are not performing well. This assessment should
support such efforts.
Signed by:
Gene L. Dodaro:
Acting Comptroller General of the United States:
[End of letter]
United States Government Accountability Office:
Washington, DC 20548:
March 2, 2009:
Congressional Committees:
The National Aeronautics and Space Administration's (NASA) extensive
portfolio of missions ranges from sending robotic vehicles to Mars, to
scientific study of Earth from space, to assembling and supplying the
International Space Station. Some of these missions, such as the Hubble
Space Telescope and NASA's earth science efforts, have literally
changed the way we view our planet and the universe. The technology
that NASA developed has resulted in numerous spin-off products that are
used across a wide range of technical and commercial fields.
However, NASA has also had its share of challenges. For example, the X-
33 and X-34 programs, which were meant to demonstrate technology for
future reusable launch vehicles, were canceled due to technical
difficulties and cost overruns after NASA spent more than $1 billion on
them. More recently, the Mars Science Laboratory, which was already
over budget, announced a two-year launch delay. Current estimates
suggest the price of this delay may be $400 million--which drives the
current project life-cycle cost estimate to $2.3 billion; up from its
initial confirmation estimate of $1.6 billion. GAO and others have also
reported on overruns on many other NASA programs over the past decade.
What is common among these and other programs is that whether they
succeed or fail, they cost more to build and take longer to launch than
planned. As a result, NASA is able to accomplish less than its plans
with the money it is allocated, and it is forced to make unplanned
trade-offs among its projects--shorting one to pay for the mistakes of
another.
Congress has expressed concern about NASA's performance and has
identified the need to standardize the reporting of cost, schedule, and
content for NASA research and development projects. In 2005, Congress
required NASA to report cost and schedule baselines--benchmarks against
which changes can be measured--for all NASA programs and projects with
estimated life-cycle costs of at least $250 million that have been
approved to proceed to implementation.[Footnote 1] It also required
that NASA report to Congress when development cost is likely to exceed
the baseline estimate by 15 percent or more, or when a milestone is
likely to be delayed by six months or more.[Footnote 2] In response,
NASA began establishing cost and schedule baselines in 2006 and has
been using them as the basis for annual project performance reports for
Congress provided in its annual budget submission each year.
The explanatory statement of the House Committee on Appropriations
accompanying the Consolidated Appropriations Act of 2008 directed GAO
to prepare project status reports on selected large-scale NASA
programs, projects, or activities. This report responds to that mandate
by assessing 18 NASA projects, each with a life-cycle cost over $250
million. The combined life-cycle cost for these 18 projects exceeds $50
billion. Each assessment is presented in a two-page summary that
analyzes the project's cost and schedule status and project challenges.
We also provide general observations about the performance of NASA's
major projects and the agency's management of those projects during
development.
NASA provided updated cost and schedule data as of December 2008 for 13
of the 18 projects.[Footnote 3] We reviewed and compared that data to
previously established baselines for each of those 13 projects. We took
appropriate steps to address data reliability.
Our approach included an examination of the phase of a project's
development and how each project was advancing within this framework.
Each project we reviewed was in either the formulation phase or the
implementation phase of the project life cycle. In the formulation
phase, the project develops and defines the project requirements--what
the project should be able to do--establishes a schedule, estimates
costs and produces a plan for implementation. In the implementation
phase, the project carries out these plans, performing final design and
fabrication as well as testing components and system assembly,
integrating these components and testing how they work together, and
launching the project. This phase also includes the period from project
launch through mission completion. We assessed each project's cost and
schedule and characterized growth in either as significant if it was
greater than the thresholds established for Congressional reporting.
Based on discussion with project officials and drawing on GAO's
established criteria for knowledge-based acquisitions[Footnote 4] and
on other GAO work on space and weapon system acquisitions, we
identified five challenges that can contribute to cost and schedule
growth in these projects: technology maturity, design stability,
complexity of heritage technology, contractor performance and
development partner performance. To assess technology maturity, we
examined the project's reported critical technology readiness levels--
a measure that NASA devised and that is now used at other agencies as
well. We looked at the technology readiness level at the time of the
project's preliminary design review, which occurs just before it enters
the implementation phase, and compared that against the level of
maturity that best practices call for at that stage to minimize risks.
To assess design stability, we examined the percentage of engineering
drawings completed or projected to be completed by the critical design
review--which is usually held about mid-way through the project's
development. We asked project officials to provide this information and
we compared it against GAO's best practices' metric of 90 percent of
drawings released by the critical design review. Finally, based in part
on our discussions with officials for the individual projects, we
identified the extent to which project cost and schedule were
negatively impacted by challenges integrating heritage--or pre-
existing--technology into their projects. We also discussed the extent
to which contractors' and development partners' challenges in
developing and delivering project hardware impacted overall project
cost and schedule.
In this review, these challenges were largely apparent in the projects
that had entered the implementation phase.
This list of challenges is not exhaustive; nor do we believe these
challenges will not change or evolve as we continue this work into the
future. Our objectives are to expand on the importance of developing a
knowledge-based acquisition strategy and to provide decision-makers
with an independent, knowledge-based assessment of individual systems
that identifies potential risks and allows them to take actions to put
projects that are early in the development cycle in a better position
to succeed. This report and the challenges we discuss in it are a
starting point for our future work in this area. The individual project
offices were given an opportunity to comment on and provide technical
clarifications on our assessments prior to their inclusion in the final
product.
We conducted this performance audit from February 2008 to March 2009 in
accordance with generally accepted government auditing standards. Those
standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives. Appendix II contains
detailed information on our scope and methodology. We do not provide
recommendations in this report.
A Sound Business Case Underpins Successful Acquisition Outcomes:
The major projects that NASA undertakes range from highly complex and
sophisticated space transportation vehicles, to robotic probes, to
satellites equipped with advanced sensors to study the earth. In many
cases, NASA's projects are expected to incorporate new and
sophisticated technologies while operating in harsh, distant
environments. Many of its projects are also one time articles, meaning
that there is little opportunity to apply knowledge gained to the
production of a second, third, or future increments of space craft.
Moreover, NASA often partners with other space-faring countries,
including several European nations, Japan, and Argentina. These
partnerships go a long way to foster international cooperation in
space, but they also put NASA projects in a vulnerable position when
partners do not meet their obligations or run into technical obstacles
they cannot easily overcome. While space development programs are
complex and difficult by nature, and most are one-time efforts, we are
convinced that NASA would benefit from a more disciplined approach to
its acquisitions. The nature of its work should not preclude NASA from
achieving what it promises when requesting and receiving funds.
The development and execution of a knowledge-based business case for
these projects can provide early recognition of challenges, allow
managers to take corrective action, and place needed and justifiable
projects in a better position to succeed. Our studies of best practice
organizations shows the risks inherent in NASA's work can be mitigated
by developing a solid, executable business case before committing
resources to a new product development.[Footnote 4] In its simplest
form, this is evidence that (1) the customer's needs are valid and can
best be met with the chosen concept, and (2) the chosen concept can be
developed and produced within existing resources--that is, proven
technologies, design knowledge, adequate funding, and adequate time to
deliver the product when needed. A program should not go forward into
product development unless a sound business case can be made. If the
business case measures up, the organization commits to the development
of the product, including making the financial investment. Our best
practice work has shown that developing business cases based on
matching requirements to resources before program start leads to more
predictable program outcomes--that is, programs are more likely to be
successfully completed within cost and schedule estimates and deliver
anticipated system performance.[Footnote 5]
At the heart of a business case is a knowledge-based approach to
product development that is a best practice among leading commercial
firms. Those firms have created an environment and adopted practices
that put their program managers in a good position to succeed in
meeting these expectations. For a program to deliver a successful
product within available resources, managers should demonstrate high
levels of knowledge before significant commitments are made. In
essence, knowledge supplants risk over time. This building of knowledge
can be described over the course of a program, as follows:
* When a project begins development, the customer's needs should match
the developer's available resources--mature technologies, time, and
funding. An indication of this match is the demonstrated maturity of
the technologies needed to meet customer needs--referred to as critical
technologies. If the project is relying on heritage--or pre-existing--
technology, that technology must be in appropriate form, fit, and
function to address the customer's needs within available resources.
* Then, about midway through the product's development, it should be
stable and demonstrate it is capable of meeting performance
requirements. The critical design review takes place at that point of
time because it generally signifies when the program is ready to start
building production-representative prototypes. If design stability is
not achieved, but a product development continues, costly re-designs to
address changes to project requirements and unforeseen challenges can
occur.
* Finally, by the time of the production decision, the product must be
shown to be producible within cost, schedule, and quality targets and
have demonstrated its reliability, and the design must demonstrate that
it performs as needed through realistic system-level testing. Lack of
testing increases the possibility that project managers will not have
information that could help avoid costly system failures in late stages
of development or during system operations.
Our best practice work has identified numerous other actions that can
be taken to increase the likelihood that a program can be successfully
executed once that business case is established. These include ensuring
cost estimates are complete, accurate and updated regularly; holding
suppliers accountable to deliver high-quality parts for their product
through such activities as regular supplier audits and performance
evaluations of quality and delivery; and holding program managers
accountable for their choices. Moreover, we have recommended using
metrics and controls throughout the life-cycle to gauge when the
requisite level of knowledge has been attained and direct decision
makers to consider criteria before advancing a program to the next
level and making additional investments.
The consequence of proceeding with system development without
establishing and adhering to a sound business case is substantial. GAO
and others have reported that NASA has experienced cost and schedule
growth in several of its projects over the past decade, resulting from
problems that include failing to adequately identify requirements and
underestimating complexity and technology maturity. For example, the X-
33 and X-34 programs both were terminated because of significant cost
increases caused by problems developing the necessary technologies and
flight demonstration vehicles. Neither program fully assessed the costs
associated with developing new, unproven technologies. Additionally, in
2005, GAO reported on the lack of an established sound business case
for NASA's Prometheus I--a project that faced challenges in identifying
preliminary requirements, establishing firm cost estimates and maturing
critical technologies. After concurring with GAO's recommendation that
NASA establish a firm business case for the project, NASA identified
more realistic requirements for Prometheus I and reduced the project's
requested funding by nearly $2.4 billion through 2010.
NASA Has Made Efforts To Improve Its Acquisitions:
In 2005, we reported that NASA's acquisition policies did not conform
to best practices for product development because they lacked major
decision reviews at several key points in the project life-cycle, which
would allow decision makers to make informed decisions about whether a
project should be authorized to proceed in the development life-cycle.
Based, in part, on our recommendations, NASA issued a revised policy in
March 2007[Footnote 6] that institutes several key decision points
(KDP) in the development life-cycle for space flight programs and
projects. At each KDP, a decision authority is responsible for
authorizing the transition to the next life-cycle phase for the
project.[Footnote 7] In addition, NASA acquisition policies also
require that technologies be sufficiently mature at the preliminary
design review, the design is stable at critical design review, and the
system can be fabricated within cost, schedule and performance
specifications. These changes brought the policy more in line with best
practices for product development. A more detailed discussion of NASA's
acquisition policy and how it relates to best practices is provided in
appendix I of this report.
Further, in response to GAO's designation of NASA acquisition
management as a "high risk" area,[Footnote 8] NASA developed a
corrective action plan to improve the effectiveness of NASA program/
project management.[Footnote 9] The approach focuses on how best to
ensure the mitigation of potential issues in acquisition decisions and
better monitor contractor performance. The plan identifies five areas
for improvement--program/project management, cost reporting process,
cost estimating and analysis, standard business processes, and
management of financial management systems--each of which contain
targets and goals to measure improvement. As part of this initiative,
NASA has taken a positive step in improving management oversight of
project cost, schedule, and technical performance with the
establishment of a baseline performance review reporting to NASA's
senior management. Through monthly reviews, NASA intends to highlight
projects that are predicted to exceed internal NASA cost and/or
schedule baselines, which are set lower than cost and schedule
baselines submitted to Congress, so the agency can take preemptive
actions to minimize the projects' potential cost overruns or schedule
delays.
While these efforts are positive steps towards achieving successful
project outcomes and ensuring that decision makers are appropriately
investing the agency's resources, they will be limited if project
officials are not held accountable for demonstrating that elements of a
knowledge-based business case are demonstrated at key junctures in
development. It is critical that project officials not only have a high
level of knowledge about a project at key junctures, but also that this
information is used by decision makers to make decisions on whether to
invest additional resources and allow a project to proceed through the
development life cycle.
Project Assessments:
We found several factors that occurred throughout the various projects
we reviewed that can contribute to project cost and schedule growth.
These factors--characterized as project challenges--were mostly present
in the projects that had reached the implementation phase of the
project life cycle. They, along with a profile of each project we
reviewed, are described in a two-page assessment for each project. The
project profile presents a general description of the mission
objectives for each of the projects; a picture of the spacecraft or
aircraft; a schedule timeline identifying key dates for the project; a
table identifying programmatic and launch information; and the baseline
year cost and schedule estimates and December 2008 cost and schedule
data.
The remainder of the assessment analyzes the project challenges and the
extent to which each project faces cost, schedule, or performance risk
due to these challenges. They are based on past GAO work on elements of
a successful acquisition business case--technology maturity, heritage
technology complexity, and design stability. Additionally, through our
review, we identified two more challenges--contractor performance and
development partner performance--that had an impact on cost and
schedule performance of the NASA projects. Contractor performance
impacts NASA's ability to deliver a project within cost and schedule
baselines because the agency depends on the expertise of the contractor
to deliver what it promises. Similarly, NASA sometimes relies on other
domestic and international organizations to provide key instruments,
the spacecraft, and/or launch services for collaborative projects; the
performance of these partners can impact NASA's performance for a
project. When a development partner cannot deliver an instrument or
integrate it on schedule, the impact is felt by NASA. Specifically,
since often there is no exchange of money between partners, the cost of
any delays to the project must be assumed by each partner.
For each individual project assessment, we provide a table showing the
challenges relevant to the project and a project status narrative. This
is followed by a narrative of the project challenges we identified
relevant to each project.
NASA project offices were provided an opportunity to review drafts of
the individual two-page assessments prior to their inclusion in the
final product. The projects provided both technical corrections and
more general comments. We integrated the technical corrections as
appropriate and characterized the general comments on the second page
of each two-page assessment.
Our Observations:
NASA provided cost and schedule data for 13 projects in the
implementation phase of the project life cycle.[Footnote 10] Ten of
those 13 projects experienced significant cost and/or schedule growth
from their project baselines.[Footnote 11] Based on our analysis,
development costs for projects in our review increased by an average of
almost 13 percent from their baseline cost estimates--all in just two
or three years--including one project's cost that increased by over 50
percent. It should be noted that a number of these projects had
experienced considerably more cost growth before they were baselined in
response to the statutory reporting requirement.[Footnote 12] Our
analysis also shows that projects in our review had an average delay of
over 11 months to their launch dates. The lack of knowledge at key
junctures during project development, as well as the complexity of
using heritage hardware--systems with characteristics similar to the
one being developed--and working relationships with contractors and
development partners contributed to the cost and schedule growth. Table
1 depicts the 13 projects we reviewed that had entered the
implementation phase, the challenges they faced or are currently
facing, and the cost and schedule changes they experienced.
Table 1: Assessment of Challenges for NASA Projects in the
Implementation Phase:
NASA Projects: Aquarius; [Shaded]
Technology Maturity: [Empty];
Design Stability: [Check];
Complexity of Heritage Technology: [Empty];
Contractor Performance: [Empty];
Development Partner Performance: [Check];
Development Cost Change: 6%;
Launch Delay (months): 10.
NASA Projects: Dawn;
Technology Maturity: [Empty];
Design Stability: [Empty];
Complexity of Heritage Technology: [Check];
Contractor Performance: [Check];
Development Partner Performance: [Empty];
Development Cost Change: -2%;
Launch Delay (months): 0.
NASA Projects: Gamma-ray Large Space Area Telescope; [Shaded]
Technology Maturity: [Empty];
Design Stability: [Check];
Complexity of Heritage Technology: [Check];
Contractor Performance: [Check];
Development Partner Performance: [Check];
Development Cost Change: 5%;
Launch Delay (months): 9.
NASA Projects: Glory; [Shaded]
Technology Maturity: [Check];
Design Stability: [Check];
Complexity of Heritage Technology: [Check];
Contractor Performance: [Check];
Development Partner Performance: [Empty];
Development Cost Change: 54%;
Launch Delay (months): 6.
NASA Projects: Herschel; [Shaded]
Technology Maturity: [Check];
Design Stability: [Check];
Complexity of Heritage Technology: [Empty];
Contractor Performance: [Empty];
Development Partner Performance: [Check];
Development Cost Change: 13%;
Launch Delay (months): 20.
NASA Projects: Kepler; [Shaded]
Technology Maturity: [Empty];
Design Stability: [Empty];
Complexity of Heritage Technology: [Check];
Contractor Performance: [Check];
Development Partner Performance: [Empty];
Development Cost Change: 25%;
Launch Delay (months): 9.
NASA Projects: Lunar Reconnaissance Orbiter;
Technology Maturity: [Empty];
Design Stability: [Empty];
Complexity of Heritage Technology: [Check];
Contractor Performance: [Empty];
Development Partner Performance: [Empty];
Development Cost Change: 0%;
Launch Delay (months): 6[A].
NASA Projects: Mars Science Laboratory; [Shaded]
Technology Maturity: [Check];
Design Stability: [Check];
Complexity of Heritage Technology: [Check];
Contractor Performance: [Empty];
Development Partner Performance: [Empty];
Development Cost Change: 26%;
Launch Delay (months): 25.
NASA Projects: NPOESS Preparatory Project; [Shaded]
Technology Maturity: [Check];
Design Stability: [Check];
Complexity of Heritage Technology: [Check];
Contractor Performance: [Empty];
Development Partner Performance: [Check];
Development Cost Change: 19%;
Launch Delay (months): 26.
NASA Projects: Orbiting Carbon Observatory; [Shaded]
Technology Maturity: [Empty];
Design Stability: [Check];
Complexity of Heritage Technology: [Empty];
Contractor Performance: [Check];
Development Partner Performance: [Empty];
Development Cost Change: 18%;
Launch Delay (months): 5.
NASA Projects: Solar Dynamics Observatory; [Shaded]
Technology Maturity: [Empty];
Design Stability: [Check];
Complexity of Heritage Technology: [Empty];
Contractor Performance: [Check];
Development Partner Performance: [Check];
Development Cost Change: 1%;
Launch Delay (months): 17.
NASA Projects: Stratospheric Observatory for Infrared Astronomy;
[Shaded]
Technology Maturity: [Empty];
Design Stability: [Empty];
Complexity of Heritage Technology: [Check];
Contractor Performance: [Check];
Development Partner Performance: [Empty];
Development Cost Change: 3%;
Launch Delay (months): 9.
NASA Projects: Wide-field Infrared Survey Explorer; • -1% 0
Technology Maturity: [Empty];
Design Stability: [Check];
Complexity of Heritage Technology: [Empty];
Contractor Performance: [Empty];
Development Partner Performance: [Empty];
Development Cost Change: -1%;
Launch Delay (months): 0.
Source: GAO analysis of NASA project data.
Note: Shading indicates project exceeded cost and/or schedule baseline.
A blank cell indicates the challenge does not apply to that particular
project or the project did not supply data or make a projection of the
data.
[A] The Lunar Reconnaissance Orbiter exceeded its schedule threshold by
5 months and 25 days, but the table shows 6 months due to rounding.
[End of table]
We did not specifically correlate individual project challenges with
specific cost and/or schedule changes in each project. The degree to
which specific challenges contributed to cost and schedule growth
varied across the projects in this review. Nonetheless, since previous
GAO work has demonstrated the impact of these challenges on cost and
schedule growth and our discussions with NASA project officials
identified the additional challenges we discuss as contributing to cost
and schedule growth, we are confident in our characterization of them
for the purpose of this specific review.
Technology Maturity:
Four of the thirteen projects in our assessment for which we received
data and that had entered the implementation phase did so without first
maturing all critical technologies.[Footnote 13] Further, three of
those four projects had also not matured their critical technologies
before continuing to assembly, integration, and testing. This means
that needed knowledge about these technologies remained unknown well
into development thereby adding potential cost and schedule risk to the
projects. For example, five of the eight critical technologies for one
instrument and three of the five critical technologies for another
instrument identified by the Herschel project office were immature when
the project moved into implementation. Almost two years later at the
critical design reviews, four of the thirteen critical technologies for
these two instruments were still immature, yet the project proceeded.
When complex development programs proceed without understanding whether
technologies can work as intended, they end up facing unanticipated
technical problems that have costly, reverberating effects on other
aspects of the program.
Design Stability:
The majority of the projects in our assessment that held a critical
design review did so without first achieving a stable design. GAO best
practices recommend completion of at least 90 percent of engineering
drawings at the critical design review to provide evidence that the
design is stable. Though NASA's acquisition policy does not specify how
the project should achieve design stability by the critical design
review, NASA's system engineering handbook adheres to GAO's metric. Of
the projects we were able to assess that had reached that point in
their life-cycle, none had achieved design stability by the time they
proceeded into assembly, integration, and testing.
All of the projects in our assessment that had reached their critical
design review and that provided data on engineering drawings
experienced some growth in the total number of design drawings after
their critical design review. Growth ranged from 8 percent to, in the
case of two projects, well over 100 percent. Some of this increase can
be attributed to change in system design after critical design review.
For some projects, design changes after critical design review were
necessary due to problems in maturing technologies or issues found
during testing. For example, the Mars Science Laboratory required
several design changes to address various issues, including redesign of
the plumbing for the propulsion system, which increased the drawing
count by 67 percent from critical design review to the time of our
review.
Complexity of Heritage Technology:
More than half the projects in the implementation phase--8 of them--
encountered challenges in integrating or modifying heritage
technologies. Additionally, two projects in formulation--Ares I and
Orion--also encountered this challenge. We found that the projects that
relied on heritage technologies underestimated the effort required to
modify them to the necessary form, fit, or function. According to NASA
officials, heritage technologies are not the same as critical
technologies because, in their opinion, critical technologies are not
based on existing--or heritage--technology. Generally, the project
officials said that the technology they were using was not considered
"new" if it had been demonstrated in a test environment or used on a
prior mission, even if there needed to be a change or customization in
configuration or design. Yet, these projects all failed to build in the
necessary resources for technology modification. For example, the
Kepler project office did not identify any critical technologies since
all had flown on earlier missions, but viewed their modification as a
design challenge for the Kepler mission. However, the project
underestimated the effort required to modify the photometry array and,
as a result, this challenge contributed to a 25 percent--or $78
million--cost overrun and Kepler's launch schedule being delayed by
nine months.
Contractor Performance:
Six of the seven projects that cited contractor performance as a
challenge also experienced significant cost and/or schedule growth.
Through our discussions with the project offices, we were informed that
contractors encountered technical and design problems with hardware
which disrupted development progress. Additionally, contractors lacked
the experience in space systems that was required for the projects,
which may be the underlying reason for these development challenges.
For example, the Dawn contractor had no experience in deep space
missions. Officials from the company acknowledged they had difficulty
developing the spacecraft wiring. They also encountered problems
developing the ion propulsion system for the spacecraft.
Contractors also faced workforce or corporate issues, such as closing
facilities, lack of resources, and management inefficiencies. For
example, the Glory project manager cited management inefficiencies and
execution problems with the instrument contractor. According to Glory
project officials at NASA, among the drivers of these management
inefficiencies were senior leadership changes, a loss of core
competencies due to plant closure, and a lack of proper decision
authority. The contractor agreed that the plant closure and the need to
re-staff were major project challenges. In this case, as with others in
our review, the contractors forfeited their contract fees or spent the
fee they had received from NASA to cover project costs.
Development Partner Performance:
Five of the thirteen projects we reviewed encountered challenges with a
development partner. In these cases, the development partner could not
meet their commitments to the project within planned timeframes. This
may have been a result of issues within the specific development
partner organization or as a result of issues faced by a contractor to
that development partner. For example, NASA is collaborating with the
European Space Agency (ESA) on the Herschel space observatory. While
NASA has delivered its two instruments to ESA, ESA has encountered
difficulties developing its instruments and has delayed Herschel's
launch by 14 months. Because of this delay, NASA has incurred about $39
million in cost growth due to the need to fund component developers for
a longer period of time than originally planned.
Assessment Of Individual Projects:
Our assessments of all 18 individual projects follow.
Aquarius:
Aquarius is a satellite mission developed by NASA and the Space Agency
of Argentina (Comisión Nacional de Actividades Espaciales, CONAE) to
investigate the links between the global water cycle, ocean
circulation, and the climate. It will measure global sea surface
salinity. The Aquarius science goals are to observe and model the
processes that relate salinity variations to climatic changes in the
global cycling of water and to understand how these variations
influence the general ocean circulation. By measuring salinity globally
for 3 years, Aquarius will provide an unprecedented new view of the
ocean's role in climate.
Figure: Artist depiction of Aquarius Project Office.
[End of figure]
Formulation to Implementation:
Formulation start (12/03);
Preliminary Design Review (6/05);
Critical Design Review (9/06);
GAO Review (12/08);
Launch Readiness Date (5/10).
Project Essentials:
NASA Center Lead: Jet Propulsion Laboratory;
International Partner: Argentina's National Committee of Space
Activities (CONAE);
Major Contractors: in-house development;
Projected Launch Date: May 23, 2010;
Launch Location: Vandenberg AFB, Calif.
Launch Vehicle: Delta II;
Mission Duration: 3 years for Aquarius mission; 5 years for SAC-D
(CONAE) mission.
Project Challenges:
* Design Stability;
* Development Partner Performance.
Table: Project Performance (then year dollars in millions):
Total Project Cost:
Baseline Est. (FY 2008): $241.8;
Latest (Dec. 2008): $253.1;
Change: 4.7%.
Formulation Cost:
Baseline Est. (FY 2008): $35.5;
Latest (Dec. 2008): $35.6;
Change: 0.3%.
Development Cost:
Baseline Est. (FY 2008): $192.7;
Latest (Dec. 2008): $204.5;
Change: 6.1%.
Operations Cost:
Baseline Est. (FY 2008): $13.6;
Latest (Dec. 2008): $13.0;
Change: -4.4%.
Launch Schedule:
Baseline Est. (FY 2008): 7/2009;
Latest (Dec. 2008): 5/2010;
Change: 10 months.
[End of table]
Project Status:
The launch for Aquarius has been delayed 10 months, from July 2009 to
May 2010 because of delays in CONAE‘s spacecraft development
activities. The launch delay prompted NASA to report to the Congress
that the Aquarius project exceeded its development schedule threshold
and caused NASA to experience a $10.7 million cost increase. Based on
the cost-sharing arrangements with CONAE, NASA will also bear its own
costs associated with future delays. NASA has continued its development
of the Aquarius instrument, which is currently scheduled for completion
in March 2009 and shipment to CONAE in June 2009 for integration with
the Argentine-developed spacecraft.
Detailed Project Discussion:
The only critical technology the project office identified was the
Aquarius instrument itself, which includes the scatterometer and the
radiometer components. The project deemed the instrument mature at the
preliminary design review because the instrument uses heritage
technologies, even though those technologies were brought together in a
different form, fit, and function for use on Aquarius. The instrument
design, however, was not stable at the critical design review (CDR) as
the Aquarius project had released only 16 percent of the engineering
drawings. Project officials told us that some detailed parts were either
not accounted for or not very mature at CDR, and they needed a follow-
on review to clear up the issue. For example, details in the design of
a connector arm of a reflector to the instrument were lagging. In
addition, project officials said that the Aquarius instrument design
was far ahead of development of CONAE‘s spacecraft, so the project
could not finalize and release all the instrument drawings until CONAE
finished the spacecraft design. To help minimize project risk in the
interim, project officials said NASA provided CONAE with an engineering
model to work with as the Argentines developed the spacecraft. All
engineering drawings have now been released.
Aquarius‘ schedule slipped 10 months, prompting NASA to report to the
Congress that the Aquarius program has exceeded its development
schedule threshold. According to project officials and budget
documents, a delay in development of the spacecraft bus by CONAE is the
primary reason for the schedule slip. Project officials said that CONAE
is using some newer and unfamiliar technologies on the spacecraft, such
as lithium-ion batteries for power storage. NASA‘s review of CONAE‘s
proposed schedule indicated that CONAE had made several high-risk
decisions in order to meet a planned launch date of September 2009.
For example, CONAE decided to begin flight model fabrication before
completing adequate testing of the engineering models. Subsequent
discussions between NASA and CONAE led to a decision to set a new
launch date of May 2010. The spacecraft will also house several
instruments for CONAE science missions. According to project officials,
those instruments all appear to be on schedule, but officials added
that none of those instruments are needed for NASA‘s Aquarius mission
and that the mission would launch without the CONAE instruments if any
were delayed.
NASA expects the Aquarius instrument to be completed in March 2009 and
held until June 2009 when it will ship to Argentina to be integrated
with the spacecraft. Since no funds are exchanged between the U.S. and
Argentina for this project, NASA bears its own costs associated with
any further delays for its portion that could occur. Project officials
indicated that the schedule slip increased NASA‘s cost by $10.7
million. They also noted that this cost increase does not include
increased launch vehicle costs because of the delay or Delta-II launch
site maintenance costs at Vandenberg Air Force Base.
Project Office Comments:
The Aquarius project provided technical comments to a draft of this
assessment, which were incorporated as appropriate. Project officials
also commented that Aquarius had changes to its baseline due to slips by
its development partner, the CONAE, and that they believe the NASA
contribution to this mission is on schedule for completion in March
2009. They added that the benefit of the international partnership, plus
the groundbreaking information about the Earth‘s climate, out weigh the
additional costs, which NASA has chosen to absorb within its budget.
Project officials said that NASA will continue to closely monitor
progress and work with its development partner to minimize impacts.
[End of Aquarius section]
Ares I Crew Launch Vehicle (CLV):
Figure: Artist depiction of Ares Project Office.
[End of figure]
NASA‘s Ares I Crew Launch Vehicle, as part of the Constellation
Program, is the next generation human spacecraft that will carry the
Orion Crew Exploration Vehicle into low Earth orbit. The mission of the
Ares I project is to deliver a safe, reliable, and affordable launch
system for space exploration. Ares I will feature a 24.5-metric ton
lift capability to carry crew to the Moon or deliver crew and cargo to
the International Space Station.
Formulation to Implementation:
Formulation start (9/05);
Preliminary Design Review (9/08);
GAO Review (12/08);
Critical Design Review (3/10);
Launch Readiness (3/15).
Project Essentials:
NASA Center Lead: Marshall Space Flight Center;
International Partner: None;
Major Contractors: Alliant Techsystems, Pratt and Whitney, Rocketdyne,
Boeing;
Projected Launch Date: March 2015;
Launch Location: Kennedy Space Center, Fla.
Launch Vehicle: Ares I;
Mission Duration: N/A.
Project Challenges:
* Complexity of Heritage Technology.
Project Performance (then year dollars in millions):
Preliminary Estimate of Project Life Cycle Cost*:
Latest (Jan. 2009): $17,000 to $20,000.
*This estimate is preliminary, as the project is in formulation and
there is still uncertainty in the value as design options are explored.
NASA uses these estimates for planning purposes. This estimate is for
the Ares I vehicle only.
Launch Schedule: 3/2015
Project Status:
Contract costs for the development the Ares I increased by $304 million
since initial award and the first manned launch has slipped from fiscal
year 2014 to fiscal year 2015. The Ares I had planned to begin
developmental flight testing in April 2009. However, delays to the
planned Hubble Space Telescope servicing mission have impacted the
project‘s ability to modify the launch pad needed to support planned
testing, resulting in at least a 3-month delay to the first Ares I
developmental flight test.
Detailed Project Discussion:
Because of the use of heritage systems and technology in system
designs, Ares I project officials said they did not identify any
critical technologies. However, we found that all three major elements
of the Ares I system”the first stage, upper stage, and upper stage
engine”face significant development challenges. The first stage draws
heavily from existing Space Shuttle systems, but requires modifications
such as incorporating a fifth segment that is likely to affect flight
characteristics. In addition, modeling indicates that thrust
oscillation within the first stage could cause unacceptable structural
vibrations throughout the Ares I and Orion vehicles which could
adversely affect crew safety if left unmitigated. NASA is considering
solutions including incorporating tuned vibration absorbers into the
Ares I first stage or adding a composite structure between the first
and second stages. Thrust oscillation was again identified as a risk
during the September 2008 preliminary design review, and the project
has scheduled another review in the fall of 2009 to fully incorporate
design solutions. The upper stage design includes a shared bulkhead
between the hydrogen and oxygen fuel tanks, even though experience from
the Apollo program shows that common bulkheads are complex and
difficult to manufacture. The J-2X upper stage engine represents a new
engine development effort that is likely to encounter problems during
development; NASA estimates that J-2X will require 29 rework cycles to
address problems, which they state is less than the number experienced
during the development of other rocket engines.
NASA has not released official cost and schedule estimates to complete
the Ares I program. NASA officials stated that these estimates will be
made available when the project moves into implementation, or at the
conclusion of the Constellation Program‘s non-advocate review. However,
the value of various development contracts for the Ares I have
increased by $304 million since initial award, and the first manned
launch has slipped from 2014 to 2015.
The project has already experienced schedule delays that they attribute
to funding instability in fiscal years 2007 and 2008 and launch pad
availability. Constellation‘s integrated risk management system also
indicates there is a high risk that funding shortfalls could occur in
fiscal years 2009 through 2012, resulting in planned work not being
completed to support schedules and milestones. Further, the delayed
Hubble Space Telescope servicing mission has caused the first planned
Ares I developmental flight test”Ares I-X”to slip at least 3 months
from April 2009 to July 2009. Since the Hubble mission will have a back
up Shuttle for crew rescue purposes, thus utilizing both launch pads,
the Ares I project cannot modify launch pad 39B for its use until the
Hubble servicing mission is complete. NASA continues to develop an
integrated schedule based on how the Hubble mission will impact pad
modifications for the Ares I-X mission, as well as joint scheduling of
a mobile launch platform and space in the Vertical Assembly Building.
Project Office Comments:
The Ares I project office provided technical comments to a draft of
this assessment, which were incorporated as appropriate. Project
officials also commented that they believe the project has made
progress in maturing the Ares I design and associated elements, and
that all planned reviews have been executed on a schedule that supports
the initial operating capability commitment. They added that the
project is responding to technical and programmatic challenges, and
they feel that all major element contracts are in place and are
performing to plan.
[End of Ares I Crew Launch Vehicle (CLV) section]
Dawn:
Figure: Artist depiction: NASA/JPL/McREL; Background: William K.
Hartmann, UCLA (artist depiction).
[End of figure]
The Dawn mission is on a journey to the two largest asteroids in our
solar system, Vesta and Ceres. Launched from Cape Canaveral in
September 2007, the Dawn spacecraft will encounter and orbit Vesta 4
years later, then travel an additional three years to reach and orbit
Ceres. The Dawn spacecraft will use solar-electric (ion) propulsion to
reach and orbit Vesta for 7 months and Ceres for 5 months while
performing scientific investigations at various altitudes and lighting
conditions. Dawn will use imaging, spectroscopy, and gravity
measurements to characterize the two asteroids---measuring their mass,
gravity fields, principal axes, rotational axes, and moments of
inertia.
Formulation to Implementation:
Formulation Start (4/01);
Implementation Preliminary Design Review (10/03);
Critical Design Review (6/04);
Launch Readiness (9/07);
GAO Review (12/08).
Project Essentials:
NASA Center Lead: Jet Propulsion Laboratory;
Partners: Los Alamos National Laboratories, German Aerospace Center
(DLR) with the Max Planck Institute for Aeronomy, Agenzia Spaziale
Italiana (ASI);
Major Contractors: Orbital Sciences Corporation;
Launch Date: September 27, 2007;
Launch Location: Cape Canaveral AFS, Fla.
Launch Vehicle: Delta II;
Mission Duration: 8 Years.
Project Challenges:
* Complexity of Heritage Technology;
* Contractor Performance.
Table: Project Performance (then year dollars in millions):
Total Project Cost:
Baseline Est. (FY 2008): $460.4;
Latest (Dec. 2008): $465.0;
Change: 1.0%.
Formulation Cost:
Baseline Est. (FY 2008): $99.3;
Latest (Dec. 2008): $106.6;
Change: 7.4%.
Development Cost:
Baseline Est. (FY 2008): $273.7;
Latest (Dec. 2008): $269.4;
Change: -1.6%.
Operations Cost:
Baseline Est. (FY 2008): $87.3;
Latest (Dec. 2008): $89.0;
Change: 1.9%.
Launch Schedule:
Baseline Est. (FY 2007): 9/2007;
Latest (Dec. 2008): 9/2007;
Change: none.
[End of table]
Project Status:
Dawn launched on September 27, 2007. The spacecraft is scheduled to
begin the survey of Vesta on August 18, 2011 and then survey Ceres
beginning February 18, 2015.
Detailed Project Discussion:
The Dawn project has been beset with funding issues. The project was
approved for early formulation in January 2001, but was delayed nine
months as NASA did not have the funds to proceed. Budget issues also
caused a delay in the project as it moved through the formulation
phase. The mission objectives were then modified to a baseline mission
to Vesta with travel to Ceres as an extended mission. During the
project‘s second confirmation review, NASA added the travel to Ceres as
a primary mission objective. The project was also told to increase
reserves to 25 percent to comply with JPL design principles which,
according to project officials, were not written when the Dawn project
was first proposed, causing the project to be descoped and under-funded
at the beginning of implementation. Project management expended $25
million in project reserves in the first year of implementation
attempting to meet a June 2006 launch date, but the use of reserves was
not conveyed to the mission directorate. In the subsequent year, the
project experienced significant cost overruns. The project stopped
development activities between October 2005 and January 2006 during a
review by an Independent Assessment Team (IAT). The IAT reported
technical issues with the project, recommended management changes, and
stated a need for an additional $57 million and a 12 to 18 month
extension to complete implementation. According to project officials,
NASA‘s Science Mission Directorate terminated the project in February
2006, but it was reinstated on appeal and resulted in a launch
readiness date slip to June 2007. Ultimately, this one year launch
delay cost the project an additional $54 million.
JPL indicated that contractor performance led to several problems
during Dawn‘s development, generally stemming from a lack of technical
and corporate experience on the part of the prime contractor with regard
to complex space systems, such as the ion propulsion system which
contractor officials agreed was new to them. The IAT noted that JPL did
not provide enough oversight of its contractor, which had no system-
level planetary project implementation experience, to assure hardware
delivery schedules would be met and software development activities
could be accomplished on time and within budget. Project officials told
us that other sub-contractors on the project also experienced
development and testing issues. For example, a sub-contractor working
on development of the ion propulsion system encountered problems that
led to deficient workmanship and component failures, while another
subcontractor had issues with development of the xenon tank for the ion
propulsion system; both the flight tank and spare failed testing. As a
result of these issues and other system level implementation issues,
the project experienced cost overruns and the overall launch readiness
date for the system slipped 15 months. Subsequently, the prime
contractor suggested forfeiting part of their contract award fees to
keep the project on cost and its mission intact, and NASA agreed.
The initial project proposal for Dawn assumed a high level of heritage
technology for the ion propulsion system from the Deep Space One
mission. According to project officials, inheritance reviews were
conducted early in the life cycle for Dawn and the design was generally
correct. A study performed during formulation should have derived that
the cost and schedule assumptions of using heritage technology were
not valid, but officials told us the study was not accurate. Problems
with the heritage technology, however, were discovered in
implementation, resulting in significant cost growth.
Project Office Comments:
The Dawn project office provided technical comments to a draft of this
assessment, which were incorporated as appropriate. Project officials
said that NASA agrees that there were funding issues but points out
that they were initially externally driven, which necessitated changes
to project scope during the project life cycle and resulted in the
prime contractor giving up their fee prior to confirmation. Project
officials also agreed that there were technical challenges faced by
both prime and by some sub-contractors, some of which were due to a
higher expectation of heritage hardware than was actually the case.
[End of Dawn section]
Gamma-ray Large Area Space Telescope (GLAST):
Figure: Gamma-ray Large Area Space Telescope (GLAST):
Source: Kennedy Space Center, Cape Canaveral, FI IMG RSC-08PD-1637.
[End of figure]
The Gamma-ray Large Area Space Telescope (GLAST) seeks to improve
understanding of the structure of the universe. By measuring the
direction, energy, and arrival time of celestial high-energy gamma
rays, GLAST will map the sky with 50 times the sensitivity of previous
missions. GLAST‘s scientific payload includes two instruments: the
Large Area Telescope (LAT) and the Gammaray Burst Monitor (GBM). The
mission has four objectives: (1) understanding the mechanisms of
particle acceleration in astrophysical environments; (2) determining
the high-energy behavior of gammaray bursts; (3) resolving and
identifying point sources with known objects; and (4) probing dark
matter and the extra galactic background light in the early universe.
Formulation to Implementation:
Formulation Start (7/99):
Preliminary Design Review (6/03);
Critical Design Review (9/04);
Launch Readiness (5/08);
GAO Review (12/08).
Project Essentials:
NASA Center Lead: Goddard Space Flight Center;
Partners: U.S. Department of Energy, France, Germany, Japan, Italy and
Sweden;
Major Contractors: Stanford University, General Dynamics;
Launch Date: June 11, 2008;
Launch Location: Cape Canaveral AFS, Fla.
Launch Vehicle: Delta II;
Mission Duration: 5 years (10 year goal).
Project Challenges:
* Design Stability;
* Complexity of Heritage Technology;
* Contractor Performance;
* Development Partner Performance.
Table: Project Performance (then year dollars in millions):
Total Project Cost:
Baseline Est. (FY 2008): $723.3;
Latest (Dec. 2008): $774.5;
Change: 7.1%.
Formulation Cost:
Baseline Est. (FY 2008): $97.4;
Latest (Dec. 2008): $98.7;
Change: 1.3%.
Development Cost:
Baseline Est. (FY 2008): $403.3;
Latest (Dec. 2008): $423.0;
Change: 4.9%.
Operations Cost:
Baseline Est. (FY 2008): $222.5;
Latest (Dec. 2008): $252.8;
Change: 13.6%.
Launch Schedule:
Baseline Est. (FY 2007): 9/2007;
Latest (Dec. 2008): 6/2008;
Change: 9 months.
[End of table]
Project Status:
GLAST successfully launched into low Earth orbit on June 11, 2008.
Detailed Project Discussion:
Prior to launch in June 2008, the GLAST project experienced several
schedule delays because of conflicts over test facilities, launch pad
time, and engineering issues. These delays resulted in NASA‘s reporting
to the Congress that the GLAST project exceeded its schedule baseline
by 8 months. Project officials told us that the spacecraft vendor gave
priority to Department of Defense projects for thermal vacuum testing at
its test facility. This action forced GLAST to be moved to the Naval
Research Labs for testing. In order to accommodate GLAST, the alternate
test facility required some minor modification. According to NASA
officials, this resulted in a 3-month delay. A busy launch schedule at
Cape Canaveral then made it difficult for GLAST to re-schedule its
launch date, contributing to the remainder of the project‘s overall
schedule slip.
Project officials said schedule slippage can also be attributed to
heritage technology engineering problems. At the project‘s preliminary
design review, GLAST had matured its one critical technology, while the
rest were considered heritage technologies. The project considers the
Large Area Telescope (LAT) a new instrument, though it is made up of
several heritage technologies. According to a project official, the LAT
has experienced both engineering design and electrical parts problems
that resulted in schedule delays and the need for additional funding.
Likewise, officials told us that a component of GLAST‘s command and
data handling system also features a new combination of heritage
technology. Because of software and hardware problems, project
officials said that the prime contractor had to bring this work, which
had been outsourced to a sub-contractor, back in-house.
The project also identified partner issues that contributed to an
increase in project cost. According to project officials, France
initially was responsible for significant instrument integration work;
however, the French were unable to complete that work and, as a result,
the project office transferred it to the Naval Research Laboratory.
This transfer increased costs by about $5 million. In addition,
officials said that Italy originally was supposed to supply the GLAST
ground station with X-band communications. However, in 2003, Italian
officials informed the project they could not keep this commitment. The
antenna on the GLAST spacecraft now uses Ku-band communications
instead. Italy also used an inexperienced contractor to produce GLAST‘s
tracking towers, a situation that resulted in contamination problems.
Project officials stated that these partner issues combined to increase
the cost of the GLAST project and contributed to the $45 million
increase.
The GLAST project‘s design was not stable at critical design review as
the project had released only 76 percent of its drawings and
experienced a 31 percent growth in the number of drawings after the
critical design review. Project officials attributed the growth to the
withdrawal of the French partners, the change to a Ku-band transmitter
and ground system, and the change in facility for producing the solar
arrays.
Project Office Comments:
The GLAST project office provided technical comments on a draft of this
assessment, which were incorporated as appropriate. Project officials
also commented that they believe the principal project challenge was
the loss of development partners, and that the eight month launch slip
was caused by contractor performance and launch vehicle development
issues. They did not consider design stability or the complexity of
heritage technology as issues for this project.
[End of Gamma-ray Large Area Space Telescope (GLAST) section]
Glory:
Figure: Glory Project Office (artist depiction).
[End of figure]
The Glory project is a low-Earth orbit satellite that will contribute
to the U.S. Climate Change Science Program. The satellite has two
principal science objectives: (1) collect data on the properties of
aerosols and black carbon in the Earth‘s atmosphere and climate systems
and (2) collect data on solar irradiance. The satellite has two main
instruments---the Aerosol Polarimetry Sensor (APS) and the Total
Irradiance Monitor (TIM)---as well as two cloud cameras. The TIM will
allow NASA to have uninterrupted solar irradiance data by bridging the
gap between NASA‘s Solar Radiation and Climate Experiment and the
National Polar Orbiting Environmental Satellite System (NPOESS)
missions.
Formulation to Implementation:
Formulation Start (9/05);
Preliminary Design Review (9/05);
Critical Design Review (7/06);
GAO Review (12/08);
Launch Readiness Date (06/09).
Project Essentials:
NASA Center Lead: Goddard Space Flight Center;
International Partner: None;
Major Contractors: Raytheon Space and Airborne Systems, University of
Colorado Laboratory for Atmospheric and Space Physics, Orbital Sciences
Corporation;
Projected Launch Date: June 2009;
Launch Location: Vandenberg AFB, Calif.
Launch Vehicle: Taurus XL;
Mission Duration: 3 years (5 year goal).
Project Challenges:
* Technology Maturity;
* Design Stability;
* Complexity of Heritage Technology;
* Contractor Performance.
Table: Project Performance (then year dollars in millions):
Total Project Cost:
Baseline Est. (FY 2008): $266.1;
Latest (Dec. 2008): $347.9;
Change: 30.7%.
Formulation Cost:
Baseline Est. (FY 2008): $70.8;
Latest (Dec. 2008): $70.5;
Change: -0.4%.
Development Cost:
Baseline Est. (FY 2008): $168.9;
Latest (Dec. 2008): $259.1;
Change: 53.4%.
Operations Cost:
Baseline Est. (FY 2008): $26.4;
Latest (Dec. 2008): $18.3;
Change: -30.7%.
Launch Schedule:
Baseline Est. (FY 2008): 12/2008;
Latest (Dec. 2008): 6/2009;
Change: 6 months.
[End of table]
Project Status:
The Glory project reported to the Congress that it exceeded its
development cost threshold by 31 percent from its baseline, requiring
the Congress to reauthorize Glory. The project is waiting for delivery
of the APS, which is now projected for February 2009. The delivery of
this instrument is over one year behind schedule. The launch date for
Glory, originally scheduled for June 2008, is now scheduled for June
2009. The launch delay may require the project to report to the
Congress that it will also exceed its development schedule baseline.
Detailed Project Discussion:
The Glory project has experienced significant delays because of late
delivery of the Aerosol Polarimetry Sensor (APS), which is based on
heritage technology. At its preliminary design review in September 2005,
the Glory project had one immature technology: the APS. At that review,
the project estimated that the APS would be delivered by September
2007. According to the APS contractor, the instrument is now forecasted
for delivery in February 2009---over one year behind schedule. The
project identified contractor performance as the top risk facing the
mission. Despite the contractor‘s performance, NASA has kept work on
the APS with the company because the project believes it is more cost
effective than starting a new inhouse development project of this
instrument. NASA estimated that an in-house development effort would
cost an additional $78 million and delay launch until February 2010.
Glory project officials stated that the APS development problems do not
stem from technical issues, but from the contractor‘s inability to plan
and execute the work. The officials outlined several causes for the
project‘s issues with the contractor, including the company
consolidating its workforce and a resulting loss of APS corporate
design knowledge. Contractor officials told us that along with moving
the APS development effort from one facility to another, they made the
decision to finish building the instrument with the new team rather
than doing a complete design analysis. They said this led directly to
cost and schedule increases as they had to perform more testing
concurrent with the development of the instrument.
At the critical design review, the project‘s design was not stable as
it had released only 70 percent of its drawings. As of GAO‘s review, 99
percent of total drawings have been released. However, Glory‘s drawing
count increased by 27 percent after the critical design review. This
increase is attributed to the modification of drawings for heritage
parts for Glory‘s unique configuration.
Since Glory was baselined in fiscal year 2008, the project‘s
development costs have increased by 31 percent. As a result, NASA has
reported to the Congress that Glory has exceeded its development cost
threshold, requiring the Congress to reauthorize the project.
Uncertainty in the project prior to its mission confirmation in 2005
delayed the launch readiness date from June 2008 to December 2008. More
recently, the project‘s scheduled launch date has slipped from December
2008 to June 2009, which could cause the project to also have to report
the slip to the Congress. According to project officials, Glory‘s
recent cost and schedule issues are driven solely by the late delivery
of the APS. The project has taken several steps to mitigate the cost
increases caused by the delayed delivery of the APS and to improve the
contractor‘s performance. The project has eliminated requirements,
simplified the instrument design, provided NASA engineering and
management resources to the contractor, and involved both NASA and
contractor executives in addressing the problems. According to
contractor officials, the company has used its award fee to cover the
costs on this project. The company has also provided its own funding to
help off-set cost overruns.
Project Office Comments:
The project office provided technical comments to a draft of this
assessment, which were incorporated as appropriate. Project officials
also commented that since filing the threshold report, NASA has
continued to pursue the baseline plan for the Glory implementation.
They added that performance at the APS instrument supplier continues to
be slow, but the instrument technical performance evaluated at each
major developmental gate has been excellent. In addition, they believe
the engineering design and the technical performance of the APS
instrument have never been issues, and the programmatic issues have all
been connected with the supplier‘s manufacturing and management.
[End of Glory section]
Global Precipitation Measurement (GPM) Mission:
Figure: GPM Project Office (artist depiction).
[End of figure]
The Global Precipitation Measurement (GPM) mission, a joint NASA and
Japan Aerospace Exploration Agency (JAXA) project, seeks to improve the
scientific understanding of the global water cycle and the accuracy of
precipitation forecasts. The GPM is composed of a core spacecraft
carrying two main instruments: a Dual-frequency Precipitation Radar
(DPR) and a GPM Microwave Imager (GMI). In addition, the GPM project
includes a second Low-Inclination spacecraft with a second GMI
instrument. The GPM builds on the work of the Tropical Rainfall
Measuring Mission, and will provide the first opportunity to calibrate
measurements of global precipitation.
Formulation to Implementation:
Formulation Start (7/02);
Preliminary Design Review (11/08);
GAO Review (12/08);
Critical Design Review (11/09);
Launch Core Spacecraft (7/13);
Launch constellation Spacecraft (11/14);
Project Essentials:
NASA Center: Goddard Space Flight Center;
International Partner: Japanese Aerospace Exploration Agency (JAXA);
Major Contractors: Ball Aerospace;
Projected Launch Date: July 2013;
Launch Location: Tanegashima Island, Japan;
Launch Vehicle: H-IIA (Japan);
Mission Duration: 3 years (5 years consumables).
Project Challenges:
* None Currently Identified.
Project Performance (then year dollars in millions):
Preliminary Estimate of Project Life Cycle Cost*:
Latest (Jan. 2009): not provided;
* NASA suggested it will provide baseline estimates for
this project when it proceeds from formulation into
implementation.
Launch Schedule:
Latest (Jan. 2009): 7/2013.
Project Status:
GPM is in the formulation phase and is expected to enter implementation
after its mission confirmation review in the spring of 2009. Recent
project budget changes show reductions in fiscal years 2009 and 2010.
These reductions caused a project re-plan and delayed the scheduled
development of the second GMI instrument by 1 year and delayed the
launch of the Low-Inclination observatory by 5 months. The re-plan
schedules a July 2013 launch of the core spacecraft at the latest as
requested by JAXA.
Detailed Project Discussion:
The GPM spacecraft and its components are being designed to be
demiseable---they will burn up during re-entry into the Earth‘s
atmosphere to limit orbital debris---which poses a challenge for the
project in the integration of the propellant management device with the
craft‘s aluminum composite propulsion tanks. Neither the propellant
management device nor the aluminum composite tanks are new
technologies, but the integration of the two is the challenge.
Currently, the integration of the propellant management device and the
aluminum composite tank is not expected to be mature until after the
preliminary design review. If the project is unable to sufficiently
mature this technology, it will use a titanium propellant management
device and/or tank that is less demiseable.
The GPM project has not reached a design review where we could assess
design stability based on our metric. The project currently has
released 17 percent of its engineering drawings, but expects to have
released only 70 percent of drawings at the critical design review.
According to project officials, the two main instruments”the JAXA-
supplied Dual-frequency Precipitation Radar (DPR) and the NASA-supplied
GPM Microwave Imager (GMI)”are based on heritage technology and
therefore are not considered critical technologies. However, the DPR
and GMI will have to be adapted to the GPM spacecraft design for this
mission. In addition, the DPR instrument includes a Ka-band radar that
the project identified as a new design.
Project officials told us that JAXA has been frustrated by NASA‘s
uncertainty over funding the GPM project and questioned NASA‘s
commitment to the project. According to a March 2008 report from NASA‘s
Inspector General, budget reductions to the GPM project in fiscal years
2005 through 2007 led to a 2-year delay in the contract for the
development and delivery of the GMI. These reductions caused the launch
of the core spacecraft to be delayed from 2007 to 2013 and the cost
estimate to rise from $600 million to over $1 billion. In early 2008,
NASA signed a launch vehicle agreement with JAXA. Subsequently, the
preliminary design and critical design reviews were scheduled for the
project.
The project‘s budget was recently reduced for fiscal years 2009 and
2010. Following these cuts, NASA directed the project to re-plan with
two constraints: 1) maintain the core spacecraft launch date of June
2013 but let the Low-Inclination spacecraft slip as necessary and, 2)
accept increased programmatic risk from low contingency funds in fiscal
year 2009. The re-plan presented maintains the core spacecraft to a July
2013 launch date requested by JAXA, but the start of work on the second
GMI instrument is delayed by one year and the launch of the
Constellation observatory is delayed by 5 months. As a result of the
reduction in funding levels, NASA considers fiscal year 2009 as a high-
risk year for the project since it now has low contingency reserves of
approximately 5 percent.
Project Office Comments:
The project office provided technical comments to a draft of this
assessment, which were incorporated as appropriate. Project officials
said that following the numerous mission delays that drove the launch
date from 2008 to the current July 2013, the sustained funding
increases the GPM project has received since the fiscal year 2008
budget has enabled steady progress towards mission confirmation and
implementation in fiscal year 2009. They added that JAXA has been
satisfied with this progress, including the formal agreement
concluded in early 2008 between NASA and JAXA on the use of the H-IIA
launch vehicle to launch the Core Observatory. Project officials
believe they will have a stable design since they plan to have necessary
hardware manufacturing drawings released by the critical design review.
[End of Global Precipitation Measurement (GPM) Mission section]
Herschel:
Figure: ESA/AOES Medialab (artist depiction).
[End of figure]
The Herschel Space observatory, a collaborative project between NASA
and the European Space Agency (ESA), will seek to discover how the first
galaxies formed and how they evolved to give rise to present day
galaxies like our own. Herschel has the largest mirror ever built for a
space telescope. At 3.5 meters in diameter, the mirror will collect
long-wavelength radiation from some of the coldest and most distant
objects in the Universe. It will be able to observe dust-obscured and
cold objects that are invisible to other telescopes. Additional targets
for Herschel will include clouds of gas and dust where new stars are
being born, disks out of which planets may form, and cometary
atmospheres packed with complex organic molecules.
Formulation to Implementation:
Formulation Start (1998);
Preliminary Design Review (7/00);
Critical Design Review (7/01);
GAO Review (12/08);
Launch Readiness (04/09);
Project Essentials:
NASA Center Lead: Jet Propulsion Laboratory:
International Partner: European Space Agency (ESA);
Major Contractors: in-house development;
Projected Launch Date: April 2009;
Launch Location: Kourou, French Guiana;
Launch Vehicle: Ariane 5 (ESA Supplied);
Mission Duration: 3 years (5 year goal).
Project Challenges:
* Technology Maturity;
* Design Stability;
* Development Partner Performance.
Table: Project Performance (then year dollars in millions):
Total Project Cost:
Baseline Est. (FY 2008): $325.4;
Latest (Dec. 2008): $276.8;
Change: -14.9%.
Formulation Cost:
Baseline Est. (FY 2008): $10.4;
Latest (Dec. 2008): $10.4;
Change: 0.0%.
Development Cost:
Baseline Est. (FY 2008): $117.0;
Latest (Dec. 2008): $131.7;
Change: 12.6%.
Operations Cost:
Baseline Est. (FY 2008): $198.0;
Latest (Dec. 2008): $134.7;
Change: -32.0%.
Launch Schedule:
Baseline Est. (FY 2008): 8/2007;
Latest (Dec. 2008): 4/2009;
Change: 20 months.
[End of table]
Project Status:
Since Herschel‘s baseline was established in 2007, ESA slipped the
Herschel launch schedule three times because of scope changes and
challenges with integration of the instruments onto the spacecraft. A
recent slip resulted in a project cost increase of $43 million and
required NASA to report to the Congress that it exceeded its schedule
baseline. The project is currently behind schedule by about 50 days, a
delay that has caused a fourth launch date slip from its current October
2008 date to April 2009.
Detailed Project Discussion:
NASA has completed the development of components for two Herschel
Instruments---the Heterodyne Instrument for the Far Infrared (HIFI) and
the Spectral and Photometric Imaging Receiver (SPIRE) instrument---and
delivered them to ESA. However, during both the preliminary and
critical design reviews, some of the critical technologies for these
elements were considered immature. At the preliminary design review
(PDR) for HIFI, five of the eight critical technologies were immature.
Later, at critical design review (CDR), two of the eight HIFI critical
technologies were still assessed as immature. SPIRE had a similar
record. At SPIRE‘s PDR, three of the five critical technologies were
assessed as being immature. Two years later at CDR, two of five SPIRE
critical technologies were still assessed as immature. Regardless of
this, the project proceeded. After delivery of NASA‘s components,
problems were found during testing of the equipment in Europe.
According to the project office, the HIFI failed in thermal cycling
during testing and SPIRE had problems with the wiring that connects its
detectors. The technical issues with the two instruments cost $3.9
million to resolve.
In addition to technology maturity issues, NASA committed to developing
components for the HIFI and SPIRE instruments before achieving design
stability for the instruments. At the CDR for both the HIFI and SPIRE
instruments, NASA had released less than 10 percent of the engineering
design drawings. According to the project office, this was primarily
due to the fact that ESA‘s interface drawings were in preliminary
format. The office also said that the lack of timeliness in the
submission of design drawings is a challenge when the project has to
depend on multiple partners for input.
Herschel‘s $43 million growth in life cycle costs can be largely
attributed to technical integration problems, which resulted in launch
delays. Those delays are also the primary cause of overall schedule
slippage. ESA‘s contractor could not complete development of its
instruments or integrate Herschel instruments in a timely manner,
prompting ESA to pull the integration work in-house. While NASA faced
some technical problems with development of components for the HIFI and
SPIRE instruments, resulting in about $3.9 million of cost growth,
project officials said the remaining increase of about $39 million is
due to the three slips in Herschel‘s launch date since the project‘s
baseline was established in February 2007 since the project must
maintain a workforce to support testing and integration activities.
Based on the 14-month delay in launch date, NASA reported to the
Congress in February 2008 that the Herschel project has exceeded its
schedule baseline. Herschel‘s launch schedule has now slipped even
further. The project office stated that Herschel is currently behind
schedule by about 50 days, which caused the launch date to slip from
its current October 2008 date to early in calendar year 2009.
Project Office Comments:
The Herschel project office provided technical comments to a draft of
this assessment, which were incorporated as appropriate. Project
officials also commented that although NASA did have some technical
issues with their hardware contributions which did cause an increase to
the NASA cost in order to correct the problems, the majority of the
cost increase to NASA has been due to technical problems on the
European side which have caused the launch to slip several times in the
last several years.
[End of Herschel section]
James Webb Space Telescope (JWST):
Figure: JWST Project Office (artist depiction).
[End of figure]
The James Webb Space Telescope (JWST) is a large, infrared-optimized
space telescope that is designed to find the first galaxies that formed
in the early universe. The focus of scientific study will include first
light, assembly of galaxies, origins of stars and planetary systems,
and origins of the elements necessary for life. JWST's instruments will
be designed to work primarily in the infrared range of the
electromagnetic spectrum, with some capability in the visible range.
JWST will have a large mirror, 6.5 meters (21.3 feet) in diameter and a
sunshield the size of a tennis court. Both the mirror and sunshade will
not fit onto the rocket fully open, so both will fold up and open once
JWST is in outer space. JWST will reside in an orbit about 1.5 million
kilometers (1 million miles) from the Earth.
Formulation to Implementation:
Formulation Start (3/99);
Preliminary Design Review (3/08);
GAO Review (12/08);
Critical Design Review (7/09);
Launch Readiness Date (6/13);
Project Essentials:
NASA Center Lead: Goddard Space Flight Center;
International Partner: European Space Agency (ESA), Canadian Space
Agency (CSA);
Major Contractors: Northrop Grumman;
Projected Launch Date: June 2013;
Launch Location: Kourou, French Guiana;
Launch Vehicle: Ariane 5 (ESA Supplied);
Mission Duration: 5 years (10 year goal).
Project Challenges:
* None Currently Identified.
Table: Project Performance (then year dollars in millions):
Total Project Cost:
Latest (Jan. 2009): Not provided.
Formulation Cost:
Latest (Jan. 2009): Not provided.
Development Cost:
Latest (Jan. 2009): Not provided.
Operations Cost:
Latest (Jan. 2009): Not provided.
Launch Schedule:
Latest (Jan. 2009): 6/2013.
* NASA suggested it will supply the baseline estimates for this project
when it provides them to Congress in the FY10 Budget Request.
[End of table]
Project Status:
The JWST project was re-planned in fiscal year 2006 after a $1 billion
cost increase”$3.5 billion to $4.5 billion”and a 2-year schedule delay
on the project. A major risk that continued to affect the project
following this replan was the low level and late phasing of contingency
funding budgeted, despite 12 percent of the $1 billion cost growth
being used to increase such funding. Although JWST passed its
preliminary design review, the project still has to address several
issues related to testing.
Detailed Project Discussion:
The JWST project was re-planned in fiscal year 2006 after a $1 billion
cost increase and a 2-year schedule delay on the project. About half of
the cost growth was because of a 1-year schedule slip, resulting from a
delayed decision to use an ESA-supplied Ariane 5 launch vehicle and an
additional 10-month slip caused by budget profile limitations in fiscal
years 2006 and 2007. Changes in requirements and a 12 percent increase
in the program‘s contingency funding accounted for the remainder of the
growth. Despite this increase in contingency funding (i.e., reserves),
the level and phasing of contingency funding budgeted for the project
continues to be a major risk. An independent review team expressed
concern over the contingency funding, stating that it is too low and
phased in too late. Further, it stated that a contingency fund of 25
percent to 30 percent would be appropriate for a project through
implementation. Goddard Space Flight Center policies also require
reserves of 25 percent through Phase D, Implementation. NASA directed
its Science Mission Directorate to address the JWST reserves issue at
the project‘s confirmation reviews in the summer of 2008. The project
budgeted for 5 years of operation for JWST, instead of 10 years of
mission operations, which, according to the JWST Deputy Associate
Director, saved the project approximately $300 million.
Prior to the re-plan in fiscal year 2006, the JWST project was set to
proceed into development with immature technologies. Because of
substantial cost growth on the project and as part of the 2006 re-plan,
NASA decided to invest additional time and resources in maturing JWST‘s
critical technologies prior to the preliminary design review. JWST held
a technology non advocate review in January 2007 to assess the maturity
of its ten critical technologies. At that time, all but one of the
critical technologies was assessed as mature, the remaining critical
technology”the cryocooler”has since been matured. Maturing critical
technologies on the project prior to entering implementation was a
significant step to reducing risk. The JWST project office also had 53
percent of its design drawings released at its preliminary design
review in March 2008 and anticipates that it will have 94 percent of
its design drawings released by the critical design review in June
2009.
Although JWST passed its preliminary design review, the project still
has to address several issues related to testing. One concern is that
the project plans to do only one test at the highest level of assembly
possible in the cryogenic vacuum chamber at the Johnson Space Center.
The review panel advised JWST to add another test cycle to its
schedule. Further, the board recommended the addition of a center of
curvature test on the Optical Telescope Element and was also concerned
that the project was not planning to test the sunshield at the highest
level of assembly in the cryogenic vacuum chamber. The project is still
working through how to address such testing issues.
Project Office Comments:
The JWST project provided technical comments to a draft of this
assessment, which were incorporated as appropriate. Project officials
said they generally agree with the assessment as presented. Project
officials commented that the specific concern expressed by the review
team was that the project's baseline plan only included one cryogenic
thermal vacuum test opportunity at the integrated level of assembly,
but the officials added that the specific tests that will be conducted
during that one cryogenic thermal vacuum test opportunity are numerous
and comprehensive. Project officials said to address the review team‘s
concern in this regard, the project has accounted for the cost of the
additional cryogenic testing, should it eventually be required.
[End of James Webb Space Telescope (JWST) section]
Kepler:
Figure: Photograph of Kepler craft.
Source: Ball Aerospace.
[End of figure]
The Kepler mission has been designed to discover Earth-like planets in
orbit around stars in our galaxy. The goal is to detect tens or even
hundreds of Earthsize planets in the habitable zones of stars similar
to our own sun. The habitable zone is the region around a star where
the temperature of a terrestrial-type planet can be expected to allow
water to exist in liquid form on the planet's surface, thereby
increasing the probability of life. Kepler will explore the structure
and diversity of planetary systems by conducting a census of extra-
solar terrestrial planets using a photometer in heliocentric orbit to
observe the dimming of starlight caused by planetary transits.
Formulation to Implementation:
Formulation Start (12/01);
Preliminary Design Review (10/04);
Critical Design Review (10/06);
GAO Review (12/08);
Launch Readiness (2/09);
Project Essentials:
NASA Center Lead: Jet Propulsion Laboratory;
International Partner: None;
Major Contractors: Ball Aerospace and Technologies Corp.
Projected Launch Date: March 2009;
Launch Location: Cape Canaveral AFS, Fla.;
Launch Vehicle: Delta II;
Mission Duration: 3.5 years.
Project Challenges:
* Complexity of Heritage Technology;
* Contractor Performance.
Table: Project Performance (then year dollars in millions):
Total Project Cost:
Baseline Est. (FY 2007): $497.5;
Latest (Dec. 2008): $594.8;
Change: 19.6%.
Formulation Cost:
Baseline Est. (FY 2007): $138.1;
Latest (Dec. 2008): $141.2;
Change: 2.2%.
Development Cost:
Baseline Est. (FY 2007): $312.8;
Latest (Dec. 2008): $390.3;
Change: 24.8%.
Operations Cost:
Baseline Est. (FY 2007): $46.6;
Latest (Dec. 2008): $63.3;
Change: 35.8%.
Launch Schedule:
Baseline Est. (FY 2007): 6/2008;
Latest (Dec. 2008): 3/2009;
Change: 9 months.
[End of table]
Project Status:
Since being baselined in fiscal year 2007, NASA has reported to the
Congress that both Kepler‘s development costs and schedule have
exceeded the baseline. During that time, Kepler‘s development costs
have increased by about $78 million---or 25 percent---and its schedule
has increased by 9 months, despite a reliance on heritage technologies.
Kepler project officials attribute the cost and schedule growth to
contractor performance problems, cost overruns, and the disruption
caused by a $35-million budget reduction in fiscal year 2005.
Detailed Project Discussion:
None of Kepler‘s technologies were identified as critical by the
project management office because all of Kepler‘s technologies have
flown on other missions and are therefore considered heritage. However,
the project office acknowledged that the customization of some of
Kepler‘s instruments, and the reliance on heritage technology has
proven to be a challenge to Kepler‘s development. Project officials
told us that Kepler‘s large photometry array added to the complexity of
the project because photometers of Kepler‘s sensitivity have not flown
before and proved more difficult to adapt than anticipated; an
adaptation that contributed to cost growth. Officials added that the
Kepler photometer requires a low noise level in its signal chain in
order to detect changes in the brightness of stars. This made
developing the electronics for the focal plane array a challenge. The
focal plane array is the largest ever flown in space and has stringent
requirements. Coupled with the high density of elements and electrical
and thermal attachments, this makes the assembly and tests of this
element a key challenge for the project.
We were unable to determine if Kepler‘s design was stable at its
critical design review. According to the project office, the prime
contractor, Ball Aerospace and Technologies Corporation, implemented a
new drawing management system called Agile, and the project did not
have any way to recover the forecast drawings count at the critical
design phase in October 2006. However, the project reports that 96
percent of its engineering design drawings have been released to the
manufacturer.
Kepler‘s total cost and overall schedule have increased significantly.
Since being baselined in fiscal year 2007, Kepler‘s development costs
have increased by about $78 million”or 25 percent”and its schedule has
increased by nine months. NASA has reported to the Congress that both
Kepler‘s development costs and schedule have exceeded the baseline. The
project office attributes the cost and schedule growth to contractor
performance problems, which occurred because the prime contractor was
unable to execute the project planned activities within the cost and
schedule they proposed, despite a reliance on heritage technology.
Contractor officials agreed that they underestimated the complexity and
the effort required to modify the existing these technologies. Both the
Kepler project manager and contractor officials also believe that a $35
million funding cut in the program because of funding constraints in
fiscal year 2005 was a significant contributor of the project‘s delays.
This funding instability, according to a NASA project official,
contributed to an overall 20-month delay in the project‘s schedule and
about $169 million in cost growth.
Both the project office and the prime contractor made changes to ensure
that the project remained executable with sufficient reserves. The
project office shortened the operations period by 6 months and accepted
additional project risk when it canceled or de-scoped several tests.
For example, the flight segment vibration test was reduced to an
acoustic test, and the vibration tests of the solar panel were removed.
Additionally, the prime contractor put new management personnel in
place and according to contractor officials, agreed to commit $7
million of its projected award fee to a cost performance incentive
that may allow the contractor to earn the fee later in the project‘s
life cycle.
Project Office Comments:
The Kepler project office provided technical comments to a draft of
this assessment, which were incorporated as appropriate. Project
officials also commented that Kepler instrument uses existing
technology components in a new and complex instrument design, and that
the contractor underestimated the complexity and effort required to
develop the instrument system and subsystems. They added that after the
2006 re-baselining of the project, the contractor continued to have
problems with the instrument development resulting in additional cost
and schedule overruns. They said the project was able to absorb this
cost increase by de-scoping elements of the program, delaying the guest
observer science program and reducing the mission duration by 6 months.
[End of Kepler section]
Landsat Data Continuity Mission (LDCM):
Figure: Artist depiction of Landsat Data Continuity Mission (LDCM).
Source: General Dynamics Advanced Information Systems.
[End of figure]
The Landsat Data Continuity Mission (LDCM), a partnership between NASA
and the U.S. Geological Service (USGS), seeks to extend the ability to
detect and quantitatively characterize changes on the global land
surface at a scale where natural and man-made causes of change can be
detected and differentiated. It is the successor mission to Landsat 7.
The Landsat data series, begun in 1972, is the longest continuous
record of changes in the Earth‘s surface as seen from space. Landsat
data is a unique resource for people who work in agriculture, geology,
forestry, regional planning, education, mapping, and global change
research.
Formulation to Implementation:
Formulation Start (10/03);
Preliminary Design Review (11/08);
GAO Review (12/08);
Critical Design Review (8/09);
Launch Readiness Date (12/12).
Project Essentials:
NASA Center: Goddard Space Flight Center;
Partner: U.S. Geological Service (USGS);
Major Contractors: Ball Aerospace and Technologies Corp., General
Dynamics Advanced Information Systems, The Hammers Company;
Projected Launch Date: December 2012;
Launch Location: Vandenberg AFB, Calif.;
Launch Vehicle: Atlas V;
Mission Duration: 5 years (10 years propellant).
Project Challenges:
* None Currently Identified.
Project Performance (then year dollars in millions);
Preliminary Estimate of Project Life Cycle Cost*:
Latest* (Jan. 2009): $730 to $800.
* This estimate is preliminary, as the project is in formulation and
there is still uncertainty in the value as design options, are
explored. NASA uses these estimates for planning purposes.
Launch Schedule:
Latest* (Jan. 2009): 12/2012.
Project Status:
The LDCM project shifted its estimated launch date from July 2011 to
December 2012 after it completed its Initial Mission Confirmation
Review in September 2008. The LDCM project is on an aggressive 39-month
development schedule for the main instrument. The LDCM instrument
payload consists of a single science instrument, the Operational Land
Imager (OLI); however, NASA is considering the addition of another
science instrument”a decision that could exacerbate the already
aggressive schedule and add cost.
Detailed Project Discussion:
The LDCM instrument payload consists of a single science instrument”the
Operational Land Imager (OLI). The project considered the addition of
two other science instruments”the Thermal Infrared Sensor (TIRS)
and the Total Solar Irradiance Sensor (TSIS). The project has decided
not to add TSIS, but will continue studying whether TIRS will be
included. The project hopes to receive funding for completion of the
TIRS instrument in spring 2009. The spacecraft is being designed to
accommodate TIRS and both the spacecraft and OLI developers are
studying the impacts of adding TIRS. According to a project official,
Goddard Space Flight Center would develop and build TIRS in-house, a
process that would take approximately 48 months. If TIRS is added to
the LDCM mission, however, it could delay launch by over a year and,
according to a project official, cost about $5 million for the redesign
of the spacecraft to accommodate the instrument. This design cost does
not include the cost of integrating the instrument onto the spacecraft.
Project officials have indicated that LDCM has already undertaken an
aggressive 39-month OLI development schedule. According to the
contractor for the OLI instrument, this aggressive schedule was
necessary because of delays in the procurement process. The LDCM
project delayed its estimated launch date from July 2011 to December
2012 after it completed its Initial Mission Confirmation Review in
September 2008. While a launch after January 2012 could jeopardize the
continuity of Landsat data, project officials said recent reliability
analyses show that the Landsat 7 satellite may be operational until
2017, lessening the likelihood of a data gap.
The project office has identified four critical technologies for the
OLI instrument. Three of the four critical technologies”the wide field
of view optics, linear arrays, and modular sensor chip assemblies”are
considered fully mature as they have been fully validated by the Earth
Observing Satellite (EO-1) mission through scene comparisons with
Landsat 7. The sensor assembly chips for the OLI are considered mature
since prototypes were included on the Advanced Land Imager that flew on
EO-1. The project does not anticipate that there will be any additional
critical technologies for the spacecraft because most of the technology
used to build the spacecraft will be commercial off-the-shelf items
that have flown on other missions.
Because LDCM has not yet reached its critical design review, we were
unable to assess design stability of the project at this time. The
project office anticipates having over 95 percent of the flight design
and manufacturing drawings complete by the critical design review
currently scheduled for August 2009. The spacecraft contract was
awarded in April 2008, and the project office anticipates releasing the
spacecraft drawings after design maturation. Formal cost and schedule
baselines will be established for the project at the Mission
Confirmation Review in 2009.
Project Office Comments:
The project office provided technical comments to a draft of this
assessment, which were incorporated as appropriate. The project office
also commented that NASA and the OLI instrument vendor are making
steady progress on the OLI instrument on the planned schedule. The
project is developing detailed schedules now to ensure sufficient
schedule reserve is applied to the critical hardware developments.
[End of Landsat Data Continuity Mission (LDCM) section]
Lunar Reconnaissance Orbiter (LRO):
Refer to PDF for Figure: Photograph of Lunar Reconnaissance Orbiter
(LRO):
Source: LRO Project Office.
[End of figure]
The Lunar Reconnaissance Orbiter (LRO) is NASA‘s first mission in the
implementation of the Vision for Space Exploration, the plan to return
to the moon and beyond. LRO‘s mission is to orbit the moon for one year
measuring lunar topography, resources, and thermal and radiation
environments. This data will be used to select a landing site for
future manned missions to the moon and to ensure astronaut safety. The
LRO has a scientific payload of six main instruments and one technology
demonstration instrument. LRO‘s launch vehicle contains a secondary
payload, the Lunar Crater Observation and Sensing Satellite (LCROSS),
which will investigate lunar surface volatiles such as water.
Formulation to Implementation:
Formulation Start (5/04);
Preliminary Design Review (2/06);
Critical Design Review (11/06);
GAO Review (12/08);
Launch Readiness (4/09).
Project Essentials:
NASA Center Lead: Goddard Space Flight Center;
Partner: Boston University, University of California Los Angeles,
Southwest Research Institute, Russian Institute for Space Research,
Arizona State University, Naval Air Warfare Center;
Prime Contractors: in-house development;
Projected Launch Date: April 24, 2009;
Launch Location: Cape Canaveral AFS, Fla.;
Launch Vehicle: Atlas V;
Mission Duration: 1 year (then science mission).
Project Challenges:
* Complexity of Heritage Technology.
Table: Project Performance (then year dollars in millions):
Total Project Cost:
Baseline Est. (FY 2008): $540.1;
Latest (Dec. 2008): $540.1;
Change: 0.0.
Formulation Cost:
Baseline Est. (FY 2008): $93.3;
Latest (Dec. 2008): $94.8;
Change: 1.6%.
Development Cost:
Baseline Est. (FY 2008): $420.8;
Latest (Dec. 2008): $422.4;
Change: 0.4%.
Operations Cost:
Baseline Est. (FY 2008): $25.8;
Latest (Dec. 2008): $22.9;
Change: -11.2%.
Launch Schedule:
Baseline Est. (FY 2008): 10/2008;
Latest (Dec. 2008): 4/2009;
Change: 6 months*.
* Actual launch delay is currently 5 months and 25 days.
[End of table]
Project Status:
LRO‘s original schedule with a launch date by the end of 2008 placed
the project on a challenging and aggressive development schedule. This
schedule is driven by the need to provide data for the Orion and Ares
I hardware designs and mission planning efforts for a human lunar
mission by 2020. The project experienced challenges modifying
instruments for the moon‘s thermal environment. These challenges, along
with a decision by the launch authority to re-prioritize the LRO launch
on its manifest, contributed to a launch slip to April 2009.
Detailed Project Discussion:
The project did not identify any critical technologies. Each of the
project‘s major instruments is based significantly on heritage
technology. However, the project manager said the project had
underestimated the difficulty of the modifications needed. For example,
the project manager said the Lunar Reconnaissance Orbiter Cameras
needed some technical work to adapt their designs for the lunar thermal
environment as well as some redesign when areas needing reinforcement
were found during testing. The Lunar Orbiter Laser Altimeter, while
similar to laser altimeters that have flown on previous Mars and
Mercury missions, had issues with the electronics that time the laser
pulses of the altimeter, which, according to the project manager, took
more time to resolve than originally expected. The Diviner Lunar
Radiometer Experiment instrument is almost a copy of an instrument on
Mars now, but experienced motor failures, which the project manager
said took extra time and money to recover from. Finally, the Lyman-
Alpha Mapping Project instrument, a copy of the Pluto Alice instrument,
was slightly delayed because of a detector failure during thermal
vacuum testing. According to the project manager, most instruments
required additional design and analysis of their thermal control
designs to operate reliably on the mission. Redesign was necessary
because the lunar environment presents a harsher thermal environment
than the environment faced by earth-orbiting missions.
The project did not measure design stability by percentage of drawings
completed at the critical design review (CDR), and therefore was not
assessed according to this metric.
Project officials said NASA gave LRO more reserve funding because of
the aggressive schedule on the project to compensate for schedule
slippages. Most challenges faced by the project occurred prior to the
confirmation review, so officials stated that the project will probably
finish at about only 3 percent above the confirmation cost estimate.
However, late delivery of instruments from project partners and a
decision by the launch authority to slip the LRO launch date both
contributed to the project‘s launch date being delayed 6 months from
October 2008 to April 2009.
Project Office Comments:
The LRO project office provided technical comments to a draft of this
assessment, which were incorporated as appropriate. Project officials
also commented that the change in launch date from December 2008 to
April 2009 was made to accommodate other launch priorities and as well
as technical problems with the launch vehicle. Project officials noted
that, while LRO‘s schedule was aggressive, schedule reserve had been
built in to accommodate late instrument deliveries and the project was
on track for a December 2008 launch. They added that the additional
time afforded by the new April 2009 launch date is being used by the
project to perform additional testing and mission simulations.
[End of Lunar Reconnaissance Orbiter (LRO) section]
Mars Science Laboratory (MSL):
Figure: Mars Science Laboratory (artist depiction).
[End of figure]
The Mars Science Laboratory (MSL) is part of the Mars Exploration
Program (MEP). The MEP seeks to understand whether Mars was, is, or can
be a habitable world. To answer this question the MSL project will
investigate how geologic, climatic, and other processes have worked to
shape Mars and its environment over time, as well as how they interact
today. The MSL will continue this systematic exploration by placing a
mobile science laboratory on the Mars surface to quantitatively assess
a local site as a potential habitat for life, past or present. The MSL
is considered one of NASA‘s flagship projects and will be the most
advanced rover ever sent to explore the surface of Mars.
Formulation to Implementation:
Formulation Start (11/03);
Preliminary Design Review (6/06);
Critical Design Review (6/07);
GAO Review (12/08);
Launch Readiness (10/11).
Project Essentials:
NASA Center Lead: Jet Propulsion Laboratory;
Partners: U.S. Department of Energy, Centre Nationale d'Etude Spatiale
(France), Russian Federal Space Agency, Centro de Astrobiologia
(Spain), Canadian Space Agency;
Major Contractors: none ” in-house development;
Projected Launch Date: October, 2011;
Launch Location: Cape Canaveral AFS, Fla.;
Launch Vehicle: Atlas V;
Mission Duration: 1 year - travel, 2 years - operations.
Project Challenges:
* Technology Maturity;
* Design Stability;
* Complexity of Heritage Technology.
Table: Project Performance (then year dollars in millions):
Total Project Cost:
Baseline Est. (FY 2008): $1642.2;
Latest (Dec. 2008): $1899.6;
Change: 15.7%.
Formulation Cost:
Baseline Est. (FY 2008): $515.1;
Latest (Dec. 2008): $515.5;
Change: 0.1%.
Development Cost:
Baseline Est. (FY 2008): $968.6;
Latest (Dec. 2008): $1225.0;
Change: 26.5%.
Operations Cost:
Baseline Est. (FY 2008): $158.5;
Latest (Dec. 2008): $159.1;
Change: 0.4%.
Launch Schedule:
Baseline Est. (FY 2008): 9/2009;
Latest (Dec. 2008): 10/2011;
Change: 25 months.
[End of table]
Project Status:
Since the project was baselined, MSL has experienced significant cost
growth”over $200 million thus far, or more than a 26 percent increase
in development costs”because of technological and engineering problems.
While the project has overcome design and weight growth issues, it
continues to face other technical challenges that contributed to MSL‘s
launch delay from October 2009 to October 2011. This launch delay will
result in about $400 million in cost growth as the project works to
resolve its remaining technical risks.
Detailed Project Discussion:
At the project‘s preliminary design review, the project assessed all
seven of critical technologies as immature resulting from late
development challenges encountered. At the critical design review a
year later, three of the seven critical technologies had been replaced
by backup technologies with two of the seven still assessed as
immature, including one of the replacement technologies. In addition,
MSL‘s design was never stabilized at the critical design review.
Several design changes were required to address various issues. For
example, the plumbing for the propulsion system was redesigned because
it was determined that MSL needed larger, rigid lines for the system
than were previously used on smaller Mars rovers. These thicker lines
inadvertently became load-bearing components, which caused the project
to redesign part of the structure to account for the loads and shift
them to MSL‘s primary structure.
MSL has relied on several heritage technologies that have had to be re-
designed, re-engineered, or replaced for use on the lab. For example,
the heatshield made of a super light-weight ablator that had flown on
previous missions was considered nearly ready at the critical design
review, but it suffered a significant setback in testing and could not
be proved for use on MSL. The project had to select a new and less
mature technology”phenolic impregnated carbon ablator (PICA). According
to the MSL project office, the impact of this change was approximately
$30 million in cost growth and a nine-month delay in delivery of the
heat shield.
Significant weight growth has occurred during MSL‘s development brining
the spacecraft to 90 percent of its mass threshold according to MSL
project officials. For example, MSL‘s project manager said that the
project wanted to implement a dry lubrication scheme with lightweight
titanium gears for the actuators, or motors that allow the lab to
function autonomously. During fabrication, however, it was discovered
that the lightweight titanium gears did not provide the durability
needed for MSL, causing the project to revert to the heavier stainless
steel gear system with wet lubricant used by prior projects. To keep
the lubricant from freezing in Martian temperatures, the project also
had to add heaters to the actuators, adding even more mass to the
rover.
The project cost has grown by over $200 million in the last year”more
than a 26 percent increase in development costs”and will increase even
more due to the launch delay from October 2009 to 2011. The project
could not meet its original schedule due to difficulty in meeting
delivery milestones for actuators, key avionics, and flight software
while maintaining its full testing program. Since Mars launch windows
are optimally aligned every 26 months, the project has to delay its
planned launch to October 2011. As a result of the launch delay,
project officials state that costs will likely grow by an estimated
$400 million bringing the project‘s life-cycle cost to $2.2 to $2.3
billion.
Project Office Comments:
The MSL project office provided technical comments on a draft of this
assessment, which were incorporated as appropriate. The project also
commented that while most of the system development is on track, MSL
cannot meet its October 2009 launch date due to a few critical elements
that are lagging. Project officials said the MSL launch is now
scheduled for the fall of 2011, which is the next opportunity for an
optimally aligned Earth-Mars transit.
[End of Mars Science Laboratory (MSL) section]
NPOESS Preparatory Project (NPP):
Figure: Photograph of NPOESS Preparatory Project (NPP):
Source: Ball Aerospace.
[End of figure]
The NPOESS Preparatory Project (NPP) is a joint mission with the
National Oceanic and Atmospheric Administration and the U.S. Air Force.
The satellite will measure ozone, atmospheric and sea surface
temperatures, land and ocean biological productivity, and cloud and
aerosol properties. The NPP mission has two objectives. First, NPP will
provide a continuation of global observations following the Earth
Observing System missions Terra and Aqua. Second, NPP will provide the
National Polar-orbiting Operational Environmental Satellite System
(NPOESS) with risk-reduction demonstration and validation for the
critical NPOESS sensors, algorithms, and ground data processing.
Formulation to Implementation:
Formulation Start (11/98);
Preliminary Design Review (1/03);
Critical Design Review (8/03);
GAO Review (12/08);
Launch Readiness (6/10).
Project Essentials:
NASA Center Lead: Goddard Space Flight Center;
Partner: National Atmospheric and Oceanic Administration and U.S. Air
Force;
Major Contractors: Northrop Grumman Electrical Systems and Ball
Aerospace and Technologies Corp.;
Projected Launch Date: June 2, 2010;
Launch Location: Vandenberg AFB, Calif.;
Launch Vehicle: Delta II;
Mission Duration: 5 years.
Project Challenges:
* Technology Maturity;
* Design Stability;
* Complexity of Heritage Technology;
* Development Partner Performance.
Table: Project Performance (then year dollars in millions):
Total Project Cost:
Baseline Est. (FY 2007): $672.8;
Latest (Dec. 2008): $794.6;
Change: 18.1%.
Formulation Cost:
Baseline Est. (FY 2007): $47.3;
Latest (Dec. 2008): $47.0;
Change: -0.6%.
Development Cost:
Baseline Est. (FY 2007): $593.0;
Latest (Dec. 2008): $703.7;
Change: 18.7%.
Operations Cost:
Baseline Est. (FY 2007): $32.5;
Latest (Dec. 2008): $43.9;
Change: 35.1%.
Launch Schedule:
Baseline Est. (FY 2007): 4/2008;
Latest (Dec. 2008): 6/2010;
Change: 26 months.
[End of table]
Project Status:
Due primarily to the late delivery of a key instrument being developed
by project partners, the NPP project has experienced nearly $111
million in development cost growth and a 26-month delay in its launch
readiness date since being baselined in fiscal year 2007. As a result,
NASA has reported to the Congress that the NPP project has exceeded
both its cost and schedule thresholds. The NPP project office is
monitoring the risk of further instrument delivery delays.
Detailed Project Discussion:
The NPP project office identified six critical technologies for the
project”the spacecraft and all five instruments. Five of the six
critical technologies were assessed as immature at the preliminary and
critical design reviews. The Clouds‘ and the Earth‘s Radiant Energy
System (CERES) instrument was the only mature technology, and according
to project officials this instrument was only added back to the mission
when it was determined that development of other instruments would
cause a significant launch delay. Many of the spacecraft‘s components
and subsystems have flown on previous missions and are therefore mature.
The NPP project office now considers all critical technologies to be
mature.
The project‘s design was unstable at the critical design review (CDR).
Two instruments being developed by the Integrated Program Office (IPO),
which is composed of National Oceanic and Atmospheric Administration
and Department of Defense officials, are the Cross-track Infrared
Sounder (CrIS) and the Visible Infrared Imaging Radiometer Suite
(VIIRS). Both had to be redesigned because of failures that were
detected during testing after the CDR. The project office said a 31
percent increase in new engineering drawings was largely attributed to
the redesign of the VIIRS and CrIS stemming from testing failures.
According to a project official, the CrIS structure development
multiple fractures during testing and needed to be stripped to its
components and rebuilt. The project official also said the VIIRS could
not meet its science requirement of detecting ocean color because of
the poor quality of its filters. The official indicated a problem
exists with the system‘s requirements and not the ability of the
contractor to produce the correct filters. An official for the
contractor building the VIIRS instrument said the original requirement
was unachievable and the filters will be improved for the second VIIRS
instrument, which will be a part of the NPOESS mission.
Since NPP was baselined in fiscal year 2007, the project‘s development
costs increased by about $111 million, or almost 19 percent, and its
schedule has increased by 26 months. As a result, NASA has reported to
the Congress that the NPP project has exceeded both its cost and
schedule thresholds. The project office attributes almost all of the
cost and schedule changes to the late delivery of the VIIRS instrument
by the project partners. The instrument is now scheduled to be
delivered in April 2009. An official for the VIIRS instrument
contractor cites the presence of multiple government customers and
ongoing requirements changes as the reasons for the delay and increase
in cost. Neither the IPO nor the NPOESS prime contractor, according to
NASA‘s NPP project manager, provided adequate oversight of the VIIRS
contractor during the development of VIIRS. While there is no
contractual relationship between NASA and the VIIRS contractor, project
officials told us NASA now has two engineers at the prime contractor‘s
facility to oversee the design and development of the instrument.
Additional delay in instrument delivery could result in observatory
integration delays, cost increases, schedule slips, and possible gaps
in data continuity.
Project Office Comments:
The NPP project office provided technical comments on a draft of this
assessment, which were incorporated as appropriate. The project also
commented that the NASA-developed instruments did not experience
challenges with design stability, rather it was NASA‘s partners‘
instruments. They added that the VIIRS performance requirements have
remained stable since the critical design review and the primary drivers
of the schedule delay are issues found during fabrication and testing
of the engineering and flight models. Project officials said the VIIRS
instrument continues to incur delays during environmental testing which
will likely result in a delay in NPP launch readiness beyond June 2010.
NASA has provided the NPOESS IPO additional expertise to help provide
more oversight to attempt to minimize additional delays and increase
the likelihood of VIIRS meeting performance goals.
[End of NPOESS Preparatory Project (NPP) section]
Orbiting Carbon Observatory (OCO):
Figure: Jet Propulsion Laboratory (artist depiction).
[End of figure]
NASA‘s Orbiting Carbon Observatory (OCO) seeks to enable more reliable
forecasts of climate change. It will make the first global measurements
of atmospheric carbon dioxide with the precision and resolution needed
to characterize production and loss rates. These measurements will
improve mankind‘s understanding of the processes that regulate
atmospheric carbon dioxide. The OCO payload consists of a single unit
instrument with three high resolution grating spectrometers. Each of
these spectrometers records the intensity of radiation over one of
three very narrow Near Infrared bands that are sensitive to the
presence of carbon dioxide and oxygen. The observatory will fly in loose
formation with other satellites to enable synergy and to complement the
science return.
Formulation to Implementation:
Formulation Start (12/03);
Preliminary Design Review (7/04);
Critical Design Review (8/06);
GAO Review (12/08);
Launch Readiness (2/09).
Project Essentials:
NASA Center Lead: Jet Propulsion Laboratory;
International Partner: None;
Major Contractors: Hamilton Sundstrand Corp. and Orbital Sciences Corp.
Projected Launch Date: February 23, 2009;
Launch Location: Vandenberg AFB, Calif.;
Launch Vehicle: Taurus XL;
Mission Duration: 2 years.
Project Challenges:
* Design Stability;
* Contractor Performance.
Table: Project Performance (then year dollars in millions):
Total Project Cost:
Baseline Est. (FY 2008): $235.2;
Latest (Dec. 2008): $273.1;
Change: 16.1%.
Formulation Cost:
Baseline Est. (FY 2008): $31.0;
Latest (Dec. 2008): $32.5;
Change: 4.8%.
Development Cost:
Baseline Est. (FY 2008): $186.4;
Latest (Dec. 2008): $219.7;
Change: 17.9%.
Operations Cost:
Baseline Est. (FY 2008): $17.8;
Latest (Dec. 2008): $20.9;
Change: 17.4%.
Launch Schedule:
Baseline Est. (FY 2008): 9/2008;
Latest (Dec. 2008): 2/2009;
Change: 5 months.
[End of table]
Project Status:
OCO‘s launch date slipped from September 2008 to February 2009, and
NASA reported to the Congress that the project‘s development cost
increased 18 percent from the baseline established in fiscal year 2008.
On February 24, 2009, OCO launched but failed to reach orbit.
Detailed Project Discussion:
The only critical technology for OCO, its three-channel grating
spectrometer, was considered mature at the mission‘s preliminary design
review. However, technical problems arose for the instrument after its
critical design review (CDR) in August 2006. Testing results showed
that the detectors used in the instrument suffer from a residual image
problem when they transition from a bright-to-dark image. This is an
inherent characteristic of the detectors and the error in data will be
corrected by ground-based software. In addition, OCO‘s design was not
stable at CDR as the project reported that it had only released 66
percent of its engineering drawings. Following CDR, the project also
experienced a 15 percent increase in the total number of drawings
expected. According to project officials, the increase was attributed
to the changes made in the system design to address structural issues.
The project has since released all of its engineering drawings.
According to project officials, the contractor developing the three-
channel spectrometer underestimated the cost to develop the instrument.
In December 2005, OCO project management began providing its own
personnel to augment the contractor‘s workforce in order to mitigate
schedule slippage. Reviews of the instrument design identified areas
that would not withstand launch and/or flight forces”a finding that
necessitated a redesign of the instrument structure. According to the
deputy project manager, the contract was modified to bring
responsibility for the instrument‘s integration and testing activity in
house. Project management stated that the contractor did not receive
its award fee because of its poor performance, but will still be
eligible for on-orbit award fees.
OCO has experienced cost increases and schedule delays, and NASA has
reported to the Congress that OCO has exceeded its development cost
baseline. According to project officials, the project did not receive
funding to begin its preliminary design phase in 2003, resulting in a
one year schedule delay and an increase to the estimated mission cost
of approximately $60 million. In addition, the movement of the
instrument work in-house in October 2006 led to an increase in
development costs and an inability to maintain the planned September
2008 launch date. NASA recently reported to the Congress an 18 percent
increase in development cost from the baseline established in fiscal
year 2008 and a schedule slip to December 2008. OCO was ready to launch
in December but a delay at Vandenberg Air Force Base pushed the launch
into 2009.
Project Office Comments:
The OCO project office provided technical comments to a draft of this
assessment, which were incorporated as appropriate. In addition,
project officials commented that they believe the instrument and
mission design were stable at the critical design review and neither
have experienced significant changes to the CDR-approved design. They
added that the project did experience problems during instrument
development, assembly, and testing which prompted the project to be
rebaselined. Since then, they stated that the project has stayed within
its planned cost and schedule and was prepared to launch in December
2008, but was delayed because of unavailability of the launch range and
launch vehicle certification issues.
[End of Orbiting Carbon Observatory (OCO) section]
Orion Crew Exploration Vehicle (CEV):
Figure: Orion Crew Exploration Vehicle (CEV) (artist depiction).
Source: Lockheed Martin Space Systems.
[End of figure]
NASA‘s Orion Crew Exploration Vehicle (CEV), as part of the
Constellation Program, is the next generation spacecraft to carry crew
and cargo to the International Space Station and to the Moon. The
Constellation Program includes the CEV and a launch system that will
replace the Space Shuttle, which is slated to retire in 2010. The five-
meter diameter Orion capsule is to be launched by the Ares I Crew
Launch Vehicle. Orion will carry up to six astronauts to the
International Space Station or four astronauts to the Moon after
linking up with a lunar lander. The capsule will return to Earth and
descend on parachutes to the surface. Orion has three main elements”the
crew module (capsule), service module/spacecraft adapter, and launch
abort system.
Formulation to Implementation:
Formulation Start (7/06);
GAO Review (12/08);
Preliminary Design Review (3rd Qtr 2009);
Critical Design Review (4th Qtr 2010);
Initial Operational Capability (3/15).
Project Essentials:
NASA Center Lead: Johnson Space Center;
International Partner: None;
Major Contractors: Lockheed Martin;
Projected Launch Date: March 2015;
Launch Location: Kennedy Space Center, Fla.;
Launch Vehicle: Ares I;
Mission Duration: Varied based on destination.
Project Challenges:
* Complexity of Heritage Technology.
Project Performance (then year dollars in millions):
Preliminary Estimate of Project Life Cycle Cost*:
Latest (Jan. 2009): $20,000 TO $29,000.
* This estimate is preliminary, as the project is in formation and
there is still uncertainty in the value as design options are explored,
NASA uses these estimates for planning purposes. This estimate is for
the Orion vehicle only.
Launch Schedule:
Latest (Jan. 2009): 3/2015.
Project Status:
NASA is currently working toward a preliminary design review (PDR) for
the Orion vehicle. As a result of several issues including unexpected
weight growth, the PDR has been delayed by at least 9 months into
fiscal year 2009. Additional schedule movement is under consideration
to allow more time for integration of preliminary design products
across the Orion organization to assure acceptable risk for completing
the PDR with the right vehicle design.
Detailed Project Discussion:
The Orion project identified three critical technologies for the
spacecraft: the phenolic impregnated carbon ablator (PICA) heat shield,
which was used on Stardust, NASA‘s comet sample return mission, landing
airbags, and landing parachutes. The project identified a backup heat
shield technology for PICA. Project officials said that both heat
shield technologies have some heritage to earlier NASA missions, but
both technologies have distinct risks. According to officials, PICA
must be built and applied to Orion in sections creating gaps between
the sections that need to be filled, similar to Space Shuttle tiles.
The backup is lighter than PICA, but more difficult to manufacture. The
PICA material is the chosen technology for the thermal protection
system, but project officials said that they will select a single
technology at PDR based on performance, how difficult it is to produce,
weight, and cost. The project expects that all technologies will be
mature by the preliminary design review. We found, however, that the
heat shield development and manufacturing schedule is at risk and may
impact Orion‘s test schedule. In addition, Orion faces challenges in
the development of the attitude control motor for the launch abort
system. While similar attitude control motors have been demonstrated
before, Orion‘s motor design is complex, and any failures during
developmental testing may cause unexpected delays.
Although the Orion project has not reached a design review where we
could assess design stability based on our metric, NASA recognizes that
continued weight growth and requirements changes are contributing to
instability in the Orion design. For example, according to agency
officials, continuing Orion weight growth led NASA to redesign the
Orion vehicle in fall 2007. As a result of engineering trade-offs that
were made during this process, NASA modified the requirement for
landing on land to landing in water, which would reduce vehicle mass.
The Orion project is still working on these issues and has not yet
finalized requirements or design.
At the time of our review, NASA had not released cost and schedule
estimates for completing the Orion project. NASA officials stated that
these estimates will be made available at the conclusion of the
Constellation Program non-advocate review, which takes places after
PDR, when all NASA projects establish an integrated cost and schedule
baseline. According to the Constellation program‘s risk database, there
is a high risk that Orion could face funding shortfalls in fiscal years
2009 through 2012, resulting in planned testing not being completed in
time to support schedule and milestones. Furthermore, schedule delays
have already occurred as a result of unexpected efforts to resolve
mass, power, and other architecture issues and because the project
needed sufficient time to attain an acceptable level of design risk.
Project Office Comments:
The Orion project office provided technical comments to a draft of this
assessment, which were incorporated as appropriate. The project office
also commented that they believe steady progress has been made in all
technology areas and appropriate technology readiness will be achieved
prior to PDR in late summer 2009. They also believe that the Orion
project has achieved stability in requirements growth and that NASA
will continue to narrow design options as the project moves toward a
confirmed baseline design. Project officials added the project is on
schedule to finalize the choice of material for the heat shield by
March 2009.
[End of Orion Crew Exploration Vehicle (CEV) section]
Solar Dynamics Observatory (SDO):
Figure: Solar Dynamics Observatory (SDO) (artist depiction).
Source: SDO Project Office.
[End of figure]
NASA‘s Solar Dynamics Observatory (SDO) will investigate how the Sun's
magnetic field is structured and how its energy is converted and
released into the heliosphere in the forms of solar wind, energetic
particles, and variations in solar irradiance. The primary goal of the
SDO mission is to understand the solar variations that influence life
on Earth and humanity‘s technological systems. It seeks to do this by
determining how the Sun‘s magnetic field is generated and structured,
and how this stored magnetic energy is released. Analysis of data from
SDO‘s three instruments”Atmospheric Imaging Assembly (AIA), Extreme
Ultraviolet Variability Experiment (EVE), and Helioseismic and Magnetic
Imager (HMI)”will improve the science needed to enable space weather
predictions.
Project Essentials:
NASA Center Lead: Goddard Space Flight Center;
International Partner: None;
Major Contractors: Stanford University, Lockheed Martin Solar
Astrophysics Laboratory, University of Colorado;
Projected Launch Date: January 2010;
Launch Location: Kennedy Space Center, Fla.;
Launch Vehicle: Atlas V;
Mission Duration: 5 years (10 year goal).
Project Challenges:
* Design Stability;
* Contractor Performance;
* Development Partner Performance.
Table: Project Performance (then year dollars in millions):
Total Project Cost:
Baseline Est. (FY 2007): $785.5;
Latest (Dec. 2008): $817.0;
Change: 4.0%.
Formulation Cost:
Baseline Est. (FY 2007): $78.0;
Latest (Dec. 2008): $85.8;
Change: 10.0%.
Development Cost:
Baseline Est. (FY 2007): $623.7;
Latest (Dec. 2008): $629.6;
Change: 0.9%.
Operations Cost:
Baseline Est. (FY 2007): $83.8;
Latest (Dec. 2008): $101.6;
Change: 21.2%.
Launch Schedule:
Baseline Est. (FY 2007): 8/2008;
Latest (Dec. 2008): 1/2010;
Change: 17 months.
[End of table]
Project Status:
SDO has experienced significant launch schedule delays. Funding cuts in
fiscal year 2005 caused the project to slip SDO‘s launch date from
April to August 2008. Subsequent test scheduling issues and spacecraft
parts problems caused a further delay until December 2008. A crowded
launch manifest has now forced a 13-month delay to January 2010.
Detailed Project Discussion:
The SDO project reported that its only critical technology”a 4K x 4K
array of charge-coupled devices (CCD) to be used in both the HMI and
AIA instruments”was mature at the project‘s preliminary design review.
The United Kingdom originally led development of the CCD camera
systems, but dropped out of the project before the preliminary design
review. Project officials also stated that SDO was purposefully
designed to use existing technology components, but recognized that
some technologies”such as the Kaband transmitter, high-speed bus, and
high-gain antenna system”required modifications to be used on SDO. For
example, the existing technology for the Ka-band transmitter required a
new design for integration with SDO. Project officials told us that
originally Northrop Grumman was to build the Ka-band transmitter, but
its development was brought in house after contractor performance
issues arose.
SDO‘s design was not stable at the critical design review (CDR).
Following this review, the project experienced nearly a 1,200 percent
increase in the number of releasable drawings expected. Project
officials said only drawings for in-house structures such as propulsion
systems, electronics, instrument ports, the high-gain antenna system,
and the spacecraft were considered at CDR. Drawings for the instruments
were not included and flight drawings were only in draft form at CDR.
Project officials indicated that flight drawings did not need to be
ready so far in advance of the project‘s launch readiness date since
there was enough time to build these components.
SDO also experienced several problems during testing of flight
hardware. The project suffered a technical setback in 2007 when the
thermal vacuum chamber being used to test the high gain antenna
overheated, resulting in the need to completely rebuild the antenna.
Several other risks to the project were identified during testing. For
example, testing identified a part on the spacecraft‘s high-speed bus
that, under certain circumstances, could cause the spacecraft to reset
itself, which could mean failure to meet science data quality and
completeness requirements.
At the time of its critical design review in April 2005, the SDO
project budget was reduced by one third for fiscal year 2005 because of
other funding priorities. As a result, the project underwent a replan
that delayed the project‘s launch readiness date from April 2008 to
August 2008. Subsequent scheduling issues for testing of the AIA
instrument and other spacecraft parts problems caused further delays
and cost increases: the launch date slipped to December 2008 resulting
in a cost increase of $18.1 million. Because of launch manifest issues,
SDO‘s launch date has since slipped to January 2010.
Project Office Comments:
The SDO project office provided technical comments to a draft of this
assessment, which were incorporated as appropriate. The project office
commented that all of the problems they found during testing have been
corrected, and that they believe the SDO design has been relatively
stable and drawing releases occurred as planned. Project officials also
said the project combined technology components in new ways in a new
type of design, but the technologies themselves were not modified. They
reported that SDO has been integrated and tested and is awaiting
launch. Officials said the current delay and resulting cost increase is
due to a crowded launch manifest.
[End of Solar Dynamics Observatory (SDO) section]
Stratospheric Observatory for Infrared Astronomy (SOFIA):
Figure: Photograph of Boeing 747SP aircraft.
Source: SOFIA Program Office.
[End of figure]
SOFIA is a joint project between NASA and the German Space Agency (DLR)
to install a 2.5 meter telescope in a specially modified Boeing 747SP
aircraft. This airborne observatory is designed to provide routine
access to the visual, infrared, farinfrared, and sub-millimeter parts
of the spectrum. Its mission objectives include studying many different
kinds of astronomical objects and phenomena, including star birth and
death; the formation of new solar systems; planets, comets, and
asteroids in our solar system; and black holes at the center of
galaxies. Interchangeable instruments for the observatory are being
developed to allow a range of scientific measurement to be taken by
SOFIA.
Formulation to Implementation:
Formulation start (10/91);
GAO Review (12/08);
Initial Operational Capability (8/09);
Full Operational Capability (9/14).
Project Essentials:
NASA Center Lead: Dryden Flight Research Center;
International Partner: German Space Agency (DLR);
Major Contractors: L3 Communications, MPC Products Corporation,
University Space Research Association;
Projected Operational Capability: August 2009;
Aircraft: Modified 747SP;
Sortie Location: Dryden Flight Research Center, Calif.
Mission Duration: 20 years of science mission flights.
Project Challenges:
* Complexity of Heritage Technology;
* Contractor Performance.
Table: Project Performance (then year dollars in millions):
Total Project Cost:
Baseline Est. (FY 2007): $2954.5;
Latest (Dec. 2008): $2580.8;
Change: -12.6%.
Formulation Cost:
Baseline Est. (FY 2007): $35.0;
Latest (Dec. 2008): $35.0;
Change: 0.0%.
Development Cost:
Baseline Est. (FY 2007): $919.5;
Latest (Dec. 2008): $946.4;
Change: 2.9%.
Operations Cost:
Baseline Est. (FY 2007): $2000.0;
Latest (Dec. 2008): $1599.4;
Change: -20.0%.
Launch Schedule:
Baseline Est. (FY 2007): 12/2013;
Latest (Dec. 2008): 9/2014;
Change: 9 months.
[End of table]
Project Status:
SOFIA plans to have its first science flight in 2009. The SOFIA project
was rebaselined in fiscal year 2007; its development costs have grown
to almost four times its original estimate. The rebaseline sought to
achieve science objectives earlier than previously planned, but
resulted in a 9 month delay in full operational capability. Cost growth
is in part because of challenges with the modification of the aircraft
used as the platform for SOFIA. Project officials said the aircraft
modification proved to be more complex job than anticipated.
Detailed Project Discussion:
We could not assess the technology maturity or the design stability of
the overall project as NASA did not provide information related to the
aircraft modification. Data provided for development of the instruments
that will fly on SOFIA generally indicates a high level of technology
maturity. Many of these technologies have already been used on ground-
based telescopes, and the early instruments are essentially finished and
have been waiting for the observatory to be completed. Similarly, we
could not assess design stability of the instruments since the drawings
were still preliminary at the critical design review.
NASA experienced challenges with the modification of the aircraft used
as the platform for the SOFIA project, which led to significant cost
overruns. Contributing to this challenge, according to project
officials, was the aircraft manufacturer‘s refusal to provide the
blueprints for the 747SP. The plane had to be reverse engineered,
making the modifications more difficult. Project officials also said
that the contractor responsible for the aircraft‘s modification and
integration had limited experience with this type of work and did not
fully understand the statement of work, further contributing to cost
overruns.
The SOFIA project also experienced problems related to the original
prime contractor‘s performance earlier in development. The SOFIA
program manager said the original prime contractor was tasked to lead
the project and NASA would purchase the raw data collected by SOFIA
from the contractor. According to another NASA official, that
contractor had neither the project management experience nor the design-
build expertise necessary for the project”a situation that contributed
to some of the SOFIA project‘s problems. Consequently, NASA brought
overall management of both development and operations of SOFIA in-house
to achieve stronger technical, cost, and schedule controls. Project
management was restructured and operational responsibility now resides
with NASA‘s Dryden Flight Research Center, while NASA‘s Ames Research
Center manages the project‘s science. The original contractor is still
under contract for some science operations and instrument development.
As a result of ongoing cost growth early in development, the SOFIA
project underwent a review in 2006. The project was slated for
cancellation in 2006, and no funds were allocated to it in that fiscal
year. However, later that year, SOFIA was reinstated. In 2007, it was
redesigned and, in July of that year, rebaselined. This new plan sought
to be more responsive to the science community and achieve science
objectives earlier than previously planned by performing science
flights while still maturing the aircraft and telescope, but resulted
in a 9-month delay in full operational capability. SOFIA‘s current
development costs are estimated to be about $950 million, almost four
times the estimated development costs in 1997.
Project Office Comments:
The project office provided technical comments to a draft of this
assessment, which were incorporated as appropriate. The project office
also commented that since its rebaselining in July 2007, the SOFIA
project has not experienced cost or schedule growth.
[End of Stratospheric Observatory for Infrared Astronomy (SOFIA)
section]
Wide-field Infrared Survey Explorer (WISE):
Figure: Wide-field Infrared Survey Explorer (WISE) (artist depiction).
Source: NASA/JPL-Caltech.
[End of figure]
The WISE mission is designed to map the sky in infrared light and
search for the nearest and coolest stars, the origins of stellar and
planetary systems, the most luminous galaxies in the universe, and most
main-belt asteroids larger than 3 kilometers. It is also intended to
create a catalog of over 300 million sources that will be of interest
to future infrared studies, including the upcoming James Webb Space
Telescope mission. During its 6-month mission, WISE will use a four-
channel imager to take overlapping snapshots of the sky. The WISE
telescope optics will be cooled below 20 degrees Kelvin to keep it
colder than the objects in space it will observe so that WISE can see
the dim infrared emission from them rather than from the telescope
itself.
Formulation to Implementation:
Formulation start (3/03);
Preliminary Design Review (7/05);
Critical Design Review (6/07);
GAO Review (12/08);
Launch Readiness (11/09).
Project Essentials:
NASA Center Lead: Jet Propulsion Laboratory;
International Partner: None;
Major Contractors: Ball Aerospace and Technologies Corporation, Space
Dynamics Laboratory;
Projected Launch Date: November 1, 2009;
Launch Location: Vandenberg AFB, Calif.;
Launch Vehicle: Delta II;
Mission Duration: 6 months.
Project Challenges:
* Design Stability.
Table: Project Performance (then year dollars in millions):
Total Project Cost:
Baseline Est. (FY 2008): $311.4;
Latest (Dec. 2008): $310.5;
Change: -0.3%.
Formulation Cost:
Baseline Est. (FY 2008): $99.5;
Latest (Dec. 2008): $99.6;
Change: 0.1%.
Development Cost:
Baseline Est. (FY 2008): $192.1;
Latest (Dec. 2008): $190.9;
Change: -0.6%.
Operations Cost:
Baseline Est. (FY 2008): $19.8;
Latest (Dec. 2008): $20.0;
Change: 1.0%.
Launch Schedule:
Baseline Est. (FY 2008): 11/2009;
Latest (Dec. 2008): 1/2009;
Change: none.
[End of table]
Project Status:
The WISE project is currently on schedule to meet its November 2009
launch date. However, the failure of a structural model of the flight
cryostat during vibration testing prompted a design change to add a
soft-ride system to the launch vehicle, a solution that cost about $2.6
million. This failure has caused the project to descope some testing in
order to regain lost cost and schedule margin.
Detailed Project Discussion:
Though the project is currently on track to meet its launch readiness
date, the WISE project encountered schedule delays early in its life
cycle. According to a project official, the project was not initially
confirmed to proceed because of cost and technical concerns. As a
result, the official said the project designed a smaller telescope and
matured the technology that had concerned the review board. The
preliminary design review for WISE was held in July 2005, and the
project had its initial confirmation review in November 2005; however,
there was a lack of funding in the NASA budget for the WISE project at
that time so the formulation phase was extended. At this point in the
project, the launch readiness date had slipped from 2008 to June 2009.
A second confirmation review was held in October 2006, at which time
the launch readiness date was set for October 2009. Although the second
confirmation review happened one year later, the launch readiness date
set at the original confirmation review only slipped 4 months since,
according to a project official, the project was able to make progress
during that year.
WISE project officials identified two mission critical technologies---
the solid hydrogen cryostat and the long wavelength infrared detector
multiplexer---both of which were assessed as mature at the project‘s
preliminary design review. The solid hydrogen cryostat is a
modification of a heritage technology. It is of similar design and
construction and manufactured by the same contractor that produced
cryostats for previous NASA missions. A project official said the
project did not encounter any challenges with the development of the
cryostat. WISE‘s design, however, was not stable at the project‘s
critical design review. At the time of that review, the project had
released only 70 percent of its engineering drawings. A project
official stated that the drawing count and additional analyses,
prototypes, and engineering models were used at the critical design
review to evaluate the project‘s design stability. The project has
since released the remainder of the engineering drawings.
The project did encounter some challenges during testing, which
impacted the spacecraft‘s design. The thermal-mass-dynamics-simulator,
a structural model of the flight cryostat, failed during structural
testing. According to a NASA official, analyses done by NASA and the
cryostat‘s contractor did not predict this problem. To mitigate this
problem, the project added a soft-ride system to the launch vehicle to
reduce loads on the cryostat. The failure also caused the project to
accept more project risk by de-scoping two test events in order to
regain reserve margin. According to the project office, the remedy cost
$2.6 million, but the overall project schedule was not affected.
Project Office Comments:
The WISE project office provided technical comments to a draft of this
assessment, which were incorporated as appropriate. Project officials
also commented that they believe that development of a complex
cryogenic instrument from heritage technology was more challenging to
the project than its design stability.
[End of Wide-field Infrared Survey Explorer (WISE) section]
Agency Comments and Our Evaluation:
We provided a draft of this report to NASA for review and comment. In
written comments, NASA recognizes that its goal is to improve its cost
estimating and schedule and indicates that it will work hard to improve
its performance.
The actions NASA has taken to address our past recommendations are
positive steps toward achieving successful project outcomes and ensuring
that decision makers are appropriately investing the agency‘s resources.
However, NASA asserts that its projects are typically high risk, one-of-
a-kind missions that do not readily fit into the knowledge-based
framework associated with best practices in system acquisition. NASA‘s
own studies and those of others have shown that the challenges
discussed in this report, as well as other project management
challenges, have plagued the agency for decades. Given the fact that
most of the projects we reviewed in this study breached congressional
thresholds within a 2- to 3-year period, we remain convinced that NASA
would benefit from a more disciplined, knowledge-based approach to its
acquisitions.
NASA sought to provide clarification and additional context to the
information we provided in our observations. The agency indicated that
the growth we reported for the 10 projects in implementation was a
forward-looking estimate, rather than actual growth. For this review,
NASA provided us baseline cost and schedule estimates for most projects
and then provided us updated estimates for those same projects. We
assume that the estimates NASA provided are projections based on costs
incurred and schedule completed to date, as well as realistic
assumptions about future costs and schedule plans.
NASA also stated that cost and schedule growth for some projects was due
to factors outside of the agency‘s control. Specifically:
1. Two NASA missions”the Lunar Reconnaissance Orbiter and Solar
Dynamics Observatory”experienced delays to their launch dates due
to U.S. launch manifest prioritization. While NASA maintains that the
launch slips for LRO and SDO were beyond its control, we believe
that greater discipline in these and other acquisitions can still
alleviate the impact of these factors. Specifically, given the launch
manifest constraints that the agency is and has been experiencing, it
would be prudent to adequately plan for such launch delays when
determining cost and schedule reserves.
2. NASA believes that Aquarius, NPP, and Herschel projects experienced
cost growth and schedule delays due to partner performance beyond its
control. We believe that having the sufficient amount of insight into
the partner‘s activities and schedules may have allowed NASA to become
aware of the issues earlier and to actively manage the issues throughout
the development process.
3. NASA stated in its comments that not all cost growth is reported
from the time of the NASA commitment to Congress for the performance,
cost and schedule of its projects, as is the case with the James Webb
Space Telescope. This project was just confirmed in the fall of 2008.
Nonetheless, NASA provided GAO data for projects as late as December
2008. Since NASA develops baseline estimates for its projects at the
confirmation review that are formal commitments, we would have expected
NASA to report that data to us in December 2008.
4. NASA stated that it underestimates the complexity of developing
first-of-a-kind missions. While we recognize the nature of NASA
projects, as stated in our report, we remain convinced that a knowledge-
based approach will allow the agency to better plan for and address
these complexities.
We are pleased that NASA recognizes our desire to assist the agency in
improving its cost and schedule estimating and look forward to
continuing to work with it to improve performance in these areas.
NASA‘s comments are reprinted in appendix I. NASA also provided
technical comments, which we addressed throughout the report as
appropriate and where sufficient evidence was provided to support
significant changes.
We will send copies of the report to NASA's Administrator and
interested congressional committees. We will also make copies available
to others upon request. In addition, the report will be available at no
charge on GAO's Web site at [hyperlink, http://www.gao.gov].
Should you or your staff have any questions on matters discussed in
this report, please contact me at (202) 512-4841 or chaplainc@gao.gov.
Contact points for our Offices of Congressional Relations and Public
Affairs may be found on the last page of this report. GAO staff who
made major contributions to this report are listed in appendix IV.
Sincerely yours,
Signed by:
Cristina Chaplain:
Director:
Acquisition and Sourcing Management:
List of Congressional Committees:
The Honorable Barbara A. Mikulski:
Chairman:
The Honorable Richard C. Shelby:
Ranking Member:
Subcommittee on Commerce, Justice, Science, and Related Agencies:
Committee on Appropriations:
United States Senate:
The Honorable Alan B. Mollohan:
Chairman:
The Honorable Frank R. Wolf:
Ranking Member:
Subcommittee on Commerce, Justice, Science, and Related Agencies:
Committee on Appropriations:
House of Representatives:
The Honorable Gabrielle Giffords:
Chairwoman:
The Honorable Pete Olson:
Ranking Member:
Subcommittee on Space and Aeronautics:
Committee on Science and Technology:
House of Representatives:
[End of section]
Appendix I: Comments from the National Aeronautics and Space
Administration:
National Aeronautics and Space Administration:
Office of the Administrator:
Washington, DC 20546-0001:
February 24, 2009:
Ms. Christina Chaplain:
Director:
Acquisition and Sourcing Management:
United States Government Accountability Office:
Washington, DC 20548:
Dear Ms. Chaplain:
NASA appreciates the opportunity to comment on the draft Government
Accountability Office (GAO) report, GAO-09-306, entitled "Assessments
of Selected Large-Scale Projects." We are pleased that GAO recognizes
NASA's efforts and the hard work of many people at NASA to mitigate
acquisition management risk and lay a foundation to reduce project cost
and schedule growth. NASA values the open and constructive
communications between the NASA and GAO teams on this effort. NASA is
dedicated to continuous improvement of its acquisition management
processes and performance.
NASA has implemented improvement initiatives enumerated in the NASA
response to the 2007 GAO High Risk Area of Contract Management.
Specifically NASA has adopted a new acquisition strategy policy which
improves its ability to manage performance risks (including adoption of
probabilistic cost and schedule estimation methods); and has
established a rigorous monthly internal Baseline Performance Review
(BPR) to track program and institutional performance and identify
corrective actions to allow performance to plan. These actions have
allowed us to more clearly identify the causes of deviations from plan
that can also be used by the GAO in future reporting.
While NASA practices many of the elements of GAO's stated business case
approach, where applicable to the Agency's investment model, the
essential attributes of NASA's project development differ from those of
a commercial or production entity. NASA projects are typically high
risk one-of-a-kind missions that require a knowledge-based approach
where experience from previous missions are modified and adapted to the
new mission. NASA's intent is to advance selected technologies and
techniques to a level of maturity that would provide an acceptable
balance of investment risk and return. This approach requires projects
to develop a plan that meets system requirements within cost and
schedule constraints and with acceptable risk by the confirmation
review, called Key Decision Point C. Senior management reviews and
assesses each project at this key decision point to determine readiness
for the project to proceed into the implementation phase and detailed
design. As mentioned above, NASA management reviews projects monthly at
the BPR to assess progress against cost, schedule, and technical
commitments. This approach is outlined in NASA's policy directive and
procedural requirements related to program/project management.
In its draft report, GAO asserts that NASA has had significant cost
and/or schedule growth from the baseline estimate in 10 of the 13
projects on which NASA provided data, using GAO's definition of
significant as "greater than the thresholds established for
Congressional reporting." Among significant points of clarification
that were not provided in the report is that the growth reported was a
forward-looking estimate that included anticipated growth through
launch, as well as actual growth to date. Based on this assumption, the
projected growth indicated in the report is possible, but not yet the
actual increase as GAO states.
Other GAO-quoted figures also require some additional context to
provide the reader with an accurate view of the cost and schedule
growth. NASA notes that the GAO report does not distinguish between
factors that result in this growth and whether they are internal or
external to NASA. Recognition of the factors that NASA controls, as
well as those that are outside the control of the Agency, would best
allow us to focus on those factors which NASA can address in order to
improve the Agency's acquisition performance. Out of the 10 projects
that exceeded Congressional growth thresholds, approximately half did
so as a result of external factors. Further, it is also important to
link specific instances of cost or schedule growth to factors that
contributed to that growth. Some examples from the draft would include:
1. Slips in the launch dates for the Lunar Reconnaissance Orbiter (LRO)
and the Solar Dynamics Observatory (SDO) are a result of delays in the
U.S. launch manifest prioritization that rightly puts national security
missions first. The launch manifest, while actively managed, is beyond
the control of NASA.
2. Cost increases and delays to missions such as NPOESS Preparatory
Project (NPP), Aquarius and Herschel are due to partner performance
beyond the control of NASA.
3. Not all cost growth is reported from the time of the NASA commitment
to Congress for the performance, cost, and schedule. This is the case
for JWST which was just confirmed in the fall of 2008.
4. One of the most prevalent internal issues is underestimating the
complexity of the development of first-of-a-kind missions, such as with
MSL. In response to past history of these issues, NASA has adopted
formal program reviews at key milestones and now includes probabilistic
cost-estimating techniques to help address these uncertainties.
To ensure that NASA and GAO share a common data set for presentation to
the Congress and the public, we have provided separate technical
comments addressing the accuracy and representation of data included in
several of the Project Summary pages of the report.
NASA recognizes that the goal is to improve our performance in
estimating cost and schedule so as to enhance our ability to explore
and utilize space for the benefit of the Nation and the world. We are
committed to continuous improvement and will work hard to continuously
measure and improve our performance. To this end we welcome the
comments from GAO regarding our performance.
Thank you for the opportunity to comment on this draft report. If you
have any questions or require additional information, please contact
Julie Pollitt at (202) 358-1580.
Sincerely,
Signed by:
Charles H. Scales:
Associate Deputy Administrator:
[End of section]
Appendix II: NASA Life-Cycle For Flight Systems Compared To A Knowledge-
Based Approach:
GAO has previously conducted work on NASA's acquisition policy for
space-flight systems, and in particular, on its alignment with a
knowledge-based approach to system acquisitions. The figure below
depicts this alignment.
Figure 1: NASA's life-cycle for flight systems compared to a knowledge-
based approach:
[Refer to PDF for image: illustration]
NASA‘s life cycle for flight systems and ground support projects:
Formulation:
Pre-phase A: Concept studies;
KDP A;
Phase A: Concept development (Pre-NAR; SDR);
KDP B;
Phase B: Preliminary design and technology completion (NAR; PDR);
KDP C;
Program start:
Implementation:
Phase C: Final design and fabrication (CDR);
KDP D;
Phase D: System assembly, integration and test, launch;
KDP E;
Phase E: Operations and sustainment;
KDP F;
Phase F: Closeout.
Knowledge-based approach:
Formulation:
Concept and technology development;
Knowledge point 1 (KP1): Technologies, time, funding, and other
resources match customer needs;
Program start:
Implementation:
Product development:
- Integration;
Knowledge point 2 (KP2): Design performs as expected;
- Demonstration;
Knowledge point 3 (KP3): Production meets cost, schedule, and quality
targets.
Production:
Key:
Management decision reviews:
Pre-NAR = preliminary non-advocate review;
NAR = non-advocate review;
KDP = key decision point.
Technical reviews:
SDR = system definition review;
PDR = preliminary design review;
CDR = critical design review.
Source: NASA data and GAO analysis.
[End of figure]
As the figure shows, NASA's policy defines a project life-cycle in two
phases--the formulation[Footnote 14] and implementation[Footnote 15]
phases, which are further divided into incremental pieces: Phase A
through Phase F. Project formulation consists of Phases A and B, during
which time the projects develop and define the project requirements and
cost/schedule basis and design for implementation, including an
acquisition strategy. During the end of the formulation phase, leading
up to the preliminary design review (PDR)[Footnote 16] and Non-advocate
review (NAR)[Footnote 17], the project team completes its preliminary
design and technology development. NASA Procedural Requirements
7120.5D, NASA Space Flight Program and Project Management Requirements,
specify that the project complete development of mission-critical or
enabling technology, as needed, with demonstrated evidence of required
technology qualification (i.e., component and/or breadboard validation
in the relevant environment) documented in a technology readiness
assessment report. The project must also develop, document, and
maintain a project integrated baseline which includes the integrated
master schedule and baseline life-cycle cost estimate. Implementing
these requirements brings the project closer to ensuring that resources
and needs match, but it is not fully consistent with Knowledge point 1
of the knowledge-based acquisition life-cycle. Our best practices show
that demonstrating technology maturity at this point in the system life-
cycle should include a system or subsystem model or prototype
demonstration in a relevant environment, not only component validation.
As written, NASA's policy does not require full technology maturity
before a project enters the implementation phase.
After project confirmation, the project begins implementation,
consisting of Phases C, D, E, and F. During phases C and D, the project
performs final design and fabrication as well as testing of components
and system assembly, integration, test, and launch. Phases E and F
consist of operations and sustainment and project closeout. A second
design review, the critical design review (CDR),[Footnote 18] is held
during the implementation phase toward the end of Phase C. The purpose
of the CDR is to demonstrate that the maturity of the design is
appropriate to support proceeding with full scale fabrication,
assembly, integration, and test. Though this review is not a formal
decision review, its requirements for a mature design and ability to
meet mission performance requirements within the identified cost and
schedule constraints are similar to knowledge expected at Knowledge
point 2 of the knowledge-based acquisition life-cycle. Furthermore,
after CDR, the project must be approved at KDP D before continuing into
the next phase.
The NASA acquisition life-cycle lacks a major decision review at
Knowledge Point 3 to demonstrate that production processes are mature.
According to NASA officials, the agency rarely enters a formal
production phase due to the small quantities of space systems that they
build.
[End of section]
Appendix III: Objectives, Scope And Methodology:
Our objectives were to report on the status and challenges faced by
several NASA systems with life-cycle costs greater than $250 million
and to discuss broader trends faced by the agency in its management of
system acquisitions.
In conducting our work, we evaluated performance and identified
challenges for each of 18 major projects[Footnote 19] included in this
report. We summarized our assessments of each individual project in two
components--a project profile and a detailed discussion of project
challenges. We did not validate the data provided by the National
Aeronautics and Space Administration (NASA). However, we took
appropriate steps to address data reliability. Specifically, we
confirmed the accuracy of NASA-generated data with multiple sources
within NASA and, in some cases, with external sources. Additionally, we
corroborated data provided to us with published documentation. We
determined that the data provided by NASA project offices were
sufficiently reliable for our engagement purposes.
We developed a standardized data collection instrument (DCI) that was
completed by each project office and returned by December 2008. Through
the DCI, we gathered basic information about projects as well as
current and projected development activities for those projects. The
cost, schedule and performance data estimates that NASA inputted were
the most recent updates as of December 2008. At the time we collected
the data, 4 of the 18 projects were in formulation and 14 were in
implementation. However, NASA only provided cost and schedule data for
13 of the projects. To further understand performance issues, we talked
with officials from each project office and NASA's Office of Program
Analysis and Evaluation (PA&E).
The results collected from each project office and PA&E were summarized
in a two-page report format providing a project overview; key cost,
contract, and schedule data; and a discussion of the challenges
associated with the deviation from relevant indicators from best
practice standards. The aggregate measures and averages calculated were
analyzed for meaningful relationships, e.g. relationship between cost
growth and schedule slippage and knowledge maturity attained both at
critical milestones and through the various stages of the project life-
cycle. We identified cost and/or schedule growth as significant where,
in either case, a project's cost and/or its schedule exceeded the
thresholds for the Congressional reporting requirement.
To supplement our analysis, we relied on GAO's body of work over the
past years that has examined acquisition issues across multiple
agencies. These reports cover such issues as contracting, program
management, acquisition policy, and cost estimating. GAO also has an
extensive body of work related to challenges NASA has faced with regard
to specific system acquisitions, financial management, and cost
estimating. This work provided the context and basis for much of the
general observations we made with regard to the projects we reviewed.
Additionally, the discussions with the individual NASA projects helped
us identify further challenges faced by the projects. Together, this
contributed to our development of a short list of challenges discussed
for each project. The challenges we identified and discussed do not
represent an exhaustive or exclusive list. They are subject to change
and evolution as GAO continues this annual assessment in future years.
Our work was performed primarily at NASA headquarters in Washington,
D.C. In addition, we visited NASA's Marshall Space Flight Center in
Huntsville, Alabama, Dryden Flight Research Center at Edwards Air Force
Base in California, and Goddard Space Flight Center in Greenbelt,
Maryland to discuss individual projects. We also met with
representatives from NASA's Jet Propulsion Lab in Pasadena, California
and three NASA suppliers.
Data Limitations:
NASA only provided specific cost and schedule estimates for 13 of the
18 projects in our review. Agency officials believe that because one
project, the James Webb Space Telescope, will not formally release its
baseline cost and schedule estimates until the fiscal year 2010 budget
submission to Congress, they are not required to provide those
estimates to GAO. For those projects that had not yet entered
implementation, NASA provided internal preliminary estimated total
(life cycle) cost ranges and associated schedules, from key decision
point B (KDP-B), solely for informational purposes.[Footnote 20] NASA
formally baselines and commits itself to cost and schedule targets for
a project with a specific and aligned set of planned mission objectives
at key decision point C (KDP-C), which follows a Non-Advocate Review
(NAR) and Preliminary Design Review (PDR). KDP-C reflects the life
cycle point where NASA approves a project to leave the formulation
phase and enter into the implementation phase. NASA explained that
preliminary estimates are generated for internal planning and fiscal
year budgeting purposes at KDP-B, which occurs mid-stream in the
formulation phase, and hence, are not considered a formal commitment by
the Agency on cost and schedule for the mission deliverables. NASA
officials contend that because of changes that occur to a project's
scope and technologies between KDP-B and KDP-C, estimates of project
cost and schedule can change significantly heading toward KDP-C.
Finally, NASA did not provide data for the Global Precipitation
Measurement Mission because NASA officials said that the agency waived
the requirement for a KDP-B review for that project until it reaches
KDP-C.
Project Profile Information on Each Individual Two-Page Assessment:
This section of the two-page assessment outlines the essentials of the
project, its cost and schedule performance and its status. Project
essentials reflect pertinent information about each project, including,
where applicable, the major contractors and partners involved in the
project. These organizations have primary responsibility over a major
segment of the project, or in some cases, the entire project.
Project performance is depicted according to cost and schedule changes
in the various stages of the project life-cycle. To assess the cost and
schedule changes of each project we obtained data directly from NASA
PA&E and from NASA's Integrated Budget and Performance documents. For
systems in implementation, we compared the latest available information
with baseline cost and schedule estimates set for each project in the
fiscal year 2007 or 2008 budget request.
All cost information is presented in nominal "then year" dollars for
consistency with budget data.[Footnote 21] Baseline costs are adjusted
to reflect the cost accounting structure in NASA's fiscal year 2009
budget estimates. For the fiscal year 2009 budget request, NASA changed
its accounting practices from full-cost accounting to reporting only
direct costs at the project level. Projects in implementation at the
time of the initial baseline requirement for the fiscal year 2007
budget request were grandfathered in using the cost estimate at the
time of the fiscal year 2007 budget request. Any project that did not
have a life-cycle cost of $250 million in fiscal year 2007, but later
increased beyond that threshold, was baselined at the next budget
request. All major projects entering implementation after the fiscal
year 2007 budget request have a baseline that reflects the project's
life-cycle cost estimate upon entering implementation. For projects in
implementation, the budget table provides a life-cycle cost estimate
broken out by phases.
The schedule assessment is based on acquisition cycle time, which is
defined as the number of months between the project start, or
formulation start, and projected or actual launch date.[Footnote 22]
Formulation start generally refers to the initiation of a project; NASA
refers to project start as Key Decision Point A, or the beginning of
the formulation phase. The preliminary design review typically occurs
during the end of the formulation phase, followed by a confirmation
review which allows the project to move into the implementation phase.
The critical design review is held during the final design of
implementation and demonstrates that the maturity of the design is
appropriate to support proceeding with full scale fabrication,
assembly, integration, and test. Launch readiness is determined through
a launch readiness review which verifies that the launch system and
spacecraft/payloads are ready for launch. The implementation phase
includes the operations of the mission and concludes with project
disposal.
We assessed the extent to which NASA projects exceeded their cost and
schedule baselines. To do this, we compared the project baseline cost
and schedule estimates with the current cost and schedule data reported
by the project office in December 2008.
Project Challenges Discussion on Each Individual Two-Page Assessment:
To assess the project challenges for each project, we submitted a data
collection instrument to each project office. We also held interviews
with each of the project offices to discuss the information on the data
collection instrument. These discussions led to identification of
further challenges faced by NASA projects. These challenges were
largely apparent in the projects that had entered the implementation
phase. We then reviewed pertinent project documentation, such as the
project plan, schedule, risk assessments, and major project reviews.
To assess technology maturity, we asked project officials to assess the
technology readiness levels (TRL) of each of the project's critical
technologies at various stages of project development. Originally
developed by NASA, TRLs are measured on a scale of one to nine,
beginning with paper studies of a technology's feasibility and
culminating with a technology fully integrated into a completed
product. (See appendix IV for the definitions of technology readiness
levels.) In most cases, we did not validate the project offices'
selection of critical technologies or the determination of the
demonstrated level of maturity. However, we sought to clarify the
technology readiness levels in those cases where the information
provided raised concerns, such as where a critical technology was
reported as immature late in the project development cycle.
Additionally, we asked project officials to explain the environments in
which technologies were tested.
Our best practices work has shown that a technology readiness level of
6--demonstrating a technology as a fully integrated prototype in a
realistic environment--is the level of maturity needed to minimize
risks for space systems entering product development. In our
assessment, the technologies that have reached technology readiness
level 6 are referred to as fully mature due to the difficulty of
achieving technology readiness level 7, which is demonstrating maturity
in an operational environment--space. Projects with critical
technologies that did not achieve maturity by the preliminary design
review were assessed as having a technology maturity project challenge.
We did not assess technology maturity for those projects which had not
yet reached the preliminary design review at the time of this
assessment.[Footnote 23]
To assess design stability, we asked project officials to provide the
percentage of engineering drawings completed or projected for
completion by the preliminary and critical design reviews and as of our
current assessment.[Footnote 24] In most cases, we did not verify or
validate the percentage of engineering drawings provide by the project
office. However, we collected the project offices' rationale for cases
where it appeared that only a small number of drawings were completed
by the time of the design reviews or where the project office reported
significant growth in the number of drawings released after CDR. In
accordance with GAO best practices, projects were assessed as having
achieved design stability if they had released at least 90 percent of
all projected drawings by the critical design review. Projects which
had not met this metric were determined to have a design stability
project challenge. Though some projects used other methods to assess
design stability, such as computer and engineering models and analyses,
we did not analyze the use of these other methods and therefore could
not assess the design stability of those projects. We could not assess
design stability for those projects which had not yet reached the
critical design review at the time of this assessment.
To assess the complexity of heritage technology, we interviewed project
officials about the use of heritage technologies in their projects.
[Footnote 25] We asked them what heritage technologies were being used,
what effort was needed to modify the form, fit, and function of the
technology for use in the new system, and whether the project
encountered any problems in modifying the technology. Heritage
technologies were not considered critical technologies by several of
the projects we reviewed. Based on our interviews, and review of cost
and schedule data from the data collection instruments, and previous
GAO work on space systems, we determined whether complexity of heritage
technology was a challenge for a particular project.
To assess whether projects encountered challenges with contractor
performance, we interviewed project officials about their interaction
and experience with contractors. We also interviewed contractor
officials from Orbital Sciences Corporation, Ball Aerospace and
Technologies Corporation, and Raytheon Space Systems about their
experiences contracting with NASA. We were informed about contractor
performance problems pertaining to their workforce, the supplier base,
and technical and corporate experience. We also discussed contract fees
and situations in which NASA and a contractor agreed that the
contractor would use their award fee to cover project cost overruns. We
assessed a project as having this challenge if these contractor
performance problems, as confirmed by NASA and, where possible, the
project contractor, caused the project to experience a cost overrun,
schedule delay, or decrease in mission capability. For projects which
did not have a major contractor, we considered this challenge not
applicable to the project.
To assess whether projects encountered challenges with development
partner performance, we interviewed NASA project officials about their
interaction with international or domestic partners during project
development. Development partner performance was considered a challenge
for the project if project officials indicated that domestic or foreign
partners were experiencing problems with project development that
impacted the cost, schedule, or performance of the project for NASA.
These challenges were specific to the partner organization or caused by
a contractor to that partner organization. For projects which did not
have an international or domestic development partner, we considered
this challenge not applicable to the project.
The individual project offices were given an opportunity to comment on
and provide technical clarifications to the two-page assessments prior
to their inclusion in the final product.
We conducted this performance audit from February 2008 to March 2009 in
accordance with generally accepted government auditing standards. Those
standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.
[End of section]
Appendix IV: Technology Readiness Levels:
Technology readiness level: 1. Basic principles observed and reported;
Description: Lowest level of technology readiness. Scientific research
begins to be translated into applied research and development. Examples
might include paper studies of a technology's basic properties;
Hardware: None (paper studies and analysis);
Demonstration Environment: None.
Technology readiness level: 2. Technology concept and/or application
formulated;
Description: Invention begins. Once basic principles are observed,
practical applications can be invented. The application is speculative
and there is no proof or detailed analysis to support the assumption.
Examples are still limited to paper studies;
Hardware: None (paper studies and analysis);
Demonstration Environment: None.
Technology readiness level: 3. Analytical and experimental critical
function and/or characteristic proof of concept;
Description: Active research and development is initiated. This
includes analytical studies and laboratory studies to physically
validate analytical predictions of separate elements of the technology.
Examples include components that are not yet integrated or
representative;
Hardware: Analytical studies and demonstration of nonscale individual
components (pieces of subsystem);
Demonstration Environment: Lab.
Technology readiness level: 4. Component and/or breadboard; Validation
in laboratory environment;
Description: Basic technological components are integrated to establish
that the pieces will work together. This is relatively "low fidelity"
compared to the eventual system. Examples include integration of "ad
hoc" hardware in a laboratory;
Hardware: Low fidelity breadboard; Integration of nonscale components
to show pieces will work together. Not fully functional or form or fit
but representative of technically feasible approach suitable for flight
articles;
Demonstration Environment: Lab.
Technology readiness level: 5. Component and/or breadboard validation
in relevant environment;
Description: Fidelity of breadboard technology increases significantly.
The basic technological components are integrated with reasonably
realistic supporting elements so that the technology can be tested in a
simulated environment. Examples include "high fidelity" laboratory
integration of components;
Hardware: High fidelity breadboard; Functionally equivalent but not
necessarily form and/or fit (size weight, materials, etc). Should be
approaching appropriate scale. May include integration of several
components with reasonably realistic support elements/subsystems to
demonstrate functionality;
Demonstration Environment: Lab demonstrating functionality but not form
and fit. May include flight demonstrating breadboard in surrogate
aircraft. Technology ready for detailed design studies.
Technology readiness level: 6. System/subsystem model or prototype
demonstration in a relevant environment;
Description: Representative model or prototype system, which is well
beyond the breadboard tested for TRL 5, is tested in a relevant
environment. Represents a major step up in a technology's demonstrated
readiness. Examples include testing a prototype in a high fidelity
laboratory environment or in simulated realistic environment;
Hardware: Prototype; Should be very close to form, fit and function.
Probably includes the integration of many new components and realistic
supporting elements/subsystems if needed to demonstrate full
functionality of the subsystem;
Demonstration Environment: High-fidelity lab demonstration or
limited/restricted flight demonstration for a relevant environment.
Integration of technology is well defined.
Technology readiness level: 7. System prototype demonstration in an
realistic environment;
Description: Prototype near or at planned operational system.
Represents a major step up from TRL 6, requiring the demonstration of
an actual system prototype in a realistic environment, such as in an
aircraft, vehicle or space. Examples include testing the prototype in a
test bed aircraft;
Hardware: Prototype. Should be form, fit and function integrated with
other key supporting elements/subsystems to demonstrate full
functionality of subsystem;
Demonstration Environment: Flight demonstration in representative
realistic environment such as flying test bed or demonstrator aircraft;
Technology is well substantiated with test data.
Technology readiness level: 8. Actual system completed and "flight
qualified" through test and demonstration;
Description: Technology has been proven to work in its final form and
under expected conditions. In almost all cases, this TRL represents the
end of true system development. Examples include developmental test and
evaluation of the system in its intended weapon system to determine if
it meets design specifications;
Hardware: Flight qualified hardware;
Demonstration Environment: Developmental Test and Evaluation (DT&E) in
the actual system application.
Technology readiness level: 9. Actual system "flight proven" through
successful mission operations;
Description: Actual application of the technology in its final form and
under mission conditions, such as those encountered in operational test
and evaluation. In almost all cases, this is the end of the last "bug
fixing" aspects of true system development. Examples include using the
system under operational mission conditions;
Hardware: Actual system in final form;
Demonstration Environment: Operational Test and Evaluation (OT&E) in
operational mission conditions.
Source: GAO and its analysis of National Aeronautics and Space
Administration data.
[End of table]
[End of section]
Appendix V: GAO Contact and Staff Acknowledgments:
GAO Contact:
Cristina Chaplain (202) 512-4841 or chaplainc@gao.gov:
Acknowledgments:
In addition to the contact named above, Jim Morrison, Assistant
Director; Greg Campbell; Richard A. Cederholm; Brendan S. Culley; Neil
D. Feldman; Leon S. Gill; Rachel L. Girshick; Kristine R. Heuwinkel;
Deanna R. Laufer; Shelby S. Oakley; Kenneth E. Patton; Sylvia Schatz;
and Letisha T. Watson made key contributions to this report.
[End of section]
Footnotes:
[1] National Aeronautics and Space Administration Authorization Act of
2005, Pub. L. No. 109-161, §103; 42 U.S.C. §16613(b), (f) (4).
[2] 42 U.S.C. §16613(d)
[3] NASA also provided preliminary estimates in the form of cost ranges
for three projects in the formulation phase. Since the values provided
were ranges, rather than specific values, we did not include these
projects in our analysis. Further, the agency did not provide schedule
baselines for these projects so we could not determine any schedule
changes they experienced.
[4] GAO. Defense Acquisitions: Key Decisions to Be Made on Future
Combat System, [hyperlink, http://www.gao.gov/products/GAO-07-376]
(Washington, D.C.: March 15, 2007); GAO. Defense Acquisitions: Improved
Business Case Key for Future Combat System's Success, [hyperlink,
http://www.gao.gov/products/GAO-06-564T] (Washington, D.C.: April 4,
2006); GAO. NASA: Implementing a Knowledge-Based Acquisition Framework
Could Lead to Better Investment Decisions and Project Outcomes,
[hyperlink, http://www.gao.gov/products/GAO-06-218] (Washington, D.C.:
December 21, 2005); GAO. NASA's Space Vision: Business Case for
Prometheus 1 Needed to Ensure Requirements Match Available Resources,
[hyperlink, http://www.gao.gov/products/GAO-05-242] (Washington, D.C.:
February 28, 2005).
[5] [hyperlink, http://www.gao.gov/products/GAO-05-242].
[6] National Aeronautics and Space Administration Procedural
Requirements 7120.5D, NASA Spaceflight Program and Project Management
Requirements (Mar. 6, 2007). (Hereinafter cited as NPR 7120.5D (Mar. 6,
2007).
[7] NPR 7120.5D, paragraph 2.4.5 (Mar. 6, 2007).
[8] GAO. High-Risk Series: An Update, [hyperlink,
http://www.gao.gov/products/GAO-07-310] (Washington, D.C.: January
2007).
[9] NASA. Plan for Improvement in the GAO High-Risk Area of Contract
Management (October 31, 2007).
[10] We also reviewed the James Webb Space Telescope, but NASA did not
provide cost or schedule data for that project even though it is in
implementation.
[11] For purposes of our analysis, significant cost and schedule growth
occurs when a project's cost and/or its schedule growth exceeds the
thresholds established for Congressional reporting.
[12] 42 U.S.C. § 16613(b).
[13] Appendix IV provides a description of the metrics used to assess
technology maturity in this review.
[14] NASA defines formulation as the identification of how the program
or project supports the Agency's strategic needs, goals, and
objectives; the assessment of feasibility, technology and concepts;
risk assessment, team building, development of operations concepts and
acquisition strategies; establishment of high-level requirements and
success criteria; the preparation of plans, budgets, and schedules
essential to the success of a program or project; and the establishment
of control systems to ensure performance to those plans and alignment
with current Agency strategies.
[15] The implementation phase is defined as the execution of approved
plans for the development and operation of the program/project, and the
use of control systems to ensure performance to approved plans and
continued alignment with the Agency's strategic needs, goals, and
objectives. NPR 7120.5D, paragraph 1.2.1 c. (Mar. 6, 2007).
[16] According to NPR 7120.5D, Table 2-6 (Mar. 6, 2007), the PDR
demonstrates that the preliminary design meets all system requirements
with acceptable risk and within the cost and schedule constraints and
establishes the basis for proceeding with detailed design. It shows
that the correct design option has been selected, interfaces have been
identified, and verification methods have been described. Full baseline
cost and schedules, as well as risk assessments, management systems,
and metrics are presented.
[17] According to NPR 7120.5D, Table 2-6 (Mar. 6, 2007), the PDR
demonstrates that the preliminary design meets all system requirements
with acceptable risk and within the cost and schedule constraints and
establishes the basis for proceeding with detailed design. It shows
that the correct design option has been selected, interfaces have been
identified, and verification methods have been described. Full baseline
cost and schedules, as well as risk assessments, management systems,
and metrics are presented.
[18] According to NPR 7120.5D, Appendix A (Mar. 6, 2007), a Non-
Advocate Review (NAR) is comprised of the analysis of a proposed
program or project by a (non-advocate) team composed of management,
technical, and resources experts (personnel) from outside the advocacy
chain of the proposed program or project. It provides Agency management
with an independent assessment of the readiness of the program/project
to proceed into implementation.
[19] According to NPR 7120.5D, Table 2-6 (Mar. 6, 2007), the CDR
demonstrates that the maturity of the design is appropriate to support
proceeding with full scale fabrication, assembly, integration, and
test, and that the technical effort is on track to complete the flight
and ground system development and mission operations in order to meet
mission performance requirements within the identified cost and
schedule constraints. Progress against management plans, budget, and
schedule, as well as risk assessments are presented.
[20] These missions include: Ares I, Landsat Data Continuity Mission,
Orion, and the Global Precipitation Measurement Mission.
[21] Due to changes in NASA's accounting structure, its historical cost
data is relatively inconsistent. As such, we used "then-year" dollars
to report data consistent with the data that NASA reported to us.
[22] Some projects reported that their spacecraft would be ready for
launch sooner than the date that the launch authority could provide
actual launch services. In these cases, we used the actual launch date
for our analysis rather than the date that the project reported
readiness.
[23] According to NASA officials, projects that were in formulation at
the time of the agency's 2007 revision of its project management policy
are required to comply with that policy. Projects that had already
entered implementation at the time of the revision were directed to
implement those requirements which would not adversely affect the
project's cost and schedule baselines.
[24] In our calculation for the percentage of total number of drawings
projected for release, we used the number of drawings released at the
critical design review as a fraction of the total number of drawings
projected, including where drawing growth occurred. So, the denominator
in the calculation may have been larger than what was projected at the
critical design review. We felt that this more accurately reflected the
design stability of the project.
[25] NASA distinguishes critical technologies from heritage
technologies. NASA officials do not believe that heritage technologies
are the same as critical technologies because they believe critical
technology does not rely on existing technology. GAO best practices
describe critical technologies as those that are required for the
project to successfully meet customer requirements, regardless of
whether or not they are based on existing or heritage technology. For
the purposes of this review, we distinguish between the two types
because NASA did not report heritage technologies as critical
technologies in our data collection instrument.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
