Space Transportation
Challenges Facing NASA's Space Launch Initiative Gao ID: GAO-02-1020 September 17, 2002In 2001, the National Aeronautics and Space Administration (NASA) began undertaking a new effort--the Space Launch Initiative (SLI)--to develop a new generation of space transportation vehicles. SLI is expected to result in development of the second generation of reusable launch vehicles, the space shuttle being the first generation. NASA plans to define basic requirements for its second-generation reusable launch vehicle--that is, what the crew size will be, what the payload capacity will be, and what designs or architectures are worth pursuing--by November 2002. However, considerable challenges must be addressed before NASA can accomplish this. First, NASA has to complete a reassessment of its overall space transportation plans. Second, NASA is currently reassessing the future of the International Space Station. The decisions it will make as part of this evaluation, such as how many crew will operate the station, will have a dramatic impact on NASA's requirements for a second-generation vehicle. Third, NASA needs to decide whether the SLI program will be developed jointly with the Department of Defense (DOD) and, if so, how can it accommodate DOD's requirements for a reusable launch vehicle. Until NASA finalizes its basic requirements for SLI, it cannot implement management controls that are essential to predicting what the total costs of the program will be and to minimizing the risks with NASA's planned initial investment of $4.8 billion. It is important for NASA to implement management controls for SLI as soon as possible, so that it can provide its managers and Congress with the information needed to ensure that the program is on track and able to meet expectations.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-02-1020, Space Transportation: Challenges Facing NASA's Space Launch Initiative
This is the accessible text file for GAO report number GAO-02-1020
entitled 'Space Transportation: Challenges Facing NASA's Space Launch
Initiative' which was released on October 02, 2002.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products‘ accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
Report to the Subcommittee on Space and Aeronautics, Committee on
Science, House of Representatives:
United States General Accounting Office:
GAO:
September 2002:
Space Transportation: Challenges Facing NASA‘s Space Launch Initiative:
GAO-02-1020:
Contents:
Letter:
Results in Brief:
Background:
Important Decisions to Be Made Before Requirements Can Be Defined:
Key Management Controls Are Not Yet Implemented:
Conclusions:
Recommendations:
Agency Comments:
Scope and Methodology:
Appendix I: Comments from the National Aeronautics and Space
Administration:
Appendix II: GAO Contact and Staff Acknowledgments:
Table:
Table 1: Activities Related to the Space Launch Initiative:
Figures:
Figure 1: NASA Illustration of the 2nd Generation Reusable Launch
Vehicle:
Figure 2: Illustration of NASA‘s Overall Space Transportation Plans:
Abbreviations:
DOD: Department of Defense
EVM: Earned Value Management
NASA: National Aeronautics and Space Administration
RFP: Request for Proposal
SLI: Space Launch Initiative
TRL: technology readiness level:
United States General Accounting Office:
Washington, DC 20548:
September 17, 2002:
The Honorable Dana Rohrabacher
Chairman
The Honorable Bart Gordon
Ranking Member
Subcommittee on Space and Aeronautics
Committee on Science
House of Representatives:
In 2001, the National Aeronautics and Space Administration (NASA) began
undertaking a new effort--the Space Launch Initiative (SLI)--to develop
a new generation of space transportation vehicles. SLI is expected to
result in the development of the second generation of reusable launch
vehicles, the space shuttle being the first generation. NASA plans to
spend $4.8 billion on the program through fiscal year 2006. SLI is part
of a broader program--known as NASA‘s Integrated Space Transportation
Plan-
-to address future space transportation needs. Under the plan, NASA
envisions making upgrades to extend the life of the space shuttle and
undertaking longer-term research and development of future
transportation
technologies and transportation systems, including space vehicles that
can
reach orbit in just one-stage.
SLI is a highly ambitious program. It will require NASA to develop and
advance new technologies, such as propulsion and airframe systems,
which in turn can potentially be used by U.S. industry to create new
business opportunities in space. The undertaking will also require a
high level of communication and coordination between a range of
partners, including private-sector contractors, academia, and the
Department of Defense (DOD). Moreover, it will require effective
controls and oversight to reduce cost, scheduling, and technical risks.
NASA‘s previous attempts to develop a new generation of space vehicles
were unsuccessful largely because NASA did not successfully implement
and adhere to critical project management controls and activities.
You requested that we assess NASA‘s progress with the Space Launch
Initiative, particularly with respect to defining requirements and
implementing management controls.
Results in Brief:
NASA plans to define basic requirements for its second-generation
reusable launch vehicle--that is, what the crew size will be, what the
payload capacity will be, and what designs or architectures are worth
pursuing--by November 2002. But considerable challenges must be
addressed before NASA can accomplish this.
First, NASA has to complete a reassessment of its overall space
transportation plans. In doing so, it must decide whether it should
continue pursuing the development of second-generation vehicles as
planned, pursue alternative ways to develop the second generation in
order to more quickly replace the space shuttle, or postpone these
efforts altogether indefinitely until there is a major breakthrough in
technology that could vastly improve performance and reduce costs. This
decision will be difficult, given the uncertainties about the
availability of technologies needed to reduce costs and enhance
performance for future space flight.
Second, NASA is currently reassessing the future of the International
Space Station. The decisions it will make as part of this evaluation,
such as how many crew will operate the station, will have a dramatic
impact on NASA‘s requirements for a second-generation vehicle. But they
will be difficult to reach, since they require NASA to come to
agreement with international partners who are concerned about planned
cutbacks to the station‘s capabilities.
Third, NASA needs to decide whether the SLI program will be developed
jointly with DOD and, if so, how it can accommodate DOD‘s requirements
for a reusable launch vehicle. So far, indications are that NASA and
DOD will share many of the same objectives for the vehicle, but there
are significant differences in priorities and requirements.
Until NASA finalizes its basic requirements for SLI, it cannot
implement management controls that are essential to predicting what the
total costs of the program will be and to minimizing risks with NASA‘s
planned initial investment of $4.8 billion. These include cost
estimates, controls designed to provide early warnings of cost and
schedule overruns, and risk mitigation plans. Moreover, there are
potential impediments to NASA‘s development and effective use of a
detailed cost estimate, including the lack of a modern integrated
financial management system. Lastly, NASA does not plan to develop
several measures that are important to assessing how the program as a
whole is making progress toward achieving its key objectives, reducing
risks, and maturing technology.
It is important for NASA to implement management controls for SLI as
soon as possible, so that it can provide its managers and the Congress
with the information needed to ensure that the program is on track and
able to meet expectations. We are making recommendations to NASA that
focus on the need to make decisions with regard to the future of NASA‘s
overall space transportation plan, the future of the space station, and
DOD‘s participation in the SLI program before setting requirements for
SLI. We are also making recommendations aimed at implementing
management controls for the SLI program.
In its comments to a draft of this report, NASA stated that it concurs
with the recommendations. NASA believes that much of the SLI program‘s
success is directly related to the implementation of project management
controls and appropriate levels of insight. NASA‘s response is included
as appendix 1.
Background:
NASA‘s Space Launch Initiative is an effort to develop and build a
second generation of reusable space transportation vehicles. (See fig.
1 and table 1.) NASA‘s current transportation vehicle, the space
shuttle, has been in use for 20 years and requires a significant
portion of NASA‘s resources to operate and maintain. The primary goals
for SLI are to reduce the risk of crew loss as well as substantially
lower the cost of space transportation so that more funds can be made
available for scientific research, technology development, and
exploration activities. Currently, NASA spends nearly one-third of its
budget on space transportation.
Figure 1: Illustration of the 2nd Generation Reusable Launch Vehicle:
[See PDF for image]
Source: NASA.
SLI is part of a broader program--known as NASA‘s Integrated Space
Transportation Plan--to address space transportation needs. Under the
plan, NASA could operate the space shuttle through 2020 and make
software and hardware upgrades to the shuttle in order to extend its
use to this point. It envisions the deployment of second-generation
cargo vehicles to begin around 2011 and crew vehicles around 2014. As
with the shuttle, NASA envisions that the second-generation vehicle
will reach orbit in two stages. NASA also anticipates building a third
generation of vehicles in 2025 and even a fourth generation in 2040.
(See fig. 2.) NASA anticipates that these vehicles would reach orbit in
one stage; travel beyond low-earth orbit to far-reaching interstellar
missions; and employ revolutionary technologies, such as (1) rocket
engines that breath oxygen from the air during the climb to orbit
rather than carrying heavy oxidizers onboard, (2) propellantless
electromagnetic propulsion, and (3) solar-powered space sails.
Figure 2: Illustration of NASA‘s Overall Space Transportation Plans:
[See PDF for image]
Source: NASA.
Building the second-generation vehicle will be a considerably complex
and challenging endeavor for NASA--from both a technical and business
standpoint. For example, NASA plans to develop and advance new
technologies for the new vehicle. These include (1) new airframe
technologies that will include robust, low-cost, low-maintenance
structure, tanks, and thermal protection systems, using advanced
ceramic and metallic composite materials, and (2) new propulsion
technologies, including main propulsion systems, orbital maneuvering
systems, main engines, and propellant management. If successfully
developed, NASA believes that these new technologies could
substantially reduce the costs to maintain and operate the vehicle and
increase its reliability.
NASA also plans to develop the new vehicle through partnerships with
private industry, academia, and DOD. It is partnering with the private
sector so that it can help create business opportunities in space, and
it is partnering with DOD because of military needs for a reusable
launch vehicle. Such partnerships will require a high level of
coordination and communication, since agreements need to be reached on
what the basic capabilities of the new vehicle will be, what designs or
architectures[Footnote 1] should be pursued, how development costs will
be shared, and what individual partner responsibilities will be.
NASA recognizes that there are substantial technical and business risks
with SLI, and it is undertaking activities aimed at reducing them. To
reduce technical risks, for example, NASA is exploring, developing, and
testing technologies to make sure that they can be used on the new
vehicle, rather than proceeding with the program with uncertainty about
whether technologies will be mature enough when they are needed. To
reduce business and costs risks, NASA‘s goal is to ensure that there is
adequate competition in the development of the architecture and that
the architectures that NASA pursues will enable convergences with
NASA‘s requirements and commercial and military missions, and will not
pursue vehicles that cannot meet program goals.
To date, NASA has already explored hundreds of concepts and
technologies that could be used for future government and commercial
launch systems and space transportation operations. The ’leap-ahead“
technologies studied include crew survival systems, advanced tanks and
airframe structures, long-life rocket engines, and thermal protection
systems. In March 2002, NASA selected 15 of the most promising
candidates to go forward into more detailed development. In November
2002, NASA plans to narrow the field down to 3 candidates and in
September 2003 to at least 2 candidates. From fiscal year 2003 through
fiscal 2006, two competing system architectures will be developed,
ready for a full-scale development decision in fiscal 2006.
NASA plans to spend $4.8 billion for the SLI program through fiscal
year 2006. This investment is to be used for (1) systems engineering
and activities concerning the definition of requirements; (2) efforts
to compete designs for the reusable launch vehicle as well as efforts
to reduce business and technical risks; (3) activities to develop and
demonstrate designs, technologies, and system-level integration issues
associated with such NASA-unique transportation elements as a crew
transport vehicle and cargo carriers; and (4) activities to develop
alternative access to the space station as well as to provide
contingency backup or relief for the space shuttle. NASA‘s efforts to
compete designs for the vehicle and reduce technical and business risks
consist of approximately $3 billion of the planned initial investment.
At the present time, no prototype vehicle integrating all new SLI
technologies is planned before NASA‘s fiscal year 2006 decision on
whether to proceed to full-scale development.
Table 1: Activities Related to the Space Launch Initiative:
Year: 2001; Month: Feb.; Event: * SLI program approved; Narrative: NASA
plans to budget $4.8 billion for the program through fiscal year 2006..
Year: 2001; Month: May; Event: * Initial contracts awarded to
22 contractors; Narrative: The contracts, valued at almost $800
million, were not intended to provide a specific vehicle design, but
rather to explore concepts and technologies that could be used for
future government and commercial launch systems and space
transportation operations. The leap-ahead technologies explored
included crew survival systems, advanced tanks and airframe structures,
long-life rocket engines, and thermal protection systems..
Year: 2002; Month: Mar.; Event: * Interim Architecture and Technology
Review; Narrative: Design concepts for the second-generation vehicle
narrowed down from hundreds to 15..
Year: 2002; Month: Nov.; Event: * Systems Requirements Review;
Narrative: This review is to focus attention on fewer space
transportation architectures and technology areas, to select three
architectures that can be pursued, and reach agreement on the
development of system requirements..
Year: 2003; Month: Feb.; Event: * Request for Proposals (RFP) to select
SLI designs; Narrative: RFPs are to focus on selecting the most
promising architectures to proceed toward a detailed preliminary
design..
Year: 2003; Month: Sept.; Event: * Contract awards for selection of
two designs; Narrative: Second phase of formulation program to further
develop concepts and risk-reduction activities..
Year: 2006; Month: [Empty]; Event: * SLI full-scale development
decision; Narrative: Decision for selecting architecture to enter full-
scale development..
Year: 2009; Month: [Empty]; Event: * Potential prototype vehicle
available; Narrative: Reusable unmanned vehicle with limited
capabilities..
Year: 2012; Month: [Empty]; Event: * Cargo vehicle available;
Narrative: Reusable unmanned vehicle with advanced engines and tanks..
Year: 2014; Month: [Empty]; Event: * Crew vehicle available; Narrative:
Reusable launch vehicle with crew capabilities.
[End of table]
Source: Discussions with and documentation from NASA.
Important Decisions to Be Made Before Requirements Can Be Defined:
According to a NASA official, NASA plans to define the basic
requirements for its second-generation space transportation--that is,
what the crew size will be, what the payload capacity will be, and what
designs or architectures are worth pursuing--by November 2002. These
decisions will have a significant impact on the cost, size, and design
of the new vehicle. For example, as the payload capacity increases, so
does the thrust requirement for the propulsion system, and in turn, the
cost to develop and build the system, as well as the cost to operate
the system.
However, NASA is facing a considerable challenge in reaching the point
to where it can finalize SLI requirements. This is primarily because
NASA must first make some difficult broader decisions regarding the
future of space transportation and other NASA projects, including (1)
whether NASA should skip development of a second-generation vehicle in
favor of concentrating on the third generation, (2) what the future of
the International Space Station will be, and (3) what DOD‘s role in
developing the new vehicle will be. Moreover, in making these
decisions, NASA will need to reach consensus with a wide range of
partners who have priorities and concerns different from NASA‘s, and it
will need to make trade-offs amid uncertainties as to the availability
of advanced technologies.
First, NASA must complete its ongoing reassessment of its overall space
transportation plan. This evaluation is being done as part of NASA‘s
development of a budget proposal for fiscal year 2004. The options NASA
is considering could have a far-reaching impact on the SLI program. One
option, in fact, involves postponing efforts to develop a new
generation of vehicles indefinitely until there is a major breakthrough
in technology that could vastly improve performance and reduce costs.
Other options being examined could drastically change NASA‘s timetable
and requirements for SLI. For example, NASA is looking at developing a
crewed vehicle more quickly than currently planned and launching it
atop an expendable rocket until it can field a reusable launch vehicle.
NASA is also considering deferring some development efforts so that it
can aggressively pursue a goal of building a relaunchable vehicle
prototype by 2009. Finalizing its decisions on where to go with the
space transportation plan will be difficult for NASA, given the
uncertainties about the availability of technologies needed to reduce
costs and achieve NASA‘s performance and safety objectives.
Second, NASA is currently reassessing the future of the International
Space Station. One decision it needs to make that could significantly
affect the SLI program is whether the station should support a crew of
seven astronauts, as originally planned, or three. Because of cost
growth, NASA plans to cut back to a crew of three, but its
international partners have not agreed yet to this decision and are
concerned that the cutback will severely undermine planned scientific
research. This decision could significantly affect the design and cost
of the second-generation vehicle. For example, with a bigger crew size,
the overall vehicle will have to be larger; a larger crew cabin and
additional backup systems will be required; and, as a result, the cost
to develop the vehicle will increase.
NASA is also planning to cut back the number of flights to the station
from seven to four per year, which could also have an impact on SLI.
But again, NASA‘s international partners have concerns about this
decision, since it would limit the deliveries of resources needed to
carry out research activities. At the same time that NASA is looking at
cutting back on space station capabilities, it is planning to extend
the life of its space shuttle to 2020 with software and hardware
upgrades. If NASA can successfully extend the life of the shuttle, it
may well find that a second-generation vehicle, which is not expected
to begin transporting crews to the space station until at least 2014,
would generally be duplicative in capability and therefore unnecessary.
NASA plans to make final decisions on the space station by November/
December 2002.
Third, while DOD and NASA have explored potential common areas of
interest for a new space transportation vehicle, DOD has not yet
formally defined its requirements. In fact, it is still uncertain
whether SLI will be a joint DOD/NASA program. A study conducted by the
Air Force and NASA earlier this year revealed that both NASA and DOD
shared similar objectives when it came to technologies needed for the
new vehicle as well as cargo lift requirements and launch architecture
elements. But there were differences with priorities and certain
requirements. The Air Force would like the vehicle to operate in
stronger winds, more precipitation, and a wider range of temperatures.
It would also like the vehicle to operate from an inland U.S. Air Force
base. NASA would like the vehicle to stay in orbit for a longer
duration and have more maneuverability because of its mission to
service the International Space Station. Additionally, NASA will have a
higher weight-delivered-to-orbit requirement for its crew vehicle.
It is apparent that some of DOD‘s objectives and priorities may not
match up with NASA‘s. For example, DOD envisions developing an
unpiloted vehicle, while NASA is focusing on a crewed design. DOD also
envisions developing a vehicle that could be capable of relaunching
within 12 to 48 hours and making as many as 20 flights in a 2-week time
frame. At this time, NASA does not share these objectives. According to
a
DOD official, DOD expects to finalize its requirements before the end
of
2002. Since DOD‘s decisions will affect the size, design, and
capabilities of the vehicle, it is important that these requirements be
known before NASA finalizes its own requirements in November 2002.
Otherwise, NASA will need to reexamine its requirements after narrowing
down potential system architectures to three possibilities. While
significant cost and operational benefits may accrue with a vehicle
capable of satisfying both DOD‘s and NASA‘s requirements, NASA will
need to guard against making compromises that might negatively affect
its goal of substantially lowering launch costs.
Key Management Controls Are Not Yet Implemented:
NASA cannot implement key management controls for the SLI program until
it defines its basic requirements. These include cost estimates,
controls designed to provide early warnings of cost and schedule
problems, as well as risk mitigation plans. Moreover, NASA does not
have plans to implement some performance measures, including ones that
would assess overall progress toward achieving key objectives, reducing
risks, and maturing technology. It is essential that these controls be
implemented quickly so that NASA can predict what the total costs of
the program will be and provide assurance that its investment in SLI is
being spent wisely.
Importance of Management Controls for SLI:
Undertaking ambitious, technically challenging efforts like SLI--which
involve multiple contractors and technologies that have to be developed
and proven--requires careful oversight and management. Importantly,
accurate and reliable cost estimates need to be developed, technical
and program risks need to be anticipated and mitigated, and performance
and readiness need to be closely monitored. Such undertakings also
require a high level of communication and coordination. Not carefully
implementing such project management tools and activities is a recipe
for failure. Without realistically estimating costs and risks, and
providing budgetary reserves needed to mitigate those risks, management
may not be in a position to effectively deal with the technical
problems that cutting-edge projects invariably face.
In fact, we found that NASA did not successfully implement and adhere
to a number of critical project management tools and activities in its
previous efforts to build new space transportation vehicles. As we
testified in June 2001, neither of NASA‘s X-33 or X-34 programs--which
attempted to build a new transportation vehicle that would reach orbit
in one stage---fully assessed the costs associated with developing new,
unproven technologies; provided for the financial reserves needed to
deal with technical risks and accommodate normal development delays;
developed plans to quantify and mitigate risks to NASA; or established
performance targets showing a clear path to an operational launch
vehicle.[Footnote 2] Underlying these difficulties were problems with
the agreements and contracts that established the relationship between
NASA and its industry partners and the eventual erosion of commercial
prospects for the development of new reusable launch vehicles. We
testified that lax management controls led to numerous problems with
both the X-33 and X-34 programs. Technical problems, for example,
resulted in significant schedule and cost overruns, which NASA was ill
prepared to deal with.
NASA has taken steps to avoid the problems it encountered with the X-33
and X-34 programs. In our testimony last year, for example, we pointed
out that NASA planned to increase the level of insight into SLI
projects by providing more formal reviews and varying levels of project
documentation from contractors depending on the risk involved and the
contract value. NASA also required that all proposals submitted in
response to its research announcement be accompanied by certifiable
cost and pricing data. Finally, NASA discouraged the use of cooperative
agreements, since these agreements did not prove to be effective
contracting mechanisms for research and development efforts where large
investments are required.
Cost Estimates Cannot Be Developed Until Requirements Are Defined:
NASA cannot develop cost estimates for the SLI program until it defines
the basic requirements for the new transportation vehicle and narrows
the field of possible architectures. Moreover, until requirements are
defined, NASA cannot determine whether the $4.8 billion already planned
for SLI through fiscal year 2006 will achieve the desired results for
the program‘s formulation. NASA guidance requires that life-cycle costs
be estimated, assessed, and controlled throughout a program‘s life
cycle.[Footnote 3] The estimates are to be prepared to support major
program reviews and the development of budget submissions.
In our recent review of the International Space Station, we reported
that NASA is facing additional challenges to developing reliable cost
estimates and effectively using them to manage programs.[Footnote 4]
For example, a recent study performed by the Rand Corporation for the
Office of Science and Technology Policy found that NASA has ’very good“
cost and risk modeling capabilities. However, the study also found that
NASA‘s in-house capabilities were not well integrated into the
program‘s planning and management and that NASA programs had been
reluctant to integrate cost estimate and control practices that were
sufficiently robust to yield confidence in budget estimates. In
addition, a task force appointed last year to conduct an independent
external review and assessment of the space station‘s cost, budget, and
management found that NASA tends to take a short-term focus on
executing programs--that is, rather than managing programs to come
within overall cost and scheduling goals, it manages them around annual
budgets. The task force cited NASA‘s culture of managing the space
station program in adherence to its annual budgets as perhaps the
single greatest factor in the space station program‘s cost growth.
We also reported that NASA‘s ability to develop good cost estimates for
programs is hampered by NASA‘s lack of a modern integrated financial
system to track and maintain data needed for estimating and controlling
costs. NASA has made it a top priority to develop and implement a new
system. However, the first major component of the system--the core
financial system--is not expected to be implemented until June 2003.
It will be important for NASA to overcome barriers that relate to
estimating costs for SLI and to use estimates to take a long-term
perspective in managing the SLI program. Without good cost information,
decision makers at NASA and in the Congress will not know whether the
$4.8 billion investment is sufficient for the early phases of the
program or how much more it will cost to actually develop and deploy
the new vehicle. Moreover, NASA managers will lack the information they
need to monitor costs, schedule, and performance.
Other Management Controls Cannot Be Implemented Without a Cost
Estimate:
Other management controls that are integral to successfully managing
the SLI program cannot be implemented until NASA has a cost estimate.
First, NASA has a system--known as Earned Value Management--intended to
help provide program managers and others with early warnings of cost
and schedule problems. But this system cannot be effectively
implemented without having baseline requirements defined or detailed
cost estimates, schedules, and timelines developed for the SLI program.
Earned Value Management goes beyond the two-dimensional approach of
comparing budgeted costs with actual costs. It also attempts to compare
the value of work accomplished during a given period with the value of
work scheduled for that period. By using the value of work done as a
basis for estimating the cost and time to complete it, the earned value
concept should alert program managers to potential problems sooner than
expenditures alone can. The communities that have a vested interest in
earned value are the (1) program managers, who are charged with overall
management responsibility for acquisition programs; (2) contractors,
who are responsible for the contract‘s successful execution; and (3)
overseers, such as acquisition executives, financial managers, contract
surveillance officials, and cost estimators, who are tasked with
tracking and estimating program costs.
Second, individual SLI activities have prepared risk mitigation plans.
However, a program official told us that these plans cannot be
quantified at the overall program cost and schedule level without
complete cost estimates. Moreover, without a cost estimate, risks can
be measured only on a year-by-year basis and not on a multiyear basis-
-for example, to 2006. Risk mitigation plans identify, assess, and
document the risks associated with the cost, resource, schedule, and
technical aspects of a project and determine the procedures that will
be used to manage those risks. In doing so, they help ensure that a
system will meet performance requirements and be delivered on schedule
and within budget. NASA‘s guidance requires that these plans be
developed during the formulation phase of a project.
Measures to Assess Performance Not Implemented:
Agencies are required to prepare annual performance plans that
establish performance goals with measurable target levels of
performance for each program activity in the agency‘s budget and to
provide a basis for comparing actual performance with performance
goals.[Footnote 5] Doing so enables agencies to gauge the progress of
programs like SLI and, in turn, to take quick action when performance
goals are not being met.
Individual SLI activities are using computer simulations to help gauge
whether the technologies under development can meet specific
performance targets. However, NASA has not yet established broader
measures that can show NASA decision makers and the Congress whether
the program as a whole is meeting objectives such as reducing the
payload cost to approximately $1,000 per pound and reducing the risk of
crew loss to approximately 1 in 10,000 missions. Like cost and risk
controls, the development of such metrics depends on NASA‘s definition
of requirements and cost estimation for SLI.
It is important that NASA develop such measures as soon as possible. We
testified last year that one problem that hampered the previous X-33
and X-34 efforts was the fact that NASA had not developed performance
targets that establish a clear path leading to a reusable launch
vehicle.
In addition, while NASA plans to assess the readiness of technology to
gauge the maturity of individual technologies, it does not plan to
verify and validate the maturity of technology for the SLI program as a
whole. To ensure that individual technologies are sufficiently mature
by NASA‘s planned full-scale development decision for 2006, NASA
intends to use technology readiness levels (TRLs). Our prior reports
have shown that TRLs, which were pioneered by NASA, are a good way to
gauge the maturity of technologies. Readiness levels are measured along
a scale of 1 to 9, starting with paper studies of the basic concept,
proceeding with laboratory demonstrations, and ending with a technology
that has proven itself on the intended product. NASA would like to
achieve
a TRL of 6 by 2006 for key technologies, such as the propulsion system.
At
this level, a prototype is tested in a relevant environment, such as a
high-
fidelity laboratory environment or in a simulated operational
environment.
Currently, most of the technology areas are at levels 3 or 4. At level
3,
analytical studies and laboratory studies are performed to physically
validate analytical predictions of separate elements of the technology.
At level 4, basic technological components are integrated to establish
that the pieces will work together.
While assessing technologies separately should help NASA decide when
and where to insert new technologies into the SLI program, it is still
important for NASA to look at the readiness of the product as a whole
because how well various components being developed will work together
is unknown.
Another measure that NASA officials told us they do not plan to
implement is one that would assess the extent or percentage of total
risks that have been reduced with NASA‘s initial $4.8 billion
investment as well as the amount of risk remaining. Program officials
told us that it would be too resource-intensive to develop this
measure. However, in the absence of such information, decision makers
at NASA and in the Congress have little assurance that the $4.8 billion
investment in SLI can fulfill the goals expected of the program, and
they have reduced confidence that what has already been spent has
placed NASA on track to meeting its primary goals.
Conclusions:
NASA aims to be able to define system requirements for SLI by November
2002. But meeting this goal may not be realistic. NASA must first
decide whether developing a second-generation vehicle to be deployed in
2014 to 2015 is still a worthwhile endeavor, given plans to extend the
life of the space shuttle and cut back on the space station‘s
capabilities, and if so, what specific direction the program should
take and how it will fit in with DOD efforts. Making such decisions
within the short time remaining before the SLI systems requirements
review will be difficult because it will require NASA to (1) resolve
differences with its space station partners, who have concerns about
planned cutbacks to the station; (2) reach consensus with DOD on
requirements and priorities, which are now considerably different; and
(3) make trade-offs as to what capabilities and technologies it should
pursue without really knowing when advancements can be achieved. When
NASA is able to resolve these challenges and finalize requirements for
SLI, it will be critical for NASA to swiftly implement effective
management to oversee the effort. Until it does so, NASA will not be
able to assure its managers and the Congress that the initial
investment is being spent wisely and that risks are being reduced, and
it will not be able to predict what the total costs of the program will
be.
Recommendations:
We recommend that the NASA Administrator do the following:
1. Reassess the schedule for defining the requirements for the Space
Launch Initiative in order to ensure that the agency takes the
following actions before making final decisions on basic requirements
and selecting three architectures to pursue: (1) complete the
reassessment of NASA‘s integrated space transportation plan, (2) reach
consensus with its international partners on the future of the space
station, and (3) reach consensus with the Department of Defense on its
role in the SLI effort.
2. If DOD is to jointly develop the second-generation vehicle, reach
consensus with DOD on priorities and objectives for SLI and factor
DOD‘s requirements into NASA‘s own.
3. After NASA completes its system requirements review, ensure that a
cost estimate is promptly developed for the SLI investment and that
this estimate is detailed and reliable enough to be used to complete
risk mitigation plans and carry out earned value management activities.
4. Ensure that NASA can demonstrate how the $4.8 billion initial
investment supports the requirements that NASA decides to pursue in
November 2002.
5. After system requirements are defined, ensure that performance
measures are developed to assess the progress of the program, as a
whole, toward (1) meeting key performance objectives, including
lowering the cost of delivering payloads to low-Earth orbit to less
than $1,000 per pound and reducing the risk of crew loss to
approximately 1 in 10,000 missions, and (2) achieving an overall
technology readiness level of 6 by 2006.
6. Ensure that measures are developed to assess the amount of risk
reduced following each year of expenditure and the amount of risk
remaining.
Agency Comments:
In written comments on a draft of this report, NASA‘s Associate Deputy
Administrator said the agency concurs with the recommendations, adding
that the program completed its first year as planned and within budget.
According to the Associate Deputy Administrator, the agency is
reassessing the Integrated Space Transportation Plan and working with
DOD to develop their requirements. Furthermore, NASA concurs with the
recommendation concerning the development of a cost estimate. However,
the agency notes that the RLV industry suffers uncertainties in
predicting costs, primarily because of a limited set of models and data
to validate those models. NASA also concurred with the need to validate
SLI requirements and the need for performance measures to assess the
progress of the program towards meeting objectives and achieving
technology readiness levels.
Our draft report contained a recommendation that NASA develop measures
to assess the percentage of risk reduced following each year of
expenditures and the percentage of risk remaining. NASA believes that
its process of narrowing potential SLI technology alternatives to the
most promising concepts meriting further funding enables the successful
candidates to be built and become operational. In that way, the SLI
program is reducing the risks inherent in an advanced research and
technology program of this magnitude. According to NASA, it will
develop a reporting mechanism that communicates the amount of risk
reduction achieved and remaining on an annual basis. Our intent is for
NASA to provide a quantifiable means of measuring progress. Thus, we
modified that recommendation accordingly.
Scope and Methodology:
To assess the coordination of requirements with DOD and other NASA
programs, we interviewed officials within the Department of Defense and
NASA. We evaluated studies conducted by DOD and NASA on efforts to
address requirements and coordination.
To address NASA‘s management controls for the SLI program, we
interviewed NASA officials regarding cost estimates and the process by
which cost information is studied and communicated throughout NASA. We
reviewed NASA‘s and the Office of Management and Budget‘s guidance
regarding earned value management and discussed the program‘s methods
for coordinating activities. We also reviewed risk mitigation plans to
determine how cost affects these plans and discussed these plans with
program officials.
To assess program performance measures, we reviewed NASA‘s policies and
procedures governing program management and the program‘s plans for
assessing performance measures. We also interviewed SLI program
officials to understand the program‘s plans to meet goals and
objectives.
To accomplish our work, we interviewed officials from NASA‘s
headquarters, Washington, D.C.; Marshall Space Flight Center, Alabama;
Johnson Space Center, Texas; Kennedy Space Center, Florida; and
Independent Program Assessment Office, Virginia; and DOD‘s Space
Command, Colorado.
We performed our review from October 2001 through August 2002 in
accordance with generally accepted government auditing standards.
Unless you publicly announce its contents earlier, we plan no further
distribution of this report until 30 days from its issue date. At that
time, we will send copies to the NASA Administrator; Director, Office
of Management and Budget; and other interested parties. We will also
make copies available to others upon request. In addition, the report
will be available at no charge on the GAO Web site at http://
www.gao.gov.
Please contact me at (202) 512-4841 if you or your staffs have any
questions about this report. Major contributors to this report are
listed in appendix II.
Allen Li
Director
Acquisition and Sourcing Management:
Signed by Allen Li
[End of section]
Appendix I: Comments from the National Aeronautics and Space
Administration:
National Aeronautics and Space Administration:
Office of the Administrator Washington, DC 20546-0001:
SEP 5 2002:
Mr. Allen Li:
Director, Acquisition and Sourcing Management Team:
United States General Accounting Office Washington, DC 20548:
Dear Mr. Li:
NASA has reviewed the draft audit report (GAO-02-1020) and concurs with
the recommendations as submitted for the Administrator. The program has
completed its first year as planned and within budget. Much of this
success is directly related to the successful implementation of
stringent project management controls coupled with appropriate levels
of insight. In parallel with the ongoing requirements definition
efforts, the program has continued to implement its strategic plan for
SLI, which will enable the program to provide the Agency with the
necessary scientific and technological data required to support the
design and development of NASA‘s future Space Transportation Systems.
The next several paragraphs provide the planned approach to implement
each of the recommendations outlined in the report.
1. Reassess the schedule for defining the requirements for SLI in order
to ensure that the Agency takes the following actions before making
final decisions on basic requirements and selecting three architectures
to pursue: (1) complete reassessment of NASA‘s integrated space
transportation plan, (2) reach consensus with its international
partners on the future of space station, and (3) reach consensus with
the Department of Defense on its role in the SLI effort.
NASA concurs with this recommendation. The Agency is reassessing the
Integrated Space Transportation Plan (ISTP) and will provide program
guidance in mid-September 2002. It is acknowledged that the importance
of the major Agency decisions cited in Recommendation 1 warrant careful
consideration, as their resolution will ultimately affect the
architecture design and associated required technologies. The Agency
continues to actively work with the DoD in the development of their
requirements and potential partnership for the development of a next
generation RLV. The DoD is expecting to complete the development of
their requirements early in FY 2003.
2. If DoD is to jointly develop the 2nd Generation vehicle, reach
consensus with DoD on priorities and objectives for SLI and factor
DoD‘s requirements into NASA‘s own.
NASA concurs with the recommendation that the Agency reach consensus
with the DoD on the requirements and missions for the next generation
of reusable launch vehicles. The analysis and activities to achieve
this consensus are ongoing and will continue through FY 2003. We are
working closely with our counterparts within the Air Force to develop a
converged set of requirements.
3. After NASA completes its system requirements review, ensure that a
cost estimate is promptly developed for the SLI investment and that
this estimate is detailed and reliable enough to be used to complete
risk mitigation plans and to carry out earned value management.
NASA concurs with this recommendation with the qualifications that, the
program‘s purpose is to achieve the necessary resolution in cost
estimation. SLI is building the analysis capability to perform reliable
development and operational RLV cost estimation for the Agency and the
industry. As you are aware, the RLV industry suffers with tremendous
uncertainties in cost predictions, primarily due to a limited set of
models and data to validate those models. The recommendation
(understandably) reiterates a general lack of confidence by our
stakeholders in the Agency‘s ability to predict costs of complex
systems, including RLV‘s. It underlines the need for the development of
this capability, and SLI continues to work this task.
4. Ensure that NASA can demonstrate how the $ 4.8 billion initial
investment supports the requirements that NASA decides to pursue in
November 2002.
NASA concurs with this recommendation. Existing NASA guidelines specify
that validation is required prior to approval of Level I requirements.
The validation process includes analyses to support the reasonableness
and achievability of the requirements.
S. After system requirements are defined, ensure that performance
measures are developed to assess the progress of the program as a whole
toward (1) meeting key performance objectives, including lowering the
cost of delivering payloads to low-Earth orbit to less than $1000 per
pound and reducing the risk of crew loss to approximately 1 in 10,000
missions and (2) achieving an overall technology readiness level 6 by
2006:
NASA concurs with the recommendation. The SLI program maintains
detailed performance measures at the project and program level,
including key milestones, progress reviews, system and subsystem tests,
product deliveries, and Government Performance and Reporting Act (GPRA)
indicators. These metrics are continually assessed and frequently
reviewed for applicability toward the program goals and objectives. The
program has successfully implemented Earned Value Management (EVM) on
the SLI program content.
6. Ensure that measures are developed to assess the percentage of risk
reduced following each year of expenditure and the percentage of risk
remaining.
NASA concurs with this recommendation with the following amplification.
The Agency will develop a reporting mechanism that will properly
communicate the amount of risk reduction achieved and remaining on an
annual basis. The process of narrowing architectures completed a major
milestone - the Interim Architecture and Technology Review - in the
second quarter of FY 2002. The next phase will continue to focus space
transportation system designs from many concepts to several of the most
promising candidates to go forward into more detailed development. As
technology trade studies are focused and validated through a rigorous
systems engineering process, the two (technology trade studies and
transportation system designs) will narrow the field to a select few
designs and define which technologies require further investment to
enable the eventual winner to be built and become operational. In this
way, the SLI is reducing the risks inherent in an advanced research and
development program of this magnitude, while fostering a fair business
environment for industry and ensuring the wise use of valuable
resources.
We are developing the detailed information that will help to quantify
the risk reduced versus the investment and provide the necessary
knowledge to meet the SLI program objectives.
Cordially,
Daniel R. Mulville:
Associate Deputy Administrator:
[Signed by Daniel R. Mulville]
[End of section]
Appendix II: GAO Contact and Staff Acknowledgments:
GAO Contact:
Jerry Herley (202) 512-7609:
Acknowledgments:
In addition to the person named above, Cristina Chaplain, Ivy Hubler,
Danny Owens, and Dana Solomon made key contributions to this report.
[End of section]
GAO‘s Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities
and to help improve the performance and accountability of the federal
government for the American people. GAO examines the use of public
funds;
evaluates federal programs and policies; and provides analyses,
recommendations,
and other assistance to help Congress make informed oversight, policy,
and
funding decisions. GAO‘s commitment to good government is reflected in
its
core values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO‘s Web site (www.gao.gov) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as ’Today‘s Reports,“ on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select ’Subscribe to daily E-mail alert for newly
released products“ under the GAO Reports heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone: Voice: (202) 512-6000
TDD: (202) 512-2537
Fax: (202) 512-6061
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800
U.S. General Accounting Office, 441 G Street NW, Room 7149
Washington, D.C. 20548:
FOOTNOTES
[1] ’Architecture“ broadly includes an Earth-to-orbit reusable launch
vehicle; on-orbit transfer vehicles and upper stages; mission planning;
ground and flight operations; and support infrastructure, both on the
ground and in orbit.
[2] See U.S. General Accounting Office, Space Transportation: Critical
Areas NASA Needs to Address in Managing Its Reusable Launch Vehicle
Program, GAO-02-826T (Washington, D.C.: June 20, 2001).
[3] See NASA Policy Directive 7120.4A, Program/Project Management and
NASA Procedures and Guidelines 7120.5A, Program and Project Management
Processes and Requirements.
[4] See U.S. General Accounting Office, Space Station: Actions Under
Way to Manage Cost, but Significant Challenges Remain, GAO-02-735
(Washington, D.C.: July 17, 2002).
[5] 31 U.S.C. §1115.
