Business Modernization
Improvements Needed in Management of NASA's Integrated Financial Management Program Gao ID: GAO-03-507 April 30, 2003The National Aeronautics and Space Administration's (NASA) nonintegrated financial management systems have weakened its ability to oversee its contractors, and its contract management has been on GAO's high-risk list since 1990. In April 2000, NASA began its Integrated Financial Management Program (IFMP), its third attempt in recent years at modernizing financial management processes and systems. GAO was asked to review whether NASA was following key best practices in acquiring IFMP components and implementing one of the first components--the core financial module.
The core financial module, if implemented as planned, may provide some improvement to NASA's accounting system environment. However, NASA is not following key best practices for acquiring and implementing IFMP. In acquiring IFMP components, NASA is facing risks in understanding dependencies among commercial components. NASA has not analyzed the interdependencies among selected and proposed IFMP components, and it does not have a methodology for doing so. For programs like IFMP, which involve building a system from commercial components, it is essential to understand the characteristics and credentials of each component to select ones that are compatible and can be integrated without having to build and maintain expensive interfaces. By acquiring IFMP components without first understanding system component relationships, NASA has increased its risks of implementing a system that will not optimize mission performance and will cost more and take longer to implement than necessary. In implementing the core financial module, NASA is facing risks in two additional areas. User needs: NASA did not consider the information needs of key system users and deferred addressing the requirements of program managers, cost estimators, and the Congress. Although this module should eliminate NASA's separate, incompatible accounting systems, little has been done to reengineer acquisition management processes. Program managers and cost estimators indicated that they will continue to rely on other means to capture the data needed to manage programs such as the International Space Station. Requirements management: NASA is relying on a requirements management process that does not require documentation of detailed system requirements prior to system implementation and testing. Over 80 percent of the requirements GAO reviewed lacked specificity, and several could not be traced among various documents. These defects also significantly impaired the testing phase of the system implementation effort. Further, NASA has not implemented metrics to help gauge the effectiveness of its requirements management process. NASA's approach will likely result in increasing amounts of time spent on costly rework and reduced progress. Unless these issues are successfully addressed, NASA is at increased risk of having IFMP become its third unsuccessful attempt to transform its financial management and business operations.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-03-507, Business Modernization: Improvements Needed in Management of NASA's Integrated Financial Management Program
This is the accessible text file for GAO report number GAO-03-507
entitled 'Business Modernization: Improvements Needed in Management of
NASA's Integrated Financial Management Program' which was released on
May 30, 2003.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Committee on Commerce, Science, and Transportation, U.S.
Senate, and the Committee on Science, House of Representatives:
April 2003:
Business Modernization:
Improvements Needed in Management of NASA's Integrated Financial
Management Program:
GAO-03-507:
GAO Highlights:
Highlights of GAO-03-507, a report to the Committee on Commerce,
Science, and Transportation, U.S. Senate, and the Committee on
Science, House of Representatives
Why GAO Did This Study:
The National Aeronautics and Space Administration‘s (NASA)
nonintegrated financial management systems have weakened its ability
to oversee its contractors, and its contract management has been on
GAO‘s high-risk list since 1990. In April 2000, NASA began its
Integrated Financial Management Program (IFMP), its third attempt in
recent years at modernizing financial management processes and
systems. GAO was asked to review whether NASA was following key best
practices in acquiring IFMP components and implementing one of the
first components”the core financial module.
What GAO Found:
The core financial module, if implemented as planned, may provide some
improvement to NASA‘s accounting system environment. However, NASA is
not following key best practices for acquiring and implementing IFMP.
In acquiring IFMP components, NASA is facing risks in understanding
dependencies among commercial components. NASA has not analyzed the
interdependencies among selected and proposed IFMP components, and it
does not have a methodology for doing so. For programs like IFMP,
which involve building a system from commercial components, it is
essential to understand the characteristics and credentials of each
component to select ones that are compatible and can be integrated
without having to build and maintain expensive interfaces. By
acquiring IFMP components without first understanding system component
relationships, NASA has increased its risks of implementing a system
that will not optimize mission performance and will cost more and take
longer to implement than necessary.
In implementing the core financial module, NASA is facing risks in two
additional areas:
* User needs. NASA did not consider the information needs of key
system users and deferred addressing the requirements of program
managers, cost estimators, and the Congress. Although this module
should eliminate NASA‘s separate, incompatible accounting systems,
little has been done to reengineer acquisition management processes.
Program managers and cost estimators indicated that they will continue
to rely on other means to capture the data needed to manage programs
such as the International Space Station.
* Requirements management. NASA is relying on a requirements
management process that does not require documentation of detailed
system requirements prior to system implementation and testing. Over
80 percent of the requirements GAO reviewed lacked specificity, and
several could not be traced among various documents. These defects
also significantly impaired the testing phase of the system
implementation effort. Further, NASA has not implemented metrics to
help gauge the effectiveness of its requirements management process.
NASA‘s approach will likely result in increasing amounts of time spent
on costly rework and reduced progress.
Unless these issues are successfully addressed, NASA is at increased
risk of having IFMP become its third unsuccessful attempt to
transform its financial management and business operations.
What GAO Recommends:
GAO is recommending that NASA develop and implement (1) a short-term
plan to identify and mitigate the risks currently associated with
relying on already deployed IFMP commercial components and (2) a
longer term strategy for acquiring additional IFMP components that
includes implementing a methodology for commercial system component
dependency analysis. NASA agreed with GAO‘s recommendation related to
a short-term plan but disagreed with many of the findings related to
user needs and requirements management. NASA also agreed with the
importance of having an approach for acquiring additional IFMP
components, but stated that it already has an effective strategy in
place. GAO reaffirms its recommendations.
www.gao.gov/cgi-bin/getrpt?GAO-03-507.
To view the full report, including the scope
and methodology, click on the link above.
For more information, contact Gregory D. Kutz at (202) 512-9095 or
kutzg@gao.gov.
[End of section]
Letter:
Results in Brief:
Background:
NASA's Acquisition Management Strategy Does Not Include Analyzing
Component Interdependencies:
Core Financial Module Does Not Fully Address Key User Information
Requirements:
NASA's Requirements Management Process for the Core Financial Module Is
Ineffective:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendixes:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Comments from the National Aeronautics and Space
Administration:
GAO Comments:
Appendix III: GAO Contacts and Staff Acknowledgments:
GAO Contacts:
Acknowledgments:
Tables:
Table 1: Five Contractors and Their Responsibilities:
Figures:
Figure 1: Space Shuttle Flight Operations Contract:
Figure 2: Example of Level of Detail Reported versus That
Required by Cost Estimators:
Figure 3: System Requirements for the ’Manage Accounts Payable“
Process:
Figure 4: Requirements for ’Validate Payment“ Subprocess:
Figure 5: Relationship between Requirements Development and
Testing:
Abbreviations:
CSC: Computer Services Corporation:
ERP: Enterprise Resource Planning:
FFMIA: Federal Financial Management Improvement Act:
IBM: International Business Machines:
IEEE: Institute of Electrical and Electronics Engineers :
IFMP: Integrated Financial Management Program:
IMCE: International Space Station Management and Cost Evaluation :
ISS: International Space Station:
NASA: National Aeronautics and Space Administration:
OMB: Office of Management and Budget :
SEI: Software Engineering Institute:
Letter April 30, 2003:
The Honorable John McCain
Chairman
The Honorable Ernest Hollings
Ranking Minority Member
Committee on Commerce, Science,
and Transportation
United States Senate:
The Honorable Sherwood L. Boehlert
Chairman
The Honorable Ralph Hall
Ranking Minority Member
Committee on Science
House of Representatives:
Much of the National Aeronautics and Space Administration's (NASA)
success depends on the work of its contractors--on which it spends
$12.7 billion, or 90 percent of its annual budget. For many years, NASA
has not effectively overseen its contracts, principally because it has
lacked accurate and reliable information on contract spending and
performance and it has placed insufficient emphasis on end results,
product performance, and cost control. Since 1990 we have identified
NASA's contract management function as an area at high risk.[Footnote
1] NASA's ability to collect, maintain, and analyze cost and
performance data has been weakened by nonintegrated, incompatible
financial management systems and processes, and uneven and nonstandard
cost-reporting capabilities. NASA made two efforts in the past to
improve its financial management processes and develop a supporting
system intended to produce the kind of accurate and reliable
information needed to manage its contracts effectively, but both of
these efforts were eventually abandoned after a total of 12 years and a
reported $180 million in spending.
In April 2000, NASA began its third attempt at modernizing its
financial management processes and systems. NASA has estimated the life
cycle cost of this effort through 2008 to be $861 million.[Footnote 2]
This effort, known as the Integrated Financial Management Program
(IFMP), is expected to produce an integrated, NASA-wide financial
management system through the acquisition and incremental
implementation of commercial software packages and related hardware and
software components.[Footnote 3]Through the proven business processes
and centralized data management capabilities embedded in these
commercial components, NASA intends to reengineer its management
operations to "do business the way business does business." The core
financial management module, which NASA considers to be the backbone of
IFMP, is currently operating at NASA headquarters and 6 of NASA's 10
centers[Footnote 4] and is expected to be fully operational in June
2003. According to NASA's business case analysis for the system, the
core financial module will provide NASA's financial and program
managers with timely, consistent, and reliable cost and performance
information for management decisions.
Given the importance of IFMP to NASA's mission performance, you asked
us to review the program. The purpose of this report is to alert you
now to concerns we have based on our work to date and to provide NASA
management with constructive recommendations for improvement that it
can initiate as soon as possible. We are continuing our work and plan
to fully respond to your request later this year.
Our work to date has focused on whether NASA has management processes
in place for effective system acquisition and implementation. This
report addresses three issues concerning the acquisition of IFMP
components and the implementation of one of the first components--the
core financial module. Specifically, we determined whether NASA (1) was
effectively evaluating the relationship among commercial systems
component options before acquiring them, (2) had adequately considered
the information needs of key users in implementing the core financial
module, and (3) had established and implemented an effective
requirements management process to support implementation of the core
financial module.
We performed our work from April 2002 through February 2003 in
accordance with generally accepted government auditing standards. We
had intended to include our assessment of a key element of NASA's
acquisition strategy--whether NASA was acquiring IFMP components in the
context of an agencywide blueprint, commonly called an enterprise
architecture--in this report. However, because NASA did not provide the
data needed to complete our assessment until after the conclusion of
our fieldwork, we plan to address NASA's enterprise architecture in a
future product. Details on our objectives, scope, and methodology are
in
appendix I.
Results in Brief:
If implemented as planned, IFMP may provide some improvement to NASA's
current accounting system environment because it should eliminate the
separate, incompatible systems that have previously been used at each
of NASA's 10 centers and should result in standardized accounting data.
However, NASA is not following key best practices for acquiring and
implementing IFMP. Specifically, NASA has not established an analytical
capability to guide and constrain its acquisition of IFMP commercial
components. Further, in implementing the core financial module
component, NASA has deferred addressing the needs of key system users
and has not properly developed detailed system requirements.
Consequently, the agency is at risk of making a substantial investment
in a system that will fall far short of its stated goal of providing
meaningful and reliable information to support effective program
management and congressional oversight.
NASA has not analyzed the interdependencies among selected and proposed
IFMP components, and it does not have a methodology for doing so. For
programs like IFMP, which involve building a system from commercial
components, it is essential to understand the characteristics and
credentials of each component in order to select ones that are
compatible and can be integrated without having to build and maintain
expensive interfaces. The alternative to such a structured and
disciplined approach to building a commercial component-based system is
trial and error, which is fraught with risk. Although NASA has already
acquired the core financial module and three other IFMP commercial
components, the agency has not performed the analysis necessary to
understand the logical and physical relationships among the component
parts it has acquired. By acquiring these IFMP components without first
understanding system component relationships, NASA has increased its
risks of implementing a system that will not optimize mission
performance, and will cost more and take longer to implement than
necessary.
For the core financial module, NASA did not consider the information
needs of key system users and deferred addressing the requirements of
program managers, cost estimators, and the Congress. Since 1990, we
have identified NASA's contract management function as an area at high
risk, in part because of the lack of effective systems and processes
for managing and overseeing its procurement dollars, producing credible
cost estimates, and providing the Congress with appropriate visibility
over its large, complex programs. However, despite these previous
problems, program managers, cost estimators, and congressional staffs
were not included in defining system requirements for NASA's core
financial module. Instead, NASA's financial managers and accountants
have primary responsibility for this process. In addition, little has
been done to reengineer acquisition management processes, particularly
with respect to the consistency and detail of budget and actual cost
data provided by contractors. Although capable of accepting the data
needed to satisfy the information needs of these key users, NASA's new
core financial module is not being implemented to accommodate this
information. According to IFMP program officials, they chose to defer
certain system capabilities and related user requirements in order to
expedite implementation of the core financial module. As a result,
program managers and cost estimators told us that they will not rely on
the core financial module and instead will continue to rely on other
systems or use other labor-intensive means to capture the data they
need to manage programs such as the International Space Station (ISS).
Further, NASA did not have an effective requirements management process
to support the implementation of the core financial module.
Specifically, NASA was relying on a systems requirements management
process that did not require documentation of detailed system
requirements prior to system implementation and testing. Although
industry best practices and NASA's own system planning documents
indicate that detailed requirements are needed to serve as the basis
for effective system testing, NASA's approach instead relied on certain
subject matter experts' knowledge of the detailed requirements
necessary to evaluate the functionality actually provided. As a result
of this approach, our review of the core financial module requirements
found that, for many of them, (1) the functionality to be delivered was
not adequately described or stated in a manner that allowed for
quantitative evaluation and (2) the traceability between the various
requirement documents was not maintained. Accordingly, the potential
for these requirements defects to result in costly rework is
significant and increases the risk that the project will not meet its
cost, schedule, and performance objectives. Because of the direct
relationship between requirements and testing, the lack of complete and
unambiguous requirements also significantly impairs the testing phase
of the system implementation effort. For example, the core financial
module could not process vendor invoices that contained over 200 line
items--a common occurrence on NASA's large contracts--because NASA did
not design an appropriate test case. If NASA had documented its
requirements, it would have recognized that a properly designed test
case had not been developed to cover this necessary functionality.
Furthermore, NASA has not effectively implemented the types of metrics
that can help the organization understand the effectiveness of its
requirements management process, such as identifying and quantifying
any weaknesses and then developing the corrective actions needed.
We are making recommendations that address the need for NASA to
(1) develop and implement a short-term plan to identify and mitigate
the risks currently associated with relying on already deployed IFMP
commercial components and to expeditiously stabilize these components'
operation capability and performance, (2) as part of the short-term
plan, develop and properly document requirements, reengineer
acquisition management processes, and fully engage stakeholders--
including program managers, cost estimators, and the Congress--in the
development of user requirements, and (3) develop a longer term
strategy for acquiring additional IFMP components that includes
implementing a methodology for commercial system component dependency
analysis.
In written comments, which are reprinted in appendix II, NASA concurred
with the need for a short-term plan but disagreed with most of our
findings related to user needs and requirements and testing. We remain
convinced that, as we have stated, NASA needs to (1) reengineer its
acquisition management processes to ensure that program and financial
managers as well as the Congress have needed budget and actual cost and
schedule data and (2) document detailed requirements to reduce, to
acceptable levels, the risks in implementing the selected processes.
NASA also agreed with the importance of having an approach for
acquiring additional IFMP components, but stated that it already has an
effective strategy in place. We did not find convincing evidence to
support NASA's contention that it is using methodologically based
dependency analysis--a best practice for implementing commercial
component-based systems--in acquiring IFMP.
Background:
NASA has a long and well-documented history of problems overseeing its
procurement dollars, producing credible cost estimates, and providing
the Congress with appropriate visibility over its large, complex
programs. We first identified NASA's contract management as an area at
high risk in 1990 because NASA lacked effective systems and processes
for overseeing contractor activities. Over the past decade, other GAO,
Inspector General, and task force reports have shown that NASA's cost
estimates lack credibility, in part because NASA does not collect the
historical cost data needed to accurately project future costs or
assess the validity of past estimates. Finally, because NASA had not
provided the Congress with adequate visibility over the ISS program,
the Congress had little advance warning when NASA reported that the
estimated cost to complete the ISS had grown by about $5 billion in 1
year.
Since we first identified NASA's contract management as an area of high
risk, we have reported that one of NASA's most formidable barriers to
sound contract management is the lack of a modern, integrated financial
management system. NASA's ability to collect, maintain, and analyze
cost and performance data has been weakened by nonintegrated,
incompatible accounting systems and processes, and uneven and
nonstandard cost-reporting capabilities. The weaknesses in NASA's
financial management systems also caused its independent auditor,
PricewaterhouseCoopers, to conclude for fiscal years 2001 and 2002 that
NASA's financial management systems do not substantially comply with
the requirements of the Federal Financial Management Improvement Act
(FFMIA). FFMIA builds on previous financial management reform
legislation by emphasizing the need for agencies to have systems that
can generate timely, accurate, and useful information with which to
make informed decisions and to ensure accountability on an ongoing
basis.
While NASA's efforts to design and implement a new financial management
system certainly move NASA forward in this area, technology alone will
not solve NASA's problems. Our reviews, as well as NASA's, show that
finance is not viewed as an integral part of NASA's program management
decision process. Moreover, an independent task force created by NASA
to review and assess ISS costs, budget, and management reached a
similar conclusion. In its November 1, 2001, report, the International
Space Station Management and Cost Evaluation (IMCE) Task Force found
that the ISS program office does not collect the historical cost data
needed to project future costs accurately and thus perform major
program-level financial forecasting and strategic planning. The task
force also reported that NASA's ability to forecast and plan is
weakened by diverse and often incompatible center-level accounting
systems and uneven and non-standard cost reporting capabilities. The
IMCE Task Force also concluded that the current weaknesses in financial
reporting are a symptom, not a cause, of the problem and that enhanced
reporting capabilities, by way of a new integrated financial management
system, will not thoroughly solve the problem. The root of the problem,
according to the task force, is that finance is not viewed as intrinsic
to NASA's program management decision process.
NASA's IFMP includes nine module projects supporting a range of
financial, administrative, and functional areas. According to NASA
officials, of the nine module projects, two are in operation, three are
currently in implementation, and four are future modules. The two
projects in operation are resume management and position description
management; the three projects in implementation are travel management,
core financial, and budget formulation; and the four future module
projects are human resources, payroll, asset management, and contract
administration.
The core financial module project, which utilizes the SAP R/3 system,
is the backbone of IFMP and will become NASA's standard, integrated
accounting system used agencywide. The other IFMP module projects will
be integrated/interfaced with the core financial module, where
applicable. The scope of the core financial module, when fully
implemented, includes: standard general ledger, budget execution,
purchasing, accounts receivable, accounts payable, and cost management.
NASA plans to implement the core financial module at all 10 NASA
centers by June 2003. The pilot for the core financial module--
conducted at Marshall Space Flight Center--was implemented in October
2002. NASA is rolling out or deploying the core financial module at the
other nine NASA centers and headquarters in three waves. The first
wave, which consisted of Glenn Research Center, rolled out in October
2002. The second wave, which consisted of NASA headquarters, Johnson
Space Center, Kennedy Space Center, and the Jet Propulsion Laboratory
rolled out in February 2003. Ames Research Center, considered a second
wave center, rolled out in April 2003. Finally, NASA plans to roll out
the third wave, which consists of Dryden Flight Research Center,
Goddard Space Flight Center, Langley Research Center, and Stennis Space
Center, in June 2003.
IFMP Acquisition Management Structure:
NASA is contracting with multiple companies to assist in the
acquisition management, integration, and implementation of its IFMP
"system of system components." As shown in table 1, five contractors
are assisting in the integration and implementation of the core
financial module. However, none of these five contractors is
responsible and accountable for successfully implementing the entire
IFMP system. Instead, NASA has structured its IFMP acquisition so that
NASA is the system integrator, meaning that NASA is responsible for
integrating multiple commercial components and ensuring that they
collectively perform in a manner that meets the defined requirements.
Table 1: Five Contractors and Their Responsibilities:
Entity: Accenture; Responsibility/function: Implement the core
financial module in accordance with agency requirements, including
interfacing it with NASA's existing systems environment.[A].
Entity: CSC (Computer Services Corporation); Responsibility/function:
Support the operations, maintenance, and administration of the new
module, including integration efforts..
Entity: IBM (International Business Machines); Responsibility/
function: Develop training and user procedures and perform security and
internal control reviews to ensure that the core financial module
complies with accounting and financial reporting standards..
Entity: SAP; Responsibility/function: Provide technical implementation
support and training on NASA's implementation of the core financial
module.[B].
Entity: Titan Systems Corporation-Civil Government Services Group;
Responsibility/function: Perform independent verification and
validation of requirements and testing processes and results, such as
tracing requirements to test cases..
Source: NASA.
[A] NASA plans to solicit additional contracts for implementation of
other selected and acquired modules.
[B] NASA has acquired SAP's enterprise resource planning package and
has thus far planned to implement the core financial and budget
formulation modules. NASA has also acquired three other commercial
software products--Travel Manager, Resumix, and Position Description
Management.
[End of table]
NASA's Acquisition Management Strategy Does Not Include Analyzing
Component Interdependencies:
A key to effectively acquiring commercial component-based systems that
are intended to support agencywide business needs, like IFMP, is
employing recognized acquisition management controls. One such control
is to acquire system components only after deliberate and comprehensive
analysis and understanding of the components' interdependencies.
Although NASA has already acquired the core financial module and three
other IFMP commercial components, the agency has not performed the
analysis necessary to understand the logical and physical relationships
among the component parts it has acquired. By acquiring these IFMP
components without first understanding system component relationships,
NASA has increased its risks of implementing a system that will not
optimize mission performance and will cost more and take longer to
implement than necessary.
When acquiring a commercial component-based system or system of
systems, such as IFMP, industry best practices[Footnote 5] recognize
the critical importance of understanding the logical and physical
relationships among the component parts. To provide for this
understanding, these practices advocate that the system integrator,
which in the case of IFMP is NASA, employ an explicit methodology,
including a risk-based process for deciding among product alternatives,
that collects and verifies information about each component's
characteristics and credentials, evaluates the dependencies and
constraints among these components, and permits informed decisions
about which products to acquire and how to implement them. This is
necessary because commercial products are built around each vendor's
functional and architectural assumptions and paradigms, such as
approaches to error handling and data access, and these assumptions and
paradigms are likely to be different among products from different
sources. Such differences complicate product integration. Further, some
commercial products have built-in dependencies with other products that
if not known can further complicate integration. For these reasons, a
structured and disciplined approach to systematically evaluating
product to product relationships is critical. The alternative to such a
structured and disciplined approach to building a commercial component-
based system is trial and error, which is fraught with risk.
In acquiring its IFMP components to date, NASA has not performed the
above-cited dependency analysis, and it does not have a methodology for
doing so. Despite this, NASA has acquired and is in the process of
implementing a commercial product (SAP's R/3 core financial module) to
meet its needs in one business area (financial management), and it has
acquired three additional commercial products from three separate
vendors that are intended to meet its needs in other business areas
(travel management, resume management, and human capital position
description needs).[Footnote 6] Beyond the four products that it has
already acquired, NASA plans to acquire an unspecified number of
additional commercial components that are intended to meet its needs in
other business areas. To integrate those separate commercial products
into a "system of system components," NASA has executed several
contracts and plans to execute more to build interfaces (hardware and
software) to permit the components to interoperate. For example, a
contractor is currently building an interface between the core
financial module of the SAP product and the travel manager product.
When acquiring and implementing commercial hardware and software
solutions, organizations can generally pursue one of two basic courses
of action. That is, an organization can opt for a single package of
already integrated software components, which is referred to as the
"best of suite" approach, or it can opt for different software
components from different vendors, which is referred to as the "best of
breed" approach. NASA is currently following the "best of breed"
approach. According to the Integration Office Deputy Program Manager,
NASA has not performed dependency analyses among the various components
acquired to date, and those being considered for later acquisition,
because NASA's initial acquisition strategy was to acquire a single
commercial solution (i.e., "best of suite") and thus it did not
consider product interoperability to be a concern. While NASA has since
adopted a "best of breed" approach, the Integration Office Deputy
Program Manager stated that it still does not plan to perform these
analyses in the future because NASA will rely upon commercial tools
that support the development of interfaces between commercial products,
which the Integration Office Deputy Program Manager claimed will make
integration easy and relatively inexpensive and negate the need for
proactive dependency analysis. However, best practices advocate that
proactive dependency analysis and evaluation are necessary for informed
decision making regardless of whether integration tools will be used,
and particularly when a "best of breed" approach is employed.
What this means is that NASA is implementing its "best of breed"
approach using trial and error. This reactive method does not allow for
adequate understanding of commercial product dependencies until the
only alternative to integrating them is building and maintaining
complex interfaces, which unnecessarily increase system acquisition and
maintenance costs, delay promised capabilities and benefits, and do not
optimize agency performance. The results of a recent study[Footnote 7]
commissioned by NASA recognize the added risk associated with the "best
of breed" approach, and thus the importance of proactive dependency
analysis and evaluation to minimize this risk. Specifically, the study
states that NASA's "best of breed" approach will result in a higher
total cost of ownership because the agency will need to (1) acquire and
maintain multiple software licenses, (2) hire and maintain technical
staff knowledgeable about each commercial product, (3) build and
maintain interfaces to integrate the various products, and (4) provide
training to system users on each commercial product.
Core Financial Module Does Not Fully Address Key User Information
Requirements:
If implemented as planned, the core financial module may improve NASA's
current system environment by eliminating the separate, incompatible
accounting systems that have been used at each of NASA's 10 centers
previously. However, the core financial module currently being
implemented does not fully address the information requirements of key
users, such as program managers, cost estimators, or the Congress. Our
previous work at leading public and private sector
organizations[Footnote 8] has shown that user involvement and
effectively reengineering business processes are major factors in
successfully implementing financial management systems. In contrast, at
NASA, key users such as program managers and cost estimators were not
involved in defining or implementing NASA's system requirements and
have played a limited role in all aspects of the implementation of the
core financial module. Instead, NASA's financial managers and
accountants have primary responsibility for this process. Consequently,
NASA has not effectively used this opportunity to reengineer the way it
does business and implement a financial management system that
addresses many of its most significant management challenges, including
improving contract management, producing credible cost estimates, and
providing the Congress with appropriate visibility over its large,
complex programs. According to IFMP officials, they chose to forgo
certain system capabilities to expedite implementation of the core
financial module and have stated that these capabilities can be added
at a later date. In the meantime, program managers and cost estimators
will continue to rely on other nonintegrated systems outside of IFMP
and use other labor-intensive means to capture the data they need to
manage programs such as the ISS.
If Implemented as Planned, Core Financial Module May Provide Some
Improvements:
The core financial module, if implemented as planned, may provide some
improvement to NASA's current accounting system environment. According
to IFMP planning documents, the core financial module should (1)
eliminate much of the inconsistent data and lack of standardization,
(2) collect agency costs and allocate those costs to cost centers,
including civil service personnel costs, and (3) maintain a standard
general ledger to provide control over financial transactions, resource
balances, and assets and liabilities. If NASA is successful, the core
financial module could reduce the extensive amount of time and
resources currently required to consolidate NASA's 10 different
reporting entities and close the books each accounting period. However,
as discussed later, our findings relating to NASA's requirements
management and testing processes may affect NASA's ability to achieve
these improvements.
Key Users Were Not Involved in the Implementation of the Core Financial
Module:
The IFMP core financial module, although technologically capable of
meeting the needs of program managers, cost estimators, and the
Congress, is not being configured to do so because these key users have
not been actively involved in the implementation of the module. Our
previous work at leading public and private sector organizations has
shown that user involvement in reengineering business processes and
establishing and implementing system requirements are major factors in
successfully
implementing financial management systems.[Footnote 9]In fact, at these
leading organizations, not only did program and business managers
participate in the design and implementation of financial management
systems, they typically were responsible for driving the effort and
played a key role in reengineering core business processes. In
contrast, at NASA, financial managers and accountants have had primary
responsibility for the implementation of the core financial module,
while the other key users mentioned above have been largely excluded.
According to IFMP planning documents, financial managers and
accountants are considered direct customers responsible for the
administrative processes that will be reengineered and automated.
Therefore, these individuals, to date, have been engaged in defining
system requirements and priorities. On the other hand, stakeholders--
including program mangers, cost estimators, and the Congress--are
described in NASA documents as the ultimate beneficiaries of system
improvements but are not expected to be actively involved in the
system's implementation. While NASA has formed teams to reengineer
portions of the agency's administrative process, these teams primarily
consisted of financial managers. As a result, NASA has neither
reengineered its core business processes nor established adequate
requirements of the system to address many of its most significant
management challenges, including improving contract management,
producing credible cost estimates, and providing the Congress with
appropriate visibility over its large, complex programs.
The Core Financial Module Will Not Provide the Information Needed to
Manage Contracts:
The core financial module is not being implemented to provide program
managers with the information they need to fully monitor the work being
performed by contractors. Based on our review of NASA's three largest
space flight programs--Space Launch Initiative,[Footnote 10] ISS, and
Space Shuttle--we found that the core financial module, as currently
planned, will not accommodate much of the information provided by
NASA's contractors and needed by program managers to monitor contractor
performance. Specifically, the core financial module is not being
implemented to
(1) accommodate the contract schedule information received from
contractors and needed by program managers to monitor contractor
performance and (2) maintain cost data at a sufficient level of detail
for certain contracts.
Core Financial Module Does Not Integrate Cost and Schedule Data Needed
by Program Managers:
To adequately oversee NASA's largest contracts, program managers need
reliable contract cost data--both budgeted and actual--and the ability
to integrate these data with contract schedule information to monitor
progress on the contract. However, because program managers were not
involved in defining system requirements or reengineering business
processes, the core financial module is not being designed to integrate
cost and schedule data needed by program managers. As a result, program
managers are resorting to using other systems that will result in
additional cost over and above IFMP costs.
The primary source of contract schedule information used by program
managers comes directly from NASA's contractors in the form of monthly
hard copy or electronic cost and schedule performance reports. NASA
tracks contract schedule status by comparing the budgeted and actual
cost of work planned with budgeted and actual cost of work completed
for specific time periods. The term "schedule" incorporates both the
concept of status of work and whether a project or task is being
completed within planned time frames. Depending on the nature of the
work being performed, the method of measuring work progress varies.
Work is measured in terms of tasks when a specific end product or end
result is produced. But when work does not produce a specific end
product or result, level-of-effort or a more time-oriented method of
measurement is used. The type of information, level of detail, and
reporting format provided by contractors are determined during the
contract negotiation process and vary from contract to contract
depending on the size, complexity, and duration of the contract. In
general, however, these reports show contractor progress against cost
and schedule targets set by the program manager and against which
contractor performance can be measured. Contractors also report any
significant variances from the targets and explain how they will be
mitigated.
NASA's program managers need this contractor information to plan and
manage their programs effectively. However, the information from cost
and schedule performance reports is not recorded in the core financial
module. Instead, NASA uses only data from monthly contractor financial
management reports, commonly referred to as NASA form 533 reports, to
update the core financial module. NASA form 533 reports contain
estimated and actual contractor cost data but, according to NASA
program managers, do not contain the data needed to adequately assess
schedule performance. According to IFMP officials, the information
needed to perform cost and schedule analysis by program managers is
outside the scope of the core financial module and IFMP. IFMP program
officials told us that they chose to forgo certain system capabilities
to expedite implementation of the core financial module and have stated
that these capabilities can be added at a later date. However, NASA
does not currently have a plan for maintaining the data contained in
cost and schedule performance reports in the core financial module or
IFMP.
Because contract schedule information is not currently maintained
through the core financial module, program managers will continue to
rely on hard copy reports, electronic spreadsheets, or other means to
monitor contractor performance. Several of NASA's programs, including
the Space Launch Initiative and the ISS, are currently using other
systems to monitor contract cost and schedule performance, but these
are stand-alone efforts and have not been part of a coordinated NASA
plan. Officials at Marshall Space Flight Center have recognized the
importance of maintaining common cost and schedule performance data in
a single integrated system that is available to all NASA managers at
all locations. As such, these officials have proposed that NASA
establish the system they currently use as a NASA-wide standard.
NASA has stated that the core financial module is expected to result in
a single, integrated financial management system that is intended to
serve the needs of its program managers. By not including the cost and
schedule information needed by program managers in the core financial
module, NASA risks operating with two sets of books--one that is used
to report to management and the Congress and another that is used to
manage NASA's programs.
Core Financial Module Will Rely on Legacy Coding Structure:
Because NASA has not fundamentally changed the way it operates by
involving key users in business process reengineering efforts, the core
financial module is not being implemented to capture cost information
at the same level of detail that it is received from NASA's
contractors. Instead of implementing an accounting code structure that
would meet the information needs of program managers, NASA has embedded
the same accounting code structure that it uses in its legacy reporting
system in the core financial module. As a result, the availability of
detailed cost data is dependent on the adequacy of NASA's legacy coding
structure. In some cases, the cost information received by NASA on
monthly contractor reports must be aggregated to a higher, less
detailed level before it is posted against the old accounting code
structure. For example, as shown in figure 1, program managers for the
Space Shuttle receive monthly contractor reports on the space flight
operations contract that track costs related to friction stir weld and
propulsion safety upgrades separately.
Figure 1: Space Shuttle Flight Operations Contract:
[See PDF for image]
Note: Amounts shown are for illustrative purposes only.
[End of figure]
However, because the NASA legacy accounting code structure embedded in
the core financial module only tracks the cost of space shuttle flight
hardware upgrades, the more detailed costs that program managers need,
such as friction stir weld and propulsion safety upgrades, are not
available through the core financial module. According to NASA
officials, the core financial module is capable of capturing this more
detailed cost data; however, due to the complexity associated with
converting detailed data from the centers' legacy systems, NASA has
deferred this capability. While this information is available to
program managers from the contractor, it is not available through the
core financial module. In fact, on this particular contract, program
managers have access to the contractor's system and, therefore, have
access to an even greater level of detail than that reported by the
contractor on hard copy reports.
On the other hand, in cases where the legacy coding structure
adequately reflects the programs' information needs, the cost data
received from contractors do not have to be aggregated prior to
posting. For example, program officials with the ISS program recently
redesigned the program's cost coding structure in order to more
precisely identify the cost of specific work. This was not done as part
of an IFMP reengineering effort, but in response to external criticisms
of the program's failure to manage its costs. Regardless of the reason,
the program's reengineering effort has to some extent improved the
usefulness of the cost data being entered into the core financial
module.
Core Financial Module Will Not Provide the Information Needed to
Prepare Credible Cost Estimates:
The core financial module, as currently planned, will not provide
sufficiently detailed data for cost estimators. Although the core
financial module is technologically capable of maintaining the detailed
data required by cost estimators, cost estimators were not involved in
defining the system requirements or reengineering business processes.
As a result, NASA has not determined the most cost-effective way to
satisfy the information needs of its cost estimators nor reengineered
its business process to ensure that their needs are met.
According to members of NASA's cost estimating community, they
typically need cost data at an even greater level of detail than that
currently being provided by NASA's contractors. The cost estimators we
spoke with told us that their requests for more detailed cost data are
often not satisfied through the contract negotiation process. For
example, as shown in figure 2, while program managers may want--and
contractors may provide--the cost of an engine fan, cost estimators
need to know more detailed information, including the cost of the
various tasks needed to make a rotor assembly, which ultimately becomes
part of the fan.
Figure 2: Example of Level of Detail Reported versus That Required by
Cost Estimators:
[See PDF for image]
[End of figure]
The lack of sufficiently detailed information for cost estimators is
due to NASA's lack of reengineering efforts for the acquisition
management process, which should have been done prior to implementing
the core financial module. Because the core financial module will not
contain sufficiently detailed historical cost data necessary for
projecting future costs, cost estimators will continue to rely on
labor-intensive data collection efforts after a program is completed.
These efforts involve searching through old hard copy and electronic
contractor reports to extract all relevant data. NASA pays its
contractors extra to provide data required but not contained in these
reports, usually at a later point in time. Data collection after the
fact is expensive but, according to some NASA officials, is more cost
effective than requiring contractors to provide detailed cost data
throughout the course of the contract. However, NASA has not done the
analysis needed to determine the appropriate mix of routinely requiring
contractors to provide detailed cost data and capturing that data in
the core financial module versus purchasing the data after a contract
is complete.
Core Financial Module May Not Provide Better Information for
Congressional Oversight:
NASA has identified the Congress as a key stakeholder and ultimate
beneficiary of system improvements. However, based on our discussions
with congressional staffs from NASA's authorizing committees, the
agency did not consult with them regarding their information needs.
Consequently, NASA cannot be sure that it is implementing a system that
will provide the Congress with the information it needs for oversight.
As discussed previously, according to IFMP planning documents,
financial managers and accountants are considered direct customers and
are responsible for defining system requirements and priorities. On the
other hand, NASA considered the Congress a stakeholder and, therefore,
did not seek input from congressional staffs in defining system
requirements.
Similar to the problems faced by program managers and cost estimators,
the core financial module may not address many of the information needs
of the Congress. To properly assess the agency's annual budget
submission and make funding decisions, the Congress needs timely,
reliable cost and schedule information on the status of large, high-
risk programs, such as the ISS. As previously described, the module
will not provide the type of cost and schedule information that program
managers need to adequately monitor the status of NASA's major programs
and may not maintain sufficient information to readily address any
special congressional needs that arise.
Nevertheless, the Congress should be able to receive somewhat better
information about NASA's finances than it has in the past because, as
previously described, the core financial module may improve some
aspects of NASA's ability to produce reliable financial information.
For example, the use of a standard general ledger will provide more
standardized accounting data and general ledger controls. As a result,
the core financial module should enable NASA to provide timelier, more
reliable high-level cost information to the Congress on some issues,
such as annual spending limits.
NASA's Requirements Management Process for the Core Financial Module Is
Ineffective:
NASA has not effectively implemented a requirements management
process[Footnote 11] to support the implementation of the core
financial module and therefore has increased the risk that the agency
will not be able to effectively identify and manage the detailed system
requirements that system developers and program managers use to
acquire, implement, and test a system. Specifically, based on
discussions with IFMP officials and a review of the process documents
related to the core financial module, we found that NASA was relying on
a requirements management process that did not require detailed
documentation of system requirements prior to system testing. Industry
best practices, as well as NASA's own system planning documents,
indicate that detailed system requirements should be documented to
serve as the basis for effective system testing. Instead, NASA's
approach relied on the expertise of certain subject matter experts to
remember the detailed requirements necessary to evaluate the
functionality actually provided.
As a result of this approach, we found that (1) for over 80 percent of
the 132 core financial module requirements we reviewed, the
functionality to be delivered was not adequately described or stated in
a manner that allowed for quantitative evaluation and (2) the
traceability among the various requirement documents was not
maintained. Accordingly, the potential for these requirements defects
to result in costly rework is significant and increases the risk that
the core financial module will not meet its cost, schedule, and
performance objectives. Because of the direct relationship between
requirements and testing, the lack of complete and unambiguous
requirements also significantly impairs the testing phase. Furthermore,
NASA has not effectively implemented the types of metrics that can help
it understand the effectiveness of its requirements management process,
such as identifying and quantifying any weaknesses in its process and
then developing the corrective actions needed.
NASA Requirements Management Process Was Not Designed to Provide
Detailed System Requirements:
Requirements are the specifications that system developers and program
managers use to acquire, implement, and test a system. Requirements
should be consistent with one another, verifiable, and directly
traceable to higher-level business or functional requirements. It is
critical that requirements be carefully defined and that they flow
directly from the organization's concept of operations (how the
organization's day-to-day operations are or will be carried out to meet
mission needs).[Footnote 12] Improperly defined or incomplete
requirements have been commonly identified as a root cause of system
failure and systems that do not meet their cost, schedule, or
performance goals. Without adequately defined requirements that have
been properly reviewed and validated, a significant risk exists that
the system will need extensive and costly changes before it will meet
NASA's needs.
As discussed previously, NASA is designing and fielding the core
financial module without having determined the specific information
needs of its key stakeholders, including program managers, cost
estimators, and the Congress. The omission of this critical step
increased the risk that the project would not effectively include all
the detailed system requirements that were needed to achieve
management's vision of a core financial management module that provides
timely, consistent, and reliable cost and performance information for
management decisions.
IFMP officials stated that their basic approach to developing the core
financial module system requirements was (1) defining high-level
requirements that could be used for making a software selection,
(2) defining the business processes that the core financial module
needed to address, (3) linking the requirements that were originally
defined for the software selection to those business processes, and (4)
using subject matter experts to determine whether the application met
the business processes envisioned by the users during their discussions
of the needed functionality. A key feature of the NASA approach is that
the detailed requirements covered in the discussion of the business
processes are not required to be documented prior to testing. Rather,
NASA depends on subject matter experts, who are assigned to ensure that
the core financial module has the needed functionality, to know the
detailed requirements necessary to evaluate the functionality actually
provided. Such an approach relies on the subject matter expert being
available throughout the process and on the expert remembering the
undocumented requirements completely and consistently. Specifically,
an individual assigned to develop a test case[Footnote 13] is relied on
to understand the detailed requirements associated with all facets of
that test case and then to ensure that the test will provide the
information needed to understand whether the functionality was actually
provided.
IFMP officials also stated that the current approach was based on
discussions with their contractors and eliminated the need for detailed
documented requirements normally associated with efforts such as IFMP.
They also recognized that this approach was somewhat inconsistent with
their own Requirements Management Framework, issued in October 2000,
which stated that "[i]n order to test the software, a more detailed
statement of a requirement or process may be required to insure [sic]
the successful completion of a test." This document also recognized
that these detailed requirements would be needed for "a more refined
testable set of requirements . . . and needed to serve as a basis for
the testing that will occur . . ." In a January 2003 report[Footnote
14] by a contractor on the lessons learned on the IFMP effort, the
contractor noted that NASA would need to develop a set of requirements
and design specifications that had been validated by the individuals
responsible for managing each process. The contractor also noted that
although such an approach delays the first phase of the project design,
it reduces the overall implementation time.
As a result of NASA's stated approach to requirements management, our
review of NASA's system requirements related to the process documents
for the core financial module found that key attributes of effective
requirements were missing for many requirements. According to the
Institute of Electrical and Electronics Engineers (IEEE)--a leading
source for defining the best practices for efforts such as this--good
requirements have several characteristics, including the
following:[Footnote 15]
* The requirements document contains all the requirements identified by
the customer, as well as those needed for the definition of the system.
* The requirements fully describe the software functionality to be
delivered. Functionality is a defined objective or characteristic
action of a system or component. For example, a system may have
inventory control as its primary functionality.
* The requirements are stated in clear terms that allow for
quantitative evaluation. Specifically, all readers of a requirement
should arrive at a single, consistent interpretation of it.
* Traceability among various requirement documents is maintained.
Requirements for projects such as IFMP can be expressed at various
levels depending on user needs. They range from agencywide business
requirements to increasingly detailed functional requirements that
eventually permit the software project managers and other technicians
to design and build the required functionality in the new system.
Adequate traceability ensures that a requirement in one document is
consistent with and linked to applicable requirements in another
document.
NASA established about 590 requirements for the core financial
module.[Footnote 16] We reviewed in detail one business process area of
this module--the "Manage Accounts Payable" process--that included 132
of these requirements. We found that (1) for over 80 percent of the 132
requirements the functionality to be delivered was not adequately
described or stated in a manner that allowed for quantitative
evaluation and (2) the traceability between the various requirement
documents was not maintained.
Requirements Were Not Specific:
For over 80 percent of the 132 "Manage Accounts Payable" requirements,
the process documents lacked the specific information necessary to
understand the required functionality that should be provided and how
to determine quantitatively, through testing or other analysis, whether
the system will meet NASA's needs. The following are examples of core
financial module requirements that lacked the necessary specificity.
* One requirement stated that the system must "[a]llow the information
contained in the system to be queried to present detailed data as
requested (such as payee information). The capability to perform a
Print Screen must be available to all user screens." This requirement
did not clearly state such items as (1) the data elements that must be
supported, (2) how the user would obtain the data definitions for the
data elements that could be used, (3) the tool or process that would be
used to perform these queries, and (4) the relationship between the
ability to perform such queries and the requirement to be able to print
the screen.
* The core financial module was required to support "multiple payment
addresses and/or bank information for a single payee." This requirement
did not clearly state the maximum number of payment address and bank
information entries that should be allowed.
* Several requirements called for the core financial module to make
accounting entries; however, these requirements did not define the
specific accounting entries that should be made.
The lack of documented requirements that are complete and unambiguous
not only increases the risk that the project's functionality goals will
not be met, but also significantly impairs the testing phase of the
system implementation efforts, as discussed later in this report.
Traceability Was Not Maintained:
NASA has adopted a four-level approach to defining its requirements--
processes, subprocesses, activities, and tasks, with processes stating
high-level requirements and tasks providing the most detailed level. In
reviewing the various requirement documents, we found that (1)
traceability was not always maintained through the various documents
and (2) the level of detail did not provide additional specificity for
a given requirement as it progressed through the hierarchy.
Traceability allows the user to follow the life of the requirement both
forward and backward through these documents and from origin through
implementation. Traceability is also critical to understanding the
parentage, interconnections, and dependencies among the individual
requirements. This information in turn is critical to understanding the
impact when a requirement is changed or deleted. Without an effective
traceability approach, it is very difficult to perform such actions as
(1) accurately determining the impact of changes and making value-based
decisions when considering requirement changes, (2) maintaining the
system once it goes into production, (3) tracking the project's
progress, and (4) understanding the impact of a defect discovered
during testing.
To illustrate these issues, we attempted to follow the hierarchy of
requirements for one of the core financial module's seven processes
through the four levels of requirements utilized by NASA for this
project. As shown in figure 3, the "Manage Accounts Payable" process
area has 132 requirements associated with nine subprocesses. However,
one of the 132 requirements, "Multiple User Access," contained in the
"Manage Accounts Payable" process, was not shown in any of the
subprocesses, and it was unclear where this requirement would be
further defined.
Figure 3: System Requirements for the "Manage Accounts Payable"
Process:
[See PDF for image]
Note: The total number of requirements shown for the subprocesses (202)
exceeds the number of requirements shown for the "Manage Accounts
Payable" process (132) because some requirements apply to more than one
activity.
[End of figure]
Our review of the nine subprocesses found that 5 of the remaining 131
requirements contained in the subprocesses were not linked to any
activity. For example, a requirement that applies to all federal
agencies and is designed to ensure compliance with the Internal Revenue
Service's 1099 reporting requirements was not included in any of the
activities. Therefore, the individuals responsible for implementing the
requirements contained in the activities would not have the full
universe of requirements they must address. Conversely, as shown in
figure 4, several of the activities for the "Validate Payment"
subprocess did not contain any related requirements. Therefore, it was
unclear whether these activities should have been associated with this
subprocess. For example, the "clear advance" and "adjust invoice"
activities did not include any requirements related to validating
payments. Further, the lack of requirements for these activities may
cause confusion for the individuals assigned to test the functionality
associated with this subprocess. We were also unable to trace the
requirements from activities to tasks, which should be the most
detailed level of requirements because, for the activities associated
with the "Validate Payment" subprocess, NASA used the same information
for the activities and tasks. In other words, the requirements for the
tasks were identical to those listed for the activities and therefore
did not provide any additional details.
Figure 4: Requirements for "Validate Payment" Subprocess:
[See PDF for image]
Note: The total number of requirements shown for the activities (22)
exceeds the number of requirements shown for the "Validate Payment"
subprocess (17) because some requirements apply to more than one
activity.
[End of figure]
As can been seen in this example, NASA was unable to maintain adequate
traceability of the requirements for this subprocess as it progressed
through the hierarchy. More important, the level of specificity
associated with these requirements did not change. Based on our review,
we generally found that the wording of a given requirement was
identical regardless of the requirement document reviewed. For example,
if a requirement was listed in a subprocess area and flowed to an
activity, the same wording was used and the needed level of specificity
to help ensure proper implementation was not available. Therefore,
although NASA appeared to have adopted a requirements hierarchy that
would facilitate the needed specificity as the requirements flowed from
subprocesses to tasks and activities, the implementation of this
approach did not address the specificity problems discussed earlier.
Accordingly, this is another factor that increases the risk that this
project will not meet its schedule, cost, and functionality goals. A
NASA contractor hired to help evaluate the implementation of the core
financial module had similar findings. For example, the contractor
found that NASA had not developed documentation that explicitly details
the relationship between lower-level requirements and requirements of
the next level.
Requirements Defects Adversely Affect Testing of the Core Financial
Module:
Because requirements provide the foundation for system testing, the
specificity and traceability defects in the system requirements
preclude NASA from implementing a disciplined testing process. That is,
requirements must be complete, clear, and well documented to design and
implement an effective testing program. Consequently, NASA is taking a
significant risk that its testing efforts will not detect significant
defects until after the system is placed into production. Industry best
practices indicate that the sooner a defect is recognized and
corrected, the cheaper it is to fix. This is especially true since NASA
is depending on the subject matter experts' knowledge, rather than
documented requirements, to ensure that the application does not have
any significant defects before the system is placed into production.
As shown in figure 5, there is a direct relationship between
requirements and testing.
Figure 5: Relationship between Requirements Development and Testing:
[See PDF for image]
[End of figure]
Although the actual testing activities occur late in the development
cycle, test planning can help disciplined activities reduce
requirements-related defects. For example, developing conceptual test
cases based on the requirements derived from the concept of operations
and functional requirements stages can identify errors, omissions, and
ambiguities long before any code is written or a system is configured.
Disciplined organizations also recognize that planning testing
activities in coordination with the requirements development process
has major benefits.
We have identified several indications that NASA's testing program has
been adversely affected by the lack of complete and specific
requirements. Although we plan to continue our review of NASA's testing
plan for IFMP implementation, we noted (1) significant defects that
appeared to be related to requirements occurred in the application
after it was placed into production and (2) several cases where NASA
did not ensure that modifications made to the application did not cause
unintended effects and that the system or component still complied with
its specified requirements after the change.
Significant Defects Appeared in Production System:
Our review of the system test defect reports for the core financial
module disclosed that several defects considered by NASA to have an
initial severity rating of critical[Footnote 17] or high[Footnote 18]
had been identified after the system was placed into production at
Marshall Space Flight Center and Glenn Research Center. Detecting such
problems after the system goes into production may lead to costly
rework due to factors such as having to reenter transactions and adjust
reports manually. Furthermore, the manual processes required to make
these adjustments may introduce data integrity errors. Our preliminary
review indicated that the root cause of many of these defects could be
linked to the lack of complete requirements. For example, see the
following:
* Shortly after the system was placed into production, NASA found that
the core financial module was not properly executing certain business
rules. An emergency fix was developed, and the defect report noted that
a long-term solution and requirements would need to be developed. It
was unclear why the subject matter experts did not include the business
rules in the test cases used to evaluate the functionality of the
application. However, we believe one cause is the lack of documented
requirements that the testers could use to develop effective test
cases.
* NASA was unable to process vendor invoices that contained over 200
line items, which, according to NASA officials, is a common occurrence
on NASA's large contracts. It was unclear why the subject matter
experts responsible for testing this functionality had not developed
the test cases to ensure that large invoices were properly processed
before the system was placed into production. IFMP officials recognized
that this was an oversight in their testing process. If NASA had
documented its requirements, it would have recognized that a properly
developed test case had not been designed to cover this necessary
functionality.
* About 3 months after the core financial module was placed into
production at one center, it was found that when the system produced
multiple bills for the same customer, only the first bill was sure to
have the proper account classification code printed. The remaining
bills often contained incorrect values since the program improperly
assumed that the account classification code would not change until the
customer changed. Since the account classification code is critical for
these types of bills, the center was required to manually make the
necessary corrections on the bills.
Adequate Regression Testing Is Not Being Performed:
Efforts such as the core financial module undergo change constantly at
this stage of their development as functionality is being added and
defects are being corrected. However, before the revised application is
released, testing needs to be performed to ensure that any
modifications have not caused unintended effects and that the system or
component still complies with its specified requirements. This practice
is commonly referred to as regression testing. An effective regression
testing program is critical for ensuring that the functionality
associated with requirements that has been validated during previous
testing efforts has not been impaired by subsequent changes in the
application.
Although NASA officials stated that they require regression testing
before deploying any changes, we found that they do not have an
effective method to ensure that adequate regression testing is being
performed or that a consistent approach is being taken in performing
such testing. According to NASA officials, the individual identifying a
defect is responsible for ensuring that the defect is corrected and for
determining the amount of testing necessary to ensure that the defect
has been corrected. For example, if a defect is identified when
executing a test case, the tester may only test the section of the case
where the error was originally identified rather than performing all
the steps in the test case. This approach increases the risks that
defects introduced into the application during enhancements or defect
corrections will not be detected until after the application is
deployed, which results in costly rework. We noted several examples
where NASA appeared to perform inadequate regression testing. These
include the following:
* After adding an interface, it was found that an application screen
for recording advances did not operate as it had before the change was
made.
* A process for recording transactions provided by the Department of
the Treasury failed after an update to the application program.
* One center was testing a certain type of invoice and received an
error message. This error was attributed to a system patch that had
been applied to the application.
IFMP officials agreed that they did not have a comprehensive regression
testing program with a consistent approach. However, they told us that
they believed that any defects would be detected by the centers as the
application progresses through its releases because they encourage each
center to completely retest the application before placing the
application into production. This approach is particularly risky in
light of the requirements defects discussed previously, which
substantially increase the risk that the testing conducted by the
centers not yet operational may not detect any negative impacts
associated with a system change. However, IFMP officials recognized
that, after all centers are in production, a regression testing program
will be needed.
A NASA contractor monitoring this project also identified potential
problems relating to regression testing. According to the contractor
performing this work, NASA has not implemented the testing tools
necessary to adequately perform the regression testing to provide NASA
reasonable assurance that changes made in a given software release do
not have any adverse consequences for future releases.
Performance Metrics Could Be Used to Assess Potential Risks of
Identified Weaknesses:
Without a well-documented set of requirements, it is impossible to
place an error in context and understand the cause of the defect--for
example, determining whether the error was caused by the underlying
requirements or by some other process failure, such as inadequate
testing or inadequate controls over system configuration. NASA has not
effectively captured the types of metrics that can help the
organization understand the effectiveness of its management processes,
such as identifying and quantifying any weaknesses in its requirements
management process. Accordingly, NASA is unable to implement a metrics
measurement process that allows it to understand (1) its capabilities
to manage the IFMP effort, (2) how its process problems will affect its
cost, schedule, and performance objectives, and (3) the corrective
actions needed to reduce the risks associated with the problems
identified.
The Software Engineering Institute (SEI) has found that metrics
identifying important events and trends are invaluable in guiding
software organizations to informed decisions. Key SEI findings relating
to metrics include the following:
* The success of any software organization depends on its ability to
make predictions and commitments relative to the products it produces.
* Effective measurement processes help software groups succeed by
enabling them to understand their capabilities, so that they can
develop achievable plans for producing and delivering products and
services.
* Measurements enable people to detect trends and to anticipate
problems, thus providing better control of costs, reducing risks,
improving quality, and ensuring that business objectives are
achieved.[Footnote 19]
A critical element in helping to ensure that a project meets its cost,
schedule, and performance goals is to ensure that defects are minimized
and corrected as early in the process as possible. Although NASA has a
system that captures the defects that have been identified during
testing, we found that the agency did not analyze its identified
defects to determine their root causes. Understanding the root cause of
a defect is critical to evaluating the effectiveness of a process. For
example, if a significant number of defects are caused by inadequate
requirements definition, then the organization knows that the
requirements management process it has adopted is not effectively
reducing risks to acceptable levels. IFMP officials stated that they do
not capture the root causes of their defects.
Our initial observations identified that the root cause of many defects
appeared to relate directly to the requirements management process. For
example, see the following:
* About a week after the system was placed into production at a center,
NASA found that it was making payments to its vendors 1 day earlier
than required by Treasury regulations. This occurred because NASA
thought that Treasury would warehouse its payments. If NASA had
researched and documented the requirements associated with payment
warehousing for cash management purposes, it would have known that
Treasury does not warehouse payments such as these.
* About 3 weeks after the system went into production, NASA found that
one of the payment processing tools was not working as required. It was
unclear whether this was caused by a requirements defect or failure to
properly test the functionality. A review of the requirements documents
relating to this functionality provided a different description of the
requirement than that included in the defect report.
* In early October 2002, NASA found that the accounting entries for
certain advance transactions were incorrect. Properly documenting the
requirements, developing a test case that ensured the requirements were
met by the application, and executing that test case after the change
was made should have detected this problem.
* After a patch was applied to the system, it was found that some code
was duplicated, which caused an error. The apparent reason for the
duplication of code was that manual adjustments were made to the code
after the patch had been applied.
By analyzing the root causes of its identified defects, NASA could
determine whether the requirements management approach it has adopted
sufficiently reduces its risks of the system not meeting cost,
schedule, and functionality goals to acceptable levels. Root cause
analysis would also help to quantify the risks inherent in the testing
process that NASA has selected for the core financial module. Because,
as discussed previously, its approach in both these areas includes
elements that are not considered industry best practices, such metrics
would be particularly important to NASA's being reasonably assured that
its processes will result in a system that meets its business needs.
Conclusions:
NASA has established the right goal for IFMP, and its ongoing
implementation of several already-acquired system components,
particularly the core financial module, may provide some improvements
to NASA's accounting data. However, implementation of these components
will only partially address NASA's information needs related to its
complex space programs and contracts because NASA has deferred
implementation of the system capabilities needed to provide this
information and has not reengineered key business processes such as
acquisition management. NASA's long-standing weaknesses in this area
have been central to our designation of NASA contract management as
high risk. Moreover, NASA's approach to acquiring and implementing IFMP
components has and will continue to introduce risk and increase the
chances that the agency will fall short of meeting its IFMP goal.
NASA faces serious near-term risks in implementing the commercial
components that it has already acquired, including the core financial
module. However, it is too far along in deploying these components to
its centers, and relying on them to support operations, to stop and
first acquire and then implement them properly. Instead, NASA will be
forced to make the best of what it has acquired and implemented,
meaning that NASA will have to stabilize the components while they are
operational by identifying and correcting requirements defects and
adequately testing the components to ensure that completed requirements
are met. Such rework of already-deployed system components is a much
more costly approach to implementing systems than adequately defining
requirements and effectively testing system capabilities before they
are deployed. However, NASA has left itself no other viable option.
In the longer term, NASA has an opportunity to avoid the mistakes it
has made to date in acquiring system components, such as the core
financial module, by first determining whether proposed components are
the best solutions to meeting the agency's corporate needs before it
acquires them. It is critically important that NASA's acquisition
management strategy for future components includes a well-defined,
risk-based methodology for understanding the dependencies among
commercial component options before it acquires any additional
components. Once these components are acquired, it is also critically
important that NASA employ effective requirements management, testing,
and performance metrics practices in implementing the components. To do
less will increase the risk of IFMP becoming NASA's third unsuccessful
attempt to transform its financial management and business operations.
Recommendations for Executive Action:
Given that NASA has already largely deployed and placed into production
the IFMP commercial components acquired to date, we recommend that the
NASA Administrator direct the Program Executive Officer for IFMP to
focus near-term attention on stabilizing the operational effectiveness
of these deployed commercial components.
Specifically, to mitigate the risks associated with relying on already-
deployed IFMP commercial components and to expeditiously stabilize
these components' operational capability and performance, we recommend
that the Administrator direct the Program Executive Officer for IFMP to
develop and implement a corrective action plan. At a minimum, this plan
should provide for:
* identifying known and potential risks,
* assessing the severity of the risks on the basis of probability and
impact,
* developing risk mitigation strategies,
* assigning accountability and responsibility for implementing these
strategies,
* tracking progress in implementing these strategies, and:
* reporting progress regularly and frequently to relevant congressional
committees.
Additionally, this plan should provide for:
* developing and properly documenting requirements that are consistent,
verifiable, and traceable, and that contain the necessary specificity
to minimize requirement-related defects;
* conducting thorough regression testing before placing modified
components into production;
* implementing a metrics program that will identify and address the
root causes of system defects;
* reengineering acquisition management processes, particularly with
respect to the consistency and detail of budget and actual cost and
schedule data provided by contractors; and:
* engaging stakeholders--including program managers, cost estimators,
and the Congress--in developing a complete and correct set of user
requirements.
To mitigate future risks, we further recommend that the Administrator
require the Program Executive Officer for IFMP to complete the
following actions before the acquisition of any additional IFMP
components:
* establish and implement a methodology for commercial system component
dependency analysis and decision making, and:
* evaluate the suitability of already acquired, but not yet
implemented, IFMP component products within the context of a component
dependency analysis methodology.
Agency Comments and Our Evaluation:
In its written comments on a draft of this report, NASA stated that it
recognized and was addressing several of the concerns we raised and had
already implemented some of the recommendations. NASA also stated that
it disagreed with some of the issues in the report. NASA's comments on
our recommendations included the following:
* With regard to our recommendation to establish and implement a
methodology for IFMP commercial system component dependency analysis
and decision-making, NASA stated that it agreed with the importance of
having an approach for acquiring additional IFMP components and
believes that it has an effective strategy already in place. We
disagree that NASA has an effective strategy because it did not provide
convincing evidence to support its position that it is using
methodologically based dependency analysis--a best practice for
implementing commercial component-based systems--in acquiring IFMP.
* Although NASA concurred with our recommendation regarding the need
for a short-term plan to mitigate the risks currently associated with
relying on already-deployed IFMP commercial components, it disagreed
with many of our findings in the areas of (1) its efforts to involve
users and reengineer its business process to ensure that the core
financial module would meet the needs of program managers and cost
estimators and (2) the need for detailed system requirements. We
continue to believe that any effort that falls short of end-to-end
business process reengineering will not result in a system that
substantially improves the data available for contract oversight and
decision-making and that documented, detailed requirements are
necessary to reduce the risks of implementing the selected processes to
acceptable levels.
Overall, NASA disagreed with our findings related to three issues--
dependency analysis, user needs, and requirements and testing--which
are addressed in the following sections. NASA also included several
technical comments, which we have addressed as appropriate throughout
the report.
Dependency Analysis:
In its written comments on our recommendation to establish and
implement a methodology for IFMP commercial component dependency
analysis and decision making, NASA stated that it agreed with the
importance of having an approach for acquiring additional IFMP
components but disagreed with our finding that it has not performed
such dependency analysis to date in acquiring four IFMP commercial
components. It also disagreed that it lacked a methodology to guide its
analysis, and subsequent decision making, for future IFMP component
acquisitions. According to NASA's comments, the agency already has an
effective strategy in place and it has followed this strategy to date
in acquiring four IFMP commercial components. NASA described this
strategy as consisting of two factors: following an enterprise resource
planning (ERP) suite integration strategy (i.e, "best of suite"
approach) and using an enterprise application integration framework and
associated tool set for integrating current and future IFMP components.
NASA said that prior to receiving our draft report, it had provided us
detailed documentation describing how it began performing its component
dependency analysis before selecting the SAP ERP product for IFMP's
core financial module. The agency also noted that in a meeting
following its receipt of our draft report, it had provided us with
clear evidence that the program began performing dependency analysis
before selecting the SAP product. Further, NASA's comments stated that
it was using an enterprise application integration tool to facilitate
product integration and ease the associated complexities of integrating
multiple products. NASA added that there were "perhaps
miscommunications" and "some misunderstanding" of its approach, and
opined that much of our concern about component dependency stems from
our belief that NASA is not following a "best of suite" approach but
rather a "best of breed" approach. To support its view that it is
following a "best of suite" approach, NASA offered several statements,
including that it is (1) on record in presentations and letters,
including responses to congressional inquiries, that it is following
"best of suite," (2) developing business cases before implementing an
IFMP module, (3) working with SAP to extend its ERP product, and (4)
following a prioritization process when considering how to introduce
functionality that the SAP ERP product does not provide.
We agree with several of NASA's comments and disagree with others.
Collectively, NASA's comments do not change our finding and
recommendation. Specifically, we do not question that NASA is using an
enterprise application integration product and that this product
facilitates integration of system components, both commercial and
legacy components. Further, we do not challenge NASA's statements
regarding its representation in presentations and briefings that it is
following a "best of suite" approach, its development of business
cases, its interactions with SAP, and its use of a prioritization
process.
However, we do challenge NASA's assertion that much of our concern is
based on our belief that it is following a "best of breed" approach. On
the contrary, our finding and recommendation do not hinge on the
distinctions between "best of breed" versus "best of suite," despite
evidence supporting our statements in the report that NASA is indeed
following a best of breed approach. Such evidence includes (1) a report
from a NASA contractor hired to provide an independent evaluation of
IFMP stating that NASA is following a "best of breed" approach, (2)
NASA's acquisition of four separate commercial products from four
vendors to satisfy the first five of nine planned IFMP system modules,
and (3) NASA's statement in its comments that additional products may
be selected in the future. We fully appreciate that when implementing
an ERP solution, other vendor products will likely be needed to fill
gaps between agency requirements and the ERP product's capabilities.
Accordingly, we state in our report that proactive, methodologically
based dependency analysis and evaluation is needed regardless of
whether an agency is following a "best of breed" or a "best of suite"
approach, although we appropriately recognize that this analysis and
evaluation is more vital in a "best of breed" effort.
Instead, our finding and recommendation is based on whether NASA is
following, and plans to follow, methodologically based dependency
analysis--a best practice for implementing commercial component-based
systems--in acquiring IFMP. In this regard, documentation that NASA
provided us during the course of our review, and that it provided
following its receipt of our draft report, both of which NASA cited in
its comments, does not offer convincing evidence that NASA is following
this best practice. For example, the documentation lacked product
descriptions and comparisons as well as any analysis of integration
requirements. Moreover, the Deputy Program Manager responsible for IFMP
integration told us during the course of our review that proactive
analysis of prospective IFMP components' dependencies had not been
performed and was not planned, and that NASA did not have a methodology
for doing such analysis. The Deputy Program Manager for IFMP
integration added, similar to NASA's comments on a draft of this
report, that the agency's use of an enterprise application integration
product and its associated tools will make integration easy and will
negate the need for proactive dependency analysis. As noted above, we
recognize that this product and tool set facilitates integration of
multiple system components. However, it does not negate the need for
dependency analysis and understanding to support informed decision-
making before integration begins. As we state in our report, without
such a proactive approach to acquiring system components, the risk of
component product incompatibilities increases, as do the challenges and
complexities that integration products and tools must overcome in
integrating the products.
User Needs:
NASA agreed that deployed and in-deployment modules do not yet meet all
the needs of program managers. NASA indicated that this was the result
of its "step-wise" approach in implementing the core financial module
first and then integrating follow-on modules at a later date. As noted
in our report, however, the deferral of many basic management functions
has resulted in critical NASA programs, such as the ISS, using other
systems to monitor contract cost and schedule performance. By not
including the cost and schedule information needed by program managers
in the core financial module, NASA risks operating with two sets of
books--one that is used to report to management and the Congress and
another that is used to manage NASA's programs. NASA disagreed with our
specific findings related to user needs in three key areas:
* NASA believes that we have understated its accomplishments and the
significance of the current capabilities delivered by the core
financial module.
* NASA took issue with our assessment of the level of detail maintained
in the core financial module, but did not comment specifically on our
recommendation that the agency reengineer its acquisition management
processes, particularly with respect to the consistency and detail of
budgeted and actual cost and schedule data provided by contractors.
* NASA disagrees with our characterization that key users were not
actively involved in the implementation of the core financial module or
defining system requirements, although NASA indicates that better
coordination was needed between program managers and the financial
management community.
First, we acknowledge again the significant effort that NASA has put
into this project. Moving from 10 separate, incompatible systems to a
single integrated financial management system is a major, complex
undertaking. However, as we discussed previously in this report, the
core financial module falls short of NASA's own representation of the
module's capabilities, which was to provide program managers with the
information required for day-to-day decision making. Specifically, it
does not provide integrated cost and schedule performance information
needed by program managers to oversee many of NASA's largest and most
complex contracts. In commenting on a draft of this report, NASA
officials stated that it was never NASA's intent to integrate schedule
data with the initial core financial module implementation. However,
IFMP planning documents (including its program plan and business case
analysis), congressional testimony by NASA's Administrator, and NASA's
own press releases clearly established an expectation that the core
financial module would remedy many of NASA's long-standing management
challenges by providing program managers and other users with
integrated financial and performance information. For example,
according to his testimony before the House of Representatives
Committee on Science on February 27, 2002, the NASA Administrator
stated that while all components of IFMP are important, the successful
completion of the core financial project will satisfy the Office of
Management and Budget requirement that the financial and performance
management systems supporting day-to-day operations are fully:
integrated.[Footnote 20] NASA responded that it is currently in the
process of engaging program managers and defining specific requirements
related to needed cost and schedule performance data.
Second, we recognize that the commercial components NASA has selected
for its core financial module are technologically capable of capturing
and maintaining the detailed cost data required by program managers and
cost estimators. However, the level of detailed cost data currently
maintained in the core financial module depends on the level of detail
provided by NASA's contractors and the coding structure embedded in the
core financial module. With respect to the level of detail provided by
contractors, we reported that NASA has not reengineered its acquisition
management processes to ensure that contractors are consistently
providing the detailed cost data needed by program managers and cost
estimators and recommended that NASA do so.
NASA did not specifically address our recommendation but stated that it
is incumbent upon program managers and cost estimators to learn and
understand the capabilities of the new module and take advantage of
them for their specific purposes. NASA's comments also indicate that
the data structure in the core financial module would be extended
beyond the current legacy capabilities (i.e., the module will be able
to record a greater level of detail) in fiscal year 2004. However,
increasing the module's capacity to store greater detail will not
ensure that the information needed by program managers and cost
estimators is requested and received from contractors and subsequently
updated in the module. Although NASA commented that it would review its
current project management process to ensure that its contractors
provide the appropriate levels of cost data, we continue to believe
that any effort that falls short of end-to-end business process
reengineering will not result in a system that substantially improves
the data available for contract oversight and decision making.
Third, we acknowledge that the IFMP implementation team made an effort
to include resource management staff from program management offices in
various process teams. However, as we discussed previously in this
report, no effort was made to include the cost estimating community in
these efforts. While program management office staff did participate,
these efforts did not address the program cost and schedule needs of
program managers or cost estimators. For example, the program
management staff with whom we spoke, who worked on three of NASA's
largest programs (ISS, Space Shuttle, and Space Launch Initiative),
viewed the core financial module as an "accounting system" that would
be used by the accountants but was not necessarily going to change the
way that program managers manage their programs. With this
understanding, it is not surprising that the core financial module does
not meet the needs of program managers or cost estimators. Implementing
an integrated financial management system that is intended to change
the way an organization does business is extremely complex and involves
cultural, organizational, and process improvements. It also means
making financial management an agency-wide priority. Our work at
leading public and private sector organizations has shown that
implementing a financial management system that meets the
organization's business needs takes more than just placing a handful of
business or line management representatives on the implementation team.
NASA's approach has resulted in a core financial module that will be of
limited value to program managers and cost estimators, who will
continue to rely on other systems or ad hoc processes to get the data
they need. As such, implementation of the core financial module to date
continues to foster the concern that, at NASA, finance is not viewed as
an integral part of NASA's program management decision process.
Requirements and Testing:
NASA generally agreed that improvements were needed in its requirements
management and testing processes and has stated that it has already
begun to make improvements. For example, NASA recognized the need to
implement a more rigorous regression testing methodology and stated
that by October 2003 it would have an improved regression testing
program. NASA also recognized that its process for tracing requirements
and testing needed improvement and stated that it planned to implement
improved capability and functionality for traceability over the next
few months.
However, according to NASA officials, they are following best practices
for implementing an ERP solution and have "defined and implemented
rigorous, closed-loop requirements and testing processes." Further,
regarding the applicability of requirements management standards, NASA
did not agree that IEEE 830-1998 was applicable to the IFMP since it
was an ERP implementation effort. NASA stated that specifying detailed
requirements for already-developed software is high risk and that other
leading industry experts have told them that NASA needed to change its
processes to conform to the capabilities of the commercial software
selected rather than attempt to change the software to conform to the
existing NASA processes. We agree with NASA's position that it needs to
change its business processes to conform to the software; however, we
do not agree with the agency's position that detailed requirements are
not needed. We continue to believe that NASA needs to properly
configure the software based on detailed requirements in a manner that
supports the business processes that have been adopted from the
selected ERP solution. Because it has not done so, we continue to
believe that NASA has not effectively implemented the types of
disciplined processes necessary to reduce this project's risks to
acceptable levels. Acceptable levels refer to the fact that any systems
acquisition effort, such as that being undertaken by NASA, will have
some requirements-related defects. However, the goal is to reduce the
risks and prevent significant requirements defects in order to limit
the negative impact of these defects on cost, timeliness, and
performance of the project.
During our review, we discussed with IFMP officials our concerns about
the lack of documented, detailed system requirements for implementing
the core financial module. In those meetings, we recognized that NASA's
approach for developing requirements was based on a business process
model and did not disagree that this approach could be used to define
how NASA would implement the necessary functionality. However, we
continue to believe that once the business processes are defined and
selected, documented, detailed requirements are necessary to reduce the
risks of implementing the selected processes to acceptable levels. As
NASA noted in its comments, its consultants also recommended that NASA
needed to "determine the requirements while putting together the design
process." Therefore, guidance provided by the IEEE standard is
applicable to the successful configuration and implementation of
commercial software packages and is useful to help gauge the
effectiveness of those efforts.
As agreed with your offices, unless you announce its contents earlier,
we will not distribute this report further until 30 days from its date.
At that time, we will send copies to interested congressional
committees, the NASA Administrator, and the Director of the Office of
Management and Budget. We will make copies available to others upon
request. In addition, the report will be available at no charge on the
GAO Web site at http://www.gao.gov.
If you or your staffs have any questions concerning this report, please
contact Gregory D. Kutz at (202) 512-9505 or kutzg@gao.gov, Randolph C.
Hite at (202) 512-6256 or hiter@gao.gov, Allen Li at (202) 512-4841 or
lia@gao.gov, or Keith A. Rhodes at (202) 512-6412 or rhodesk@gao.gov.
Key contributors to this report are acknowledged in appendix III.
Signed by:
Gregory D. Kutz
Director
Financial Management and Assurance :
Allen Li
Director
Acquisition and Sourcing Management:
Randolph C. Hite
Director
Information Technology Architecture
and Systems Issues:
Keith A. Rhodes
Chief Technologist
Applied Research and Methods
[End of section]
Appendixes:
Appendix I: Objectives, Scope, and Methodology:
To determine whether the National Aeronautics and Space Administration
(NASA) is effectively managing the Integrated Financial Management
Program (IFMP) acquisition, we reviewed relevant program-level
acquisition management documentation to obtain an understanding of
NASA's plans and strategy, including the program overview, program-and
project-level management plans, the acquisition strategy,
implementation and integration plans, briefing materials on the
agency's plans to develop an information architecture, and a report on
IFMP lessons learned by NASA's consultant, Gartner, Inc. We also
interviewed various program officials, including the Program Executive
Officer for IFMP, the IFMP Program Director, the IFMP Deputy Program
Director, the Core Financial Project Manager, the Integration Office
Deputy Program Manager, and the Chief Information Officer to clarify
our understanding of the agency's strategy and obtain current
information on the status of the agency's efforts. Specifically, we
inquired as to NASA's basis for selecting already-acquired commercial
products and its plans for selecting future modules. We then compared
NASA's plans and activities to relevant best practices, Office of
Management and Budget (OMB) requirements, federal guidance, and NASA
procedures and guidance.
We had also intended to include our assessment of a key element of
NASA's acquisition strategy--whether NASA was acquiring IFMP components
in the context of an enterprise architecture--in this report. However,
because NASA did not provide the data needed to complete our assessment
until after the conclusion of our fieldwork, we plan to address NASA's
enterprise architecture in a future product.
To determine whether NASA had adequately considered the information
needs of key users in implementing the core financial module of IFMP,
we reviewed IFMP documents discussing the business case and properties
of the core financial module and spoke with IFMP implementers at
Marshall Space Flight Center--the lead center on this project--and NASA
headquarters. To determine whether the data requirements, as
established by the IFMP implementers, would address NASA's known
problems with cost control and cost tracking, we spoke with program
managers involved in three of NASA's largest programs and other NASA
program and business management staff at three centers--Marshall Space
Flight Center, Johnson Space Center, and Glenn Research Center. We also
reviewed prior work on NASA's cost problems, including the report by
the International Space Station Management and Cost Evaluation Task
Force, which reviewed the recent cost growth in that program and
identified causes and necessary actions.
In addition to speaking with program managers and their staffs, we
spoke with cost estimators at the three centers mentioned above as well
as Langley Research Center and NASA headquarters. We also spoke with
center staff who oversee and support earned value management for
programs that use that tool, and with the congressional staffs of
NASA's authorization committees. We asked them about the extent to
which they had been asked by IFMP implementers for input on their data
needs, the extent to which they had been involved in IFMP's design and
implementation, and whether they had been briefed by IFMP implementers
on the capabilities of the core financial module.
To determine what kind of cost information program managers use to
oversee their programs, we reviewed selected large, cost-type contracts
for NASA's three largest space flight programs--the International Space
Station, the Space Shuttle, and the Space Launch Initiative project
(intended to develop technologies for the next generation replacement
for the Space Shuttle.) For all three of these programs, cost control
and cost tracking have been issues of concern. The three programs
together involve most of NASA's work in the human space flight area,
which accounts for most of the agency's spending. These programs range
from relatively new (Space Launch Initiative) to quite mature (Space
Shuttle) programs and require the procurement of a wide range of goods
and services. Each of these programs is being run at multiple centers,
involves the work of multiple contractors, and uses cost-type
contracts[Footnote 21] that run for multiple years.
For the contracts we selected, we spoke to responsible personnel about
how costs are tracked and monitored, including the level of detail
provided by contractors, the format in which cost data are available,
and how contract cost data reporting requirements are developed. We
also obtained and reviewed copies of contractor financial management
reports and cost and schedule performance reports that we compared with
contract or program work breakdown structures, as well as contract cost
data reporting requirements and statements of work. We analyzed and
discussed with agency officials how all these documents and reports
related to each other and to the work breakdown structure. We also
discussed how the reported information was used by the programs and the
extent to which that information would be included in the core
financial module. We did not, however, evaluate whether the information
currently received from contractors, or represented as needed by
program managers and cost estimators, was adequate for management
purposes.
IFMP's core financial module is intended to address known problems with
NASA's program cost accounting and with its financial reporting. We did
not, however, review how the core financial module will address the
agency's financial reporting issues, including property accounting and
budgetary information, and whether the module will reduce the time and
resources needed to close the books each accounting period and reduce
the number of postclosing adjusting entries. We plan to review and
report on these issues at a later date.
To assess whether NASA had established and implemented an effective
requirements management process to support implementation of the core
financial module, we wanted to determine whether NASA had effectively
implemented (1) the disciplined processes that can reduce project risks
to acceptable levels for its requirements management process and (2)
the types of metrics to identify and quantify any weaknesses in its
requirements management process. To accomplish these objectives, we:
* reviewed various requirements documents produced for the core
financial module project, including the over 500 contract requirements
used to acquire the SAP software;
* performed an in-depth review and analysis of the 132 requirements,
which represent about 22 percent of the contract requirements,
developed for the "Managing Accounts Payable" process to determine
whether they had the attributes normally associated with good
requirements and whether these requirements traced between the various
requirements documents;
* reviewed NASA's procedures for defining its requirements management
framework and compared these procedures to its current practices;
* reviewed business processes, problem defect reports, test conditions,
test cases, and test execution logs contained in Accenture's Method
Delivery Management system--NASA's project management tool; and:
* reviewed guidance published by the Institute of Electrical and
Electronics Engineers (IEEE), Software Engineering Institute (SEI), and
publications by experts to determine the attributes that should be used
for developing good requirements and for identifying and quantifying
performance metrics.
To augment these document reviews and analyses, we interviewed
officials from NASA headquarters, Marshall Space Flight Center, and
NASA's independent verification and validation contractor--Titan
Systems Corporation. In addition, we discussed with NASA officials the
processes they used to measure the effectiveness of their requirements
management process and compared NASA's process to those used by
disciplined organizations. In order to determine the processes that can
be used to help an organization understand the effectiveness of its
processes, we used information from IEEE, SEI, and subject matter
experts.
We conducted our work at NASA headquarters in Washington, D.C.;
Marshall Space Flight Center in Huntsville, Alabama; Glenn Research
Center in Cleveland, Ohio; Johnson Space Center in Houston, Texas;
Langley Research Center in Hampton Roads, Virginia; and Goddard Space
Flight Center in Greenbelt, Maryland. We received written comments on a
draft of this report from the NASA Deputy Administrator. These comments
are addressed in the "Agency Comments and Our Evaluation" section of
this report and are reprinted in appendix II. We performed our work
from April 2002 through February 2003 in accordance with generally
accepted government auditing standards.
[End of section]
Appendix II: Comments from the National Aeronautics and Space
Administration:
National Aeronautics and Space Administration:
Office of the Administrator Washington, DC 20546-0001:
March 25, 2003:
Mr. Gregory D. Kutz Director:
Financial Management and Assurance United States General Accounting
Office Washington, DC 20548:
Dear Mr. Kutz:
Thank you for the opportunity to review and comment on the draft report
entitled, Business Modernization: Improvements Needed in Management
ofNASA's Integrated Financial Management Program (GAO-03-507). We
appreciate the GAO's continued interest in this vital program and
desire to see this undertaking successfully completed.
One of the lessons learned from our previous Integrated Financial
Management Program (IFMP) effort was that we should not try to develop
and implement the ultimate functionality of the system all at once. In
its previous effort, NASA did not adopt a scaled deployment strategy
which is a recommended best practice approach for large-scale
Enterprise Resource Planning (ERP) implementation such as our IFM
Program. That effort failed due, in part, to trying to immediately
satisfy all functional requirements by developing a complete all-
encompassing custom system.
Although we are recognizing and addressing several of the GAO's
concerns raised in the report, including already implementing some of
the proposed recommendations, we are also in disagreement with some of
the issues identified in the report. Summarized below is our responses
to each individual recommendation. The detail of our responses is found
in the enclosure to this letter.
Recommendation #1: Develop and implement a short-term plan to identify
and mitigate the risks currently associated with relying on already
deployed IFMP commercial components.
NASA concurs with this recommendation.
Related to this recommendation is an implication that IFMP's operations
are highly unstable due to inadequate requirements and testing
processes. This assertion was based on a "snapshot in time" taken by
the GAO when the Core Financials (CF) module was still in stabilization
phase after the initial "go live" at the first two Centers, Marshall
Space Flight Center (MSFC) and Glenn Research Center (GRC). The larger
challenges encountered during the initial days of operation under the
CF module at those two Centers have not been repeated in the recent "go
live" at our Wave 2 Centers, Johnson
Space Center (JSC), Kennedy Space Center (KSC), NASA Headquarters, and
the NASA Management Office at the Jet Propulsion Laboratory. While
there have been operational issues identified with the Wave 2
deployment, they have been more quickly resolved through concerted data
clean up and/or follow up user instruction. Illustratively, JSC was
able to eliminate their payment backlog within 2 weeks of their "go
live" date. Additionally, the volume of trouble tickets for Wave 2 was
significantly reduced (65 percent less per day average for the first 21
days of operations) in relation to the MSFC and GRC deployment.
In its concern of NASA's requirements processes, the report relies on
the requirements management standard IEEE 830-1998. However, this
standard does not appear to be directly applicable to our program. The
upfront scope of the published standard states, "This recommended
practice is aimed at specifying requirements of software to be
developed but also can be applied to assist in the selection of in-
house and commercial software products. However, application to
already-developed software could be counterproductive." NASA is
implementing already developed, commercial software, and, as stated
earlier, is now following best practices for an ERP implementation in
this current effort. The IFM program has defined and implemented
rigorous, closed-loop requirements and testing processes, and is using
commercial tools and metrics to manage these processes. Though the
capabilities in place provide for requirements testing traceability,
NASA agrees that they can be cumbersome, unwieldy, and in some cases,
opaque and that improvements should and will be made. The IFMP will be
implementing those improvements over the next few months on the current
Budget Formulation Project and, following GAO's recommendation, will
also require that future implementors and integrators use improved
capability and functionality for traceability in predeployment process
testing.
With respect to regression testing, IFMP's maturity in this area is
consistent with most organizations at this phase of implementation.
However, NASA recognizes the need for a more rigorous regression
testing methodology as the Core Financial project moves from
implementation to long-term operational support. Following our final
Wave 3 implementation, IFMP will be implementing a plan, coordinated
with its IV&V contractors, to ensure that structured regression testing
methods are employed on the Agencywide system to fully identify and
capture all facets of specific process-related issues. This plan will
be in place by October 2003.
Going forward, IFMP will also continue to improve its requirements and
testing processes and associated plans. However, there might still be
differences between NASA and the GAO with respect to how ERP
implementations should be managed. I propose
that NASA and the GAO meet jointly with industry-leading ERP experts to
better understand and ensure that IFMP is indeed following best
practices.
Finally, in the relationship between the new Core Financials system and
the level of detail for cost estimation, our new system has been
configured to accept many additional elements beyond those available in
the existing Agency coding structure. The use of
these additional capabilitie is a function of the Centers and Program/
Project management constituencies' ability and interest, including cost
estimators, to learn and understand the capabilities of the new module
and to take advantage of them for their specific purposes. We do
acknowledge that this education has not been consistently applied
throughout the Agency. NASA has committed to immediately review its
current project management processes to ensure that management, at the
initiation of new projects, rigorously defines and implements the
appropriate levels of cost reporting based on future analytical needs,
and better coordinate with the financial management community to ensure
that the Core Financials system is appropriately configured.
Recommendation #2: Develop and implement a longer-term strategy for
acquiring additional IFMP components that includes implementing a
methodology for commercial system component dependency analysis.
NASA agrees with the importance of having a long-term approach for
acquiring additional IFM components and believes it has an effective
strategy already in place.
Detailed documentation had been previously provided to GAO describing
how the IFM program began performing component dependency analysis
prior to the final ERP product (SAP) selection for its Core Financials
module, and that NASA was developing an Enterprise Application
Integration (EAI) architecture to be used as a framework for both
managing the integration process and addressing associated
complexities. The risks associated with integration were identified
early in the process, and appropriate mitigation strategies were
developed and implemented. As noted in the report, the need for
extensive dependency analysis and associated integration risks can be
mitigated by using an ERP suite integration strategy (Best of Suite)
rather than trying to integrate third-party products (Best of Breed).
However, we take exception to the assertion that NASA is not following
that approach. NASA has been following this approach and is, in fact,
recognized by SAP as a "platinum" customer, SAP's highest level used as
a reference for other customers.
The third-party products identified in the report, as evidence that
NASA is not using a "Best of Suite" strategy, provided required
functionality that either did not exist in a SAP suite model or, if
available from SAP, did not meet published Federal requirements.
Codependency with the Core Financials functionality was fully evaluated
and ranked before final product selection was performed. Furthermore,
NASA has, and continues to follow, a rigorous process for gap
identification and management. Consequently, our Core Financials
implementation required only 15 extensions (which are unique software
capabilities, necessary to meet customer-driven business needs,
developed and
maintained using SAP tools), a single third-party product (a credit
card tracking and reconciliation tool), and no modifications to the
COTS core software. In comparison to widely used industry benchmarks,
NASA's has implemented few additional custom capabilities relative to
its basic COTS configuration which helped reduce operational risk.
The new e-Gov initiatives and other Federal mandates to cross-service
applications (e.g., e-Payroll) have the potential to weaken our "Best
of Suite" strategy. Nevertheless, NASA has not deviated from its "Best
of Suite" strategy, wherever feasible. This is most clearly evidenced
by the fact that every major product selection since selecting SAP for
its Core Financials module has resulted in extending the SAP suite
rather than adopting "Best of Breed" third party applications. These
selections include Business Warehouse (for reporting), Strategic
Enterprise Management (for Budget Formulation), and Asset Management.
Furthermore, we have initiated efforts with SAP to improve their
products in support of future projects where we identified gaps in
functionality.
We would like to also point out that since FY 2000, (which was when
this critical program was reformulated), the IFM program has fully and
successfully deployed its Resume Management and Position Description
Management modules Agencywide, and has implemented its Travel
Management module at nine of 10 Centers (the 10TH Center will be
operational next month). Since becoming operational in late 2001, the
Resume Management system (NASA STARS) has been recognized as a model in
the Federal Government for staffing and recruiting systems, has
received numerous accolades, and is the benchmark that Office of
Personnel Management (OPM) has selected for the Recruitment One e-Gov
initiative.
Our "backbone" system module for Financial Management, Core Financials,
has been implemented at six Centers and is on schedule to complete
implementation at the remaining Centers this coming June. In comparison
to similar ERP implementations of the size and complexity of our Core
Financial project where each Center is actually a new implementation
due to their historical incompatibilities, NASA's rollout to date has
been relatively "quiet". Accenture, the implementation contractor for
the Core Financial Project, has implemented SAP on over 750
engagements. Their characterization of the most recent implementation
at JSC is that it was probably the smoothest they've ever witnessed.
Finally, it should be noted that our IFM program is on budget and on
schedule. Please do not hesitate to contact me, or the Program's PEO,
if you have any further questions about this response.
Frederick D. Gregory
Deputy Administrator:
Signed by Frederick D. Gregory:
Enclosure:
NASAS ACQUISITION MANAGEMENT STRATEGY DOES NOT INCLUDE ANALYZING
COMPONENT DEPENDENCIES:
Much of the GAO's concern in this area stems from their belief that
NASA is following a "best of breed" approach. We believe there is some
misunderstanding and, perhaps, miscommunications with the GAO on NASA's
strategy with respect to "best of breed" versus "best of suite." Cited
below are several statements related to NASA's business systems
strategy, which is still considered "best of suite":
* Before IFMP acquired the Core Financials module in late FY 2000,
there was a desire for a "best of suite" strategy; however, there was a
degree of uncertainty as to how successful that could be given unique
Federal requirements, and the variability of offerings from potential
ERP providers. Our "best of suite" strategy was finalized after
selecting SAP. The IFMP is on record with numerous documents
(presentations and letters) documenting this strategy, including
several responses to Congress.
* Even with a "best of suite" strategy, IFMP has processes in place to
confirm that an ERP suite can still meet NASA's requirements.
Primarily, the Program develops business case analyses (BCA's) to
assess areas such as alternatives, costs, risks, and benefits, before
starting a project (module implementation). This is a prudent business
practice that IFMP performs, and that OMB, GAO, and other external
oversight organizations expect the Program to perform. It is therefore
possible that, moving forward, a feasible alternative could eventually
be selected outside of a "best of suite" model.
Even with the "suite" strategy, NASA recognizes that the ERP product
may need further evolution to meet Federal needs. This is evidenced by
NASA's partnership with SAP to help the software provider better
understand Federal requirements and to evolve their Budget Formulation
and Human Resources functionality. IFMP would not continue to work with
SAP in this partnership arrangement if it did not have intentions to
extend the suite for NASA's use.
* The COTS software used in the three projects identified by GAO
provides functionality not found in ERP suite software (i.e., SAP).
This "functionality not found" is termed "gap." NASA has established a
prioritization process for filling gaps that assess:
* Alternative configuration approaches to achieve targeted
functionality;
* Adapt NASA business processes consistent with COTS
capability;
* Use functionality extensions based on SAP tools and user exits; 0
Integrate third-party products; and finally,
* Modify the basic software base.
Although this process does allow the potential to fill functionality
gaps using third party software solutions, our prioritization ensures
that such an approach is adopted only if there are critical losses of
functionality. This process necessarily involves a detailed evaluation
of data dependencies. Furthermore in the report, the three products
selected have limited or no integration requirements with the Core
Financial function.
These selection decisions do not imply that the program deviated from
"best of suite". In the larger groupings of finance, budgeting, asset
management, and Human Resources, NASA still has a "best of suite"
strategy. The new e-Gov initiatives and other Federal mandates to
cross-service applications (e.g., e-Payroll) might potentially impact
NASA's overall "best of suite" model. Nevertheless, to date, NASA has
not yet deviated from its established "best of suite" strategy.
* Since selecting SAP for Core Financials, IFMP has made three major
"selection" decisions which have extended the SAP suite. These are
Business Warehouse (for reporting), Strategic Enterprise Management
(for Budget Formulation), and Asset Management.
The report also contends that NASA has not performed a detailed enough
dependency analysis, and does not have a methodology for doing so, and
lacking such analysis will lead to a more complicated and risk-prone
integration. We believe this GAO concern was addressed at a March 10
meeting that occurred after the initial report draft. At this meeting,
the GAO was provided detailed documentation tracing the evolution of
NASA's current Enterprise architecture and the linkage to the
architecture that was employed in the acquisition, priorities, and
decisions made by the IFMP. Included in this documentation was clear
evidence that the program began performing dependency analysis prior to
ERP product (SAP) selection (Core Financials module), and was
developing an Enterprise Application Integration (EAI) architecture
which would serve as a framework for both managing the process of
integration and for easing the associated complexities. The risks
associated with the integration were recognized, and appropriate
mitigation strategies were followed. To date, the Core Financials
system has been successfully integrated with one third party product
and 35 legacy systems.
In summary, the GAO's concern about complex product integration is
mitigated in two ways: (1) IFMP does have a "best of suite" strategy
focused around the SAP product; and (2) IFMP has developed an EAI
framework and is already successfully using this EAI tool set
supporting requirements for legacy system interfaces and third-party
product integration.
CORE FINANCIALS MODULE DOES NOT FULLY ADDRESS KEY USER INFORMATION
REQUIREMENTS:
If Implemented as Planned, the Core Financials Module May Provide Some
Improvements:
The report understates the significance of the current capabilities
delivered by the Core Financials module.
* The Core Financials module has clear improvements over NASA's
previous accounting systems providing plan and actual financial
expenditures with seamless visibility across Center and organization
lines. This has been a fundamental limitation on Program and Project
management for large activities that cross Center lines.
* The module also has real-time processing and reporting capabilities
that provide immediate access to key financial information and
eliminates the separate databases for procurement and financial
activities which had existed previously.
* Finally, this module is the backbone of the IFMP, establishing the
SAP suite, creating a business warehouse for enhanced data access and
information delivery and an integration architecture to tie in legacy
systems, external data repositories, and other software products. When
it becomes an Agencywide operation in June, it will enable NASA to
transition to full cost management practices.
As an enabler of NASA's full cost initiative, the Core Financials
system will provide the tools for applying full cost practices that
will help lead to more efficient, optimal use of institutional
resources such as:
* Justification of Institutional resources based on real-time project
requirements;
* Elimination of "free" resources. Program and project
managers will have the insight necessary in defining more accurately
institutional capabilities; and:
* Funding for service pool allocations based on demand/consumption
rather than with a parametric formula used with old program support
accounts.
As stated in the report, the Core Financials system will eliminate
inconsistent data through single data entry and provide for data
standardization. The module will provide NASA with an integrated, on-
line access to standard information across different business
processes, NASA sites, and programs. This is a major leap forward for
NASA. Furthermore, the Core Financials system is a highly scalable and
configurable commercial solution that is flexible enough to respond to
the ever present and continuously evolving Financial Management and
Reporting requirements.
Key Users Were Not Involved in the Implementation of the Core
Financials Module:
The report observes that since financial managers were the primary
source of the subject matter expertise for the Core Financial Project,
NASA did not reengineer its financial management processes to support
broader management challenges. We disagree with this assumption.
Representatives from our project resource management and procurement
communities were heavily involved in the requirement definition and
process design activities for the Core Financials module. These
communities participated in baselining the original requirement set and
processes, in adapting these requirements and processes to the
realities and best practices inherent in the selected Commercial Off-
the-Shelf (COTS) software products, and in the implementation of those
resulting processes at the Centers.
When the IFMP set out on its current effort to acquire and implement a
core financial system for the Agency, it had an extensive
experiencebase in defining NASA's requirements for such a system. NASA
began its work by updating its requirements from previous efforts to
focus on the specific scope that the Core Financial Project was charged
with fulfilling. The scope of the Core Financial module was guided by
lessons learned from the previous effort, as well as recommendations
from a NASA OIG audit report on the reasons for failure of the prior
effort. The JFMIP Framework for Federal
Financial Management Systems was also used to guide the scope
definition for the project. The latest JFMIP Core Financial System
Requirements document was used to update the requirement set to match
the latest Federal financial system guidance. NASA also revised its
core financial requirement set to incorporate lessons learned from the
previous efforts, and to confirm that the new requirement sets were in
line with the scope of the project. Per the direction of the IFMP
Steering Council, the new requirements set were to include budget
execution, purchasing, accounts payable, accounts receivable, cost
management, standard general ledger, and fixed assets.
To finalize these new Core Financial requirements, NASA assembled an
Agencywide team composed of representatives from across the financial,
resource/project management, procurement, and asset management
communities. This Agencywide Process Team started out with
approximately 40 team members and quickly grew to over 50 core team
members with another 100 members participating on a part time basis
throughout the Agency Design and Pilot Center implementation. Within
this group of subject matter experts, the budget execution and cost
management subprocess teams included participants from the project
resource management community. The purchasing subprocess team was
heavily leveraged with representatives from the procurement management
community.
During the Pilot Center implementation, Core Financial team members met
with the Project office representatives and focus groups on numerous
occasions to understand their information needs. Detailed working
sessions were conducted as the team developed the final inventory of
interfaces required for legacy Center systems, many of which were
resident in the project offices across the Center. By working with
representatives from each of these offices, the Core Financial team was
able to ensure that the information needs of these organizations would
be provided through the reporting capability in the SAP Business
Warehouse solution being delivered as part of the Core Financial
implementation. The team met on a scheduled basis with project office
representatives to understand their current use of the legacy coding
structure as they planned for the conversion of legacy data into the
new financial classification structure in SAP. In addition, the team
conducted focus group discussions with key Center project resource
management representatives to understand the priorities of the project
office community among the functional drivers defined for the Core
Financial Project. Further, the team utilized an anonymous electronic
polling tool to gather objective input and rankings from the
participants. These prioritized project functional drivers, as
documented in the Business Case Analysis for the Core Financial
Project, have been used throughout the design and implementation effort
to support issue resolution and to weigh advantages and disadvantages
among alternative solutions.
As the Core Financial project began Agency rollout, each NASA Center
formed an implementation team for the Core Financial deployment. These
teams included participation from the resources and project management
communities. Of particular significance is the degree of participation
by the International Space Station (ISS) Program Office at JSC. The JSC
Core Financial Implementation Team included a full-time member from the
ISS Program Office. This team member was instrumental in
ensuring that Space Station reporting requirements were considered in
the implementation at JSC.
The Core Financial Modules Will Not Provide the Information Needed to
Manage Contracts:
The report states that "the core financial module is not being
implemented to ....... maintain cost data at a sufficient level of
detail for certain contracts." However, the Core Financials module has
significant ability to capture contract costs at very detailed levels
of a work breakdown structure (WBS). The SAP system has been configured
to capture approximately 30 additional elements above those
accommodated in the existing Agency coding structure. However, as noted
correctly in the report, the use of these additional capabilities is
incumbent on the Center and Program/Project management communities,
including cost estimators, to understand the possibilities and to take
advantage of them for their specific purposes. NASA's current policy
for contractor cost capture:
(NPD 9501.1G ) also recognizes this upfront coordination, stating that
"reporting requirements necessary for the management of a project will
be determined as early as possible in the project planning stage, as a
team effort involving all organizations which will play a role in
monitoring the contract during the performance period. The cognizant
procurement, technical, project, financial, and resources management
offices shall be a part of the reporting structure development and
approval process.":
Acknowledging that contractor cost capture has not been consistently
applied, NASA has committed to immediately review its current project
management processes to ensure that project management, at the
initiation of new projects, rigorously review and establish the
appropriate levels of cost reporting based on future analysis needs
(e.g., cost estimating), and coordinate with their financial
counterparts to affirm that the Core Financial system is configured
appropriately.
The Core Financial Module Does Not Integrate Cost and Schedule Data
Needed by Program Managers:
Though NASA agrees with the GAO that schedule integration is needed
with the Core Financial module, it was never NASA's intent to integrate
schedule data with the initial Core Financial implementation. One of
the lessons learned from the previous effort is that we cannot develop
and implement all required functionality at once. This step-wise
approach is a fundamental best practice of large-scale ERP
implementations which NASA did not follow on that previous effort. That
effort failed. Trying to satisfy all requirements needs for an initial
rollout of an ERP system bears a level of risk that was judged
unacceptable by the Program. For this reason, NASA's approach was to
first implement the Core Financial module. Once this financial
"backbone" is built (to be complete this summer), NASA plans to
integrate the follow-on modules needed to complete the overall IFM
Enterprise system environment.
Part of this planned follow-on is an evaluation of additional project
management capabilities. Schedule integration capability is inherent in
the Project Systems module of the SAP product on which the Core
Financial system is built. Additionally, interfaces are available to
integrate SAP with NASA's two primary schedule tools, Primavera and
Microsoft Project. In addition, SAP has very powerful mechanisms for
capturing and reporting performance metrics related to earned value.
Over the past year, NASA has performed two separate studies related to
the use of SAP's Project Systems. Both studies confirmed the robust
capabilities inherent in the tool, as well as its ability to interface
with leading third-party schedule tools. The IFMP has been working
closely with the Office of the Chief Engineer and the Program
Management Committee Working Group (PMCWG) in defining the requirements
for a pilot on one or two new projects using Project Systems
capabilities. This process is currently underway and is due to
completed this summer.
Currently, it is indeed correct that the deployed and in-deployment
modules do not yet meet all the needs of the Program management
community. Our Program and Project management activities have a need to
use information not only from Budget Execution (Core Financials), but
also integrate data from Budget Formulation, Human Capital, Asset
Management and Contract Administration to get the full benefits of the
IFM system. Those modules are on schedule to be deployed as planned.
The Core Financials Module Will Rely on Legacy Coding Structure:
The heading within this section of the report could imply that NASA
will continue to rely on its legacy coding and that the system will not
meet Project Manager needs for lower levels of detail. As the GAO
recognized in its report, "...the core financial module is capable of
capturing this more detailed cost data, however, due to the complexity
associated with converting detailed data from the center's legacy
systems, NASA has deferred this capability." It is true that these
capabilities, although designed into the system, have not been fully
used during the FY 2003 transition year. The major constraint in FY
2003 is that Centers are "going live" throughout the year and the
primary driver for the level of detail is the conversion of existing
legacy systems. To attempt to transform the legacy data into a lower
level of detail as part of the conversion process would destroy the
integrity of the information being converted as well as the audit
traceability from the legacy system to SAP. Since part of the year will
utilize the legacy systems and the other part will utilize the new
systems, it is not appropriate to change the level of detail partway
through the fiscal year. In FY 2004, however, NASA will operate fully
under the new financial module, and the data structures can and will be
extended beyond the legacy capabilities. These decisions will be driven
by the needs of Program and Project managers. Further, the data
structure is scheduled to be adapted to implement the forthcoming full
cost accounting and reporting requirements.
For future processing, the configuration of the SAP system provides
extensive capabilities beyond those available today in the current
Agencywide coding structure (AWCS). The SAP system has been configured
to capture approximately 30 additional elements beyond those
accommodated in the existing Agency coding structure. Currently, NASA
has fielded a Full Cost Implementation team to define the budget
structure to be used in FY 2004, and to be configured in the Core
Financial system.
This team consists of nine working groups, includes many Program and
Project managers and representatives, as well as cost estimators, and
works closely with the IFMP to ensure
that the configuration will be ready to receive and execute the budget
under the new environment.
Core Financial Module Will Not Provide The Information Needed To
Prepare Credible Cost Estimates:
The report observes that "...the core financial module will not contain
sufficiently detailed historical cost data necessary for projecting
future costs, cost estimators will continue to rely on labor-intensive
data collection efforts after a program is completed." As noted earlier
in this response, it was never NASA's intent to convert low levels of
historical cost data from legacy systems into the new financial system.
The Core Financial system is capable of tracking costs at a very
detailed level; however, as explained previously, NASA cannot fully
transition lower levels of reporting until:
FY 2004. At that time, the project manager will ensure that the
appropriate levels of costs are captured using NASA's contractor cost
reporting (i.e., 533 process) so that, going forward, the project can
be effectively managed and an appropriate level of detail is available
to support cost estimators' needs.
With respect to the detail required for cost estimating, the
granularity and fidelity of the cost estimates within NASA vary by
program phase, purpose of the cost estimate, and the availability of
estimating resources. Some estimates are done at the lower levels of
the VWBS, as suggested by GAO. However, other estimates are done at a
much higher levels of the WBS. For example, commitment estimates in
NASA are performed near the Preliminary Design Review (PDR) at which
the designs are typically at the "fan, compressor, turbine level"
(e.g., GAO Report Figure 2) or, at most, one level below.
As previously noted, the SAP system has specifically been configured to
capture many additional elements above those accommodated in the
existing Agency coding structure. The use of these additional
capabilities is in some cases a function of the Center and Program/
Project management communities, including cost estimators, to
understand the possibilities and to take advantage of them for their
specific purposes. Recognizing that this has not been consistently
applied within this Agency, NASA has committed to review its current
project management processes (e.g., NPG 7120.5) to ensure that project
management, at the initiation of new projects, rigorously review and
establish the appropriate levels of cost reporting based on future
analysis needs, and coordinate with their financial counterparts to
affirm that the Core Financial system is configured appropriately.
Even with an appropriate cost capture system in place, there might be
occasions when cost estimators require levels of detail or specific
data elements that are not stored in its financial systems. It is
NASA's position that it is more cost effective to research these
information needs from various sources, rather than attempt, upfront,
to forecast the universe of information required for cost estimators
and then develop the means to capture that information into a single
source system.
Core Financial Module May Not Provide Better Information for
Congressional Oversight:
We acknowledge that NASA did not specifically consult with
congressional staff concerning the specifics for the design of the new
system. They have been clear in laws (e.g., the Chief Financial Officer
Act), report language, briefings, and hearings concerning their
expectations for the results of IFMP. This direction has established a
number of specific requirements embodied in the Joint Financial
Management Improvement Program Federal requirements and enabling
certification process. These requirements were mandatory design
requirements for NASA. In addition, key system features were confirmed
with Agency process owners who respond to congressional inquiries and
other external regulatory Agency requirements on a daily basis. Based
on the GAO concern, we will increase our efforts to interact with
congressional staff on a regular basis and to ensure that their
requirements are identified and addressed.
NASA'S REQUIREMENTS MANAGEMENT PROCESS FOR THE CORE FINANCIAL MODULE IS
INEFFECTIVE:
NASA Requirements Management Process Was Not Designed to Provide
Detailed System Requirements:
The report appears to base its review of the Program's requirements
processes on a standard which is primarily aimed at software
development projects akin to other NASA efforts. In this instance, NASA
is using already developed commercial software. The report refers to
IEEE 830-1998 (Recommended Practice for Software Requirements
Specifications) in which the upfront scope states: "This recommended
practice is aimed at specifying requirements of software to be
developed but also can be applied to assist in the selection of in-
house and commercial software products. However, application to
already-developed software could be counterproductive.":
Lessons learned from other organizations like Microsoft, Northrop
Grumman, and Apple computers are very consistent. To be successful with
ongoing ERP implementations, NASA needs to change its processes to fit
the capabilities of the system, not the other way around. Using
traditional detailed process requirements driving a high level of
modifications to the core software is extremely high risk. In fact,
recommendations from a recent Gartner report (the same document cited
in the GAO report) confirm this: "Rather than flushing out requirements
and then asking the implementor to implement them - the requirements
can change, or are so strongly set that they require a rigorous design
process to meet them even though they don't meet best practices - work
with the integrator or someone who knows SAP and mutually determine the
requirements while putting together the design process.":
Part of any organization's ERP implementation effort includes an
adjustment and paradigm shift regarding project approach and
methodology. This is particularly true when compared to "custom-
developed" software implementations. One reason to pursue a COTS
implementation, such as ours, is the benefit of proven software
functionality and vendor support/maintenance. The SAP R/3 product is
widely used in many commercial industries (over 12,000 customers at
20,000 installations worldwide). The core R/3
foundation has been enhanced and extended to meet U.S. Federal
Government specifics, and has successfully been JFMIP certified. JFMIP
uses an extensive qualification test to determine whether a software
package complies with all mandatory core Federal accounting
requirements.
Requirements Were Not Specific:
The two requirements noted by the report are JFMIP requirements that
were used in the COTS acquisition. In the first requirement, related to
the ability to query information contained in the system, the report
has concerns about the level of specificity because the requirement did
not clearly state the data elements to be supported in the query. NASA
established that the SAP system has basic query capabilities. As part
of the implementation services acquisition effort, NASA delivered
additional requirements for reporting (over 100 reports were defined
including report purpose, selection options, data elements, report
media, and frequency). As part of the configuration effort, NASA
subject-matter experts established detailed reporting requirements
that not only include what data was to be addressed but, also, whether
or not the query would be done through formal reporting or "ad hoc"
queries for analytical purposes. Those configurations were then
validated by each of the 10 Centers implementing the system.
Traceability Was Not Maintained:
An Independent Validation and Verification (IV&V) contractor (Titan
Corp) was utilized on the Core Financial project with the stated
objective to minimize the risk of the SAP software implementation and
maximize confidence in operational readiness. The IV&V's ongoing review
focused on requirements traceability and testing and included an
analysis of the configuration of the Core Financial solution. A recap
of the IV&V summary of Level IV to V Configuration Traceability
Analysis is included below:
"In terms of the Level IV to Level V configuration traceability
analysis, IV&V found, with one exception, that the key SAP
configuration and master data elements are configured to support NASA
H.2 requirements. The exception is the SAP "demonstration " code values
for the key SAP fields of plant code and storeroom, which IV&V strongly
recommends be removed from the production configuration.
IV&V commends the Accenture and NASA teams on their approach,
recommendations and choices regarding SAP configuration options.
Unnecessary complexities were carefully avoided, resulting in a very
streamlined and straightforward SAP configuration, reserving a great
deal of f exibility for the future.
IV&V also notes information regarding NASA centers has been configured
in a consistent and parallel fashion within the various underlying SAP
configuration/master data tables, that tables added to SAP specifically
for NASA use are referentially consistent with appropriate SAP master
validation tables, and that the use of custom code for the NASA
deployment has been minimized. ":
However, on a negative note, the IV&V also noted that traceability
"... was found to be maintained in a fragmented manner in the Accenture
provided Lotus Notes Method Delivery Management (MDM) system, by a
linkage of both levels of requirements through business processes. The
IV&V Team recommends that future IFMP implementations develop
documentation that explicitly details the relationship between lower
level requirements and the requirements of the next higher level. ":
Although NASA has maintained adequate traceability, as noted by the
IV&V, and can identify the linkages between the various requirements
and testing levels, the method in which this is performed is cumbersome
and needs improvement. NASA agrees with this recommendation and has
already taken steps to improve traceability within the Budget
Formulation Project, noting lessons learned and areas for improvement.
Additionally, IFMP will ensure that future implementors/integrators of
IFMP modules have improved capabilities for traceability.
Requirements Defects Adversely Affect Testing Of The Core Financial
Module:
The report emphasizes the direct relationship between requirements and
testing, stating that "NASA's testing program has been adversely
affected by the lack of complete and specific requirements." Figure 5
in this section illustrates the report's stated relationship between
requirements and testing. In this figure, the report points out that
integration testing and unit testing "are normally handled by the COTS
vendor"; but does not reflect the fact that NASA has and is performing
extensive unit testing, through each Center's Conference Room Pilots
(CRP's), and System and Integration Testing (SIT). The Core Financial
Project has in place a rigorous, closed-loop testing program to ensure
that NASA's configuration of the core SAP software, as well as
interfaces to existing legacy systems, are functioning as intended.
Initially, NASA completed a series of 3 prototype tests of SAP
functions and configuration, consisting of over 490 business scenarios.
NASA further tested the configured and interfaced solution through 5
rounds of SIT testing at MSFC and 4 rounds at GRC. For the MSFC
testing, over 100 subject matter experts from across the Agency
participated in the Agency level testing. In preparation for the SIT
tests, NASA's requirements were decomposed into over 1700 lower level
test conditions. These test conditions were validated in 114 test cases
executed 325 times at the two Centers. Two rounds of SIT testing were
conducted against data converted into SAP from legacy systems at both
MSFC and GRC. And although, the Project felt that the testing completed
for first two Centers was extensive and thorough, additional test
execution from each Wave implementation team was planned and required.
Each additional team and test effort provides additional confirmation,
additional details and scenario variations of business processing, plus
provides additional insight and suggestions for continuous improvement.
These subsequent test efforts also ensure that the Center teams
validate the "Center-specific" components (e.g., configuration build
out of Center defined values, Center interfaces, and Center converted
data). As the Center resources' knowledge and understanding grows, the
Center is better positioned to support the new system and provide
assistance to other business end-users. They are then also able to
provide input
for enhancement and suggestions for future additional capabilities. The
information gathered and incorporated based on actual production
operations is also used to improve the subsequent implementation
efforts and provide guidance and suggestions to the Wave Centers.
This extensive level of testing exceeded the requirements found in the
Joint Financial Management Improvement Program (JFMIP) as part of their
extensive qualification test to determine whether a software package
complies with all mandatory core Federal accounting requirements.
NASA's selection of the SAP software to satisfy its Core Financial
requirements was predicated, in part, upon the successful certification
testing completed by the U.S. Department of Treasury in accordance with
the Joint Financial Management Improvement Program (JFMIP). The intent
of this certification is to ensure that software implemented by Federal
agencies in the area of financial management meets minimum Federal
requirements. In addition to this certification, SAP core software has
been implemented by over 20,000 clients worldwide. This is a proven
software product supported by a robust software development vendor with
over 80percent of the market share in the Enterprise Resource Planning
(ERP) space.
When implementing ERP software products, not only do the design and
implementation approaches change compared to "custom-developed"
efforts, the specifics of the different testing phases change somewhat
as well. Since the primary software functions are pre-defined and
validated, the customer focus should be on a different type of testing
and verification. Specifically this focus should be on the execution of
the new business processes (utilizing the new tools and system
functions) and on the verification of any customer-specific components
(such as an interface or custom report) to ensure the total solution
meets the business requirements.
Significant Defects Appeared in Production System:
The report underlines critical and high severity problems discovered
after the Core Financial system was placed in production at MSFC and
GRC, noting that the "defects could be linked back to the lack of
complete requirements." These problems, which resulted in some delays
in payment processing, stemmed mostly from issues related to the
internal posting of the software that were not previously evident
during the testing efforts. The issues were not related to a lack of
complete requirements. These posting issues occurred when invoices
against very large, multifunded, service contracts were being
processed. Some documents had hundreds of funding citations. During
NASA's testing efforts, test conditions were included to address
multifunded service contracts; however, they could not simulate
processing for the volume of line items encountered in a production
environment. These problems have all been identified and corrected and
were not experienced by any of the Wave 2 Centers. The payment backlogs
at MSFC and GRC have been reduced to that expected from normal business
operations. In addition, one of the Wave 2 Centers, JSC, was able to
process their payment backlog within their first 14 days of operations
on the Core Financial system.
Adequate Regression Testing Is Not Being Performed:
Based on interaction with other customers who have implemented large
ERP solutions and discussions with a leading ERP consulting firm, NASA
believes that the maturity of its planning in this area regression
testing is consistent with most organizations at this phase in their
implementation. Nevertheless, as noted in the report, NASA recognizes
the need for a more rigorous regression testing methodology as the Core
Financial project moves from implementation into long term sustaining
support. The IFMP Competency Center, which is the organization
responsible for Core Financials operational support, is in the process
of developing a sustaining support release strategy that defines major,
minor, and emergency releases and the associated types of regression
testing required for different types of releases. They are also
investigating the use of automated regression tools to determine if
they can deliver efficiencies in the execution of regression scripts
and allow for more comprehensive regression testing.
As the IFMP continues with implementation, it has defined and applied
processes for testing every facet of the system. The report notes that
in large ERP implementations, such as this one, it is typical for the
new application to undergo constant change as it is introduced into the
organization and stabilized. NASA's approach has been to process
reported system defects and change requests using a methodology that
allows for the application of high-priority fixes or enhancements, with
all other changes allocated to the next scheduled "wave" of NASA Center
deployments. Each wave deployment includes three phases of system
integration testing that serve as a fairly comprehensive regression
test for that release of the software. Those changes or fixes that must
be applied because of business criticality are tested based on the
applicable business scenarios. Complete regression testing of all
possible requirements impacted by these types of changes is not
feasible or practical during this stage of the implementation. The
report cites three examples where system changes have negatively
impacted system functionality. NASA acknowledges that these types of
problems have occurred, but believe that they are isolated cases and
that the continued improvement of system performance indicates that
these problems have not been pervasive.
The GAO report cites one example of a SAP-provided patch that broke
system functionality. Though there have been some issues with SAP
patches, the IFMP Competency Center continues to improve its ability to
seamlessly deploy patches. As an example, after the Wave 1 Centers were
in production with SAP, and prior to bringing up the Wave 2 Centers,
the team implemented SAP patch numbers 7, 8, and 9. These patches
contained several hundred individual repairs to the standard and public
sector SAP R/3 application code. The patches were deployed using a
staged approach, being first introduced into a separate "sandbox"
landscape, where preliminary testing was performed. The patches were
subsequently applied to a stand-alone quality assurance system, where
the team executed a formalized set of regression test cases. During
this round of testing, several significant problems were identified,
including the reoccurrence of a purchase order "short dump" issue that
had been corrected in an earlier vendor patch, as well as a problem
with the business warehouse extractors. Due to these issues, the
application of the patch to the development instance was postponed.
Further testing was performed. After the issues were addressed, the
patch was applied over a weekend to the
production instance. There was only one issue that affected production
after the patch application, which turned out to be a vendor problem
that was quickly addressed by SAP. There was no impact to any custom
developed objects or interfaces.
Performance Metrics Could Be Used To Assess Potential Risks Of
Identified Weaknesses:
The report states that "NASA is unable to implement a metrics
measurement process that allows it to understand (1) its capabilities
to manage the IFMP effort, (2) how its process problems will affect its
cost, schedule, and performance objectives, and (3) the corrective
actions needed to reduce the risks associated with the problems
identified. The IFM Program does use metrics as part of its management
processes. Metrics are extensively used on the Core Financial project
to track, analyze, and report problems to all levels of management,
including the Core Financial Project Steering Committee. Metrics on
problem reports graphically depict problems by priority, status,
category, closure rates, etc. Other metrics utilized on a recurring
basis (weekly, monthly, quarterly) include cost plan versus actual,
cost trending, schedule and accomplishment hit rates, and others.
Within the context of this discussion, the report further asserts that
NASA does not appropriately analyze its defects. We respectfully
disagree with the GAO on this point. IFMP does have structured testing
and problem analysis processes in place. Additionally, the project team
works closely with SAP and its other software vendors to obtain a
clearer understanding of the specific causes of problems to ensure that
systemic problems do not arise. This is evidenced through the recent
Wave 2 implementation. While there have been operational issues
identified with the Wave 2 cutover, they have been quickly resolved
through data clean up and/or follow up user instruction. In this
instance, JSC was able to catch up their payment backlog in the new
system within 2 weeks of their go live date. The volume of trouble
tickets for Wave 2 is significantly reduced (65% less per day average
for first 14 days of operations) from those seen in the early days of
the MSFC and GRC deployment (see Figure 1 below).
New Trouble Tickets Since Wave 1 Go-Live (10/29/02):
[See PDF for image]
[End of figure]
The following are GAO's comments on the National Aeronautics and Space
Administration's (NASA) letter dated March 25, 2003.
GAO Comments:
1. See the "Agency Comments and Our Evaluation" section of this report.
2. Although NASA indicates that it has extended the SAP suite to
include Business Warehouse (for reporting) and Asset Management, it did
not provide us any documentation to support these selections during the
course of our fieldwork.
3. We did not assess the deployment and operation of the three modules
to which NASA referred. We understand that the NASA Inspector General
has recently begun a review of the Travel Management module.
4. The scope of our work did not include a review of the Integrated
Financial Management Program (IFMP) budget or schedule. We plan to
address these issues in a future product.
5. As stated in this report, although the core financial module will
provide some improvement to NASA's current accounting system
environment, certain system capabilities have been deferred and will
not be available when the system becomes an agencywide operation in
June. Without an effort to reengineer NASA's acquisition management
processes, it is unlikely that detailed cost information will be
available to meet the needs of program managers, cost estimators, and
the Congress. Thus, NASA's assertion that it will be able to transition
to "full cost management practices" in June of 2003 is questionable.
6. NASA's comments refer to the "Accounts Payable" process illustrated
in figure 3 of the report. While we did not verify or evaluate the
extent to which the additional requirements to which NASA refers in its
response were established or validated, the accounts payable
requirements, as described, do not provide for quantitative evaluation
to determine whether the system meets NASA's needs. Furthermore, the
additional requirements did not provide the needed clarification for
the requirement cited in our report related to the ability to query
information. Moreover, given that NASA added new requirements for
reporting, it is unclear whether the existing accounts payable
requirement was to provide some other query functionality not included
in the other general reporting requirements.
7. As noted in our report, requirements provide the foundation for
system testing. In meetings with NASA, we acknowledged that the
"process-centric" approach that the agency adopted was an acceptable
methodology for understanding how the processes supported by the
selected enterprise resource planning (ERP) solution could be
implemented at NASA. However, we believe that this approach still
requires the development and documentation of the necessary
requirements to fully understand the functionality to be provided by a
given process. Without such requirements, a disciplined testing process
is very difficult to implement since requirements are a fundamental
attribute of an effective testing process. As discussed in our report,
we continue to believe that the lack of an effective requirements
management process hampered NASA's testing efforts since significant
defects in the production system should have been detected before
system implementation.
Although NASA stated that it will repeat its testing efforts at each
center implementing the system, without adequately documenting its
requirements and ensuring that the testing process adequately tests
those requirements, it does not have reasonable assurance that the
testing process will identify significant defects before a center is
converted to the production system. For example, NASA stated that it
had developed over 1,700 test conditions. However, it was not until the
system was placed into production that NASA identified several
significant weaknesses, as discussed in our report. We continue to
believe that NASA will not have reasonable assurance that it has
adequately tested the system until it (1) documents its requirements
and (2) develops test conditions that fully test those requirements.
8. As noted in our report, discussions with IFMP officials recognized
that a test case was not properly developed to test large contracts
that contained over 200 line items--a common occurrence according to
IFMP officials--and that this was an oversight in their testing
process. Had NASA developed and documented a detailed requirement for
this functionality and then mapped its test conditions against those
requirements, it would have recognized that it had not developed a test
condition to properly demonstrate and test the functionality prior to
the system going into production. Properly processing these types of
payments may have enabled NASA to reduce the impact of the payment
backlog.
9. As noted in our report, NASA does not have metrics that properly
analyze the cause of the defects so that it can improve its processes.
For example, although NASA was able to show the number of defects that
were related to subsequent implementation, referred to as the second
wave, it did not have information that could be used to analyze whether
these defects were caused by, for example, requirements or testing
problems or by not adequately correcting prior defects. Therefore,
although NASA states that it has a structured testing and problem
analysis process in place, we continue to believe that the examples
provided in NASA's comments do not provide the data necessary to
identify the causes of defects or assess the effectiveness of processes
such as the requirements management and testing processes. As noted in
our report, these types of data can be used to prevent or anticipate
problems before they occur, resulting in less rework.
:
[End of section]
Appendix III: GAO Contacts and Staff Acknowledgments:
GAO Contacts :
Diane Handley, (404) 679-1986
Cynthia Jackson, (202) 512-5086
Chris Martin, (202) 512-9481:
Acknowledgments:
Staff members who made key contributions to this report were Erin
Baker, Molly Boyle, Felicia Brooks, Francine DelVecchio, Michael
Giannone, Jamie Haynes, Richard J. Herley, Kristi Karls, LaTonya
Miller, Maria Storts, Eric Trout, Teresa Tucker, Carrie Wilson, and
Jenniffer Wilson.
(120148):
FOOTNOTES
[1] At that time, we began a special effort to review and report on the
federal program areas that our work had identified as high risk because
of vulnerabilities to waste, fraud, abuse, and mismanagement. We first
issued our High-Risk Series in December 1992 and have continued to
include NASA's contract management as an area of high risk since. See
U.S. General Accounting Office, High-Risk Series: NASA Contract
Management, GAO/HR-93-11 (Washington, D.C.: December 1992) and High-
Risk Series: NASA Contract Management, GAO-03-119 (Washington, D.C.:
January 2003).
[2] For this estimate, NASA has defined life cycle costs to include
implementation efforts through fiscal year 2008 and major upgrades,
plus operation and support costs for each system module for the first 2
years after the module goes live.
[3] The system is to consist of nine modules: core financial
management, resume management, travel management, position description
management, human resource management, payroll, budget formulation,
contract administration, and asset management.
[4] NASA is comprised of its headquarters offices, nine Centers located
throughout the country, and the Jet Propulsion Laboratory. The Jet
Propulsion Laboratory is operated by the California Institute of
Technology, but for purposes of this report, we treat the Jet
Propulsion Laboratory as a center.
[5] See for example, Tricia Oberndorf, Lisa Brownsword, and Carol A.
Sledge, Ph.D., An Activity Framework for COTS-Based Systems, Technical
Report CMU/SEI-2000-TR-010 (Pittsburgh, Pa.: Software Engineering
Institute, Carnegie Mellon University, October 2000).
[6] NASA has acquired the following commercial products: (1) SAP AG's
R/3, version 4.62,
(2) Gelco's Travel Manager, version 8.0, (3) Resumix, version 6, and
(4) Avue Digital Services' Position Description Management, which is a
subscription service.
[7] Gartner, Inc., A Report for NASA: IFMP Lessons Learned and Key
Considerations for Future Module Projects, January 20, 2003.
[8] U.S. General Accounting Office, Executive Guide: Creating Value
Through World-class Financial Management, GAO/AIMD-00-134 (Washington,
D.C.: April 2000).
[9] GAO/AIMD-00-134.
[10] During the time of our review, NASA was pursuing a program--known
as the Space Launch Initiative--to build a new generation of space
vehicles to replace its aging space shuttle. This was part of NASA's
broader plan for the future of space travel--known as NASA's Integrated
Space Transportation Plan. On October 21, 2002, NASA postponed further
implementation of the program to focus on defining the Department of
Defense's role, determining future requirements of the ISS, and
establishing the agency's future space transportation needs. In
November 2002, the administration submitted to the Congress an
amendment to NASA's fiscal year 2003 budget request to implement a
new Integrated Space Transportation Plan. The new plan makes
investments to extend the space shuttle's operational life and
refocuses the Space Launch Initiative program on developing an
orbital space plane--which provides crew transfer capability to and
from the space station--and next generation launch technology.
[11] According to the Software Engineering Institute, requirements
management is a process that establishes a common understanding between
the customer and the software project manager regarding the customer's
business needs that will be addressed by a project. A critical part of
this process is to ensure that the requirements development portion of
the effort documents, at a sufficient level of detail, the problems
that need to be solved and the objectives that need to be achieved.
[12] According to Institue of Electrical and Electronics Engineers
Standard 1362-1998, a concept of operations document is normally one of
the first documents that is produced during a disciplined development
effort since it describes system characteristics for a proposed system
from the user's viewpoint. This is important since a good concept of
operations document can be used to communicate overall quantitative and
qualitative system characteristics to the user, developer, and other
organizational elements. This allows the reader to understand the user
organizations, missions, and organizational objectives from an
integrated systems point of view.
[13] A test case is a series of actions, performed serially, in
parallel, or in some combination, that creates the desired test
conditions. Rex Black, Managing the Testing Process: Practical Tools
and Techniques For Managing Hardware and Software Testing (Redmond,
Wash.: Microsoft Press, 1999).
[14] Gartner, Inc.
[15] IEEE 830-1998.
[16] NASA originally identified about 590 requirements. However, 51 of
these were deleted and 86 were deferred.
[17] NASA defines critical defects as those that "impact the ability to
move forward or complete an entire business function or task, and
impacts multiple business functions, multiple users and/or locations.
[It] presents a failure that has no workaround or alternative."
[18] NASA defines high defects as those that have "a significant impact
on the completion of a business function or task, however, activities
can continue as far as the next function. A limited number of business
functions, business users, or locations are impacted, and may be an
impact to only one."
[19] William A. Florac, Robert E. Park, and Anita D. Carleton,
Practical Software Measurement: Measuring for Process Management and
Improvement (Pittsburgh, Pa.: Software Engineering Institute, Carnegie
Mellon University, 1997).
[20] The Chief Financial Officers Act of 1990 and subsequent related
financial management reform legislation, among other things, set
expectations for agencies to develop and deploy more modern financial
management systems, produce sound cost and operating performance
information, and design results oriented reports on the government's
financial condition by integrating budget, accounting, and program
information.
[21] Cost-reimbursement contracts are often the most appropriate type
for developmental items or those for which the exact price of the goods
or services being purchased cannot be definitely known prior to
contract award. This type of procurement instrument places greater risk
on the government than contracts based on firm fixed prices.
GAO's Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to daily E-mail alert for newly
released products" under the GAO Reports heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office
441 G Street NW,
Room LM Washington,
D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.
General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.
20548:
