NASA Procurement
Use of Award Fees for Achieving Program Outcomes Should Be Improved Gao ID: GAO-07-58 January 17, 2007Cost-plus-award-fee contracts accounted for almost half of the National Aeronautic and Space Administration's (NASA) obligated contract dollars for fiscal years 2002-2004. Since 1990, we have identified NASA's contract management as a high-risk area--in part because of a lack of emphasis on end results. Congress asked us to examine (1) the extent NASA's guidance on award fees addresses problems previously identified with the use of award-fee contracts and (2) whether NASA follows its guidance in using award fees to achieve desired outcomes. We reviewed the top 10 dollar value award-fee contracts active from fiscal years 2002 through 2004.
NASA guidance on the use of cost-plus-award-fee (CPAF) contracts provides criteria to improve the effectiveness of award fees. For example, the guidance emphasizes outcome factors that are good indicators of success in achieving desired results, cautions against using numerous evaluation factors, prohibits rollover of unearned fee, and encourages evaluating the costs and benefits of such contracts before using this contract type. However, NASA does notalways follow the preferred approach laid out in its guidance. For example, some evaluation criteria contained input or process factors, such as program planning and organizational management. Moreover, some contracts included numerous supporting subfactors that may dilute emphasis on any specific criteria. Although the Federal Acquisition Regulation and NASA guidance require considering the costs and benefits of choosing a CPAF contract, NASA did not perform such analyses. In some cases there appears to be a significant disconnect between program results and fees paid. For example, NASA paid the contractor for the Earth Observing System Data and Information System Core System 97 percent of the available award fee despite a delay in the completion of the contract by over 2 years and an increase in the cost of the contract of more than 50 percent. NASA officials expressed satisfaction with the results of the contracts we reviewed, and this was further evidenced by the extent of fee paid. NASA's satisfaction was based on its evaluations of contractor performance against criteria established in the award-fee plan. While NASA's evaluations would indicate generally good contractor performance, that performance did not always translate into desired program outcomes. That disconnect raises questions as to the extent NASA is achieving the effectiveness it sought through the establishment of guidance on the use of award fees. NASA has not evaluated the overall effectiveness of award fees and does not have metrics in place for conducting such evaluations.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-07-58, NASA Procurement: Use of Award Fees for Achieving Program Outcomes Should Be Improved
This is the accessible text file for GAO report number GAO-07-58
entitled 'NASA Procurement: Use of Award Fees for Achieving Program
Outcomes Should Be Improved' which was released on February 16, 2007.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Chairman, Committee on Science, House of Representatives:
United States Government Accountability Office:
GAO:
January 2007:
NASA Procurement:
Use of Award Fees for Achieving Program Outcomes Should Be Improved:
GAO-07-58:
GAO Highlights:
Highlights of GAO-07-58, a report to the Chairman, Committee on
Science, House of Representatives
Why GAO Did This Study:
Cost-plus-award-fee contracts accounted for almost half of the National
Aeronautic and Space Administration‘s (NASA) obligated contract dollars
for fiscal years 2002-2004. Since 1990, we have identified NASA‘s
contract management as a high-risk area”in part because of a lack of
emphasis on end results. You asked us to examine (1) the extent NASA‘s
guidance on award fees addresses problems previously identified with
the use of award-fee contracts and (2) whether NASA follows its
guidance in using award fees to achieve desired outcomes. We reviewed
the top 10 dollar value award-fee contracts active from fiscal years
2002 through 2004.
What GAO Found:
NASA guidance on the use of cost-plus-award-fee (CPAF) contracts
provides criteria to improve the effectiveness of award fees. For
example, the guidance emphasizes outcome factors that are good
indicators of success in achieving desired results, cautions against
using numerous evaluation factors, prohibits rollover of unearned fee,
and encourages evaluating the costs and benefits of such contracts
before using this contract type.
However, NASA does not always follow the preferred approach laid out in
its guidance. For example, some evaluation criteria contained input or
process factors, such as program planning and organizational
management. Moreover, some contracts included numerous supporting
subfactors that may dilute emphasis on any specific criteria. Although
the Federal Acquisition Regulation and NASA guidance require
considering the costs and benefits of choosing a CPAF contract, NASA
did not perform such analyses.
In some cases there appears to be a significant disconnect between
program results and fees paid. For example, NASA paid the contractor
for the Earth Observing System Data and Information System Core System
97 percent of the available award fee despite a delay in the completion
of the contract by over 2 years and an increase in the cost of the
contract of more than 50 percent.
NASA officials expressed satisfaction with the results of the contracts
we reviewed, and this was further evidenced by the extent of fee paid.
NASA‘s satisfaction was based on its evaluations of contractor
performance against criteria established in the award-fee plan. While
NASA‘s evaluations would indicate generally good contractor
performance, that performance did not always translate into desired
program outcomes. That disconnect raises questions as to the extent
NASA is achieving the effectiveness it sought through the establishment
of guidance on the use of award fees. NASA has not evaluated the
overall effectiveness of award fees and does not have metrics in place
for conducting such evaluations.
What GAO Recommends:
We recommend NASA improve its current use of award fees by
reemphasizing tying award-fee payments to desired outcomes, limiting
the number of factors used in contractor evaluations as its guidance
recommends, and by using this contract type only when justified by a
consideration of costs and benefits. We also recommend that NASA
develop metrics for measuring the effectiveness of award fees,
establish a system for collecting data on the use of award-fee
contracts, and regularly examine the effectiveness of award fees in
achieving desired outcomes. In commenting on a draft of this report,
NASA concurred with all three recommendations.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-58].
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Ann M. Calvaresi-Barr,
(202) 512-4841, calvaresibarra@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
NASA's Award-Fee Policy Addresses Many Cost-Plus-Award-Fee Contracting
Issues Identified as Problematic:
NASA Has Not Always Followed the Preferred Approach Laid Out in Its
Award-Fee Guidance:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Summary Description of the 10 NASA Contracts GAO Reviewed:
Appendix III: Comments from the National Aeronautics and Space
Administration:
Appendix IV: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Award Fees as a Negotiated Percentage of Contract Value:
Table 2: Percentage of Available Award Fee Pool Paid on Contracts
Examined:
Abbreviations:
CPAF: cost-plus-award-fee:
CSOC: Consolidated Space Operations Contract:
DOD: Department of Defense:
ECS: EOSDIS Core System:
EOSDIS: Earth Observing System Data and Information System:
ETM+: Enhanced Thematic Mapper Plus:
FAR: Federal Acquisition Regulation:
FDO: fee determination official:
IEMP: Integrated Enterprise Management Program:
ISS: International Space Station:
JPL: Jet Propulsion Laboratory:
MSES: Mechanical System Engineering Services:
NASA: National Aeronautics and Space Administration:
PEB: performance evaluation board:
SEAT: Science, Engineering, Analysis, and Test:
United States Government Accountability Office:
Washington, DC 20548:
January 17, 2007:
The Honorable Bart Gordon:
Chairman:
Committee on Science:
House of Representatives:
The Federal Acquisition Regulation (FAR) states that cost-plus-award-
fee (CPAF) contracts are intended to motivate excellent contractor
performance in areas such as quality, timeliness, technical ingenuity,
and cost-effective management.[Footnote 1] During the early 1990s, the
National Aeronautics and Space Administration (NASA) Inspector General
and NASA internal studies raised concerns about NASA's use of CPAF
contracts.[Footnote 2] As a result, NASA developed specific guidance to
improve the effectiveness of award fees. The CPAF contract type
continues to be used extensively by NASA for obtaining both goods and
services, accounting for almost half of NASA contract dollars for
fiscal years 2002 through 2004.[Footnote 3] Given this, you requested
that we examine NASA's use of award-fee contracts and determine (1) the
extent NASA's guidance addresses the problems previously identified
with the use of award-fee contracts and (2) whether NASA follows its
guidance in using award fees to achieve desired outcomes.
To address these objectives, we reviewed a sample of NASA CPAF
contracts that were active from fiscal years 2002 through
2004.[Footnote 4] We reviewed contract files, obtained information from
program and contracting officials through the use of a structured
questionnaire, and discussed the application of award-fee criteria with
NASA officials involved in the award-fee process. Our work was
conducted between August 2005 and October 2006 in accordance with
generally accepted government auditing standards. For a complete
description of our scope and methodology, see appendix I.
Results in Brief:
NASA regulations and guidance on the use of cost-plus-award-fee
contracts address many of the issues and problems identified by NASA
and the NASA Inspector General and provide criteria for appropriately
using award-fee contracts. For example, NASA encourages tying fees to
outcomes, prohibits the rollover of unearned award fee, authorizes the
use of interim fees on end item contracts until final product delivery,
and encourages the use of performance-based contracts for the
procurement of services and supplies. Further, because of the cost and
administrative burden associated with managing this type of contract,
the FAR and NASA's Award Fee Contracting Guide recommend evaluating the
costs and benefits of using a cost-plus-award-fee contract before
committing to this contract type.
NASA did not consistently implement key aspects of the agency's
guidance on major award-fee contracts that we reviewed. NASA's award-
fee guide states that while it is sometimes valuable to consider input
factors when evaluating contractor performance, it is NASA's preference
to use outcome-based criteria, and each of the contracts we reviewed
had some outcome based criteria. However, some criteria used to
evaluate performance were process or input factors, such as awarding
portions of the fee for the quality and effectiveness of the
contractor's scheduling system, and program planning and organizational
management. Other contracts used numerous subfactors for evaluating
contractor performance, which, according to NASA's award-fee guide, can
dilute emphasis on any specific criteria. For example, one contract
specified 3 primary performance evaluation factors, but included 96
subfactors for fiscal year 2004 and 108 subfactors for fiscal year
2005. Also, although the FAR and NASA's award-fee guide specify
consideration of the costs and benefits before using a CPAF contract
because of the cost and administrative burden associated with these
contracts, no analysis of costs and benefits was conducted for the
contracts we reviewed. Finally, NASA officials expressed satisfaction
with the results of the contracts we reviewed. NASA's satisfaction was
based on its evaluations of contractor performance against criteria
established in the award-fee plan. While NASA's evaluations would
indicate generally good contractor performance, such performance did
not always translate into desired program outcomes. That disconnect
raises questions as to the extent NASA is achieving the effectiveness
it sought through the establishment of guidance on award fees.
Specifically, our analysis showed that NASA paid significant amounts of
available fee on all of the contracts we reviewed, including those end
item contracts that did not deliver a capability within initial cost,
schedule, and performance parameters. For example, NASA paid the
contractor for the Earth Observing System Data and Information System
Core System 97 percent of the available award fee despite a delay in
the completion of the contract of more than 2 years and an increase in
the cost of the contract of more than 50 percent. Since revising its
guidance, NASA has not evaluated the effectiveness of award fees in
achieving program results and does not have metrics in place for
measuring the effectiveness of award fees.
We are recommending that NASA improve its current implementation of
cost-plus-award-fee contracts by reemphasizing its current guidance on
tying award-fee payments, particularly on major end item contracts, to
outcome factors and limiting the number of subfactors used in
evaluations and to use this contract type only when justified by a
consideration of costs and benefits. Further, we are recommending that
NASA develop metrics for measuring and a mechanism for evaluating the
effectiveness of award fees in achieving desired outcomes. In
commenting on a draft of this report, NASA concurred with all three
recommendations.
Background:
In January 2004, the President announced a new "Vision for Space
Exploration" calling for human and robotic missions to the Moon, Mars,
and beyond. Over the next two decades, NASA plans to spend over $100
billion to develop a number of new capabilities, supporting
technologies, and facilities that are critical to enabling space
exploration missions. Development of the critical capabilities and
technologies will be largely dependent on NASA contractors, who
constitute more than two-thirds of NASA's workforce.[Footnote 5]
According to NASA officials, 87 percent of NASA's $16.6 billion budget
for fiscal year 2006 was spent on work performed by its contractors.
Since 1990, we have designated NASA's contract management as a high-
risk area. This is based primarily on NASA's lack of a modern
integrated financial management system that can provide reliable
information on contract spending and performance as well as NASA's lack
of emphasis on end results, product performance, and cost control. For
example, our most recent high-risk report stated that while NASA has
taken actions to improve its contract management function, it continues
to face considerable challenges in implementing its contracts
effectively.[Footnote 6]
NASA is organized under four mission directorates--Aeronautics
Research, Exploration, Science, and Space Operations--each of which
covers a major area of the agency's research and development efforts.
The agency is composed of NASA headquarters, 10 field centers, and the
contractor-operated Jet Propulsion Laboratory.[Footnote 7]
Cost-Plus-Award-Fee Contracts:
NASA and other federal agencies can choose among numerous contract
types for acquiring goods and services that can differ in part
according to the nature of the fee that agencies offer to the
contractor for achieving or exceeding specified objectives or goals.
According to the FAR, a CPAF contract is appropriate to use when it is
difficult to measure key elements of performance. It is widely used to
procure nonroutine services such as the development of new systems.
Typically, award-fee contracts emphasize several aspects of contractor
performance, such as schedule performance, technical performance, and
cost control. Because development and administration of award-fee
contracts involve substantially more effort over the life of a contract
than other types of contracts, the FAR and NASA's Award Fee Contracting
Guide specify that the expected benefits of using an award-fee contract
must exceed the additional administrative effort and cost
involved.[Footnote 8]
The theory behind CPAF contracts is that although the government
assumes most of the cost risk, it retains control over most or all of
the contractor's potential profit as leverage. On CPAF contracts, the
award fee is often the only source of potential fee for the contractor.
According to the NASA FAR Supplement and NASA's Award Fee Contracting
Guide[Footnote 9], these contracts can include a base fee of anywhere
from 0 to 3 percent of the estimated value of a nonservice contract.
However, NASA's regulations and guide do not allow the use of a base
fee on service contracts. Table 1 shows the percentage of award fee
available on the contracts we examined. (See app. II for a description
of these contracts.)
Table 1: Award Fees as a Negotiated Percentage of Contract Value:
Contract: Jet Propulsion Laboratory[A];
Award-fee percentage: 1.5.
Contract: International Space Station[B];
Award-fee percentage: 11.0.
Contract: Consolidated Space Operations;
Award-fee percentage: 10.0.
Contract: Joint Base Operation and Support;
Award-fee percentage: 8.0.
Contract: Science, Engineering, Analysis, and Test;
Award-fee percentage: 6.0.
Contract: Engineering and Technical Support for Life Sciences;
Award- fee percentage: 5.0.
Contract: Program Information Systems Mission Services;
Award-fee percentage: 6.0.
Contract: Earth Observing System Data and Information System Core
System;
Award-fee percentage: 9.9.
Contract: Mechanical Systems Engineering Services;
Award-fee percentage: 9.2.
Contract: Landsat-7;
Award-fee percentage: 7.0.
Sources: NASA submissions to GAO and contract and award-fee
documentation; GAO (analysis).
Note: Half of the 10 contracts we reviewed also included other types of
fee or incentives including performance incentives and fixed fees.
[A] Although this contract for the operation and management of the Jet
Propulsion Laboratory is a CPAF contract, it differs from the other
contracts we reviewed in that the contractor, the California Institute
of Technology (Caltech), is a nonprofit educational institution.
According to the NASA FAR Supplement 1815.404-471-6(a), it is NASA's
policy not to pay profit or fee on contracts with educational
institutions. This contract is an exception. According to NASA
officials, the fee paid becomes part of Caltech's investment in the
institution's educational programs and the infrastructure supporting
the research efforts made by Caltech.
[B] Eleven percent is the maximum award fee available as negotiated in
the December 2003 International Space Station contract extension.
[End of table]
NASA's Use of Cost-Plus-Award-Fee Contracts:
NASA relies heavily on CPAF contracts. This contract type accounted for
48 percent of obligated contract dollars and 7.7 percent of contract
actions from fiscal years 2002 through 2004. By comparison, between
fiscal years 1999 and 2003, award-fee contracts accounted for 13
percent of the contract dollars and 3.4 percent of contract actions at
the Department of Defense (DOD). A CPAF contract includes an estimate
of the total cost of what is being contracted for, may include a fee
with a possible base amount fixed at the inception of the contract, and
includes an award amount that is intended to motivate excellence in
contract performance. The award fee is paid based upon the government's
periodic judgmental evaluations of contractor performance.
Award-Fee Criteria:
When developing evaluation plans, NASA's award-fee guide indicates that
evaluation plans may include outcomes, outputs, inputs, or a
combination of these elements. NASA's guide expresses a preference for
outcome factors. It notes that while it is sometimes valuable to
consider input and output factors when evaluating contractor
performance, outcome factors are better indicators of success relative
to the desired result.
* An outcome factor is an assessment of the results of an activity
compared to its intended purpose. Outcome-based factors are the least
administratively burdensome type of performance evaluation factor, and
should provide the best indicator of overall success. Outcome-based
factors should therefore be the first type of evaluation factor
considered for use, and are often ideal for nonroutine efforts.
* An output factor is the tabulation, calculation, or recording of
activity or effort and can be expressed in a quantitative or
qualitative manner. Output factors may be more desirable for routine
efforts, but are administratively more burdensome than outcome factors
due to the tabulation, calculation, or recording requirements. When
output factors are used, care should be taken to ensure that there is a
logical connection between the reported measures and the program's
mission, goals, and objectives.
* Input factors refer to intermediate processes, procedures, actions,
or techniques that are key elements influencing successful contract
performance. These may include testing and other engineering processes
and techniques; quality assurance and maintenance procedures;
subcontracting plans; purchasing department management; and inventory,
work assignment, and budgetary controls.
Evaluation of Award-Fee Contracts:
For CPAF contracts, NASA personnel conduct periodic, typically
semiannual evaluations of contractor's performance against the criteria
specified in a performance evaluation plan. During the course of the
evaluation period, performance monitors track contractor performance,
and once the period is over they assess the performance and report to
the performance evaluation board (PEB). The PEB considers the reports
as well as any other pertinent information and prepares a report for
the fee determination official (FDO) with findings and recommendations.
The contractor is given an opportunity to provide a self-assessment of
its performance during the evaluation period, which is often a written
report. The FDO may meet with the PEB to discuss the report, after
which a final determination is made in writing as to the amount of fee
to be paid. The FDO provides the determination to the contracting
officer and a copy of the related document to the contractor.
Acquisition Environment Can Affect Acquisition Outcomes:
When discussing award-fee contracts, it is important to acknowledge the
acquisition environment in which they are used. Award fees are intended
to motivate excellent contractor performance, which should result in
excellent program outcomes. However, award fees should not be used to
make up for factors internal or external to the acquisition environment
that hinder the success of acquisition outcomes. These factors may
include inadequate resources and financial management systems, lack of
knowledge prior to starting the acquisition, or unsound acquisition
practices. We have reported that in some cases, NASA's failure to
define requirements adequately and develop realistic cost estimates
resulted in projects costing more, taking longer, and achieving less
than originally planned.[Footnote 10] The persistence of these problems
in NASA contract management is not only indicative of undisciplined
processes or practices such as these, but may also reflect the fact
that the design, development, and production of major space systems are
extremely complex technical processes that must operate within complex
budget and political processes. Even properly run programs can
experience problems that may arise from unknowns, such as technical
obstacles and changes in circumstances. Only a few things need to go
wrong to cause major problems, and many things must go right for a
program to be successful.
NASA's Award-Fee Policy Addresses Many Cost-Plus-Award-Fee Contracting
Issues Identified as Problematic:
The NASA FAR Supplement and NASA's Award Fee Contracting Guide address
many of the issues and problems identified by NASA on the use of award-
fee contracts and provide criteria for appropriately using such
contracts. Much of the guidance on award-fee contracting was issued in
response to weaknesses in CPAF contracting practices identified by NASA
internal reviews and NASA's Office of Inspector General in the early
1990s.[Footnote 11] Those weaknesses included the awarding of excessive
fees with limited emphasis on acquisition outcomes (end results,
product performance, and cost control); rollover of unearned fee; use
of base fee; and the failure to use both positive and negative
incentives. NASA updated its award-fee guide in 1994, 1997, and 2001 to
explain and elaborate on its award-fee policy. The 2001 revision also
reflects the FAR's additional emphasis on using performance-based
contracts.
NASA's award-fee guide emphasizes tying fees to outcome factors. The
guide states that outcome-based factors are the least administratively
burdensome type of evaluation factor and should provide the best
indicator of overall success. The award-fee guide warns against
micromanaging performance and diluting the emphasis of criteria by
spreading the potential award fee over a large number of performance
evaluation factors. Instead, the guide recommends selecting broad
performance evaluation factors, such as technical factors, project
management, and cost control supplemented by a limited number of
subfactors under these factors.
Cost control is required to be a key performance evaluation factor in
award-fee performance evaluation plans, largely because of past
performance issues in which contractors were paid millions of dollars
in fees on contracts that were experiencing hundreds of millions of
dollars in cost overruns.[Footnote 12] The NASA FAR Supplement states
that cost control shall be no less than 25 percent of the total
weighted evaluation factors when explicit evaluation factor weightings
are used. The NASA FAR Supplement states that emphasis on cost control
should be balanced against other performance requirement objectives,
and the contractor should not be incentivized to pursue cost control to
the point that overall performance is significantly degraded.
NASA's regulations prohibit rolling over unearned fee to subsequent
evaluation periods for service contracts. For such contracts, each
interim evaluation and the last evaluation are final. Another key
element of the current award-fee regulations is an increased emphasis
on overall contractor performance and the end product, rather than on
incremental progress. NASA requires conducting interim evaluations on
end item contracts until final product delivery to monitor performance
prior to contract completion and establish the basis for making interim
payments. At the end of the contract, a final evaluation is conducted
and the contractor's total performance is evaluated against the award-
fee plan to determine total earned award fee. For example, the
contractor may be evaluated and paid an interim fee once every 6 months
until the product is delivered. During the final evaluation, the
contractor's performance is evaluated to determine total earned award
fee. The final evaluation may result in the contractor retaining the
fee previously awarded or receiving additional or less fee than
previously awarded and thus refunding a portion of the fee to the
government. The final evaluation provides NASA the opportunity to make
an award-fee decision based on actual quality, total cost, and ability
to meet the contract schedule at the point the final product is
delivered.
Further, under the award-fee policy in effect prior to the 1994 and
subsequent revisions to the guidance, base fee was allowed on all CPAF
contracts. NASA's current regulations prohibit the use of base fee on
service contracts and restrict the use of base fee on end item
contracts, such as for hardware. When base fee is used, it is not to
exceed 3 percent of estimated contract costs and it should only be paid
if the final award-fee evaluation is satisfactory or better. We note
that base fee, which was paid on two of the three end item contracts we
reviewed, did not exceed 3 percent, and none of the seven service
contracts included base fee.
Another issue addressed by NASA's regulations is the use of both
positive and negative performance incentives in its CPAF contracts. The
NASA FAR Supplement provides that award-fee contracts with primary
deliverables of hardware and with a total estimated cost and fee of
greater than $25 million require both kinds of incentives based on
measurements of hardware performance against objective
criteria.[Footnote 13] Performance incentives are separate and distinct
from award fee and measure contractor performance up to delivery and
acceptance. Performance incentives are designed to reward contractors
when performance of delivered hardware is above minimum contract
requirements. For example, if the government establishes a specified
level of objective performance for a product that the contractor
exceeds, the contractor can be paid a performance incentive in addition
to the award fee already paid. If the contractor just meets this
measure, it cannot receive an additional performance incentive and
keeps the award fee already paid. If the contractor fails to meet the
measure, however, it must pay a negative performance incentive fee that
reduces or eliminates the entire award fee.
To address inconsistencies among NASA centers in how they evaluate
contractor performance, the current award-fee regulations also provide
a uniform rating system to be used for all NASA award-fee contracts. It
includes adjectival ratings as well as a numerical scoring system of 0-
100. Scores of 61-70 percent are considered satisfactory, and the
regulations specify that contractors receiving a rating of less than 61
percent will not receive any fee. A contractor is not to be paid any
base fee or award fee for less than satisfactory overall performance.
NASA's award-fee guide encourages the use of performance-based
contracts for the procurement of services and supplies. The guide
states that constructing performance-based contracts that clearly
define performance requirements, include easily understood performance
standards, and have an objective incentive mechanism will result in
contractors having a clearer understanding of the government's
expectations and will ultimately facilitate enhanced contractor
performance.
Finally, because of the cost and administrative burden associated with
administering award-fee contracts, the FAR and NASA's award-fee guide
specify consideration of the costs and benefits of using a CPAF
contract before committing to this contract type. Through an evaluation
of the administrative costs versus the expected benefits, the
contracting officer should be able to assess whether the benefits the
government gains through a CPAF contract will outweigh the additional
costs of overseeing and administering the contract. The award-fee guide
provides an example of how to calculate the administrative cost and
states that benefits could be measured in dollars saved through cost
control or enhanced technical capability.
NASA Has Not Always Followed the Preferred Approach Laid Out in Its
Award-Fee Guidance:
Although the revisions in NASA's regulations and guidance on award-fee
contracts address many weaknesses previously identified, the contracts
that we reviewed did not always demonstrate use of award fees by the
centers in the way that NASA prefers as outlined in its guidance. Some
performance evaluation plans or reports included input evaluation
factors, which are not the best indicators of success relative to the
desired result, although they are allowed by the guidance. Other
contracts included numerous subcategories for evaluating the contractor
that can lessen the importance of any particular subcategory and reduce
the leverage of the award fee on any particular criterion. Also,
although the FAR and NASA's award-fee guide calls for a consideration
of the costs and benefits of using cost-plus-award-fee contracts
because of the cost and administrative burden involved, we found no
examples of a documented analysis of costs and benefits. Finally, NASA
officials expressed satisfaction with the results of the contracts
based on their evaluations of contractor performance against criteria
established in the award-fee plan. Those evaluations would indicate
generally good performance. However, that performance did not always
translate into desired program outcomes. NASA paid a majority of the
available award fee on all of the contracts we reviewed, including
those end item contracts that did not deliver a capability within
initial cost, schedule, and performance parameters.[Footnote 14] That
disconnect raises questions as to the extent NASA is achieving the
effectiveness it sought through the establishment of guidance on the
use of award fees. Further, NASA has not evaluated the overall
effectiveness of award fees in promoting program outcomes and does not
have metrics in place for measuring their effectiveness in achieving
program outcomes.
Some Evaluation Factors Were Not Outcome Factors as Preferred by NASA
Guidance:
Some performance evaluation subfactors included in performance
evaluation plans or reports were not outcome oriented. NASA's award-fee
guide states that while it is sometimes valuable to consider input and
output factors when evaluating contractor performance, it is NASA's
preference when feasible to tie fees to evaluation factors that are
based on outcomes because outcome-based factors provide the best
indicator of overall success. The award-fee guide recommends selecting
broad performance evaluation factors, such as technical factors,
project management, and cost control, and cautions that factors related
to intermediate processes, procedures, and actions may cause the
contractor to divert its attention from the overall desired outcome.
The guide states that those types of factors, while allowed, are not
always true indicators of the contractor's performance and should be
relied on with caution. Further, with service contracts, input factors
may be of little or no value as a basis for evaluation. While the
contracts we reviewed generally used outcome factors as part of the
evaluation of performance, some supporting subfactors that formed the
basis of the ratings for performance measured compliance with process
or input factors that may not provide the best indicators of success
relative to the desired results.
For example, a part of the award fee on the Mechanical System
Engineering Services (MSES) contract was to be awarded for program and
business management performance. There were five subfactors under this
primary performance factor. Two of these subfactors, program planning
and organizational management and business management were input
subfactors. These two input subfactors measure contractor processes or
inputs, but do not focus on final results. Subfactors in the Landsat-7
contract included input subfactors such as responsiveness of the
contractor's corporate management, quality and effectiveness of the
contractor's scheduling system, and prudent utilization of manpower and
timely removal of manpower upon completion of tasks.
Some Contracts Did Not Limit the Number of Subfactors for Evaluating
Contractor Performance as NASA Guidance Recommends:
The NASA award-fee guide cautions that spreading the potential award
fee over a large number of performance evaluation factors dilutes
emphasis on any particular performance evaluation criterion, increases
the prospect of any one item being too small and thus overlooked, and
increases the administrative burden. It encourages broad performance
evaluation factors such as technical factors, project management, and
cost control, which should be supplemented by only a limited number of
subfactors describing significant evaluation elements over which the
contractor has effective management control. Our analysis showed that a
large number of subfactors were used to evaluate contractor performance
for some contracts.
For example, the Jet Propulsion Laboratory (JPL) contract, which
includes both service and product deliverables defined in task orders
under the contract, uses three primary performance evaluation factors
for measuring contractor performance--programmatic, scientific, and
engineering; institutional management; and support to outreach
initiative programs. Although the JPL performance evaluation plan
characterizes award-fee subfactors as representing major areas of
emphasis during the performance period, the award-fee subfactors used
to support the broad performance evaluation factors were numerous--96
subfactors were used to evaluate the contractor's performance in fiscal
year 2004, and 108 subfactors were used in fiscal year 2005.[Footnote
15] The Engineering and Technical Support for Life Sciences contract
used three broad performance evaluation factors also--technical
performance, schedule performance and contract management, and cost
control--but evaluated the contractor on numerous supporting subfactors
identified as tasks or subtasks in the contractor performance
evaluation reports. For example, on one task order under this contract,
performance evaluation reports for various evaluation periods showed as
many as 50 different subtasks used to evaluate the contractor's
performance for the primary evaluation criteria: (1) technical
performance and (2) schedule performance and contract management.
The Landsat-7 contract also included a number of subfactors. Contractor
performance under this contract was evaluated in several different
areas each time the performance evaluation board met. Technical
performance and program management were grouped together in one primary
performance evaluation factor, and business management and cost
performance were grouped together in the other primary performance
evaluation factor. There were 9 subfactors under technical performance
and 12 subfactors under program management, including quality and
effectiveness of the contractor's scheduling system. Under business
management and cost performance, 17 evaluation subfactors and elements
were to be considered, including compliance with general contract
provisions and clauses and weekly scheduling of teleconferences to
determine schedule status. In addition to the number of subfactors that
fell under the two primary performance evaluation factors, there were
nine additional evaluation criteria, including resourcefulness,
communication, and responsiveness.
NASA Did Not Perform Cost-Benefit Analyses:
Although the FAR and NASA's award-fee guide require consideration of
the costs and benefits of using a CPAF contract before committing to
this contract type to determine whether the benefits outweigh the
additional cost and administrative burden of managing the contract, we
found no instances where a documented cost-benefit analysis had been
done for any of the contracts under our review. According to the
guidance, since award-fee contracts require additional administrative
effort, they should be used only when the contract values, performance
period, and expected benefits are sufficient to warrant that additional
management effort. Careful selection of the most appropriate contract
type and careful tailoring should prevent a situation in which the
burden of administering the award fee is out of proportion to the
improvements expected in the quality of the contractor's performance
and in overall management. In addition, CPAF contracts can be
particularly costly and burdensome for NASA to administer because of
contract reporting and review requirements. Major cost drivers include
the number of award-fee periods, performance monitors, and performance
evaluation board members necessary for implementing the award-fee
process. For example, according to NASA's Award Fee Contracting Guide's
conservative estimate, it would cost about $387,000 to administer the
award-fee process over the life of a 5-year contract. The guide notes
that the estimate does not represent all associated administrative cost
that might arise. Although NASA procurement officials acknowledged that
formal cost-benefit analyses were not prepared, some officials referred
to determination and findings statements or acquisition strategy
meeting documents associated with specific contracts as providing some
evidence of consideration given to whether or not CPAF contracts should
be used.
Award-Fee Payments at Times Did Not Reflect Program Outcomes:
While NASA officials expressed satisfaction with the results of the
contracts, in some cases there appeared to be a disconnect between the
fee paid and program results. NASA paid most of the available fee on
all of the contracts we reviewed--including on projects that showed
cost increases, schedule delays, and technical problems. The total
estimated value of the 10 contracts we reviewed was more than $31
billion. NASA paid between 80 and 99 percent of the maximum award fee
possible on these contracts. The average was 90 percent, which equated
to almost a billion dollars in total award fees paid under the 10
contracts. Table 2 shows the percentage of award fee paid for each of
the 10 contracts we reviewed.
Table 2: Percentage of Available Award Fee Pool Paid on Contracts
Examined:
Contract: Jet Propulsion Laboratory[A];
Award-fee percentage: 90.
Contract: International Space Station[A];
Award-fee percentage: 92.
Contract: Consolidated Space Operations;
Award-fee percentage: 80.
Contract: Joint Base Operation and Support[A];
Award-fee percentage: 90.
Contract: Science, Engineering, Analysis, and Test;
Award-fee percentage: 80.
Contract: Engineering and Technical Support for Life Sciences;
Award-fee percentage: 91.
Contract: Program Information Systems Mission Services;
Award-fee percentage: 89.
Contract: Earth Observing System Data and Information System Core
System;
Award-fee percentage: 97.
Contract: Mechanical Systems Engineering Services;
Award-fee percentage: 90.
Contract: Landsat-7;
Award-fee percentage: 99.
Sources: NASA contract and award-fee documentation; GAO (analysis).
[A] These contracts were still active at the conclusion of our audit
work. The remaining 7 contracts were either closed or were in the
process of being closed.
Note: Numbers are rounded to the nearest whole percentage.
[End of table]
NASA officials expressed satisfaction with contract results, which was
further evidenced by its evaluations of contractor performance against
criteria established in the award-fee plan. While NASA's evaluations
would indicate generally good performance, such performance did not
always translate into desired program outcomes. That disconnect raises
questions as to the extent NASA is achieving the effectiveness it
sought through the establishment of guidance on the use of award-fees.
On the end item contracts we reviewed, although there were some periods
in which NASA paid a lesser percentage of the available fee, NASA
ultimately paid more than 90 percent of the available fee based on its
evaluation of contractor performance against criteria in the award-fee
plan even when those contracts did not deliver capability within
initial cost, schedule, and performance parameters. For example:
* The prime contractor for the International Space Station (ISS) has
received 92 percent of the total award fee available--$425.3 million--
although the cost increased by 131 percent, from $5.6 billion to $13
billion, in part due to increased contract scope and delays caused by
the Columbia accident, but also contractor cost overruns. In addition,
the contractor estimates that it will incur an additional $76 million
in overruns by the time the contract is completed. Further, the
completion date for space station assembly under the prime contract was
delayed by 8 years. In some cases these delays were caused by actions
not within the control of the contractor, such as problems with the
shuttle program and actions by the international partners.
* The contractor for the Earth Observing System Data and Information
System (EOSDIS) Core System (ECS) was paid 97 percent of the available
award fee--$103.2 million--despite a delay in the completion of the
contract by more than 2 years and an increase in the cost of the
contract from $766 million to $1.2 billion. Technical problems,
schedule delays, and cost control problems led to a major restructuring
of the contract.
* The Landsat-7 contractor was paid 99 percent of the available award
fee or more than $17 million.[Footnote 16] The original contract was
managed by the Air Force but was subsequently transferred to NASA and
rebaselined. The cost of the contract when transferred to NASA and
rebaselined was $342.7 million. The Landsat-7 launch was delayed by 9
months and although the original scope of the work under the contract
was significantly reduced[Footnote 17], the cost of the contract
increased. By the time the contract was complete, costs had risen 20
percent to $409.6 million.
While some NASA officials pointed out that problems encountered on
these contracts were at times outside the control of the contractor,
difficulties such as these with achieving program results have resulted
in NASA contract management being considered a high-risk area by GAO.
We did not review these contracts to determine responsibility for
undesirable results and therefore make no conclusion as to whether the
fee paid was appropriate on each particular contract. However, the high
fees paid on contracts where programs experienced disappointing results
raise questions as to the effectiveness of award fees as a tool for
obtaining desired program outcomes.
For the service contracts we reviewed, NASA officials reported that
they were satisfied with the results and quality of services provided.
While we could not assess these contracts against cost, schedule, and
performance outcomes as we could with the end item contracts, we did
assess the award-fee criteria used in these contracts against NASA
guidance. Here we found instances of process and input-oriented
subfactors and the inclusion of numerous subfactors in evaluating
performance. Further, we found no evidence that a cost-benefit analysis
had been performed prior to choosing the contract type. Taken together,
this is not the preferred approach according to NASA guidance, which
raises questions as to the degree to which performance outcomes--
getting the quality of service desired--was actually the basis for
judging contractor performance and awarding fee.
NASA Has Not Assessed the Effectiveness of Award Fees in Achieving
Program Outcomes or Developed Metrics for Conducting Such Evaluations:
NASA views CPAF contracts as a viable and often preferred mechanism for
acquiring the types of goods and services that the agency procures.
NASA's satisfaction with the results of these contracts is evidenced by
the level of fee paid on all of the contracts we reviewed and is based
on NASA's evaluation of compliance with criteria contained in its award-
fee plans. However, the agency has not evaluated the overall
effectiveness of award fees in promoting desired outcomes. As noted,
NASA developed its new policies on award-fee contracts because the
agency and its Office of Inspector General found that it was paying
excessive fees with limited emphasis on acquisition outcomes. However,
according to NASA officials, the agency has not completed any
assessments of the effectiveness of award fees since the award-fee
policy was restructured in the 1990s, nor has it developed metrics or
performance measures to conduct such evaluations. Further, NASA lacks
an agencywide system with the capability of compiling and aggregating
award-fee information and for identifying trends and outcomes.
According to NASA officials, even NASA's modern Integrated Enterprise
Management Program (IEMP) will not provide this capability. Thus, NASA
cannot meaningfully judge how well award fees are improving or can
improve contractor performance and program outcomes.
Conclusions:
NASA could better link its award fees to desired results by making
greater use of outcome factors, its preferred criteria for evaluating
award fee contracts. While NASA has established policies and guidance
that provide an appropriate framework for their use, the agency has not
always used award fees as preferred by its guidance. To the extent that
NASA uses input evaluation factors and numerous subfactors for
evaluating performance, NASA may be diluting the leverage of award fees
in achieving desired results. Our review raises questions as to the
extent NASA is achieving the effectiveness it sought through the
establishment of guidance on the use of award fees. However, NASA has
not evaluated the overall effectiveness of its implementation of award
fees.
Recommendations for Executive Action:
We are making the following three recommendations to increase the
likelihood that the award fees NASA pays incentivize high performance
from its suppliers.
* We recommend that the NASA Administrator reemphasize to the NASA
centers the importance of tying award-fee criteria to desired outcomes
and limiting the number of subfactors used in evaluations.
* To ensure that cost-plus-award-fee contracts are used only when their
benefits outweigh the costs, we recommend that the NASA Administrator
direct the centers to consider costs and benefits in choosing this
contract type by requiring documentation explaining how the perceived
benefits will offset the additional cost associated with its
administration as required by the FAR.
* Finally, we recommend that the NASA Administrator require the
development of metrics for measuring the effectiveness of award fees,
establish a system for collecting data on the use of award-fee
contracts, and regularly examine the effectiveness of award fees in
achieving desired acquisition outcomes.
Agency Comments and Our Evaluation:
In commenting on a draft of this report, NASA concurred with our
recommendations and indicated that it would reemphasize its current
guidance as recommended, address the issues raised by the report in
training, and cover those issues in its internal reviews of procurement
operations at the individual Space Centers. In terms of our
recommendation to develop metrics for measuring the effectiveness of
award fees and establish a system for collecting data on the use of
award-fee contracts, NASA concurred and indicated it would explore the
best way to develop and use metrics for evaluating the effectiveness of
award fees and set up a system for collecting data on award-fee
contracts. NASA said it planned to contact the Department of Defense to
obtain information on its process, since DOD is also developing such a
data collection system and metrics for measuring the effectiveness of
award fees. NASA also provided technical comments on the draft, which
have been incorporated as appropriate.
As agreed with your office, unless you announce its contents earlier,
we will not distribute this report further until 30 days from its date.
At that time, we will send copies to interested congressional
committees and the NASA Administrator. We will also make copies
available to others upon request. In addition, the report will be
available at no charge on the GAO Web site at [Hyperlink,
http://www.gao.gov].
If you or your staff have any questions concerning this report, please
contact me at (202) 512-4841 or calvaresibarra@gao.gov. Contact points
for our Offices of Congressional Relations and Public Affairs may be
found on the last page of this report. Key contributors to this report
are acknowledged in appendix IV.
Signed by:
Ann M. Calvaresi-Barr:
Director:
Acquisition and Sourcing Management:
[End of section]
Appendix I: Scope and Methodology:
Scope and Methodology:
Our objectives were to determine (1) the extent the National
Aeronautics and Space Administration's (NASA) guidance addresses the
problems previously identified with the use of award-fee contracts and
(2) whether NASA follows its guidance in using award fees to achieve
desired outcomes.
We selected 10 NASA cost-plus-award-fee (CPAF) contracts to review. Our
selection was based on contract data from the Federal Procurement Data
System. We extracted information on all NASA contracts active between
fiscal years 2002 and 2004 that were coded as CPAF. To ensure the
validity of the database from which we drew our contracts, we confirmed
the contract type of each of the 10 contracts we selected through NASA
contracting officers and contract documentation. The contracts we
selected were the top 10 dollar value contracts active from fiscal
years 2002 through 2004. These contracts account for about $7.6
billion, or 44 percent, of obligated cost-plus-award-fee-dollars for
the 3-year period.
To determine the extent NASA's guidance addresses the problems
previously identified with the use of award-fee contracts and whether
NASA follows its guidance in using award fees to achieve desired
outcomes, we interviewed responsible program and procurement officials
at NASA headquarters and six NASA centers. We also reviewed the Federal
Acquisition Regulation (FAR), the NASA FAR Supplement, and NASA's Award
Fee Contracting Guide. We conducted a literature review and examined
previous reports, studies, and analyses done by GAO, NASA, the NASA
Inspector General, or others that included information related to
NASA's use of award fees and other relevant issues.
Additionally, we reviewed contract files, obtained information from
program and contracting officials through the use of a structured
questionnaire, and discussed the application of award-fee criteria with
NASA officials involved in the award-fee process. The contract
documents we reviewed contained information related to the development
and implementation of the award fee. This information included the
basic contract and statement of work, acquisition planning documents,
award-fee modifications, performance evaluation plan documentation
describing fee criteria for specific evaluation periods, contractor
self-assessments, performance evaluation board reports, and fee
determination documents. We used this information to corroborate and
supplement the information provided by NASA officials in response to
structured questionnaires we prepared and interviews we conducted. We e-
mailed the questionnaires and received written responses for all 10 of
the contracts. We conducted structured interviews with contracting and
program officials concerning the development, implementation, and
effectiveness of the award-fee structure for some of the contracts.
To accomplish our work, we visited NASA headquarters in Washington,
D.C. We also visited and held teleconferences with Goddard Space Flight
Center in Greenbelt, Maryland, responsible for managing 3 of the
contracts we reviewed; Johnson Space Center in Houston, Texas,
responsible for managing 3 of the contracts; and Marshall Space Flight
Center in Huntsville, Alabama, responsible for managing 1 of the
contracts. We held teleconferences with officials at the Jet Propulsion
Laboratory in Pasadena, California; Kennedy Space Center in Cape
Canaveral, Florida; and Ames Research Center in Moffett Field,
California, responsible for managing 1 contract each under our review.
Our work was conducted from August 2005 through October 2006 in
accordance with generally accepted government auditing standards.
[End of section]
Appendix II: Summary Description of the 10 NASA Contracts GAO Reviewed:
NAS5-60000-Earth Observing System Data and Information System Core
System-Goddard Space Flight Center:
NAS5-60000 was an end item hardware cost-plus-award-fee contract
between NASA and Hughes Applied Information Systems Incorporation.
Raytheon Information Systems Company acquired Hughes in December 1999
and became the prime contractor. The contract, currently closed, was
managed by Goddard Space Flight Center. The 10-year research and
development contract was awarded in March 1993 for the development and
operation of the Earth Observing System Data and Information System
Core System. The period of performance on the contract actually ended
in April 2005, and the contract has since been closed. According to
Goddard Space Flight Center procurement officials, the desired program
outcome or objective of the contract was to develop a technically
capable system to process data from NASA's satellites at a reasonable
cost. Procurement officials stated that the Earth Observing System Data
and Information System Core System, a state-of-the-art data-processing
system, is currently dedicated to the processing and dissemination of
NASA Earth Science satellite data.
NAS15-10000-International Space Station-Johnson Space Center:
NAS15-10000 is an end item hardware cost-plus-award-fee contract
between NASA and the Boeing Company. The contract, currently active, is
managed by the Johnson Space Center. A letter contract was awarded in
November 1993 and was definitized in January 1995 as a cost-plus-
incentive-fee award-fee contract. In October 1999, during a
restructuring of the contract, the cost-plus-incentive-fee award-fee
contract was converted to a cost-plus-award-fee contract. The contract
was extended in December 2003, partially because of the Columbia
accident. This planned 10-year contract is for the design, development,
manufacture, and on-orbit assembly of the U.S. on-orbit segment of the
International Space Station. The contract also included provisions for
a level of effort that included (1) sustaining engineering, (2) multi-
element integrated testing, (3) logistics and maintenance-post
production support, (4) technical definition of contract changes, and
(5) other engineering support. According to Johnson Space Center
procurement officials, the desired program outcomes or objectives of
this contract are (1) completion of the U. S. on-orbit segment,
delivery, and on-orbit acceptance of the space station; (2) sustaining
engineering of the U.S. on-orbit segment hardware and software and
common hardware and software provided to international partners/
participants and payloads; (3) post-production support of the U.S. on-
orbit segment hardware and common hardware provided to the
international partners/participants; and (4) space station end-to-end
subsystems management for the majority of the subsystems and specialty
engineering disciplines.
NAS5-32633-Landsat-7 Spacecraft-Goddard Space Flight Center:
NAS5-32633 was an end item hardware cost-plus-award-fee contract
between NASA and Lockheed Martin Missiles and Space. The contract,
currently closed, was managed by Goddard Space Flight Center. The
research and development contract was initially awarded by the Air
Force in October 1992 and transferred to NASA in May 1994. The contract
was for the design, development, fabrication, integration, test, and
pre-and post-launch support of the Landsat-7 spacecraft. Landsat-7 was
launched in April 1999; the contract was completed in 2005. The purpose
of the Landsat-7 satellite is to obtain continuous remotely sensed,
high-resolution imagery of the earth's surface for environmental
monitoring, disaster assessment, land use and regional planning,
cartography, range management, and oil and mineral exploration.
According to Goddard Space Flight Center procurement officials, the
desired program outcome or objective of the contract was to develop an
operational satellite that met the science requirements of users and
the laws requiring the data be obtained at a reasonable cost.
NAS8-60000-Program Information Systems Mission Services-Marshall Space
Flight Center:
NAS8-60000 was a cost-plus-award-fee service contract between NASA and
Computer Sciences Corporation. The contract, managed by the Marshall
Space Flight Center, was in the process of being closed as of June
2006. It was awarded in May 1994, and covered a 2-year period of
performance, but included options to extend the period of performance
for an additional 6 years--through April 30, 2002. The contract was
extended three times, with the period of performance ending on March
30, 2004. The primary purpose of the contract was to provide services
in the area of program information system mission services. The
contractor's responsibilities were to manage, be responsible for, and
provide information services to meet requirements of the Information
Systems Services Office and its customers. According to Marshall Space
Flight Center procurement officials, the desired program outcome or
objective of the contract was to provide services including operating
and maintaining existing equipment and software; gathering, analyzing,
defining, and documenting systems requirements; and planning,
designing, developing, acquiring, integrating, testing, and
implementing new systems or enhancements to existing systems.
NAS2-14263-Engineering and Technical Support for Life Sciences-Ames
Research Center:
NAS2-14263 was a cost-plus-award-fee service contract between NASA and
Lockheed Martin Engineering and Science Company, defined under task
orders. The contract, managed by Ames Research Center, was in the
process of being closed as of June 2006. Its period of performance
ended in September 2003. The 5-year research and development contract
was awarded in May 1995 for the provision of engineering and technical
support services for Ames Research Center life sciences. The work to be
performed included engineering and technical support for life sciences
projects, including space shuttle life sciences payloads, other life
science payloads, the Space Station Biological Research Project, ground-
based life sciences research, and advanced life support technology
development. According to Ames Research Center procurement officials,
the desired program outcome or objective of the contract was to achieve
support for space life science projects, life sciences research, and
related technology.
NAS9-19100-Science, Engineering, Analysis, and Test-Johnson Space
Center:
NAS9-19100 was a cost-plus-award-fee service contract between NASA and
Lockheed Martin with indefinite delivery, indefinite quantity task
orders; performance-based; and level-of-effort provisions. Following
the merger of Lockheed and Martin in 1995, NASA consolidated two
existing contracts to form NAS9-19100 with an effective date of October
1, 1996. The contract, managed by Johnson Space Center, was in the
process of being closed as of June 2006. The period of performance
ended in January 2005. The contract included requirements related to
hardware, government-furnished crew equipment, facilities, laboratory
maintenance, life sciences, flight hardware, and support for the
science and engineering requirements of the Space Shuttle Program and
the International Space Station Program. According to Johnson Space
Center procurement officials, the desired program outcomes or
objectives of the contract were to provide engineering and science
support to all engineering directorates at Johnson Space Center as well
as support both the science and engineering requirements of the shuttle
and space station programs.
NAS9-98100-Consolidated Space Operations Contract-Johnson Space Center:
NAS9-98100 was a cost-plus-award-fee service contract between NASA and
Lockheed Martin Space Operations Company, with task orders and level-
of-effort provisions. The contract, which was in the process of being
closed as of June 2006, was managed by the Johnson Space Center. It was
awarded on September 25, 1998, with a basic 5-year period of
performance and an option for an additional 5-year period. NASA chose
not to exercise the option for the second 5-year period of performance.
The contract required (1) developing an integrated operations approach
to spacecraft design, operations, and data processing that minimized
cost and the support infrastructure required to conduct space
operations; (2) obtaining a highly capable and accountable contractor
that would be responsible for providing space operations mission and
data services; and (3) providing a contract and management structure
that would enable outsourcing, commercialization, or privatization of
some or all service under the contract. According to Johnson Space
Center procurement officials, the desired program outcomes or
objectives of the contract were to (1) provide excellent quality and
reliable mission and data services at a significantly reduced cost; (2)
move end-to-end mission and service responsibility and accountability
to industry; (3) implement an integrated architecture that reduces
overlap, eliminates unnecessary duplication, and reduces life cycle
costs; (4) define streamlined processes that minimize intermediaries
required to define requirements and deliver services; and (5) adopt
private sector commercial practices and services.
NAS10-99001-Joint Base Operation and Support-Kennedy Space Center:
NAS10-99001 is a cost-plus-award-fee service contract between NASA and
Space Gateway Support. The contract, currently active, is managed by
Kennedy Space Center. The contract was awarded on October 1, 1998, for
a basic 5-year period of performance and included an option for an
additional 5 years. NASA exercised that option on October 1, 2003. The
purpose was to provide for base operations support at NASA's Kennedy
Space Center and the Air Force's Cape Canaveral Air Force Station, as
well as specific requirements at Patrick Air Force Base and Florida
Annexes into one consolidated contract. In addition to NASA and the Air
Force, other primary customers include the Navy, Department of
Interior, Spaceport Florida, and commercial customers such as Boeing,
Lockheed Martin, Orbital Science, and Astrotech. According to Kennedy
Space Center procurement officials, the desired program outcomes or
objectives of the contract are to (1) enhance safety for the public and
on-site workforce; (2) provide protection of human, national, and
environmental resources; (3) provide high-quality and responsive
service to customers; (4) reduce the cost of doing business for NASA
and the Air Force; (5) provide flexibility to respond to new
requirements and unplanned events; (6) improve supportability and
reliability through innovative methodologies and concepts; (7) provide
common support practices and systems; and (8) increase small business
subcontracting goals.
NAS5-01090-Mechanical System Engineering Services-Goddard Space Flight
Center:
NAS5-01090 is a cost-plus-award-fee service contract between NASA and
Swales and Associates, with a line item for indefinite delivery,
indefinite quantity task orders. The contract, currently active, is
managed by Goddard Space Flight Center. NAS5-01090 was awarded in
January 2001 with a period of performance of 5 years and 30 days.
According to Goddard Space Flight Center procurement officials, the
period of performance was extended and was currently scheduled to end
on August 15, 2006. The purpose of the contract is to provide
engineering services for the study, design, development, fabrication,
integration, testing, verification, and operations of space flight and
ground system hardware and software, including development and
validation of new technologies to enable future science missions.
According to Goddard Space Flight Center procurement officials, the
desired program outcomes or objectives of the contract were to obtain
high-quality performance, desired results, and output.
NAS7-03001-Jet Propulsion Laboratory:
NAS7-03001 is a cost-plus-award-fee contract between NASA and the
California Institute of Technology, a private nonprofit educational
institution, which establishes the relationship for the operation of
the Jet Propulsion Laboratory (JPL) federally funded research and
development center. The contract, currently active, is a 5-year
research and development contract awarded in November 2002 for the
operation and management of JPL. The contract allows for extension or
decrease to the initial period of performance in 3-or 9-month
increments. JPL is a NASA-owned facility as well as an operating
division of Caltech. Caltech has operated JPL as a federally funded
research and development center since 1959 to meet certain government
research and development needs, which, according to the contract, could
not be met as effectively by existing government resources or normal
contractor relationships. The contract includes both service and
product deliverables, which are defined in task orders issued under the
contract. The contract encompasses a large number of discrete programs
and projects--approximately 500 active task orders. According to NASA
procurement officials, the desired program outcomes or objectives of
the contract are specific performance requirements defined in task
orders issued under the contract. The contract encompasses support of
exploration of the solar system, including earth-based investigations,
investigations and studies to support NASA missions in the fields of
earth science and astrophysics and astrobiology, as well as development
of supporting fundamental technologies.
[End of section]
Appendix III: Comments from the National Aeronautics and Space
Administration:
National Aeronautics and Space Administration:
Office of the Administrator:
Washington, DC 20546-0001:
January 12, 2007:
Ms. Ann Calvaresi-Bare:
Director:
Acquisition and Sourcing Management:
United States Government Accountability Office:
Washington, DC 20548:
Dear Ms. Calvaresi-Barr:
The National Aeronautics and Space Administration (NASA) appreciates
the opportunity to comment on your draft report entitled "NASA
Procurement: Use of Award Fees for Achieving Program Outcomes Can Be
Improved" (GAO-07-58). The information provided in your report should
help us make better use of award-fee contracts.
In the draft report, GAO recommends that the NASA Administrator take
the following three actions:
Recommendation 1: Reemphasize to the NASA Centers the importance of
tying award-fee criteria to desired outcomes and limiting the number of
sub factors used in evaluations.
Concur - NASA concurs with this recommendation. NASA's award-fee
guidance already expresses a preference for tying award fees to
evaluation factors that are based on outcomes as opposed to inputs or
outputs. NASA's guidance also cautions against spreading the potential
award fee over a large number of performance evaluation factors that
dilute the emphasis on any particular performance evaluation criteria.
NASA will reemphasize its guidance, address it in training, and cover
this area in its ongoing internal reviews of Center procurement
operations.
Recommendation 2: To ensure that cost-plus-award-fee contracts are only
used when their benefits outweigh their costs, direct the Centers to
consider costs and benefits in choosing this contract type by requiring
documentation explaining how the perceived benefits will offset the
additional costs associated with its administration, as required by the
Federal Acquisition Regulation (FAR).
Concur - NASA concurs with this recommendation. NASA guidance already
requires that the award-fee contract type only be selected when it is
demonstrated that the costs of administration of award-fee contracts
are offset by the perceived benefits. Although the FAR does not require
written documentation of the cost benefit analysis, NASA will amend its
FAR supplement to do so. NASA will reemphasize this existing
requirement, address it in training, and cover this area in its ongoing
internal reviews of Center procurement operations.
Recommendation 3: Require the development of metrics for measuring the
effectiveness of award fees, establish a system for collecting data on
the use of award-fee contracts, and regularly examine the effectiveness
of award fees in achieving desired acquisition outcomes.
Concur - NASA concurs with this recommendation. Although we understood
from the auditors that no other agency currently performs the
recommended activities, NASA will explore the best way to develop and
use metrics for evaluating the effectiveness of award fees, as well as
set up a system for collecting data on the use of award-fee contracts.
This system will allow NASA to measure whether the fees awarded are
commensurate with the contract goals achieved. The auditors stated that
the Department of Defense (DoD) was in the process of developing
metrics and a system for collecting data on its award-fee contracts.
NASA will contact DoD to obtain information on its processes and assess
their compatibility with NASA's processes.
Although NASA concurs with GAO's draft report recommendations,
technical comments will be provided under separate cover.
Thank you for the opportunity to respond to this draft report.
Sincerely,
Signed by:
Shana Dale:
Deputy Administrator:
[End of section]
Appendix IV: GAO Contact and Staff Acknowledgments:
GAO Contact:
Ann M. Calvaresi-Barr, (202) 512-4841:
Staff Acknowledgments:
In addition to the individual named above, Thomas Denomme, Assistant
Director; James Beard; Shirley Johnson; Julia Kennon; Heather Barker
Miller; Kenneth Patton; Sylvia Schatz; and Robert Swierczek made key
contributions to this report.
FOOTNOTES
[1] FAR Part 16.405-2(a)(2).
[2] In December 2005, we issued a report on the use of award and
incentive fees at the Department of Defense that identified many of the
same issues as had been identified by NASA. GAO, Defense Acquisitions:
DOD Has Paid Billions in Award and Incentive Fees Regardless of
Acquisition Outcomes, GAO-06-66 (Washington, D.C., Dec. 19, 2005).
[3] Fiscal year 2004 was the last year for which award-fee contract
data were available in the Federal Procurement Data System when we
began our work.
[4] The sample was 10 contracts that were the top 10 dollar value award-
fee contracts in terms of obligations active in that time period. The
estimated value of the 10 contracts totaled over $31 billion as of June
2006, and the contracts accounted for 44 percent of obligated CPAF
dollars for the 3-year period. Three of the contracts were for end
items, while 7 were service contracts.
[5] This workforce composition also includes NASA grantees.
[6] GAO, High-Risk Series: An Update, GAO-05-207 (Washington, D.C.:
January 2005).
[7] NASA's field centers include Ames Research Center, Dryden Flight
Research Center, Glenn Research Center, Goddard Space Flight Center,
Johnson Space Center, Kennedy Space Center, Langley Research Center,
Marshall Space Flight Center, NASA Shared Services Center, Stennis
Space Center, and the Jet Propulsion Laboratory, which is a federally
funded research and development center and operated by the California
Institute of Technology.
[8] FAR Part 16.404(b)(1) and 16.405-2(b)(1)(iii).
[9] NASA's regulatory award-fee policy is found in the NASA FAR
Supplement. We refer to the NASA FAR Supplement as NASA's "regulations"
throughout this report. NASA's Award Fee Contracting Guide, dated June
2001, states that the purpose of the guide is to explain and elaborate
on NASA's regulatory policy, providing examples and dealing with
practical concerns that could not be addressed in the NASA FAR
Supplement.
[10] GAO, NASA: Long-Term Commitment to and Investment in Space
Exploration Program Requires More Knowledge, GAO-06-817R (Washington,
D.C.: July 17, 2006).
[11] These weaknesses in NASA CPAF contracting practices were cited in
our 1994 report, GAO, NASA Procurement: Challenges Remain in
Implementing Improvement Reforms, GAO/NSIAD-94-179 (Washington, D.C.,
Aug. 18, 1994).
[12] GAO/NSIAD-94-179.
[13] An exception is made for those awarded under the commercial item
procedures of FAR Part 12.
[14] By initial cost, schedule, and performance parameters, we mean
those parameters agreed to at initial contract award.
[15] In comments on our draft report, NASA officials told us that the
number of subfactors used to evaluate the contractor's performance in
fiscal year 2006 was 57, and the number of subfactors planned for
fiscal year 2007 will be further reduced to 45.
[16] The 99 percent of award fee paid on the Landsat-7 contract
includes 6 years of on-orbit incentive fee, which is based on the
satellite meeting the government's requirements.
[17] Under the original contract, managed by the Air Force, the
contractor was to assemble the satellite and its payload---the Enhanced
Thematic Mapper Plus (ETM+) instrument. The original estimated cost of
the contract, before it was transitioned to NASA and rebaselined, was
$372.4 million. When the contract was transferred from the Air Force to
NASA, the subcontract for the ETM+ instrument was split into a separate
prime contract.
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site (www.gao.gov). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site.
To have GAO e-mail you a list of newly posted products every afternoon,
go to www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125
Washington, D.C. 20548:
Public Affairs:
Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548:
