Chemical Demilitarization
Additional Management Actions Needed to Meet Key Performance Goals of DOD's Chemical Demilitarization Program Gao ID: GAO-08-134 December 6, 2007Destruction of the nation's remaining stockpile of chemical weapons in a safe, efficient, and timely manner is essential to meet Chemical Weapons Convention treaty obligations and to reduce the risk of a potential catastrophic event. The Department of Defense (DOD) established the Chemical Demilitarization Program to manage the destruction of the remaining stockpile. GAO was asked to evaluate the (1) progress DOD and the Army have made in addressing GAO's prior recommendations to strengthen program management, (2) reasonableness of schedule milestones, (3) reliability of cost estimates, and (4) effectiveness of efforts to provide monetary incentives to the systems contractors. GAO reviewed relevant planning documents, schedules, cost estimates, and contracts; interviewed program and contractor officials; and visited chemical agent destruction sites.
DOD and the Army have taken steps in addressing GAO's prior recommendations to strengthen program management by establishing an overall strategy and supporting implementation plan, but some key elements, such as annual performance measures for some key goals, including interim destruction goals, are not fully developed. Moreover, actions DOD and the Army have taken to identify and mitigate the risk of future program schedule extensions and cost growth have not been effective because the Chemical Materials Agency's risk management process has not been fully developed or integrated with DOD's risk management process. As a result, managers lack an integrated and systematic approach to evaluate and manage risk. Recently achieved destruction rates may indicate that adjusted schedule milestones are overly conservative. Moreover, the program's schedule was extended in 2005 in response to the slower-than-anticipated destruction rates experienced since 2003, when more optimistic milestones were adopted. GAO's review of processing rates achieved at the sites since the revised schedule determined that most sites are significantly ahead of the revised, extended program estimates for recently completed munitions campaigns. For example, Umatilla, Oregon, completed destruction of one of its rocket munitions by August 2007, about 12 months earlier than forecasted. GAO was not able to verify the accuracy of the program's cost estimates because of shortcomings in the underlying cost data, such as undefined facility closure requirements and unstable baseline costs. GAO determined that the usefulness of the earned value management tool to provide visibility over the program's cost performance is limited because of these shortcomings. The program's projected cost growth is largely attributable to longer schedules and increased costs associated with facility closure estimates. GAO estimated that a 1-month schedule extension would cost an additional $9.5 million. Also, schedules used for the closure cost estimates for destruction sites have increased, from 12 months to about 30 months at some sites, but they have not been finalized or independently reviewed. Since 2003, the Army has actively managed criteria for determining monetary fees--largely award fees--it provided systems contractors to incentivize their performance. These fees had emphasized safety and environmental compliance, and Army officials acknowledged that award fees did not successfully control schedule and cost growth. In 2006, after audit results and DOD direction, and while extending the schedule, the Army reinstated schedule as a performance measure. The award fee plan allows site project managers flexibility to weigh performance measures within a range, and GAO determined that the high end of the range for management was generally selected, rather than for schedule and cost. Most criteria in the program's award fee plans are subjective and are not linked to objective outcomes as much as possible, as called for in DOD guidance. Given the program's maturity, GAO found that the Army could apply more weight to performance measures, such as schedule and cost, and add objective criteria in these plans.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-08-134, Chemical Demilitarization: Additional Management Actions Needed to Meet Key Performance Goals of DOD's Chemical Demilitarization Program
This is the accessible text file for GAO report number GAO-08-134
entitled 'Chemical Demilitarization: Additional Management Actions
Needed to Met Key Performance Goals of DOD's Chemical Demilitarization
Program' which was released on January 16, 2008.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
United States Government Accountability Office:
GAO:
December 2007:
Chemical Demilitarization:
Additional Management Actions Needed to Meet Key Performance Goals of
DOD's Chemical Demilitarization Program:
Chemical Demilitarization:
GAO-08-134:
GAO Highlights:
Highlights of GAO-08-134, report to congressional requesters.
Why GAO Did This Study:
Destruction of the nation‘s remaining stockpile of chemical weapons in
a safe, efficient, and timely manner is essential to meet Chemical
Weapons Convention treaty obligations and to reduce the risk of a
potential catastrophic event. The Department of Defense (DOD)
established the Chemical Demilitarization Program to manage the
destruction of the remaining stockpile. GAO was asked to evaluate the
(1) progress DOD and the Army have made in addressing GAO‘s prior
recommendations to strengthen program management, (2) reasonableness of
schedule milestones, (3) reliability of cost estimates, and (4)
effectiveness of efforts to provide monetary incentives to the systems
contractors. GAO reviewed relevant planning documents, schedules, cost
estimates, and contracts; interviewed program and contractor officials;
and visited chemical agent destruction sites.
What GAO Found:
DOD and the Army have taken steps in addressing GAO‘s prior
recommendations to strengthen program management by establishing an
overall strategy and supporting implementation plan, but some key
elements, such as annual performance measures for some key goals,
including interim destruction goals, are not fully developed. Moreover,
actions DOD and the Army have taken to identify and mitigate the risk
of future program schedule extensions and cost growth have not been
effective because the Chemical Materials Agency‘s risk management
process has not been fully developed or integrated with DOD‘s risk
management process. As a result, managers lack an integrated and
systematic approach to evaluate and manage risk.
Recently achieved destruction rates may indicate that adjusted schedule
milestones are overly conservative. Moreover, the program‘s schedule
was extended in 2005 in response to the slower-than-anticipated
destruction rates experienced since 2003, when more optimistic
milestones were adopted. GAO‘s review of processing rates achieved at
the sites since the revised schedule determined that most sites are
significantly ahead of the revised, extended program estimates for
recently completed munitions campaigns. For example, Umatilla, Oregon,
completed destruction of one of its rocket munitions by August 2007,
about 12 months earlier than forecasted.
GAO was not able to verify the accuracy of the program‘s cost estimates
because of shortcomings in the underlying cost data, such as undefined
facility closure requirements and unstable baseline costs. GAO
determined that the usefulness of the earned value management tool to
provide visibility over the program‘s cost performance is limited
because of these shortcomings. The program‘s projected cost growth is
largely attributable to longer schedules and increased costs associated
with facility closure estimates. GAO estimated that a 1-month schedule
extension would cost an additional $9.5 million. Also, schedules used
for the closure cost estimates for destruction sites have increased,
from 12 months to about 30 months at some sites, but they have not been
finalized or independently reviewed.
Since 2003, the Army has actively managed criteria for determining
monetary fees”largely award fees”it provided systems contractors to
incentivize their performance. These fees had emphasized safety and
environmental compliance, and Army officials acknowledged that award
fees did not successfully control schedule and cost growth. In 2006,
after audit results and DOD direction, and while extending the
schedule, the Army reinstated schedule as a performance measure. The
award fee plan allows site project managers flexibility to weigh
performance measures within a range, and GAO determined that the high
end of the range for management was generally selected, rather than for
schedule and cost. Most criteria in the program‘s award fee plans are
subjective and are not linked to objective outcomes as much as
possible, as called for in DOD guidance. Given the program‘s maturity,
GAO found that the Army could apply more weight to performance
measures, such as schedule and cost, and add objective criteria in
these plans
What GAO Recommends:
GAO recommends that DOD and the Army develop interim destruction goals,
approaches, and milestones; establish time frames to complete its risk
management approach; develop realistic schedule and closure cost
estimates; finalize and independently review cost estimates; and
determine whether a greater emphasis can be placed on schedule and
cost, and develop more objective award fee performance evaluation
criteria. DOD concurred or partially concurred with 12 of GAO‘s 13
recommendations.
To view the full product, including the scope and methodology, click on
[hyperlink, http://www.GAO-08-134]. For more information, contact Davi
D'Agostino at (202) 512-5431 or dagostinod@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
DOD and the Army Have Taken Steps to Strengthen Program Management, but
Have Not Fully Developed an Integrated Approach for Measuring
Performance and Managing Risk:
Recently Improved Destruction Rates for Chemical Munitions May Indicate
Overly Conservative Schedule Milestones at the Four Incineration Sites:
Shortcomings in Underlying Cost Data Make Earned Value Management
Results Unreliable and Limited Our Ability to Determine Cost Estimate
Reliability:
The Army Has Actively Managed Award Fees and Could Further Emphasize
Schedule and Cost and Link Award Fees to More Objective Performance
Measures:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Two Recently Introduced Performance-Based Incentive Fees:
Appendix III: Comments from the Department of Defense:
Appendix IV: GAO Contact and Staff Acknowledgments:
Related GAO Products:
Tables:
Table 1: Status of DOD's Chemical Agent Stockpile Sites and Chemical
Agent Destroyed as of September 30, 2007:
Table 2: Comparison of Some Key Destruction Objectives Identified in
DOD's Strategic Plan and CMA's Strategic Plan:
Table 3: Comparison of Performance Measures Identified in the PART, OSD
Strategic Plan, CMA Strategic Plan, and Selected Acquisition Report:
Table 4: Comparison of Estimated Operation Completion Dates:
Table 5: Number of Days Saved per Site since 2005 Program Office
Schedule Estimates:
Table 6: Potential Operational Cost Savings Associated with Days Saved
since 2005 Schedule Estimate:
Table 7: Number of Months of Improvement in 2007 Program Current
Estimate for Operations Completion versus 2005 Army Program Office
Estimate and 2005 DOD Independent Estimate:
Table 8: CMA Assessed Confidence in Achieving DOD Schedule Milestone
and April 29, 2012, Treaty Deadline:
Table 9: Changes in Estimated Total Program Cost:
Table 10: Comparison of GAO's Independent Estimates to Complete
Operations and Closure Activities with the Systems Contractor's and
Program Manager's Estimates:
Table 11: Percentage of Award Fees Earned at the Incineration Sites and
Newport, from Fiscal Year 2003 to Fiscal Year 2006:
Table 12: Performance Measures and Weights in Award Fee Plans, Fiscal
Years 2003-2007:
Figures:
Figure 1: U.S. Stockpile of Chemical Agents and Munitions:
Figure 2: Umatilla Chemical Agent Disposal Facility, Umatilla Chemical
Depot, Oregon:
Figure 3: Delivery of a Ton Container to the Processing Building at the
Newport Chemical Agent Disposal Facility, Newport Chemical Depot,
Indiana:
Figure 4: Intermodal Storage Containers at the Newport Chemical Agent
Disposal Facility, Newport Chemical Depot, Indiana:
Figure 5: 2005 Program Office Schedule Estimates by Processing Rates,
Downtime, and Risk:
Figure 6: Comparison of the Performance Baseline with the Planned
Budget and Systems Contractors' Estimate at Completion:
Figure 7: Comparison of the Monetary Incentive Fees Earned by Systems
Contractors at the Incineration Sites and at Newport during Fiscal Year
2006:
Abbreviations:
ACWA: Assembled Chemical Weapons Alternatives:
APB: Acquisition Program Baseline:
CAIG: Cost Analysis Improvement Group:
CMA: Chemical Materials Agency:
CWC: Chemical Weapons Convention:
DOD: Department of Defense:
DPPBI: Director's Programmatic Performance Based Incentive:
EAC: estimate at completion:
EVM: earned value management:
OMB: Office of Management and Budget:
OSD: Office of the Secretary of Defense:
PART: Program Assessment Rating Tool:
SAR: Selected Acquisition Report:
United States Government Accountability Office:
Washington, DC 20548:
December 6, 2007:
The Honorable Adam Smith:
Chairman:
The Honorable Mac Thornberry:
Ranking Member:
Subcommittee on Terrorism and Unconventional Threats and Capabilities:
Committee on Armed Services:
House of Representatives:
The Honorable Jim Saxton:
House of Representatives:
Destruction of the nation's remaining stockpile of lethal chemical
weapons in a safe, efficient, and timely manner is essential to meeting
U.S. treaty obligations under the Chemical Weapons Convention
(CWC)[Footnote 1] and to reducing the risk to the public and the
environment of a potential catastrophic event. In 1985, Congress
directed the Department of Defense (DOD) to destroy the U.S. stockpile
of lethal chemical agents and munitions in the United States with the
requirement that DOD was to ensure maximum protection for the
environment, the general public, and the personnel involved in the
destruction efforts. DOD designated the Department of the Army as its
executive agent, and the Army established the Chemical Demilitarization
Program to destroy the stockpile at nine storage sites.[Footnote 2] The
United States and more than 180 other countries are signatories to the
CWC, which went into effect in 1997. The CWC prohibits the use of
chemical weapons and mandates a deadline of April 29, 2012, for the
United States to destroy the existing stockpiles.[Footnote 3]
DOD's Chemical Demilitarization Program has not experienced a chemical
event since its inception in 1985 that resulted in an environmental
catastrophe or death, but the program has experienced some potentially
serious events and has been plagued by numerous schedule revisions,
cost growth, and long-standing management weaknesses. The Chemical
Demilitarization Program is managed as a major defense acquisition
program and, as such, has cost and schedule milestones that are
contained in an Acquisition Program Baseline (APB). As of September 30,
2006, Congress had appropriated about $15.8 billion for the program. In
2005, DOD reported that the program had exceeded its milestones under
the 2003 APB in excess of Nunn-McCurdy thresholds for cost and schedule
performance[Footnote 4] for the second time in 3 years,[Footnote 5] and
new schedule milestones and cost estimates were being developed. These
revisions increased DOD's estimated costs for the entire Chemical
Demilitarization Program by 33 percent from about $24 billion in 2003
to about $32 billion in 2005. Moreover, the Army's revised cost
estimates for the portion of the program responsible for destroying the
stockpile, the Chemical Stockpile Elimination Project,[Footnote 6]
increased by more than 53 percent from about $15.8 billion in 2003 to
about $24.2 billion in 2005. A large percentage of this increase was
attributable to revised government cost estimates for the destruction
facility contracts. In April 2006, DOD announced that the program will
not meet the extended CWC deadline of April 29, 2012, for total
destruction of its stockpile, and it estimated that only 66 percent of
the stockpile will be destroyed by then. DOD also announced that it
would seek resources to complete destruction as close to the CWC
deadline as possible and separately requested authority to provide
additional monetary incentives if systems contractors operating the
incineration sites could accelerate their destruction activities to
safely exceed, meet, or finish destruction near the CWC deadline.
Moreover, the program's remaining four incineration sites--Anniston
Chemical Agent Disposal Facility, Anniston Chemical Activity
(Anniston), Alabama; Pine Bluff Chemical Agent Disposal Facility, Pine
Bluff Chemical Activity (Pine Bluff), Arkansas; Tooele Chemical Agent
Disposal Facility, Deseret Chemical Depot (Tooele), Utah; and Umatilla
Chemical Agent Disposal Facility, Umatilla Chemical Depot (Umatilla),
Oregon--were forecasted to miss the CWC deadline for the first
time.[Footnote 7]
In the past, the program has had long-term problems in meeting all of
the schedule milestones for destroying the stockpile, in part because
of the lengthy environmental permitting processes prior to operation
start-up and unexpected conditions of the aging munitions. The
stockpile destruction program managed by the Chemical Materials Agency
(CMA) has now reached a more mature phase, and CMA and its systems
contractors have more experience in incinerating each type of chemical
agent and munition in its stockpile as well as neutralizing bulk nerve
and blister agents.
DOD, the Army, CMA, and the systems contractors that operate the
destruction facilities face a difficult and hazardous task, as well as
the need to overcome some significant technical challenges, in
destroying the U.S. stockpile of lethal chemical agents while ensuring
the safety of workers, the public, and the environment. Recognizing
these difficulties and the unique program risks, DOD uses a contract
that reimburses the systems contractors for all reasonable, allowable,
and allocable costs. To encourage defense contractors to perform in an
innovative, efficient, and effective way in areas deemed important to
the program's success, DOD gives its systems contractors the
opportunity to earn millions of dollars through monetary incentives
known as award and incentive fees.
We reported in 2003 that DOD and the Army had not developed an overall
strategy and risk management approach for the chemical demilitarization
program to guide the program with a set of principles and a means to
manage risk, understand its evolution and implementation, and determine
whether initiatives are achieving their desired results.[Footnote 8]
The absence of an overall strategy and risk management approach left
the program without a clear road map to monitor program performance and
meet the goal of destroying the chemical stockpile in a safe and timely
manner. We recommended that DOD and the Army adopt practices that
leading organizations embrace for effectively implementing and managing
programs, to include promulgating a comprehensive mission statement,
long-term and annual performance goals, measurable performance
indicators, and evaluation and corrective action plans. DOD concurred
with our recommendations and stated that it would establish these
management tools. A list of related GAO products regarding chemical
demilitarization and storage and disposal of the chemical weapons
stockpile is included at the end of this report.
We also reported that overall DOD programs have paid contractors large
amounts of award fees on acquisitions that are falling behind schedule,
overrunning costs, and experiencing significant technical
problems.[Footnote 9] In addition, we recommended, among other things,
that DOD instruct the military services to move toward more outcome-
based award fee criteria that are both achievable and promote
accountability for acquisition outcomes.[Footnote 10] DOD concurred
with the recommendation and, in a March 2006 policy memorandum,
addressed desired outcomes and the role the award fee should play in
the overall acquisition strategy. The memorandum stated that "while
award fee contracts are used when it is neither feasible nor effective
to devise predetermined objective performance targets, it is imperative
that award fees be tied to identifiable interim outcomes, discrete
events or milestones, as much as possible."
This report responds to a request from the House Committee on Armed
Services' Subcommittee on Terrorism and Unconventional Threats and
Capabilities that we conduct a review of DOD's Chemical
Demilitarization Program. Specifically, this report evaluates the (1)
progress DOD and the Army have made in addressing our prior
recommendations to strengthen program management, whether this progress
has been sufficient to correct past program weaknesses, and the
effectiveness of actions DOD and the Army have taken to identify and
mitigate the risk of future program schedule extensions and cost
growth; (2) reasonableness of the methodology used to determine
schedule milestones; (3) reliability of its cost estimates; and (4)
effectiveness of DOD's and the Army's use of monetary incentives to
improve systems contractors' performance.
To assess the progress DOD and the Army have made in implementing GAO's
prior recommendations to strengthen program management, whether it has
been sufficient, and the effectiveness of actions taken to identify and
mitigate the risk of future program schedule extensions and cost
growth, we obtained and reviewed strategic and implementation plans and
interviewed cognizant officials to identify whether key elements were
present, such as a program mission statement, long-term goals and
objectives, delineation of roles and responsibilities of DOD and Army
offices, and near-term performance measures. To assess the
reasonableness of the methodology used to determine schedule milestones
as well as the reliability of cost estimates, we reviewed current
program estimates, destruction schedules, earned value management
(EVM)[Footnote 11] data, and other documents. We also obtained and
reviewed the program's risk management plans and related documents. In
addition, we identified the issues that had caused delays and
ascertained approaches being used to reduce the potential for delays in
the future. To analyze the effectiveness of DOD's and the Army's use of
monetary incentives to improve systems contractors' performance, we
obtained and reviewed contract documents on the award fees offered to
the systems contractors, and interviewed DOD and Army officials as well
as systems contractors' representatives. We obtained and reviewed the
incentive agreement plan and incentive fees for the Aberdeen Chemical
Agent Disposal Facility (Aberdeen), Maryland, and Newport Chemical
Agent Disposal Facility, Newport Chemical Depot (Newport), Indiana, to
determine the challenges those sites faced in negotiating the incentive
fee agreements. Also, we interviewed officials with the U.S. Army
Sustainment Command, CMA, and CMA and systems contractors at Tooele and
Umatilla. In addition, we met with DOD and Army program officials and
interviewed officials at two of the chemical agent destruction sites.
This report focuses primarily on the seven CMA-managed chemical
destruction sites--Aberdeen, Anniston, Johnston Atoll in the Pacific
Ocean, Newport, Pine Bluff, Tooele, and Umatilla. We selected these
sites for our review because most are in the operational phase,
represent about 82 percent of the remaining stockpile of chemical
agents to be destroyed, and have reached a level of maturity in
processing a variety of types of munitions and agents that will help
meet CWC goals. Two sites--Aberdeen and Johnston Atoll--have already
completed operations and have either been closed or are being closed.
This report does not focus on the two chemical agent destruction sites
that are part of the Assembled Chemical Weapons Alternatives (ACWA)
program--Blue Grass Chemical Agent-Destruction Pilot Plant, Blue Grass
Chemical Activity (Blue Grass), Kentucky, and Pueblo Chemical Agent-
Destruction Pilot Plant, Pueblo Chemical Depot (Pueblo), Colorado--
because they are currently in the design and construction phase to
destroy chemical agents, report directly to DOD rather than to the
Army, and because these facilities are not forecasted to begin
destruction operations before the CWC deadline, although the stockpiles
at these two sites are required to be destroyed by the deadline as
well.
We examined the reliability of the data used in this report by
verifying EVM system certification, applying analytical tests to the
EVM data, and examining EVM system effectiveness in providing
meaningful performance measurement to program management. We determined
that the EVM system as used at each site was not reliable, which is
discussed more fully in the report. We conducted this performance audit
from May 2006 through July 2007 in accordance with generally accepted
government auditing standards. A more thorough description of our scope
and methodology is provided in appendix I.
Results in Brief:
While DOD and the Army have taken steps toward addressing our prior
recommendations to strengthen program management by establishing
strategic and supporting implementation plans, they have not fully
developed an integrated approach for measuring performance or for
managing risk.[Footnote 12] DOD issued a strategic plan in June 2005,
and CMA issued a strategic plan in July 2005 and a risk management plan
in May 2006. While these plans identify challenges and contain
performance measures to monitor progress, we found that the plans lack
schedule goals for key phases. Neither DOD's 2006 baseline nor its
April 2006 and April 2007 Selected Acquisition Reports (SAR) to
Congress include schedule milestones extending beyond 2012. The program
further lacks annual performance goals linked to long-term strategic
goals, such as meeting the extended April 29, 2012, CWC deadline.
Without this linkage, program officials will lack the information on
interim progress needed to determine whether corrective action is
needed to safely destroy the stockpile by the treaty deadline.
Furthermore, while CMA's integrated risk management plan identified a
process for conducting risk management, the agency has not developed
the tools needed to implement its approach, such as a risk trade-off
analysis. For example, the program's plans identified contamination of
the mustard agent stockpile at one site as a key risk but lacked a
method for analyzing that risk or managing its potential effects on
cost and schedule. The later discovery of the extent of the
contamination delayed planned operations by more than 12 months. Also,
our analysis showed that DOD's and the Army's processes for identifying
risk are not integrated across sites or organizations, and CMA's
identified risks have not been fully aligned with those identified in
DOD's strategic plan. Consequently, program managers are hampered in
their information sharing and their progress in safely destroying
chemical agents, as well as in their ability to systematically evaluate
and manage risk. Finally, unlike more useful annual performance plans
we have reviewed, which include historical performance trends and
multiyear future goals, CMA's annual performance plan has a single-year
focus. We are recommending that the Secretary of the Army incorporate
historical trends and multiyear future goals in its annual performance
plan; develop interim destruction goals, approaches, and milestones;
and establish a time frame for implementing its risk management
approach, including integration across sites and with DOD.
Recently achieved processing rates have considerably exceeded
forecasted rates at the operational sites and indicate that DOD's
revised schedule milestones may be too conservative to be a realistic
measure. DOD extended its Chemical Demilitarization Program schedule in
2005 in response to the slower actual destruction rates experienced
than had been estimated when more optimistic milestones were adopted in
2003. As we have previously reported,[Footnote 13] best practices for
schedule development call for assumptions that are both up-to-date and
reasonable. Using a new schedule methodology, and as a result of
factoring in slower processing rates and greater risk, CMA extended
schedules at each site between 62 months and 80 months beyond the
baselines established in 2003. The extensions prompted budget and
acquisition reviews, including an independent cost and schedule
assessment, which in turn led DOD to further extend the program's
schedule milestones for completing destruction operations and closure
activities. Our analysis of recently achieved processing rates showed
that at the time of our review, the sites were between 2 months and 39
months ahead of the revised estimates for munitions destruction.
Revised schedule estimates published in the April 2007 SAR further
reflect processing rates that considerably exceed forecasts.
Nevertheless, DOD has no current plans to reassess the program schedule
milestones, as such reassessments are usually prompted by schedule
slippages. Higher destruction rates than scheduled indicate that an
independent reassessment of the program's scheduled milestones may be
warranted, thus assuring DOD and Congress of the reasonableness of the
baseline against which its progress is being measured and of its costs
as the current schedule is not providing DOD and Congress with a
reasonable baseline against which to measure progress and cost. We are
recommending that the Secretary of Defense direct the Cost Analysis
Improvement Group to assess the reasonableness of schedule milestones,
and that DOD periodically review the program's processing rates to
determine whether the milestones are still reasonable and include
schedule milestones for completing destruction operations and facility
closure activities in the program's baseline and acquisition report.
Shortcomings in the underlying data used in the program's EVM showed
that the cost estimate is unreliable. Our review showed that
performance baselines lacked validity and had not been stable because
all requirements, such as closure costs, are not fully identified and
included in program cost estimates and because large amounts of
additional costs are added to the performance baseline annually. In
addition, systems contractors' cost estimates were significantly lower
than the Army's cost estimates, due largely to the Army's expectations
of longer operational periods and greater closure requirements. The
program's cost increases are largely attributable to extended schedules
and facility closure estimates. Best practices indicate that cost
estimates should be comprehensive, and DOD guidance further reflects
the need for valid and stable performance baselines from which to make
projections and perform trend analysis. DOD and Congress currently lack
accurate estimates for budgeting for and funding all activities through
facility closure. Similarly, given the significant differences between
the contractors' and Army's cost estimates, an independent cost review
would provide DOD and Congress assurance that cost estimates are
realistic. We are recommending that the Army define closure
requirements and develop accurate and realistic closure cost estimates;
define all contract requirements, including authorized unpriced work
associated with facility closure activities, and establish a time frame
for placing these requirements on contract; and that an independent
review of the cost estimates be done and linked with reasonable
schedule milestones.
According to the Army and based on our analysis, award fees--the
predominant fees--offered to contractors have thus far not
satisfactorily resulted in the control of schedule performance outcomes
and cost growth. Army officials indicate that the majority of the award
fee incentives are dedicated to achieving high standards for safety,
surety, and environmental compliance in recognition of the lethal
nature of the chemical agent stockpile and the congressional mandate
for maximum protection of the workforce, the public, and the
environment. The Army has actively reviewed and adjusted criteria for
determining award fees it has provided to the systems contractors to
incentivize aspects of their performance. Based on our analysis, from
2003 through 2006, the award fee plans have heavily emphasized safety
and environmental compliance and did not have a separate performance
measure for schedule until 2006. Army officials acknowledged that award
fees have not satisfactorily resulted in the control of schedule and
cost growth. In response to our previous recommendations, in March
2006, DOD issued policy guidance directing that award fees be tied to
identifiable interim outcomes, discrete events, or milestones as much
as possible. CMA issued its award fee policy in September 2006, and our
review of this policy indicated that CMA did not clearly or fully
incorporate the concept of tying "award fees to identifiable interim
outcomes, discrete events or milestones as much as possible," as
directed in the March 2006 DOD guidance. In addition, the current award
fee plan for the five chemical demilitarization sites we reviewed
allows government site project managers flexibility in weighting each
performance measure within a set range. We found that managers have
selected the high end of the range for management, and the low end of
the range for schedule and cost, when determining award fee
percentages. Moreover, many performance measures in the award fee plans
contain subjective award fee criteria, and award fees for the most part
are not linked to objective or measurable criteria. Our analysis showed
that it is possible for CMA to further link award fee criteria in this
program to objective or measurable outcomes in at least three
performance measures. Given the maturity of the operations phase at the
incineration sites and Newport, although not legally required, we
identified opportunities, from a management perspective, to apply more
weight to performance measures, such as schedule and cost, to better
align CMA's award fee policy with DOD's policy, and to make greater use
of objective, measurable criteria in award fee plans to increase the
likelihood of meeting the April 29, 2012, treaty deadline. Two recently
established and planned performance-based incentive fees for the
incineration sites are focused on schedule. One fee was established in
2005 to encourage and reward collaboration among the incineration site
systems contractors with the goal of improving the overall program
schedule and cost performance. This fee is awarded based on sites'
collective performance and, in fiscal year 2007, this incentive fee
focused mostly on schedule performance associated with achieving the
CWC treaty deadline. The other planned incentive fee, as authorized by
Congress, is structured to accelerate destruction operations and
facility closure at the incineration sites. We could not determine how
effective these incentives might be due to the collective manner in
which the fee to encourage and reward collaboration is earned, its
relative size, and recent focus on objective schedule performance, and
because the other planned incentive fee has not yet been implemented.
We are recommending that the Army (1) determine whether greater weight
can be applied to key performance measures, such as schedule and cost,
in award fee plans; (2) better align its award fee policy with DOD's
March 2006 guidance on award fees; and (3) link award fee criteria, as
much as possible, to performance measures that focus on identifiable
interim outcomes, discrete events, or milestones.
We provided a draft of this report to DOD in October 2007 for its
review and comment. In written comments on a draft of this report, DOD
concurred or partially concurred with 12 of our 13 recommendations. DOD
did not concur with our recommendation regarding defining closure
requirements and developing accurate and realistic closure cost
estimates for each of DOD's chemical demilitarization sites. DOD
recommended that we delete it, since DOD officials believe that a
subsequent recommendation defining all contract requirements, including
those associated with facility closure activities, was more
encompassing. Based on DOD's comments, rather than deleting the
recommendation, we clarified it and the related recommendation to more
clearly distinguish the difference between the two recommendations, and
consolidated it with our two other recommendations related to defining
contract requirements. DOD also stated that the department will
continue to use acquisition management tools and discipline to ensure
the destruction of the United States' chemical weapons stockpile in a
safe and secure manner, while being economical and meeting the U.S.
commitments under the CWC. DOD identified a number of actions that it
already has initiated in response to our recommendations. DOD also
provided us with technical comments, which we incorporated in the
report as appropriate. DOD's response is reprinted in appendix III.
Background:
In 1985, Congress directed DOD to destroy the U.S. stockpile of lethal
chemical agents and munitions in the United States.[Footnote 14] DOD
designated the Department of the Army as its executive agent for the
program, and the Army established the Chemical Demilitarization
Program, which was charged with the destruction of the original
stockpile at nine storage sites. Figure 1 shows the locations of the
nine storage sites; the number of items, such as projectiles, rockets,
mortars, land mines, and bombs, originally stored at each site; and the
number of items remaining to be destroyed as of September 30, 2007.
Figure 1: U.S. Stockpile of Chemical Agents and Munitions:
This figure is a map showing stockpile of chemical agents and
munitions.
Umatilla Chemical Depot, Oregon;
Number of items: 220,603;
Remaining items: 65,060.
Deseret Chemical Depot, Utah;
Number of items: 1,138,785;
Remaining items: 122,361.
Newport Chemical Depot, Indiana;
Number of items: 1,690;
Remaining items: 579.
Aberdeen Proving Ground, Maryland;
Number of items: 1,817;
Remaining items: 0.
Blue Grass Army Depot, Kentucky;
Number of items: 101,764;
Remaining items: 101,764.
Anniston Army Depot, Alabama;
Number of items: 661,531;
Remaining items: 446,712.
Pine Bluff Arsenal, Arkansas;
Number of items: 123,095;
Remaining items: 32,684.
Pueblo Chemical Depot, Colorado;
Number of items: 780,078;
Remaining items: 780,078.
Johnston Atoll, Pacific Ocean;
Number of items: 412,798;
Remaining items: 60.
[See PDF for image]
Source: GAO analysis of CMA data.
[End of figure]
The remaining U.S. stockpile, as of June 2007, consists of two nerve
agents--GB and VX[Footnote 15]--and a blister agent--mustard[Footnote
16]--which currently are stored at seven sites in the continental
United States. Incineration was initially selected as the method to
destroy the stockpile at all nine sites, and destruction began in 1990
at Johnston Atoll. This location was selected to be the operational
prototype facility to test the destruction process prior to it being
conducted in the continental United States. Figure 2 shows an example
of a chemical agent disposal facility like those used at the four
operational incineration sites (Anniston, Pine Bluff, Tooele, and
Umatilla).
Figure 2: Umatilla Chemical Agent Disposal Facility, Umatilla Chemical
Depot, Oregon:
This figure is a photograph of Umatilla Chemical Agent Disposal
Facility, in Umatilla Chemical Depot, Oregon.
[See PDF for image]
Source: Umatilla Chemical Agent Disposal Facility.
[End of figure]
Subsequent to the decision to use incineration to destroy the remaining
stockpile, Congress directed DOD to evaluate the possibility of using
alternative technologies to incineration.[Footnote 17] In 1994, the
Army initiated a project to develop nonincineration technologies for
use at the two bulk-agent-only sites at Aberdeen and Newport. These
sites were selected in part because their stockpiles were relatively
simple--each site had only one type of agent and this agent was stored
in bulk-agent (ton) containers, which are used to store various
chemical agents at the chemical agent destruction sites.[Footnote 18]
Figure 3 shows a ton container being transported to the processing
building at Newport.
Figure 3: Delivery of a Ton Container to the Processing Building at the
Newport Chemical Agent Disposal Facility, Newport Chemical Depot,
Indiana:
This figure is a photograph of the delivery of a ton container to the
processing building at Newport Chemical Agent Disposal Facility, in
Newport, Indiana.
[See PDF for image]
Source: Newport Chemical Agent Disposal Facility.
[End of figure]
In 1997, DOD approved pilot testing of a neutralization technology at
these two sites. Figure 4 shows the type of hydrolysate-filled
intermodal storage containers used at Newport. The hydrolysate is
awaiting final disposition.[Footnote 19]
Figure 4: Intermodal Storage Containers at the Newport Chemical Agent
Disposal Facility, Newport Chemical Depot, Indiana:
This figure is a photograph of intermodal storage containers at the
Newport Chemical Agent Disposal Facility in Newport Chemical Depot,
Indiana.
[See PDF for image]
Source: Newport Chemical Agent Disposal Facility.
[End of figure]
Also in 1996, Congress directed DOD to evaluate the use of alternative
technologies and suspended incineration planning activities at Blue
Grass and Pueblo, two sites with assembled weapons.[Footnote 20]
Furthermore, Congress directed that these two sites be managed in a
program independent of the Army's Chemical Demilitarization Program and
report to DOD directly instead of the Army.[Footnote 21] Thus, the ACWA
program was established.
Table 1 shows DOD's nine chemical agent stockpile sites; the primary
destruction method used (i.e., incineration or neutralization) and the
secondary disposal method used for the treatment of the resulting
wastewater, if applicable; the types of munitions destroyed, such as
projectiles, rockets, mortars, land mines, and bombs in addition to ton
containers; the types of agents (GB, VX, and mustard) and their status
(e.g., destroyed, ongoing destruction, or planned destruction); the
original agent tonnage; the tonnage destroyed; the percentage of the
stockpile destroyed; the remaining agent tonnage; and the status of
each facility (closed, operational, or under construction) as of
September 30, 2007.
Table 1: Status of DOD's Chemical Agent Stockpile Sites and Chemical
Agent Destroyed as of September 30, 2007:
Site: Johnston Atoll;
Destruction method: Incineration;
Munition types destroyed: Projectiles, land mines, mortars, and ton
containers;
Type of agent and status[A]: GB, VX, and mustard (destroyed);
Original agent tonnage: 705[B];
Tonnage destroyed: 705;
Percentage of stockpile destroyed: 100;
Remaining agent tonnage (as of September 2007): 0;
Facility status: Closed.
Site: Tooele;
Destruction method: Incineration;
Munition types destroyed: Ton containers, mines, rockets, projectiles,
spray tanks, and mortars;
Type of agent and status[A]: GB and VX (destroyed) and mustard
(ongoing);
Original agent tonnage: 13,617;
Tonnage destroyed: 9,292;
Percentage of stockpile destroyed: 68;
Remaining agent tonnage (as of September 2007): 4,325;
Facility status: Operational.
Site: Anniston;
Destruction method: Incineration;
Munition types destroyed: Rockets and projectiles;
Type of agent and status[A]: GB (destroyed), VX (ongoing), and mustard
(planned);
Original agent tonnage: 2,254;
Tonnage destroyed: 726;
Percentage of stockpile destroyed: 32;
Remaining agent tonnage (as of September 2007): 1,528;
Facility status: Operational.
Site: Umatilla;
Destruction method: Incineration;
Munition types destroyed: Rockets, bombs, and projectiles;
Type of agent and status[A]: GB (ongoing) and VX and mustard (planned);
Original agent tonnage: 3,719;
Tonnage destroyed: 1,016;
Percentage of stockpile destroyed: 27;
Remaining agent tonnage (as of September 2007): 2,703;
Facility status: Operational.
Site: Pine Bluff;
Destruction method: Incineration;
Munition types destroyed: Rockets and ton containers;
Type of agent and status[A]: GB (destroyed) and VX and mustard
(planned);
Original agent tonnage: 3,850;
Tonnage destroyed: 484;
Percentage of stockpile destroyed: 12;
Remaining agent tonnage (as of September 2007): 3,366;
Facility status: Operational.
Site: Aberdeen;
Destruction method: Neutralization and off-site treatment of
wastewater;
Munition types destroyed: Ton containers;
Type of agent and status[A]: Mustard (destroyed) and off-site
wastewater treatment (complete);
Original agent tonnage: 1,622;
Tonnage destroyed: 1,622;
Percentage of stockpile destroyed: 100;
Remaining agent tonnage (as of September 2007): 0;
Facility status: Closed.
Site: Newport;
Destruction method: Neutralization and off-site treatment of
wastewater;
Munition types destroyed: Ton containers;
Type of agent and status[A]: VX (ongoing) and off-site wastewater
treatment (ongoing)[C];
Original agent tonnage: 1,269;
Tonnage destroyed: 587;
Percentage of stockpile destroyed: 46;
Remaining agent tonnage (as of September 2007): 682;
Facility status: Operational.
Site: Pueblo;
Destruction method: Neutralization and on-site biotreatment of
wastewater;
Munition types destroyed: Projectiles and mortars;
Type of agent and status[A]: Mustard (planned);
Original agent tonnage: 2,611;
Tonnage destroyed: 0;
Percentage of stockpile destroyed: 0;
Remaining agent tonnage (as of September 2007): 2,611;
Facility status: Destruction pilot plant under construction.
Site: Blue Grass;
Destruction method: Neutralization and on-site water oxidation of
wastewater;
Munition types destroyed: Rockets and projectiles;
Type of agent and status[A]: GB, VX, and mustard (planned);
Original agent tonnage: 523;
Tonnage destroyed: 0;
Percentage of stockpile destroyed: 0;
Remaining agent tonnage (as of September 2007): 523;
Facility status: Destruction pilot plant under construction.
Total;
Destruction method: [Empty];
Munition types destroyed: [Empty];
Type of agent and status[A]: [Empty];
Original agent tonnage: 30,170;
Tonnage destroyed: 14,432;
Percentage of stockpile destroyed: 48;
Remaining agent tonnage (as of September 2007): 15,738;
Facility status: [Empty].
Source: DOD.
Note: The Non-Stockpile Chemical Materiel Project is the CMA
organization responsible for all chemical warfare materiel that resides
outside of a declared stockpile. This materiel consists of chemical
agent identification sets, which consist of glass ampoules, vials and
bottles of chemical agent, and actual chemical weapons. The project
provides centralized management and direction to DOD for the disposal
of nonstockpile chemical warfare materiel in a safe, environmentally
sound, and cost-effective manner. It is responsible for destroying the
nerve agents GB and VX in projectiles, vials, bottles, glass ampoules,
and ton containers, As of June 2007, the Non-Stockpile Chemical
Materiel Project had destroyed 563 tons of chemical agent, which is
counted toward meeting the CWC 45 percent destruction milestone.
[A] The stockpile includes two nerve agents (GB and VX) and a blister
agent (mustard).
[B] The site at Johnston Atoll destroyed 1,326 tons of agent before the
CWC entry into force in 1997. Overall, the site at Johnston Atoll has
destroyed a total of 2,031 tons of agent.
[C] On January 5, 2007, CMA began shipping Newport VX hydrolysate to a
commercial wastewater treatment facility in Port Arthur, Texas. Because
the VX hydrolysate is being transported to Texas for final treatment,
the United States receives credit for destruction of the Newport
stockpile under CWC. Of an estimated 1.8 million gallons of VX
hydrolysate at Newport, as of June 2007, approximately 33 percent has
already been shipped to Texas for incineration.
[End of table]
In 1997, the United States ratified the CWC, which prohibits the use of
these weapons and mandates the elimination of existing stockpiles by
April 29, 2012, which represents a 5-year extension from the initial
deadline.[Footnote 22] The CWC also contains a series of interim
deadlines applicable to the U.S. stockpile[Footnote 23] for destroying
1 percent, 20 percent, and 45 percent of the stockpile. The United
States met the 1 percent and 20 percent destruction deadlines, but did
not meet the original 45 percent destruction deadline. The United
States received an extension from April 2004 to December 2007 for the
45 percent deadline. As of June 2007, program officials reported that
45 percent of the U.S stockpile had been destroyed, 6 months earlier
than the interim CWC deadline for destroying that amount, and the
entire stockpile at two of the nine sites has been eliminated.
DOD designated the Chemical Demilitarization Program as a major
acquisition program.[Footnote 24] As such, the overall program cost and
schedule--and its milestones for major phases of the program--are
identified in the APB and are reported to Congress in the SAR.
Additionally, the Army, through its CMA, developed program estimates
that include schedule milestones to support its day-to-day management
responsibilities of the program. Finally, systems contractors that
operate the chemical agent destruction facilities have established
schedule milestones for completing major phases identified in their
individual contracts with the Army.
From 1998 through 2005, the total program cost estimates increased by
113 percent, resulting in the program breaching approved program
baselines in 2001 and 2005. As a result, the program was required to
certify to Congress in each instance that new cost estimates were
reasonable and that the management structure is adequate to manage and
control costs. The bulk of these program costs will go to systems
contractors that construct and operate the facilities that destroy the
chemical weapons.
DOD and the Army Have Taken Steps to Strengthen Program Management, but
Have Not Fully Developed an Integrated Approach for Measuring
Performance and Managing Risk:
DOD and the Army have taken steps in addressing our prior
recommendations to strengthen program management by establishing an
overall strategy and supporting implementation plans, but some key
elements related to performance measures and risk management are not
fully developed or implemented. DOD and the Army have identified a
number of risks across the program that could affect the program's cost
and schedule, and CMA has developed a risk management plan, but the
program's risk management approach has not been effective because it
has not been fully implemented or integrated across the program.
Moreover, while DOD and the Army have established key program goals,
performance measures have not always been linked to these goals.
DOD Addressed GAO's Prior Recommendations by Establishing Strategic and
Supporting Implementation Plans:
DOD has taken a number of actions to address our past recommendations
that it establish strategic and supporting implementation plans. We
recommended in 2003 that DOD and the Army develop an overall strategy
and implementation plan for the chemical demilitarization program that
would articulate a program mission statement, identify the program's
long-term goals and objectives, delineate the roles and
responsibilities of all DOD and Army offices, and establish near-term
performance measures.[Footnote 25] We also recommended that DOD and the
Army implement a risk management approach that anticipates and
influences internal and external factors that could adversely affect
program performance. DOD concurred with our recommendations and has
strengthened program management by establishing an overall strategy and
supporting implementation plans.
In response to our 2003 recommendations, DOD and the Army have
developed various strategic and supporting plans. For example, in June
2005, DOD published the Strategic Plan for Destruction of Lethal
Chemical Agents and Munitions.[Footnote 26] The purpose of this plan is
to serve as an overarching plan for destruction of the U.S. stockpile
of lethal chemical agents and munitions and nonstockpile items. It
applies to both portions of the program implemented by CMA as well as
the ACWA program. (In this report, we only address the parts of the
plan pertaining to the CMA stockpile program.) This plan included many
of the elements that we recommended, such as a program mission
statement to enhance national security by eliminating chemical warfare
munitions while protecting the workforce, the public, and the
environment and meeting requirements specified in the CWC. It also
included overall goals and objectives and performance measures.
Also, in July 2005, CMA published the U.S. Army's Chemical Materials
Agency Strategic Plan (2005-2010). This plan encompasses strategies for
all CMA organizational elements and programs--including the chemical
stockpile storage program, the Non-Stockpile Chemical Materiel Project,
the Elimination of Chemical Weapons program, and the Pine Bluff Arsenal
industrial base--together with those of the AWCA program. According to
the CMA strategic plan, its purpose is to be a road map for the safe
elimination of the U.S. chemical weapons stockpile. This strategic plan
articulated program goals and objectives and also described a number of
supporting implementation plans that we recommended should be
developed, including annual performance plans that would provide
performance measures and an integrated risk management plan that would
address the program's challenges.
In its July 2005 strategic plan, CMA identified 11 supporting plans
that were directly or indirectly related to the stockpile program.
According to CMA officials, 9 of the 11 supporting plans had been
developed, as of July 2007. These plans include the CMA resource
management plan, a risk management plan, and a risk management
directorate plan.
The DOD strategic plan articulated four key goals for the Chemical
Demilitarization Program: (1) eliminate the stockpile while protecting
the workforce, the public, and the environment, and comply with U.S
obligations under the CWC; (2) employ sound business practices to
ensure efficient life cycle management of program resources; (3)
institute a program of continuous improvement regarding safety,
environment protection, efficient operations and maintenance, and
facility closure to mitigate risks and to ensure compliance with
statutory, regulatory, and policy requirements and decisions; and (4)
maintain communications with the public and local and national
officials. Performance objectives with the associated performance
metrics were developed to address these goals. Table 2 contains some
key objectives to meet goals contained in the DOD and CMA strategic
plans.
Table 2: Comparison of Some Key Destruction Objectives Identified in
DOD's Strategic Plan and CMA's Strategic Plan:
DOD's Strategic Plan: Achieve CWC extended 45 percent destruction
milestone by December 31, 2007, while ensuring the safety of the
workers, the public, and the environment;
CMA's Strategic Plan: Destroy the U.S. stockpile of lethal chemical
agents and munitions while meeting the CWC deadlines and containing
total life cycle cost by using an integrated risk management approach.
DOD's Strategic Plan: Achieve CWC extended 100 percent destruction
deadline by April 29, 2012, while ensuring the safety of the workers,
the public, and the environment;
CMA's Strategic Plan: No objective linked to meeting CWC extended
deadline by April 29, 2012.
DOD's Strategic Plan: Manage life cycle costs to the program within
fiscal resources and consistent with the APB;
CMA's Strategic Plan: No objective linked to meeting the APB.
DOD's Strategic Plan: Develop and institute a risk management culture
that ensures that no chemical agent releases, exposures, and accidents
occur across the entire demilitarization community;
CMA's Strategic Plan: Meet and maintain Army, state, and federal safety
and security standards and environmental laws, regulations, and permit
conditions.
Source: GAO analysis of DOD and CMA data.
[End of table]
CMA also jointly issued the program's Fiscal Year 2007 Annual
Performance Plan in September 2006.[Footnote 27] The performance plan
contains, among other things, annual goals for meeting the CWC's
December 2007 interim destruction deadline. However, Tooele is the only
site that has approved annual plans, which it is implementing. CMA
officials stated that they will request annual performance plans from
all of the sites for fiscal year 2008. Moreover, the plans lacked some
attributes that would make them more useful to decision makers and to
Congress. We have previously found that performance plans can be more
effective if they (1) use intermediate goals and measures to show
progress or contribution to intended results, particularly if it may
take years to meet the intended results; (2) show baseline and trend
data for past performance; and (3) permit an agency to convey what it
expects to achieve in the long term by including multiyear performance
goals in its performance plan.[Footnote 28] With baseline and trend
data, the more useful performance plans provided a context for drawing
conclusions about whether performance goals are reasonable and
appropriate. By including multiyear performance goals, a performance
plan can provide congressional and other decision makers with an
indication of the incremental progress the agency expects to make in
achieving results. CMA's performance plan only contains goals related
to the specified fiscal year and does not show previous years' trends
or show how achievement of the specified goals would contribute to the
program's long-term goal.
Risk Management Approach Is Not Fully Developed or Integrated, and Some
Key Program Goals Lack Clearly Linked Performance Measures:
DOD has partially implemented our recommendations to establish
performance measures and implement a risk management approach that
anticipates and influences internal and external factors that could
adversely affect program performance. Specifically, CMA has not yet
completed all the risk management tools identified in its risk
management plan or integrated its risk management efforts across sites
and organizations. Further, while DOD did establish performance
measures as we recommended, some key program goals still lack clearly
linked performance measures. Moreover, the program is using four
different performance measurement approaches and measurements that are
not well-aligned with one another. Without annual performance goals and
measures linked with long-term strategic goals, program officials will
lack information on the interim progress being achieved and whether
corrective action is needed to safely destroy the stockpile by the
treaty deadline.
CMA Has Made Progress in Developing an Integrated Risk Management
Approach but Has Not Completed Its Risk Management Tools and
Integration Efforts:
As identified in its U.S. Army Chemical Materials Agency Integrated
Risk Management Plan, CMA has developed a process to identify program-
level risks and reports them as part of the Army's Probability of
Program Success Report.[Footnote 29] The program office, in conjunction
with the site project managers, also has developed a "risk landscape,"
which is used to capture risks that could affect the operating and
closure schedules. However, no programwide approach exists to integrate
the risk management efforts at the site, CMA, and DOD levels to ensure
that all risks are being addressed in an integrated manner. For
example, CMA officials acknowledged that risks that are included in the
Probability of Program Success Reports and the site risk landscapes
were developed independently of each other and thus may not be fully
integrated. Also, while the sites were developing risk plans that are
at different stages of maturity, CMA has not yet developed a single set
of risk management procedures to be used by the site-level project
managers and site systems contractors when developing these plans.
Lastly, the risks identified in DOD plans and those identified by CMA
were not identified as part of an integrated process. CMA officials
stated that the risks it identified are an aggregate of some site risks
and CMA program-level risks, whereas the risk identified by DOD in its
strategic plan represents a general analysis of the major factors that
the senior leadership has selected as appropriate for long-term
attention. CMA officials stated that CMA is working to establish a
formal process for aligning risks among all planning and reporting
documents.
Other steps that are needed to fully implement this approach have not
been completed. For instance, CMA identified other tools to aid in
implementing an overarching risk management program to complete the
steps in this process, but these tools have not been fully developed.
These tools include a risk trade-off analysis and a risk management
information system database. A risk trade-off analysis is normally
included in this risk planning step. According to CMA officials, while
risk trade-off analyses are discussed in their risk management plan,
the program has not formally used this analysis in assessing program
risk, and thus this analysis has not been available to help decision
makers consider how to address program challenges. For example, while
the condition of the stockpile of mustard agent at Tooele was
identified as a key program risk, a process was not in place to perform
a comprehensive analysis to consider the risks to the program's cost
and schedule if assumptions about the level of mercury contamination
were understated. Because the extent of the problem was not discovered
sooner, needed facility modifications were delayed, extending the
planned operations schedule by 12 months.
CMA has also not completed the development of its integrated risk
information management system, which would involve the collection,
documentation, and reporting of risk data in support of the steps in
the overall risk management process. According to CMA, an initial
requirement analysis for the development of an integrated risk
information system determined that initially considered in-house
solutions would require considerable additional development. A market
survey will be conducted to consider external solutions. While a
comprehensive risk management approach could manage risks, and CMA is
still in the process of developing such an approach, CMA has not
established time frames for the completion and implementation of its
risk management approach. Without clear expectations for fully
developing and implementing its risk management approach within a set
time frame, CMA could continue to be limited in its efforts to control
some of the issues that have been affecting the program.
Some Key Program Goals Lack Clearly Linked Performance Measures:
While DOD and the Army have made progress in establishing program
goals, we found that some program goals were not linked to one another
or to key performance measures, and making such linkages is a practice
of leading organizations.[Footnote 30] For example, in its fiscal year
2007 annual performance plan, CMA had developed an annual performance
measure linked to the interim CWC destruction goal of 45 percent by
December 2007. However, neither DOD nor CMA nor the Army has developed
annual performance measures that are linked to achieving the ultimate
goal of destroying the total stockpile at each remaining site by the
April 29, 2012, treaty deadline. Additionally, while the Army has
recently established annual site goals for chemical agent tons
destroyed, these targets have been linked either to Army or DOD
schedules that far exceed the extended April 29, 2012, treaty deadline.
Without this linkage, it is difficult for program officials to track
the progress that the program is making toward its goals, and program
officials have little assurance that program activities are linked to
program goals.
The program is using four different performance measurement approaches:
the SAR, the Office of Management and Budget's (OMB) Program Assessment
Rating Tool (PART),[Footnote 31] and DOD's and CMA's strategic plans.
As this is a major acquisition program, some performance measures
related to cost and schedule are contained in the SAR. However, while
DOD and CMA officials cited the schedule milestones in the APB as the
schedule the program is accountable to, neither the 2006 approved APB,
the April 2006 SAR, nor the April 2007 SAR contained schedule milestone
dates beyond April 2012, but rather stated that schedule milestones are
"to be determined." DOD officials stated that the dates are not
included in the APB because it was approved about 1 week before DOD
officially announced to Congress that all sites would not meet the
extended April 29, 2012, treaty deadline. Although 18 months (as of
October 2007) have elapsed since DOD made that announcement, the dates
have still not been included in these documents. DOD and CMA officials
stated that the milestone dates are still being used for performance
tracking despite not being included in the most recent SARs. While the
program may have a measurement for tracking progress, its exclusion
from the SAR limits program oversight by Congress.
The program also developed additional performance measures beyond those
used in the acquisition reporting system. For example, the program
developed performance measures in (1) the Office of the Secretary of
Defense's (OSD) strategic plan, (2) OMB's PART, and (3) CMA's strategic
plan. However, these measures have not always been linked to each other
and have not always been linked to key program goals. For example,
while the achievement of destruction of the stockpile by the treaty
deadline is an overall program goal in both the DOD and CMA strategic
planning documents, neither DOD nor CMA has developed annual
performance measures that are linked to achieving that goal even though
they had established goals linked to the interim 45 percent destruction
goal.
Leading organizations seek to establish clear hierarchies of
performance goals and measures and link these goals and measures for
each successive organizational level to the overall program
goal.[Footnote 32] Our analysis of the various performance measures
also determined that they were not always linked to one another or tied
to key program goals. For example, while OSD identified 14 performance
measures that support the four goals contained in the strategic plan
that directly relate to the Chemical Demilitarization Program's
stockpile program, CMA's performance measures are not always aligned
with OSD's performance measures. Also, while OSD identified performance
measures that are similar to those tracked by OMB in the PART, two
performance measures in the PART are not included in OSD's strategic
plan. These two measures were the planned cost per ton of agent
destroyed against actual cost and the statistical confidence of meeting
the 45 percent CWC deadline. The SAR contained performance measures
related to safety and environmental performance, such as those related
to chemical exposures and releases, but did not have performance
measures related CWC tonnage goals. Table 3 compares measures
identified in the PART, the OSD strategic plan, the CMA strategic plan,
and the SAR.
Table 3: Comparison of Performance Measures Identified in the PART, OSD
Strategic Plan, CMA Strategic Plan, and Selected Acquisition Report:
OMB PART measures: Number of chemical agent exposures;
OSD strategic plan: No chemical agent exposures as defined in the APB;
CMA strategic plan: Number of chemical exposures as defined in the APB;
SAR (CMA and Newport): Number of chemical agent exposures.
OMB PART measures: Number of chemical agent releases;
OSD strategic plan: No chemical releases as defined in the APB;
CMA strategic plan: Number of chemical agent releases as defined in the
APB;
SAR (CMA and Newport): Number of chemical agent releases.
OMB PART measures: Tons of chemical agent destroyed against CWC 45
percent destruction deadline;
OSD strategic plan: Percentage of actual amount of stockpiled chemical
materiel destroyed against CWC 45 percent destruction deadline;
CMA strategic plan: No tonnage destroyed performance measure against
CWC 45 percent destruction deadline;
SAR (CMA and Newport): No CWC tonnage destroyed performance measure.
OMB PART measures: Tons of chemical agent destroyed against CWC 100
percent destruction deadline;
OSD strategic plan: Percentage of actual amount of stockpiled chemical
materiel destroyed against CWC 100 percent destruction deadline;
CMA strategic plan: No tonnage destroyed performance measure against
CWC 100 percent destruction deadline;
SAR (CMA and Newport): No CWC tonnage destroyed performance measure.
OMB PART measures: Recordable incidence rate;
OSD strategic plan: Recordable incidence rate;
CMA strategic plan: Recordable incidence rate;
SAR (CMA and Newport): Meet Army, state, and/or federal requirements.
OMB PART measures: Percentage of stockpile destroyed against CWC 45
percent destruction deadline;
OSD strategic plan: No related measure;
CMA strategic plan: Statistical confidence in achieving CWC 45 percent
destruction deadline;
SAR (CMA and Newport): No related measure.
OMB PART measures: Planned cost per ton of agent destroyed against
actual cost;
OSD strategic plan: No related measure;
CMA strategic plan: Plan cost per ton of agent destroyed against actual
cost;
SAR (CMA and Newport): No related measure.
Source: GAO analysis of DOD and CMA data.
[End of table]
This lack of integration between key program goals and performance
measures hampers program officials' ability to effectively track the
progress that the program is making because the performance plan lacks
destruction goals, approaches, and milestones that are directly linked
to overall program goals for safely meeting CWC deadlines.
Recently Improved Destruction Rates for Chemical Munitions May Indicate
Overly Conservative Schedule Milestones at the Four Incineration Sites:
Since DOD extended the program office schedules and DOD schedule
milestones in 2005, actual processing rates have considerably exceeded
the forecasted rates at the four incineration sites, indicating that
schedule milestones being used to measure the program's progress may be
too conservative to be realistic. DOD's Chemical Demilitarization
Program schedule has been extended in recent years largely because of
slower-than-expected processing rates for destroying chemical agent-
filled munitions. Program office schedules were extended in 2005
because of a greater recognition of slower processing rates for
destroying chemical munitions than schedule milestones adopted in the
more optimistic 2003 APB. In addition, the program office employed a
new schedule forecasting methodology that also contributed to the
schedule extensions because it accounted for more equipment downtime
and for the risks that have delayed processing in the past. The
schedule extensions exceeded the existing 2003 baseline and led DOD to
adopt new schedule milestones in 2006. Finally, program confidence
levels for meeting the April 29, 2012, CWC milestone are low.
Slower-Than-Anticipated Processing Rates and Time Frames Led to
Schedule Extensions:
The program office schedules have been extended since 2003 largely
because forecasted processing rates for destroying chemical agent-
filled munitions were reduced based on the program's analysis of
historical processing rates being slower than anticipated. Our review
of best practices indicates that estimating assumptions should be
realistic and backed by historical data to minimize uncertainty and
risk.[Footnote 33] Best practices also stipulate that estimates are
accurate when they are not overly conservative or optimistic. Best
practices also recommend that when underlying data or other assumptions
change, cost and schedule estimates should be revised to reflect the
current status. According to program officials, previous program office
schedule estimates developed in the early 2000s were based on
assumptions about processing rates derived largely from peak processing
experiences at Johnston Atoll, which began operations in 1990, and
early munitions processing campaigns at Tooele--the first two sites in
operation. According to CMA officials, program office schedule
estimates used from 2001 to 2003 were derived from using the 5 best
weeks of these two sites' production rates. The program's assumption
was that the follow-on sites, which were similarly designed, would be
able to achieve these rates because of the program's years of
experience with the various munitions and agents. Program officials
said that in practice those rates were achievable only for short
periods and were not sustainable because of problems with individual
munitions, such as difficulty in draining the agent (much slower than
assumed), equipment jams, and processing restrictions specified in
facility operating permits.
New Schedule Methodology Includes More Time for Downtime and Processing
Risks:
The program office adopted a new schedule development methodology in
2004 that not only used modeling to determine processing rates, but
gave greater recognition to the issues that cause delays, such as plant
downtime days for equipment maintenance and the inclusion of risks,
such as those associated with the unexpected condition of the stored
munitions that further extended the schedule. While previous schedule
estimates were point estimates, in that a single processing rate was
developed and used for each campaign and the individual campaigns were
added together, statistical modeling was used to develop a distribution
of different schedule outcomes. Processing rates were evaluated by
munitions type, such as projectiles, rockets, and mines, to develop a
histogram of daily processing outcomes.[Footnote 34] These results were
used as part of the modeling inputs. For current estimates, unlike past
estimates, schedule estimators also collected data on issues that had
caused processing delays in the past and identified possible impacts on
the schedule and probabilities of occurrence.
Data were also collected on past program incidents that had caused
sites to remain idle for extended periods, and the estimators also
developed probabilities for these "downtime events." The model was run
to produce a distribution of 5,000 schedule outcomes. The median
result--referred to as the 50 percent confidence level--was selected as
the program office schedule estimate. Selection of the median result
was also considered conservative, since the previous program office
schedules have used the mode, which is the outcome that happens most
frequently. According to schedule estimators, the median result is
typically to the right of or longer than the mode.
Thus, when the schedule estimates were updated in 2005 using the median
schedule outcomes, they were not only more conservative, but also
considered more realistic. Figure 5 shows how processing rates,
downtime, and risk contributed to the schedule estimate.
Figure 5: 2005 Program Office Schedule Estimates by Processing Rates,
Downtime, and Risk:
This figure is a bar graph showing 2005 program office schedule
estimates by processing rates, downtime, and risk.
[See PDF for image]
Source: CMA.
[End of figure]
Program Office Schedule Extensions Led to Revised Program Baseline:
When these schedule estimates exceeded the schedule milestones
contained in the April 2003 APB and resulted in major changes to the
life cycle cost estimates, an acquisition-related process was initiated
to develop new schedule milestones. The new schedule estimate was
approved by the Army Cost Review Board in May 2005 and resulted in the
new Army Cost Position. Concurrently, DOD was also conducting a program
review through its Defense Acquisition Board process that would result
in new schedule estimates to be included in the program's Approved
Acquisition Program Baseline. As part of this process, the DOD's Cost
Analysis Improvement Group (CAIG) developed independent schedule
estimates.[Footnote 35] CAIG's schedule forecasts were also based on
modeling to select the median outcome. Unlike those based on the
program rebaselining that occurred from 2001 to 2003, the program
office schedule estimates and those developed by CAIG were relatively
close in most cases. The schedule milestones in the 2003 APB are
compared with those in the 2005 Army and DOD schedule estimates in
table 4.
Table 4: Comparison of Estimated Operation Completion Dates:
Site: Anniston;
2003 Acquisition Program Baseline: May 2011;
2005 Army program office estimate: September 2016;
Difference from 2003 Acquisition Program Baseline: 64 months;
2005 DOD independent estimate: June 2016;
Difference from 2003 Acquisition Program Baseline: 61 months;
Program current estimate: May 2015.
Site: Newport;
2003 Acquisition Program Baseline: January 2006;
2005 Army program office estimate: October 2011;
Difference from 2003 Acquisition Program Baseline: 75 months;
2005 DOD independent estimate: February 2012;
Difference from 2003 Acquisition Program Baseline: 73 months;
Program current estimate: October 2009.
Site: Pine Bluff;
2003 Acquisition Program Baseline: November 2009;
2005 Army program office estimate: January 2015;
Difference from 2003 Acquisition Program Baseline: 62 months;
2005 DOD independent estimate: November 2015;
Difference from 2003 Acquisition Program Baseline: 72 months;
Program current estimate: June 2013.
Site: Tooele;
2003 Acquisition Program Baseline: February 2008;
2005 Army program office estimate: October 2014;
Difference from 2003 Acquisition Program Baseline: 80 months;
2005 DOD independent estimate: September 2015;
Difference from 2003 Acquisition Program Baseline: 91 months;
Program current estimate: November 2014.
Site: Umatilla;
2003 Acquisition Program Baseline: January 2011;
2005 Army program office estimate: June 2017;
Difference from 2003 Acquisition Program Baseline: 77 months;
2005 DOD independent estimate: March 2017;
Difference from 2003 Acquisition Program Baseline: 74 months;
Program current estimate: December 2014.
Source: GAO analysis of DOD and CMA data.
[End of table]
Recently Improved Processing Rates May Make the Current Schedule
Milestones Overly Conservative, but Program Confidence Levels for
Meeting Treaty Milestones Are Low:
Our review of processing rates that have been achieved at the sites
since the program office estimates were developed in 2005 determined
that most sites are significantly ahead of the program office estimate
for recently completed munition campaigns, indicating that schedule
milestones being used to measure the program's progress may be too
conservative to be realistic. For example, the 2005 program office
schedule forecasted that Umatilla would complete destruction of its
91,442 sarin nerve agent rocket munitions by August 2007. Those
operations were actually completed in August 2006--about 12 months
ahead of the program office 2005 estimate. This processing rate was
achieved despite delays to the site operator internal schedule related
to the unanticipated fires associated with some of the rockets.
Additional progress made through May 2007 has placed Umatilla about
1,184 days ahead of 2005 program office schedule estimate. We found
similar occurrences at the other incineration sites, and actual and
near-term projected performance by sites reveals that significant
schedule progress has been made since the 2005 schedule-estimating
process. Table 5 shows actual and projected schedule savings as
compared to the 2005 program office estimated schedule for campaigns
recently completed or under way.
Table 5: Number of Days Saved per Site since 2005 Program Office
Schedule Estimates:
Site: Anniston;
Cumulative days saved: 443;
Actual or projected days saved: Actual;
End date of last completed campaign completed to date: March 8, 2007;
Campaigns ongoing and completed or started and completed from May 2005
through July 2007: GB 8-inch projectiles;
GB 155-mm projectiles;
GB M360 105-mm projectiles;
VX M55 rockets/warheads.
Site: Newport;
Cumulative days saved: 658;
Actual or projected days saved: Projected;
End date of last completed campaign completed to date: December 2,
2009[A];
Campaigns ongoing and completed or started and completed from May 2005
through July 2007: VX agent neutralization.
Site: Pine Bluff;
Cumulative days saved: 375;
Actual or projected days saved: Actual;
End date of last completed campaign completed to date: May 19, 2007;
Campaigns ongoing and completed or started and completed from May 2005
through July 2007: GB M55 rockets/ warheads.
Site: Tooele;
Cumulative days saved: 61;
Actual or projected days saved: Actual;
End date of last completed campaign completed to date: August 17, 2006;
Campaigns ongoing and completed or started and completed from May 2005
through July 2007: VX land mines;
Agent changeover.
Site: Umatilla;
Cumulative days saved: 1,184;
Actual or projected days saved: Actual;
End date of last completed campaign completed to date: January 24,
2007;
Campaigns ongoing and completed or started and completed from May 2005
through July 2007: GB M55 rockets/ warheads;
GB MC-1 bombs;
GB MK-166 bombs;
GB 8-inch projectiles;
GB 155-mm projectiles.
Site: Aberdeen;
Cumulative days saved: 750;
Actual or projected days saved: Actual;
End date of last completed campaign completed to date: February 13,
2006;
Campaigns ongoing and completed or started and completed from May 2005
through July 2007: Ton container cleanout.
Total;
Cumulative days saved: 3,471;
Actual or projected days saved: [Empty];
End date of last completed campaign completed to date: [Empty];
Campaigns ongoing and completed or started and completed from May 2005
through July 2007: [Empty].
Source: GAO analysis of Army data.
[A] Figures listed above for Anniston, Pine Bluff, Tooele, and Umatilla
represent actual savings. Figures for Newport are projected savings for
current ongoing campaigns based on the historical processing rates at
those sites. The Newport forecast for completing the current campaign
is December 2009.
[End of table]
We calculate that to date, almost $1.3 billion in cost savings has
accrued based on schedule savings since the 2005 program office
estimates were established. Table 6 shows actual and projected schedule
savings as compared to the 2005 estimated schedule savings for
campaigns recently completed or under way.
Table 6: Potential Operational Cost Savings Associated with Days Saved
since 2005 Schedule Estimate:
Dollars in millions.
Anniston;
Cumulative days saved: 443;
Potential cost savings: $178.
Newport;
Cumulative days saved: 658;
Potential cost savings: 227[A].
Pine Bluff;
Cumulative days saved: 375;
Potential cost savings: 144.
Tooele;
Cumulative days saved: 61;
Potential cost savings: 25.
Umatilla;
Cumulative days saved: 1,184;
Potential cost savings: 452.
Aberdeen;
Cumulative days saved: 750;
Potential cost savings: 236.
Total;
Cumulative days saved: 3,471;
Potential cost savings: $1,262.
Source: GAO analysis of Army data.
Note: Time savings were converted to a cost by using the average fiscal
year 2005 and fiscal year 2006 budgets expressed in dollars per day.
[A] Figures listed above for Anniston, Pine Bluff, Tooele, and Umatilla
represent actual savings. Figures for Newport are projected savings for
current ongoing campaigns based on the historical processing rates at
those sites. The Newport forecast for completing the current campaign
is December 2009.
[End of table]
The April 2007 SAR for the CMA sites (including Newport) reported new
program office schedule estimates, which were a significant improvement
from the Army's 2005 program office schedule estimates that generated
the program's rebaselining and also were significantly ahead of the
independent DOD schedule estimates that are used as the program's
unofficial schedule baseline and were developed by CAIG. Table 7 shows
the number of months that the estimated schedule for completing
operations has improved from 2005 to 2007.
Table 7: Number of Months of Improvement in 2007 Program Current
Estimate for Operations Completion versus 2005 Army Program Office
Estimate and 2005 DOD Independent Estimate:
Site: Anniston;
2005 Army program office estimate: September 2016;
2005 DOD independent estimate: June 2016;
2007 program current estimate: May 2015;
Improvement from 2005 program office estimate: 16 months;
Improvement from 2005 independent DOD estimate: 13 months.
Site: Newport;
2005 Army program office estimate: October 2011;
2005 DOD independent estimate: February 2012;
2007 program current estimate: October 2009;
Improvement from 2005 program office estimate: 24 months;
Improvement from 2005 independent DOD estimate: 28 months.
Site: Pine Bluff;
2005 Army program office estimate: January 2015;
2005 DOD independent estimate: November 2015;
2007 program current estimate: June 2013;
Improvement from 2005 program office estimate: 19 months;
Improvement from 2005 independent DOD estimate: 29 months.
Site: Tooele;
2005 Army program office estimate: October 2014;
2005 DOD independent estimate: September 2015;
2007 program current estimate: November 2014;
Improvement from 2005 program office estimate: (1 month);
Improvement from 2005 independent DOD estimate: 10 months.
Site: Umatilla;
2005 Army program office estimate: June 2017;
2005 DOD independent estimate: March 2017;
2007 program current estimate: December 2014;
Improvement from 2005 program office estimate: 30 months;
Improvement from 2005 independent DOD estimate: 27 months.
Source: GAO analysis of DOD and CMA data.
[End of table]
In addition to its annual schedule update process, CMA tracks on a
monthly basis the confidence levels of achieving various program
schedule estimates. CMA tracks these levels after updating its model to
incorporate updated processing rates and any changes to estimates for
downtime or risk. These updates have shown upward trends in the
confidence level, which is the percentage of schedule outcomes that
meet or exceed a given date. As previously discussed, the 2005 program
office estimate reflected a 50 percent confidence level. By the end of
fiscal year 2006, which was about 18 months after the 2005 program
office estimates were developed, only Tooele, at 48 percent, was
slightly below the 50 percent confidence level. The other three sites'
confidence levels for achieving the 2005 program office estimate for
completing operations had increased significantly, ranging from 71
percent at Anniston to 83 percent at Umatilla. Moreover, confidence
levels based on monthly updates conducted in April 2007 showed that the
program office estimates for meeting the DOD schedule milestones were
as high as 98 percent for completing operations, but confidence levels
were very low for meeting the April 29, 2012, treaty deadline, as shown
in table 8.
Table 8: CMA Assessed Confidence in Achieving DOD Schedule Milestone
and April 29, 2012, Treaty Deadline:
Percentages: Site: Anniston;
Program's assessed confidence in achieving operations schedule
milestone: 73;
Program's assessed confidence in sites completing destruction by April
29, 2012, CWC milestone: 5.
Percentages: Site: Newport;
Program's assessed confidence in achieving operations schedule
milestone: 98;
Program's assessed confidence in sites completing destruction by April
29, 2012, CWC milestone: 98.
Percentages: Site: Pine Bluff;
Program's assessed confidence in achieving operations schedule
milestone: 81;
Program's assessed confidence in sites completing destruction by April
29, 2012, CWC milestone: 19.
Percentages: Site: Tooele;
Program's assessed confidence in achieving operations schedule
milestone: 42;
Program's assessed confidence in sites completing destruction by April
29, 2012, CWC milestone: 1.
Percentages: Site: Umatilla;
Program's assessed confidence in achieving operations schedule
milestone: 84;
Program's assessed confidence in sites completing destruction by April
29, 2012, CWC milestone: 2.
Source: CMA.
[End of table]
While the program office has updated its schedule estimates to reflect
the latest processing rates and changes to other assumptions, resulting
in schedule improvements in excess of 18 months in most cases,
independent schedule estimates have not been developed since the CAIG
2005 estimates because these estimates are usually performed in
conjunction with program rebaselinings because of schedule or cost
growth. Also, when schedules or other assumptions change, cost and
schedule estimates should be revised to reflect current status. Because
these steps have not been taken, DOD and Congress can have little
confidence that the program is being held accountable to a reasonable
baseline to measure program progress and that program costs are
reasonable.
Shortcomings in Underlying Cost Data Make Earned Value Management
Results Unreliable and Limited Our Ability to Determine Cost Estimate
Reliability:
Because of shortcomings in the underlying data, the usefulness of the
EVM data to make cost projections is limited, which makes results
unreliable, and our ability to independently verify and validate the
reliability of the cost estimates was restricted. The Chemical
Demilitarization Program's projected cost growth is largely
attributable to longer schedules and increased costs associated with
facility closure estimates. Shortcomings in the underlying data include
invalid and unstable performance baselines. The performance baselines
are problematic because all requirements, such as closure costs, are
not fully identified and included in program cost estimates and because
large amounts of additional costs are added to the performance baseline
annually. Moreover, there were significant differences between the
systems contractor cost estimates through completion and the government
cost estimates through completion because the government anticipates
longer operational periods and greater closure requirements.
Projected Cost Growth Is Largely Attributable to Longer Schedules and
Increased Facility Closure Estimates:
The Chemical Demilitarization Program's continuing growth in cost
estimates is largely attributable to longer schedules and the increased
costs associated with plant closure. Longer schedules led to increases
in cost estimates for sites currently in the operations phase because
of additional labor costs, which are a high percentage of site costs.
In addition, according to CMA officials, labor costs are very expensive
for this program because of the need for highly specialized labor that
cannot be easily replaced. According to these officials, each staff
member, for example, must pass the Personal Reliability Program, which
provides specialized training and security clearances for working at
the chemical deconstruction plants. Program officials have stated that
for this reason, it is often more cost-effective during plant shutdowns
or extended delays to retain this specialized workforce than to lay off
and rehire new staff. This workforce costs approximately $9.5 million
per month based on an average monthly spend rate across the five sites
we examined, which means that a 1-month schedule extension could
increase the life cycle cost estimates by about that amount. Table 9
shows the increase in program cost as the schedule was extended.
Table 9: Changes in Estimated Total Program Cost:
Then-year dollars in billions.
Total life cycle cost estimate;
2003 APB: $19.6;
2005 Army cost position: $29.2;
Increase: $9.6;
April 2007 SAR: $28.6.
Source: U.S. Army.
[End of table]
While the Chemical Demilitarization Program Is Using Earned Value
Management as a Tool, Some Underlying Data Make Results Unreliable:
EVM is an important program management tool required by DOD and OMB
that measures performance by comparing the value of work accomplished
with the value of work planned and the cost of the work performed.
Using EVM data, early warning of schedule delays and cost overruns can
be determined. EVM uses contractor-reported data to examine variances
reported in contractor performance reports between actual cost and time
of performing work tasks and the budgeted or estimated cost and time.
As a result, DOD and OMB policy and guidance require the use of EVM to
measure program performance.[Footnote 36]
The Chemical Demilitarization Program uses EVM[Footnote 37] as a tool
that allows both government and systems contractor managers to have
visibility over cost performance on their contracts. However, the
ability of the program's EVM system to reliably manage and forecast the
program's cost estimates may be affected by large variances between the
systems contractors' estimated cost to complete operations and closure
versus the government's estimate; systems contractors' baselines that
have not been carefully controlled, leading to high percentages of
authorized work that is unpriced for long periods of time; and limited
identification of requirements for closure costs at individual sites.
Because of the shortcomings in the underlying cost data, the usefulness
of the EVM data to make cost projections is limited and our ability to
independently verify and validate the reliability of the cost estimates
was restricted. In other words, the mechanics of the EVM system are
working properly; however, we found that the performance management
baseline does not accurately reflect the remaining life cycle costs of
the Chemical Demilitarization Program. DOD's EVM implementation guide
states that "the "to-go" plan should reflect a realistic schedule of
how the remaining work actually is to be done and the new budget should
be adequate and reflect a realistic estimate, remaining program risks,
and contain an appropriate amount of management reserve."[Footnote 38]
According to best practices, an EVM system should be devoid of data
errors and anomalies that may distort the analysis.[Footnote 39] In
addition, any cost or schedule variances should be explained and
corrective actions identified. Furthermore, the EVM system should have
a valid and stable performance baseline from which to make projections
and do trend analysis.[Footnote 40] In the case of the Chemical
Demilitarization Program, the cost account managers are inserting their
costs into the system properly, system certification verifies that the
EVM reported costs match the costs in the accounting system, the EVM
reports are being prepared with minimal errors, and cost account
managers are explaining any variances. However, the actual data are
measured against a baseline that does not reflect all of the work to be
performed, such as facility closure activities, which distorts cost and
schedule performance metrics. This undermines the effectiveness of the
EVM system in providing management with accurate and realistic metrics
for making informed management decisions.[Footnote 41]
Performance Baselines Lacked Validity and Have Been Unstable:
Our review showed that the performance baselines lack validity and have
not been stable because CMA and site systems contractors have not fully
identified all requirements, such as closure costs, and included them
in program cost estimates and because large amounts of additional
costs[Footnote 42] are added to the performance baseline annually. EVM
guidelines state that once a contract is awarded, the contractor should
establish a performance baseline that will serve as a time-phased
budget plan for accomplishing work and measuring contract
performance.[Footnote 43] This is essentially the spending plan for the
program and should closely equate to the planned budget, which is the
value of work planned through the project's completion. The performance
baseline should include all costs associated with completing the
program, including direct and indirect labor, material, and other
direct costs associated with the authorized work. However, we found
that each of the sites was lacking a stable and reliable EVM
performance baseline, which hindered our ability to perform an
independent EVM analysis.[Footnote 44]
While the planned budget and the performance baseline should closely
equate, we found that the performance baseline is significantly less
than the planned budget (close to $2 billion less) because most sites
have significant amounts of work that are authorized by the systems
contractors and the Army but are not yet placed on contract. For
example, the April 2007 SAR reports that while the contract value at
Umatilla is $1.2 billion through December 2006, an additional $700
million has been authorized but is pending negotiation. Also, the
Estimate at Completion (EAC) cost estimate, which represents all
estimated costs through the end of the contracts, is $4 billion more
than the performance baseline. Figure 6 shows the extent that planned
budget and the EAC exceed the performance baseline for all sites
combined over a 19-month period from August 2005 to February 2007.
Figure 6: Comparison of the Performance Baseline with the Planned
Budget and Systems Contractors' Estimate at Completion:
This figure is a combination line graph comparing the performance
baseline with the planned budget and systems contractors' estimate at
completion. The X axis is the year and month, and the Y axis represents
dollars in millions. One line represents planned budget, another
represents estimate at completion, and another represents performance
baseline.
[See PDF for image]
Source: GAO analysis of Army data.
[End of figure]
The performance baseline does not include all requirements, such as
closure costs, which are not fully identified and included in program
cost estimates. Best practices state that a cost estimate should be
comprehensive in order to be considered credible.[Footnote 45] Systems
contractor officials stated that in many instances what was included in
their estimated completion costs were placeholder amounts, which they
believed to be unrealistic. According to program office officials, the
government and the systems contractors have not yet completely defined
the requirements for closure, so a realistic closure cost estimate is
not currently available. However, since two sites--Johnston Atoll and
Aberdeen--have already been closed historical data are now available to
develop more accurate closure estimates at the remaining
sites.[Footnote 46] For example, based on the Army's experience of
destroying blister and nerve agents at Johnston Atoll, closure costs
for other destruction sites are increasing from initial estimates. As a
result of incorporating these actual data, assumptions for conducting
closure activities have increased from 12 months to about 30 months at
some sites.
The performance baseline also is unstable because large amounts of
additional costs are added to the performance baseline
annually.[Footnote 47] Each fiscal year, a portion of the unpriced work
that is part of the planned budget is negotiated (priced), and it is
moved into the performance baseline. The negotiated funding is often
greater than what was used as a "placeholder" budget, increasing the
contract baseline to more than what was included as unpriced work. For
example, the fiscal year 2006 negotiated increase in the contract value
at Anniston was $6 million more than the reduction in authorized
unpriced work. Program office officials explained that the performance
baseline at each site is constantly changing because of yearly contract
renegotiations[Footnote 48] for future operations and closure costs.
Program officials attribute the yearly changes to the contracts using
Operating and Maintenance funds, which expire within 1 fiscal year.
This situation created the need to put on contract a year's worth of
work in conjunction with each fiscal year's appropriation.
CMA officials stated that they are making efforts to stabilize the
baseline by placing the full life cycle costs associated with all
activities at each site. In its technical comments on a draft of this
report, DOD stated that three of the five CMA sites (Anniston, Pine
Bluff, and Umatilla) had completed contract performance baselines and
negotiations were ongoing at the two remaining sites (Newport and
Tooele). DOD expected that the negotiations for these two sites would
be completed by June 30, 2008. Once this happens, CMA officials believe
that there will be less of a need for annual negotiations and more
realistic performance baselines should result. Until program management
has a performance baseline that reflects the current plan to complete
remaining work and follows a disciplined process for incorporating
changes, management will not be able to benefit from the EVM system's
ability to reveal actual program performance and detect program risks.
Program Manager-Estimated Completion Costs Are Significantly Higher
Than Completion Costs Generated from EVM Data, Suggesting Data Are
Incomplete:
It is a best practice to develop a range of independent EACs using
commonly accepted EVM equations because doing so informs management of
the likely range of costs for completing the program so appropriate
action can be taken.[Footnote 49] In addition, DOD guidance states that
remaining work should reflect a realistic schedule and budget that also
reflect remaining risks.[Footnote 50] Finally, EVM guidelines state
that program managers should, at least on a monthly basis, rely on EVM
data to generate EACs that are derived based on the cost of work
completed to date along with an estimate of what it will cost to
complete the rest of the program.[Footnote 51] By relying on past
performance and using well-defined EVM indices, a variety of EACs can
be generated. However the EACs generated are only as good as the
underlying data in the EVM system.
Based on our analysis, program cost estimates often exceeded the worst
case of our range of independently developed completion costs estimates
because of shortcomings in the underlying data of the EVM systems
maintained at the chemical demilitarization sites. It is a best
practice to develop a range of independent cost estimates using
commonly accepted EVM equations. Doing so informs management of the
likely range of costs for completing the program so that appropriate
action can be taken. We found that when we developed independent EACs
using the systems contractor's EVM data and commonly accepted EVM
forecasting equations in all cases the systems contractors' EACs are
either at the upper end of or higher than our estimated EAC range based
on our analysis shown in table 10. In addition, the program manager's
EAC was far outside of our projected range and, in many cases, was
almost twice as high as our EAC. For example, our estimated cost range
for Anniston based on the EVM data was from $1.6 billion to $1.7
billion. The systems contractor's most likely cost estimate was $100
million above the upper limit of our range, and the program manager's
estimate was $1.4 billion higher than the upper range limit. We
attribute this difference to an understated performance baseline, which
is critical to the estimating process. Table 10 compares our
independent EAC to that of the systems contractor's and program
manager's EACs.
Table 10: Comparison of GAO's Independent Estimates to Complete
Operations and Closure Activities with the Systems Contractor's and
Program Manager's Estimates:
Then-year dollars in billions.
Anniston;
Original contract cost: $0.6;
Current planned budget: $1.7;
Systems contractor's most likely EAC: $1.8;
GAO's independent EAC range: $1.6-$1.7;
2006 Selected Acquisition Report program manager's EAC: $3.1.
Newport;
Original contract cost: 0.3;
Current planned budget: 1.0;
Systems contractor's most likely EAC: 1.1;
GAO's independent EAC range: 0.9-1.1;
2006 Selected Acquisition Report program manager's EAC: 1.8.
Pine Bluff;
Original contract cost: 0.5;
Current planned budget: 1.5;
Systems contractor's most likely EAC: 1.9;
GAO's independent EAC range: 1.4-1.5;
2006 Selected Acquisition Report program manager's EAC: 2.4.
Tooele;
Original contract cost: 0.2;
Current planned budget: 1.6;
Systems contractor's most likely EAC: 2.4;
GAO's independent EAC range: 1.4-1.6;
2006 Selected Acquisition Report program manager's EAC: 3.3.
Umatilla;
Original contract cost: 0.5;
Current planned budget: 1.8;
Systems contractor's most likely EAC: 1.9;
GAO's independent EAC range: 1.6-1.8;
2006 Selected Acquisition Report program manager's EAC: 3.8.
Source: GAO analysis of Army data.
[End of table]
In addition, the large difference between the program manager's EAC and
our independent EACs indicates that there is more work that is being
considered by the program manager. More work, in turn, will affect
future costs and requirements than is predicted by the EVM data and,
therefore, calls into question the usefulness of the EVM data to
effectively manage the program.[Footnote 52] The ability to respond
quickly to program problems depends on having early visibility into
what is causing them. Thus, if the EVM performance baseline is missing
some of the remaining work, then accurate progress assessments will not
be available to provide management with a better picture of program
status. This lack of insight may hinder decision making and program
success because problems may go undetected.
Without accurate and realistic closure requirements used to develop
completion cost estimates, DOD and Congress will not be able to
accurately identify and adequately budget for all activities through
facility closure. Similarly, without an independent cost review once
the cost estimates are complete and schedule milestones have been
updated and reviewed, DOD and Congress will not have assurance that
cost estimates for the closure of chemical demilitarization sites are
realistic.
The Army Has Actively Managed Award Fees and Could Further Emphasize
Schedule and Cost and Link Award Fees to More Objective Performance
Measures:
Our review of the Army's use of monetary fees to incentivize certain
aspects of the systems contractors' performance at the five sites in or
near beginning operations in 2003 showed that the contract monetary fee
structures at the incineration sites differed from those at Newport's
neutralization site. Award fees are predominant at the incineration
sites while incentive fees are predominant at Newport. We found that
while the Army has actively reviewed and adjusted the types of monetary
fees used and the criteria and weighting for determining monetary fees,
there be may be opportunities to further emphasize schedule and cost.
For example, during the period we reviewed, award fees have emphasized
safety and environmental compliance at each site, but have not resulted
in the control of schedule and cost growth. In fiscal year 2006, after
several external organizations recommended that it do so, the Army
reinstituted the use of a distinct award fee performance measure for
schedule.[Footnote 53] However, site project managers often weighted
schedule at the lower end of the specified range. Also, while DOD has
published guidance that award fees should be tied to identifiable
interim outcomes, discrete events, or milestones as much as possible,
we found that several performance measures, such as management and
cost, had evaluation factors that were all or mostly all subjective and
were not clearly linked to a measurable outcome. In addition to award
fees, in 2005, the Army established the Director's Programmatic
Performance Based Incentive (DPPBI), a performance-based incentive fee
to encourage the incineration site systems contractors to establish a
collaborative relationship with the goal to share lessons learned in
areas that would result in schedule reduction across all sites. Before
fiscal year 2007, this fee was based on subjective schedule performance
measures. By fiscal year 2007, DPPBI had evolved to focus mostly on the
sites' collective schedule performance linked to the CWC treaty
deadline. It is unclear how effective this incentive fee will be in
motivating individual systems contractors' schedule performance due to
the collective manner in which the fee is earned, its relative size,
and recent focus on objective schedule performance. An additional
incentive fee, authorized by section 923 of the National Defense
Authorization Act for Fiscal Year 2007, provides up to $165 million at
each incineration site for the completion of destruction and facility
closure by or close to the CWC deadline. This incentive fee, as
planned, is structured to accelerate destruction operations and
facility closure at the incineration sites. Because this fee has not
yet been implemented, we have no basis at this time to determine how
effective this incentive might be at controlling schedule.
Contract Monetary Fee Structures at the Incineration Sites Differ from
Those at Newport's Neutralization Site, and Award Fees Are Predominant
at the Incineration Sites:
CMA uses cost-plus-award-fee contracts with incentive fees to encourage
and reward the systems contractors to perform in an innovative,
efficient, and effective way in areas deemed important to the program's
success. Under these contracts, the systems contractors' allowable,
incurred costs are reimbursable to the extent prescribed in the
contract. According to CMA officials, the Chemical Demilitarization
Program systems contracts are cost reimbursable, based on the fact that
it is not possible to accurately describe the requirements in
sufficient detail to address the myriad risks and uncertainties
associated with destroying aging and deteriorating munitions that are
filled with chemical agent. As a result of this high-risk environment,
the government accepts most of the cost risk. Thus, when problems
arise, such as rocket fires or mercury contamination, government and
systems contractor resources are focused on fixing the problems or
mitigating the risk rather than fixing blame.
The base fee for this program is a fixed amount of money negotiated at
the beginning of each year of contract operations, and is usually paid
on a monthly basis to the systems contractor for its performance,
though this fee is not tied to specific performance criteria as are the
award and incentive fees. According to program officials, the rationale
for providing a base fee is due to the program's complexity and to
provide the systems contractors with a level cash flow for financial
viability. Award fees are intended to motivate systems contractor
performance in areas that are susceptible to judgmental and qualitative
measurement and evaluation and, as a result, award fee criteria and
evaluations tend to be subjective. Typically, award fee contracts
emphasize multiple aspects of contractor performance in a wide variety
of areas, such as quality, timeliness, technical ingenuity, and cost-
effective management.[Footnote 54] Finally, incentive fee contracts use
what is considered an objective evaluation of the systems contractor's
performance to adjust the fee paid, which usually involves applying a
fee determination formula specified in the contract.
The contracts at each of the four operational incineration sites--
Anniston, Pine Bluff, Tooele, and Umatilla--include (1) a base fee
(representing 3 percent of the negotiated total cost of the contract),
which is a fixed amount not subject to performance evaluation; (2) an
award fee that is 7 percent of the negotiated contract costs, which
will be determined through performance evaluations and paid from an
award fee pool; and (3) DPPBI, a performance-based incentive fee
established in 2005 and paid from an available fee pool amount that is
2 percent of negotiated contract cost. The contract at the remaining
neutralization site--Newport--has a base fee (3 percent), an award fee
(3 percent), and a performance-based incentive fee (6
percent).[Footnote 55] All five contracts had performance-based
incentive fees added after initial contract award.
The types and percentages of fees offered to the systems contractors to
safely destroy the chemical weapons stockpile differ and emphasize
different performance measures. For example, at the incineration sites,
the award fee is the predominant fee and, from fiscal year 2003 to
fiscal year 2007, this fee has largely emphasized safety and surety,
environmental compliance, and management goals. Conversely, at the
neutralization site, the performance-based incentive fee is the
predominant fee, which has emphasized schedule goals. Figure 7 compares
the monetary incentive fees earned by the systems contractors at the
incineration sites and at Newport. As shown in figure 7, in fiscal year
2006, CMA paid the systems contractors at the incineration sites a
combined total of about $39.9 million in fees, which consisted of about
$31.1 million (or 78 percent) in award fees and about $8.8 million (or
22 percent) in performance-based incentive fees. In contrast, CMA paid
the neutralization site system contractor at Newport about $14.7
million in monetary fees, which consisted of about $2.5 million (or 17
percent) in award fees and upwards of about $12.1 million (or 83
percent) in performance-based incentive fees. In fiscal year 2006, the
award fees earned at each of the incineration sites were almost three
times higher than that earned at Newport.
Figure 7: Comparison of the Monetary Incentive Fees Earned by Systems
Contractors at the Incineration Sites and at Newport during Fiscal Year
2006:
This figure is a combination of two pie charts comparing the monetary
incentive fees earned by systems contractors at the incineration sites
and at Newport during fiscal year 2006.
Incineration sites' monetary fees for fiscal year 2006L $33.9 million:
Award fee: 78%;
Incentive fee: 22%.
Newport monetary fees for fiscal year 2006:
Award fee: 83%;
Incentive fee: 17%.
[See PDF for image]
Source: U.S. Army Sustainment Command.
Note: GAO did not independently verify the award and incentive fee data
provided by the U.S. Army Sustainment Command.
[End of figure]
By predominately using performance-based incentives at Newport, the
Army has aligned more of its monetary fees with objective outcomes.
However, Army officials said that it is difficult to determine how
effective the usage of performance-based incentive fees has been in
motivating the systems contractors to meet schedule and cost milestones
because of the number of extensions made to the original incentive
agreement. The extensions were made to account for the technical
challenges encountered by the systems contractors during the
neutralization process. After the terrorist attacks of September 11,
2001, due to Army concerns about security vulnerabilities, CMA
restructured the contracts to increase the emphasis on performance-
based incentives at the Newport and Aberdeen neutralization sites to
incentivize the systems contractors to destroy the remaining stockpile
of lethal chemical agent on an accelerated schedule.[Footnote 56]
Army Actively Managed Award Fees and May Be Able to Further Emphasize
Schedule and Cost:
Since 2003, the period of our analysis, the Army has actively reviewed
and adjusted criteria for determining monetary fees--largely award
fees--it provided to the systems contractors to incentivize aspects of
their performance. CMA's award fee plans generally use performance
measures that could be expected to incentivize the desired acquisition
outcomes. CMA includes broad aspects of systems contractor performance,
such as management performance, and CMA also views these as keys to a
successful program. From 2003 through 2006, CMA used safety and surety,
environmental compliance, cost, and management as the key performance
measures. Until fiscal year 2006, however, CMA did not emphasize
schedule as a distinct performance measure in the award fee
plans.[Footnote 57]
Army Emphasized Safety and Environmental Compliance in Award Fee Plans
and Did Not Successfully Incentivize Schedule and Cost Performance:
From fiscal year 2003 through fiscal year 2006, the incineration sites
and Newport had not been able to maintain original production schedule
milestones and had experienced significant cost growth. Although the
management performance measure included some criteria for schedule,
during the period from fiscal year 2003 through fiscal year 2005,
schedule was not a distinct performance measure in the award fee plan,
and safety and environmental compliance were heavily weighted. CMA
officials acknowledged in the 2007 incentive agreement plan that while
award fees have succeeded in motivating the systems contractors to
maintain high levels of safety and environmental compliance at each
site,[Footnote 58] they have not satisfactorily resulted in the control
of schedule and cost growth. Many events and constraints contributed to
the schedule and cost growth, both within and beyond the systems
contractors' control. CMA officials stated that the process to
determine the degree to which such issues cause variances to baselines
is never clear, but generally, over the course of a year, there may be
25 percent of the problems that are clearly outside of the control or
responsibility of the systems contractor, 10 percent that are clearly
within the control of the systems contractor, and 65 percent that fall
into the "gray area" where it is difficult to determine whether the
problem is clearly outside or within the control of the systems
contractor.
From fiscal year 2003 through fiscal year 2006, the Army paid award
fees to the systems contractors between 70 percent and 96 percent of
the available award fee pools primarily for good-to-outstanding
performance in safety and surety, environmental compliance, and
management, which collectively constituted most of the basis for
determining the award fee. Table 11 shows both the actual amount and
the percentage of the total available award fee pool each site was paid
by the Army over the period from fiscal year 2003--when all five sites
were either in or preparing for the operations phase--to fiscal year
2006. Based on our analysis of this period, the Army paid the systems
contractors at three of the four incineration sites award fees between
86 percent and 96 percent of the total award fee pool available during
this period. The systems contractor at Tooele was the exception,
earning only 70 percent of the total available award fee pool,
according to the U.S. Army Sustainment Command, because of
unsatisfactory ratings due to the lack of real progress in planning and
preparing for the mustard campaign and in not meeting government
expectations for managing costs.
Table 11: Percentage of Award Fees Earned at the Incineration Sites and
Newport, from Fiscal Year 2003 to Fiscal Year 2006:
Chemical demilitarization facility: Anniston;
Amount of award fee pool (2003-2006): $24,536,587;
Amount of award fee earned (2003-2006): $23,671,150;
Percentage of award fee earned (2003-2006): 96.
Chemical demilitarization facility: Pine Bluff;
Amount of award fee pool (2003-2006): 22,564,501;
Amount of award fee earned (2003-2006): 19,633,254;
Percentage of award fee earned (2003-2006): 87.
Chemical demilitarization facility: Umatilla;
Amount of award fee pool (2003-2006): 26,235,256;
Amount of award fee earned (2003-2006): 22,638,970;
Percentage of award fee earned (2003-2006): 86.
Chemical demilitarization facility: Newport;
Amount of award fee pool (2003-2006): 10,532,717;
Amount of award fee earned (2003-2006): 8,734,713;
Percentage of award fee earned (2003-2006): 83.
Chemical demilitarization facility: Tooele;
Amount of award fee pool (2003-2006): 27,917,118;
Amount of award fee earned (2003-2006): 19,414,301;
Percentage of award fee earned (2003-2006): 70.
Chemical demilitarization facility: Total;
Amount of award fee pool (2003-2006): $111,786,179;
Amount of award fee earned (2003-2006): $94,092,388;
Percentage of award fee earned (2003-2006): [Empty].
Source: U.S. Army Sustainment Command.
[End of table]
Anniston, for example, earned $23.7 million (or 96 percent) of the
award fee during this period. According to an award fee evaluation
board report, while the Anniston systems contractor worked aggressively
to implement the schedule without compromising worker or public safety,
the systems contractor faced challenges related to disciplined
operations.[Footnote 59] The report noted that certain destruction
campaigns at Anniston were characterized by schedule delays that
resulted primarily from equipment reliability problems and, at times,
inefficient planning and execution of repair efforts. The report
further noted that while the efficiency of repairs was at times
problematic, the systems contractor could not have prevented the
equipment failure. The report also credited the systems contractor's
risk mitigation efforts with significantly reducing the risk of a 6-
month potential site shutdown that would have carried with it a
corresponding cost impact of $75 million. Thus, CMA was able to justify
paying the systems contractor nearly the full amount of available award
fee. Also, during one period, Umatilla's systems contractor earned $2.5
million (or 80 percent) of the available award fee pool while it had
destroyed 36 percent of the nerve agent rockets planned for that
period. According to an award fee evaluation board report, the
processing shortfall was attributable to the systems contractor's lack
of implementation of planned and emergent work in an efficient and
effective manner, and was estimated to increase the planned operations
schedule by 77 days.[Footnote 60] We have no basis for saying that the
systems contractors should not have received their award fees, since
they generally were evaluated as having met the award fee criteria,
which at the time did not have schedule as a distinct performance
measure.
CMA officials have acknowledged that the existing award fee process has
not satisfactorily resulted in the control of schedule and cost growth.
CMA developed the schedule performance measure in response to Army
Inspector General recommendations, and DOD guidance suggested that CMA
focus more attention on schedule criteria. The Army's reinstatement of
a distinct schedule performance measure coincided with the program's
schedule extensions.[Footnote 61] Also, in fiscal year 2004, according
to CMA, the Army Inspector General recommended that CMA develop
quantifiable metrics and both the Army Inspector General and the
Defense Contract Management Agency recommended that CMA place greater
weight on cost and schedule control. CMA documents indicate that the
schedule performance measure was developed after these recommendations
and the Under Secretary's memorandum.
Army Has Adjusted Award Fee Plans' Weighted Performance Measures and
Could Further Increase the Weight of Performance Measures, Such as
Schedule and Cost:
The determination of award fee performance measures and related weight
factors is part of CMA's award fee plan template development process,
which is used as a guideline by site project managers to develop
specific site award fee plans. The template describes specific
criteria, standards, and procedures, and provides a range of weight for
each performance measure to be used in the award fee plan to assess the
systems contractors' performance and to determine the amount of award
fee earned. The template allows the site project managers flexibility
to select a weighted factor for each performance measure within the CMA
template's range based upon the site project manager's discretion. For
example, the award fee plan's fiscal year 2007 template had a range of
10 percent to 20 percent for weighting management performance, 20
percent to 30 percent for safety and surety, 20 percent to 25 percent
for environmental compliance, 10 percent to 15 percent for cost
performance, and 10 percent to 20 percent for schedule performance. The
Army adjusted these ranges in fiscal year 2006, adding the schedule
performance measure, increasing the template's range for management,
and reducing the template's ranges for environmental compliance and
safety and surety performance measures.
Table 12 shows historical data on the award fee plans' performance
measures and the relative weights selected by each of the incineration
sites and Newport as well as the adjustments made from fiscal year 2003
to fiscal year 2007.
Table 12: Performance Measures and Weights in Award Fee Plans, Fiscal
Years 2003-2007:
[See PDF for image]
Source: U.S. Army Sustainment Command.
Notes: Data on the award fee plans included fiscal year 2003 through
fiscal year 2007 to represent the period of time the sites were in or
preparing for the operations phase. In addition, we evaluated the
performance measures and related weights for those fiscal years since
the previous APB was established in April 2003. N/A indicates that no
weight was assigned to the particular award fee plan performance
measure at that site for that particular fiscal year.
[A] Site project managers can select weights outside of suggested
ranges with CMA approval. While the award fee plan's fiscal year 2007
template had a range of 20 percent to 30 percent for weighting safety
and surety, Tooele's weight for safety and surety for fiscal year 2007
was 35 percent.
[End of table]
As shown in table 12, government site project managers have weighted
safety and surety and environmental compliance from a weight factor
that ranged from 30 percent to 40 percent in fiscal year 2003 to a
weight factor from 20 percent to 35 percent in fiscal year 2007. Also,
the award fee plans for the incineration sites and Newport have safety
"killer clause" provisions, whereby a systems contractor could lose all
or part of its award fee if there is an agent release, an individual or
worker is exposed to chemical agent, or there is a release of agent
outside of the engineering controls.
In fiscal year 2006, CMA added the schedule performance measure,
increased the template's range for management, and reduced the
template's range for environmental compliance and safety and surety. We
found, based on our analysis, that the site project managers
consistently placed schedule and cost performance at lower relative
weights than other factors, including management performance. For
example, in fiscal years 2006 and 2007, the maximum weight allowable by
CMA for schedule performance was 20 percent, but the average weight
assigned by the site project managers in the award fee plans was only
13.75 percent for the incineration sites. Conversely, the site project
managers weighted other performance measures, such as management, on
average much closer to CMA's maximum suggested weights.
Program officials stated that the weights assigned to the different
performance measures are based on the relative significance of the
criteria to the program. Due to the lethal nature of the material that
must be processed, Congress has mandated that the program ensure
maximum protection to the workforce, the public, and the environment.
Accordingly, as program officials point out, safety/surety carry the
highest weighting followed closely by environmental compliance.
However, given the maturity of the operations phase, there may be
opportunities, from a management perspective, for the Army to further
adjust weights for performance measures in the award fee plan template
to further incentivize key acquisition outcomes, such as schedule and
cost, at the incineration sites. The five incineration sites have been
preparing for or were in operation since at least 2003 and, along with
Newport and Aberdeen, have eliminated over 45 percent of the chemical
munitions stockpile. In addition, the Army has stated that it has
reduced much of its safety risk by destroying the bulk of its most
lethal and volatile chemical munitions. For example, the Tooele site
has mostly ton containers with mustard agent left to destroy, which is
considered less risky than other agents. Moreover, Army officials have
pointed out that the systems contractors have implemented safety
improvement plans that have incorporated appropriate protocols and
standards into their daily operations. As a result, from a management
perspective, there may be opportunities to further emphasize schedule
and cost without sacrificing safety and environmental compliance.
It Is Possible to Make Greater Use of Objective Award Fee Criteria and
Evaluation Factors:
Based on our review of the fiscal year 2007 award fee plan's
performance measures and related evaluation criteria for the sites,
most of the award fee criteria and evaluation factors are not objective
and measurable. We determined that it is possible to make CMA's award
fee plans more effective in achieving certain acquisition outcomes
through greater use of objective award fee criteria. We previously
recommended that DOD instruct the military services to move toward more
outcome-based award fee criteria that are achievable and promote
accountability for and are directly linked to achieving desired program
outcomes.[Footnote 62] In March 2006, DOD issued award fee guidance
stating that it is imperative that award fees be tied to identifiable
interim outcomes, discrete events, or milestones as much as
possible.[Footnote 63] We acknowledge that award fee contracts are
expected to use subjective criteria to the extent that objective
criteria are not feasible. However, we identified additional
opportunities, from a management perspective, for CMA to make greater
use of objective award fee criteria and evaluation factors to
incentivize contractors to achieve desired acquisition outcomes, such
as safely destroying the chemical stockpile on schedule and within
cost.
The award fee plans for this program make use of both subjective and
objective criteria and our analysis of the fiscal year 2007 award fee
plan showed that in most cases, the criteria are subjective. Three
performance measures--safety, environmental compliance, and schedule--
use a mix of objective and subjective performance evaluation criteria,
although we noted these evaluations can also factor in subjective
assessments. The award fee plan states that in evaluating performance
in these areas, a number of subjective attributes will be considered,
but there are no specific weights associated with the attributes.
Rather, they are considered guides to evaluate the broad spectrum of
potential systems contractor performance. The schedule performance
measure was, for the most part, objective and linked to an outcome--the
destruction of chemical munitions. For example, the 2007 schedule
performance category used site-specific destruction tonnage goals
related to meeting CWC deadlines and the APB schedule. On the other
hand, three other performance measures--cost, management, and surety--
are evaluated using all subjective criteria. While we recognize that
using subjective criteria to assess systems contractor performance is
acceptable, we identified opportunities for the Army to use more
objective award fee criteria when evaluating contractor performance.
We developed some illustrative examples to show how certain elements of
CMA's subjective criteria taken from the 2007 award fee template could
be revised to objectively measure systems contractor performance. We
recognize that there are other examples that could be used to
illustrate this point and do not suggest that these examples
necessarily be used. To develop these examples, we used as criteria our
analysis and prior work.[Footnote 64] For example, the current award
fee plan contains the following subjective criterion for evaluating
management performance: "Demonstrated ability to practice proactive
management to identify and anticipate problems and implement effective
countermeasures prior to adverse impact." Illustrative examples of
potential objective criteria that could address these criteria are
"Develop and implement an integrated risk management plan by [a
specified date]" and "Ensure that [a specified percentage] of the risks
identified have a countermeasure available and implemented within [a
specified number of days] of the mustard container campaign."
Similarly, the current award fee plan contains the following subjective
criterion for evaluating cost performance: "Cost management systems
(accounting, scheduling, planning, and budgeting) are all efficiently
and effectively integrated, and produce consistent, reliable, and
accurate data." An illustrative example of a potential objective
criterion that could address at least a portion of these criteria is
"Ensure that [a specified percentage] of the cost and schedule
management systems data accurately depicts contract cost status within
the assessment period." Moreover, the current award fee plan contains
the following subjective criterion for evaluating surety performance:
"Appropriate treatment and disposal of chemical agents and chemical
agent standards." An illustrative example of a potential objective
criterion that could address at least a portion of these criteria is
"Ensure that [a specified percentage] of chemical agents are
appropriately treated and disposed of under the chemical agent
standards."
After the March 2006 guidance on award fees was issued by the Under
Secretary of Defense for Acquisition, Technology, and Logistics, CMA
issued a policy on award fees in September 2006. CMA's policy stated
that "due to the complexity of the program and the toxic nature of the
material being processed, essential elements of performance cannot
effectively or feasibly be measured by predetermined objective metrics.
Therefore, award fee criteria are established that allow assessment of
performance through the use of objective and/or subjective metrics."
Our review of this policy indicated that CMA did not clearly or fully
incorporate the concept of tying "award fees to identifiable interim
outcomes, discrete events or milestones as much as possible" as called
for in the March 2006 guidance. Although not legally required, we
believe that tying award fees as much as possible to more objective,
measurable evaluation criteria is a sound business and management
practice, and that CMA could do more to fully incorporate and implement
the March 2006 DOD policy guidance.
Recently Established and Planned Performance-Based Incentive Fees Are
Focused on Schedule, but Their Effectiveness Is Uncertain at This Time:
The Army has recently offered monetary fees intended to provide
additional incentives to improve schedule performance. In 2005, the
Army added to its fee structure the DPPBI, which is an annual
performance-based incentive fee that is structured to encourage the
systems contractors at the four incineration sites to work
collaboratively to improve various aspects of their performance. That
is, while each individual site is evaluated and scored separately, the
individual scores are averaged to determine a composite score that
results in all sites earning the same percentage of the available DPPBI
fee pool. Since the initial evaluation period in 2005, the fee amount
available for the DPPBI has been 2 percent of the estimated fee-bearing
contract costs--relatively small when compared to the award fee
available to the incineration sites. The 2 percent DPPBI is in addition
to the 10 percent available through base and award fees.
The evaluation factors for assessing systems contractor performance for
the DPPBI have changed over time and are divided between objective
criteria and a subjective assessment of systems contractor performance
during the fiscal year. Three criteria were established for the initial
DPPBI evaluation period of January 1 through September 30, 2005: risk
mitigation, production assessment, and lessons learned. In the initial
evaluation period, the contractors collectively were deemed to have
earned 51 percent of the 2005 DPPBI pool or $3,313,954 out of
$6,497,949. In fiscal year 2006, the DPPBI criteria were changed to put
more emphasis on performance outputs and less on processes and
procedures. The fiscal year 2006 DPPBI criteria were established to
encourage the contractors to collectively reduce project schedules
while maintaining a high standard of surety, safety, and environmental
compliance. The criteria also were established to increase CMA's
confidence levels that the sites could achieve CWC destruction
deadlines, although discrete goals were not specified. For the fiscal
year 2006 period of performance, the contractors earned about 93
percent or $8,771,200 out of the $9,411,159 in the DPPBI pool for that
period.
Prior to fiscal year 2007, the DPPBI was not linked to objective
schedule performance criteria linked to the April 29, 2012, CWC treaty
deadline. Beginning in fiscal year 2007, the DPPBI fee evaluation was
linked to achieving discrete schedule milestones associated with
achieving the CWC April 29, 2012, deadline. For fiscal year 2007, 70
percent of the total DPPBI fee available to be earned (or up to a
maximum of 1.4 percent of the total negotiated contract costs,
depending on average collective performance) was based on specified
schedule attainment targets for each site. The remaining 30 percent of
the available DPPBI fee for fiscal year 2007 is based on subjective
criteria related to the contractors' performance in the application of
lessons learned, risk management, programmatic procurement, and other
collaborative approaches to reduce costs and accelerate destruction of
the stockpile. This fee would be in addition to the portion of the
award fee targeted at schedule performance, now about 1 percent of
total negotiated contract costs at most incineration sites. This
additional emphasis on schedule is a positive development, but it
remains unclear to us how effective this incentive fee will be in
motivating individual systems contractors' schedule performance due to
the collective manner in which the fee is earned, its relative size,
and recent focus on objective schedule performance. It is particularly
unclear to us how individual contractors might be motivated during
periods when the underperformance of one or more systems contractors
could greatly reduce the potential to earn the fees by the others.
Finally, the Army plans to use a recently authorized incentive fee of
up to $165 million per site in an effort to further motivate the
systems contractors to accelerate destruction and closure activities.
Although it had not been placed on contract at the time of our review,
CMA drafted a plan to offer incentive fees under the authority
contained in section 923 of the National Defense Authorization Act for
Fiscal Year 2007.[Footnote 65],[Footnote 66] This plan would allow
systems contractors at the four incineration sites the opportunity to
earn fees in addition to the 12 percent available to be earned
annually. As currently planned, systems contractors will be able to
earn incentive payments for completion of destruction operations tied
to dates around the final CWC deadline and for the acceleration of
facility closure activities, but the effectiveness of this approach is
uncertain. Under CMA's plan, it would offer the additional incentive
fee of up to $150 million to each site--up to $100 million for
destruction operations and up to $50 million for facility closure
activities completed within specified target ranges. Bonuses totaling
$15 million would also be available to each site if all sites complete
destruction operations before the treaty deadline and accelerate
facility closure activities.
CMA's plan, which would have to be implemented under each of the
contracts, has a number of positive features. For example, the plan
recognizes, as does the statute, that the enterprise of striving to
capture these incentives is to be conducted consistent with the
existing obligation to provide for maximum protection for the
environment, the general public, and the personnel who are involved in
the destruction of the chemical agents and munitions. The potential
effectiveness of this particular planned incentive at motivating the
systems contractors to complete destruction operations around the
treaty deadline is also uncertain. This incentive arrangement has not
yet been placed on contract and thus it is too early to assess its
potential success. Additional details about the two recently
established incentive fees are in appendix II.
Conclusions:
Destroying the nation's remaining stockpile of chemical weapons in a
safe, efficient, and timely manner is essential to improve the
potential for the United States to meet CWC obligations and to reduce
the risk to the public and the environment of a potential catastrophic
event associated with lethal chemical stockpiles. We recognize that
maintaining high levels of safety and environmental compliance are
inextricably linked to success in meeting schedule and cost goals. We
also recognize that it is highly important that program management,
schedule and cost estimates, and monetary incentives be soundly
structured, well aligned with program goals and desired acquisition
outcomes, reasonable, and reliable. While DOD has taken specific
actions to strengthen program management, improving the focus of annual
performance plans beyond a single year and refining and finalizing
planned actions in the risk management area could greatly enhance the
usefulness of these management tools for DOD, CMA, the sites, and
Congress. In addition, without a fully developed and implemented risk
management approach, CMA cannot ensure that it is fully anticipating
and mitigating risks going forward. Without such improvements, it would
be difficult for program officials to accurately track progress,
effectively initiate timely actions to mitigate risks, and increase the
potential for program activities to further contribute to the overall
program goal of safely destroying the chemical weapons stockpile in a
safe and environmentally compliant manner.
Similarly, it is important that DOD, CMA, and Congress have reasonable
schedule and cost estimates with reliable supporting data and other
assumptions, including when they change significantly, either
increasing or decreasing funding requirements. With most of the
chemical agent destruction sites significantly ahead of the 2005
program office schedule, the program milestones appear to be overly
conservative, regardless of unexpected risks, such as the rocket fires
at Umatilla. As a result, the program schedule milestones may no longer
be a realistic baseline for measuring program performance and warrant
further review and independent assessment to give DOD and Congress
greater confidence that the program is being held accountable to a
reasonable baseline to measure the program's progress. In addition, the
Chemical Demilitarization Program's revised APBs and its most recent
SARs--key management and oversight documents--lack up-to-date and
reviewed schedule milestone information. Furthermore, DOD needs
accurate and realistic closure cost estimates and other cost inputs for
each of DOD's chemical demilitarization sites to accurately and
adequately budget for the program, make the EVM systems useful for
program oversight, and arrive at reliable cost estimates.
Finally, from a management perspective, CMA could improve its use of
award fees to incentivize the systems contractors to try to meet the
CWC treaty deadline in a safe and environmentally compliant manner,
particularly at the incineration sites where award fees are
predominant. We identified opportunities for CMA to continue and
further advance its recent efforts to emphasize schedule and cost and
to incorporate more objective measures in its award fee evaluation
criteria as called for in DOD guidance on award fees. Given the
maturity of the operations phase, the experience gained by the systems
contractors, and the composition of the remaining stockpile, CMA may be
able to adjust the award fee plan to provide better interim incentives
to accelerate the systems contractors' performance, and do so in a safe
and environmentally sound manner. Doing so would also have the benefit
of reducing the security vulnerabilities and other risks of storing
lethal chemical agents any longer than necessary, which creates
additional safety and environmental hazards. We recognize that to the
extent that objective criteria are not feasible, using subjective
criteria to determine award fees is reasonable and expected, and DOD
guidance directs that award fee criteria be tied to performance
measures, such as identifiable interim outcomes, discrete events, or
milestones, as much as possible. We identified opportunities for CMA to
improve the effectiveness of its monetary incentives by better aligning
its award fee policy and evaluation criteria to DOD's March 2006
guidance.
Recommendations for Executive Action:
We are making 13 recommendations for executive action. They are as
follows.
To strengthen program management and increase the likelihood that more
progress can be made in destroying chemical agents, we recommend that
the Secretary of Defense direct the Secretary of the Army to take the
following three actions:
* incorporate baseline and trend data for past performance and
multiyear performance goals for the future in its annual performance
plan;
* develop interim destruction goals, approaches, and milestones that
are directly linked to overall program goals for meeting CWC deadlines;
and:
* establish a time frame for completing and implementing its risk
management approach, including integration across sites and with DOD.
To provide more realistic cost estimates for chemical demilitarization
sites in light of recent experience and information gained, we
recommend that the Secretary of Defense direct the Secretary of the
Army to update the 2005 program schedule milestones for each of DOD's
chemical demilitarization sites to reflect recent site processing rates
and then adjust the cost estimate accordingly.
To help ensure that the program baseline schedule is current and
achievable and to improve the accuracy and reliability of program
manager estimates for measuring the Chemical Demilitarization Program's
progress, we recommend that the Secretary of Defense direct the CAIG to
establish a periodic schedule to review the program's processing
history to determine whether the schedule milestones are still
reasonable.
To increase the ability of decision makers to measure the Army's and
CMA's performance against schedule growth, we recommend that Secretary
of Defense direct the Under Secretary of Defense for Acquisition,
Technology, and Logistics to include in the Chemical Demilitarization
Program's APB and SAR the updated and reviewed schedule milestones for
completing destruction operations and facility closure activities for
each of the chemical agent destruction sites.
To provide more realistic cost estimates for closure of chemical
demilitarization sites and to establish, stabilize, and maintain an
accurate, valid, and current performance management baseline, we
recommend that the Secretary of Defense direct the Secretary of the
Army to take the following three actions:
* coordinate with the systems contractors in defining closure
requirements and developing accurate and realistic closure cost
estimates for each of DOD's chemical demilitarization sites' contract
performance measurement baseline;
* define all contract requirements, including authorized unpriced work
associated with facility closure activities, and develop comprehensive
cost estimates through closure at each chemical destruction site; and:
* establish a time frame for completing the negotiation for placing all
authorized work that has not been priced on contract and ensuring that
the negotiations take place.
After the cost estimates are complete and schedule milestones have been
updated and reviewed, we recommend that the Secretary of Defense direct
the CAIG to conduct an independent cost review of the cost estimates.
To continue to further the Army's efforts to improve the potential to
meet the CWC deadline for destroying chemical weapons safely and within
environmental compliance and reduce the risk of storing lethal chemical
agents any longer than necessary, we recommend that the Secretary of
the Defense direct the Secretary of the Army to determine for each of
the four incineration site contracts whether greater weight can be
applied to performance measures, such as schedule and cost, in award
fee plans to further incentivize the systems contractors to increase
the destruction rate of chemical agents in a safe and environmentally
sound manner.
To further link monetary incentives and key acquisition outcomes, and
to provide a means for objectively measuring the systems contractors'
performance during the assessment period as much as possible, we
recommend that the Secretary of Defense direct the Secretary of the
Army to take the following two actions:
* better align its award fee policy with DOD's March 2006 guidance on
award fees and:
* link more of its award fee criteria to performance measures that
focus on identifiable interim outcomes, discrete events, or milestones.
Agency Comments and Our Evaluation:
In written comments on a draft of this report, DOD concurred or
partially concurred with 12 of our 13 recommendations. DOD did not
concur with our recommendation regarding defining closure requirements
and developing accurate and realistic closure cost estimates for each
of DOD's chemical demilitarization sites. DOD recommended that we
delete it, since DOD officials believe that a subsequent recommendation
defining all contract requirements, including those associated with
facility closure activities, was more encompassing. Based on DOD's
comments, rather than deleting the recommendation, we clarified it and
the related recommendation to more clearly distinguish the difference
between the two recommendations, and consolidated it with our two other
recommendations related to defining contract requirements. DOD also
stated that the department will continue to use acquisition management
tools and discipline to ensure the destruction of the United States'
chemical weapons stockpile in a safe and secure manner, while being
economical and meeting the U.S. commitments under the Chemical Weapons
Convention. DOD identified a number of actions that it already has
initiated in response to our recommendations. DOD also provided us with
technical comments, which we incorporated in the report, as
appropriate. DOD's comments are reprinted in appendix III.
Regarding our four recommendations related to strategic planning tools,
DOD concurred with our recommendation that it incorporate baseline and
trend data for past performance and multiyear performance goals for the
future in its annual performance plan. DOD stated that CMA currently is
incorporating these items into its fiscal year 2008 Annual Performance
Plan and expects this effort to be completed by December 31, 2007. DOD
partially concurred with our recommendation that it provide direction
to develop interim destruction goals, approaches, and milestones that
are directly linked to overall program goals for meeting CWC deadlines.
In the spirit of our recommendation, DOD stated that it would ensure
that all future CMA performance plans incorporate annual goals and
approaches leading toward the April 29, 2012, destruction deadline
through direction issued annually in support of its existing strategic
governance process and that additional direction would not be needed.
We support this planned action, which, if implemented properly, would
accomplish the same effect as our recommendation. DOD concurred with
our recommendation that the Army establish a time frame for completing
and implementing its risk management approach, including integration
across sites and with DOD. DOD stated that it expects an approved risk
management approach to be completed by June 30, 2008. DOD concurred
with our recommendation to include in the Chemical Demilitarization
Program's APB and SAR the updated and reviewed schedule milestones for
completing destruction operations and facility closure activities for
each of the chemical agent destruction sites. DOD stated that it would
request administrative changes to remove placeholder terminology from
these documents and replace them with the CAIG's schedule estimates.
Regarding our two recommendations related to schedule and cost
estimates, DOD partially concurred with our recommendation that it
update the 2005 program schedule milestones for each of DOD's chemical
demilitarization sites to reflect recent site processing rates and then
adjust the cost estimate accordingly. DOD stated that it has gained
only limited information during the past year regarding processing
rates, unknown risks, and the potential for a chemical event, and thus
does not believe it would be prudent to adjust cost and schedule
estimates or rebaseline at this time. However, DOD will evaluate the
cost and schedule estimates for the entire program during the upcoming
Fiscal Year 2010-2015 Program Objectives Memorandum build process and
program review, and will determine whether an update is warranted. DOD
concurred with our recommendation to establish a periodic schedule for
the CAIG to review the program's processing history to determine
whether the schedule milestones are still reasonable. DOD stated that
it conducts a biannual program review of the Chemical Demilitarization
Program and that during the next review in the spring of 2008, it would
evaluate the program's cost and schedule estimates. If significant
differences exist between the program's baseline and the revised
estimates, then DOD would request a CAIG review that could result in
new cost and schedule estimates. However, we believe that our report
provides clear and sufficient evidence demonstrating that there are
differences between the program's current schedule and cost estimates
and DOD's baseline to warrant a CAIG review and update as we
recommended.
Regarding our four recommendations related to contract requirements,
based on DOD's comments, we consolidated and clarified two of our
recommendations. DOD suggested that we delete the recommendation that
CMA coordinate with the systems contractors in defining closure
requirements and in developing accurate and realistic closure cost
estimates for each of DOD's chemical demilitarization sites, since the
subsequent recommendation is more encompassing. Instead, we clarified
this recommendation by adding "for each of DOD's chemical
demilitarization sites' contract performance measurement baseline"
[emphasis added], and consolidated it with our two other
recommendations related to defining contract requirements discussed
below. As discussed in our report, we continue to believe that CMA and
the systems contractors must coordinate their efforts to develop
accurate and realistic closure cost estimates. In addition, we
consolidated our recommendation that after the cost estimates are
complete and schedule milestones have been updated and reviewed, the
CAIG conduct an independent cost review of the cost estimates with our
two recommendations that are discussed below. DOD partially concurred
with our recommendation to define all contract requirements, including
those associated with facility closure activities, and develop
comprehensive cost estimates through closure at each chemical
destruction site. Based on DOD's comments, we clarified this
recommendation by adding "define all contract requirements, including
authorized unpriced work associated with facility closure activities"
[emphasis added]. DOD stated that since June 2007, three of the five
CMA destruction sites have completed contract performance baselines
that include contract requirements for operations, secondary waste
processing, closure, and known risk. Negotiations are under way to
complete the contract performance baseline for the Tooele, Utah, site
and the contract requirements and cost estimates for the Newport,
Indiana, site by June 30, 2008. DOD concurred with our recommendation
to establish a time frame for completing the negotiation for placing
all authorized work that has not been priced on contract and ensuring
that the negotiations take place. DOD stated that it expects
negotiations to be completed by June 30, 2008. Finally, DOD partially
concurred with our recommendation that the CAIG conduct an independent
review of the program's cost estimates stating that the program's
biannual review in support of the budget submission will determine if a
CAIG independent assessment is warranted. We continue to believe that
providing an independent review of the program's cost estimates could
provide greater assurance that cost estimates are reasonable.
With regard to our three recommendations related to incentive fees, DOD
concurred with our recommendation to determine for each of the four
incineration site contracts whether greater weight can be applied to
performance measures, such as schedule and cost, in award fee plans to
further incentivize the systems contractors to increase the destruction
rate of chemical agents in a safe and environmentally sound manner. DOD
also concurred with our recommendation to better align CMA's award fee
policy with DOD's March 2006 guidance on award fees, stating that this
action had been completed on September 7, 2007. However, when a copy of
the revised policy was requested, DOD stated that its response
incorrectly referenced that this action has been completed. As such,
DOD will direct a revision of the CMA policy that aligns it with the
DOD guidance on award fees, and the action is to be completed by March
31, 2008. DOD partially concurred with our recommendation to link more
of its award fee criteria to performance measures that focus on
identifiable interim outcomes, discrete events, or milestones. DOD
stated that it would conduct a review of CMA award fee plans, to be
completed by March 31, 2008, and would then determine whether further
direction and action is required. While this review represents a
positive step, our report shows that there are several areas in which
DOD could better link more of its award fee criteria with performance
measures that focus on identifiable outcomes and we believe that DOD
should fully implement this recommendation.
As agreed with your offices, unless you publicly announce the contents
of this report earlier, we plan no further distribution of it until 30
days from the date of this letter. We will then send copies of this
report to the Chairmen and Ranking Members of the Senate Committee on
Armed Services; Senate Committee on Homeland Defense and Governmental
Affairs; Subcommittee on Defense, Senate Committee on Appropriations;
House Committee on Oversight and Government Reform; and Subcommittee on
Defense, House Committee on Appropriations, and other interested
congressional parties. We are also sending copies of this report to the
Secretary of Defense; the Secretary of the Army; the Director, Office
of Management and Budget; and the Director, U.S. Army Chemical
Materials Agency. We also will make copies available to others on
request. In addition, this report will be available at no charge on the
GAO Web site at [hyperlink, http://www.gao.gov].
If you or your staff have any questions concerning this report, please
contact me at (202) 512-5431 or dagostinod@gao.gov. Contact points for
our Offices of Congressional Relations and Public Affairs may be found
on the last page of this report. GAO staff who made key contributions
to this report are listed in appendix IV.
Signed by:
Davi M. D'Agostino:
Director, Defense Capabilities and Management:
[End of section]
Appendix I: Scope and Methodology:
To assess the progress the Department of Defense (DOD) and the Army
have made in implementing our prior recommendations to strengthen
program management, whether it has been sufficient, and whether actions
taken to identify and mitigate the risk of future program schedule
extensions and cost growth have been effective, we obtained and
reviewed strategic and implementation plans and interviewed cognizant
officials to identify whether key elements were present, such as a
program mission statement, long-term goals and objectives, delineation
of roles and responsibilities of DOD and Army offices, and near-term
performance measures.
To assess the reasonableness of the methodology used to determine
schedule milestones as well as the reliability of cost estimates, we
reviewed current program estimates, destruction schedules, earned value
management (EVM) data, and other documents. We also obtained and
reviewed the program's risk management plans and related documents. In
addition, we also identified the issues that had caused delays and
ascertained approaches being used to reduce the potential for delays in
the future.
The Army's Chemical Materials Agency (CMA) provided us with EVM reports
submitted to the Army by the systems contractors for each of the
chemical demilitarization facilities spanning the period from December
2000 through February 2007. EVM reports were provided electronically in
the form of Deltek wInsightTM transfer files for Cost Performance
Report Formats 1 through 4 and separate Adobe®, MS Word, or MS Excel
files for Format 5.
Our approach in assessing the adequacy of the systems contractors' EVM
reporting was to analyze the data provided according to EVM analysis
tasks and DOD earned value implementation guidance. The GAO EVM
analysis tasks are a set of 10 analysis tasks developed by GAO that are
used to assess the adequacy of EVM. These 10 analysis tasks were
performed on each of the EVM reports provided to us. Guidance and
procedures outlined within the DOD Earned Value Management
Implementation Guide were also referenced to assess EVM system and
reporting adequacy.
Our approach in assessing the Army's ability to assess, synthesize, and
incorporate systems contractor historical and estimated cost,
technical, and schedule performance into the current program estimate
was to align historical contract performance data as documented within
the EVM reports with program estimate and baseline data documented
within Selected Acquisition Reports, the President's budget, and the
Fiscal Year 2005 Army Cost Position.
In performing this assessment, we also conducted interviews with
government and systems contractor officials at CMA headquarters in
Aberdeen, Maryland; Tooele Chemical Agent Disposal Facility, Deseret
Chemical Depot (Tooele), Utah; and Umatilla Chemical Agent Disposal
Facility, Umatilla Chemical Depot (Umatilla), Oregon.
To analyze the effectiveness of DOD's and the Army's use of monetary
incentives to improve systems contractors' performance, we obtained and
reviewed contract documentation related to the implementation of the
contracts' incentives, including the basic contract and modifications,
award fee plans, award fee determining official decisions, award fee
evaluation board reports, and pre-and postnegotiation memorandums, and
the performance-based incentive fees, including the Director's
Programmatic Performance Based Incentive (DPPBI) fee. We also
interviewed DOD and Army officials to determine the types of award and
incentive fees offered at the incineration sites and at Newport
Chemical Agent Disposal Facility, Newport Chemical Depot (Newport),
Indiana.
To evaluate the effectiveness of the award fees in controlling costs
and schedule, we reviewed the award fee plans from fiscal year 2003 to
fiscal year 2007 to identify the types of performance measures and
weight factors used to incentivize the systems contractors to safely
destroy the chemical weapons stockpile. We also looked at the extent
the criteria used for the performance measures were linked to
measurable outcomes. To identify the percentage of award fees paid to
the systems contractors at the incineration sites and Newport, we
collected and reviewed data on the award fee payments, including the
amount available for the contactors to earn, the amounts earned from
fiscal year 2003 through fiscal year 2006, and the guidance for
improving the use of these fees. We used data on the award fee plans
from fiscal year 2003 through fiscal year 2007 because they represented
the period of time the sites were in the operations phase.
To evaluate the extent documentation was prepared for adjustments to
the award fee plans, we reviewed CMA's Integrated Process Team
documents from fiscal year 2003 to fiscal year 2007 to identify
adjustments made by the various stakeholders, such as the CMA Director,
U.S. Army Sustainment Command, CMA Program and Policy Office,
incinerator site project managers, and systems contractor site general
managers. We evaluated whether Army officials provided complete
documentation supporting their rationale for adjusting performance
measures and related weights throughout the process. We also evaluated
whether CMA prepared adequate supporting documentation to enable
reviewers to understand and evaluate the process that was used, and
whether it contained, among other things, an adequate discussion of
alternative performance measures or the combination of award and
incentive fees that it considered, tested, and rejected and the reasons
for rejecting them.
To evaluate the effectiveness of the authority contained in section 923
of the National Defense Authorization Act for Fiscal Year 2007, we
obtained and reviewed the incentive agreement plan and incentive fees
for the Aberdeen Chemical Agent Disposal Facility (Aberdeen), Maryland,
and Newport to determine the challenges those sites faced in
negotiating the incentive fee agreements. Also, we interviewed
officials with the U.S. Army Sustainment Command, CMA, and CMA and
systems contractors at Tooele and Umatilla.
We also met with DOD and Army program officials and interviewed
officials at two of the chemical agent destruction sites. Finally, this
report focuses primarily on the seven CMA-managed chemical destruction
sites--Aberdeen; Anniston Chemical Agent Disposal Facility, Anniston
Chemical Activity, Alabama; Johnston Atoll in the Pacific Ocean;
Newport; Pine Bluff Chemical Agent Disposal Facility, Pine Bluff
Chemical Activity, Arkansas; Tooele; and Umatilla--because most are in
the operational phase and represent about 82 percent of the remaining
stockpile of chemical agents to be destroyed, and have reached a level
of maturity in processing a variety of types of munitions and agents
that will help meet Chemical Weapons Convention (CWC) goals. This
report does not focus on two sites--Aberdeen and Johnston Atoll--that
have already completed operations and have either been closed or are
being closed. It also does not focus on the two chemical agent
destruction sites that are part of the Assembled Chemical Weapons
Alternatives program--Blue Grass Chemical Agent-Destruction Pilot
Plant, Blue Grass Chemical Activity (Blue Grass), Kentucky, and Pueblo
Chemical Agent-Destruction Pilot Plant, Pueblo Chemical Depot,
Colorado--that currently are in the design and construction phase to
destroy chemical agents, report directly to DOD rather than to the
Army, and are not expected to help meet CWC goals.
We examined the reliability of the data used in this report by
verifying EVM system certification, applying analytical tests to the
EVM data, and examining EVM system effectiveness in providing
meaningful performance measurement to program management. We found that
EVM systems at each site had been certified. The results of our
analytical tests found the data to be reasonably free of data errors
and anomalies. However, when we examined the effectiveness of the EVM
system to provide meaningful performance measurement, we found
insufficient reliability to address our objectives.
We conducted this performance audit from May 2006 through July 2007 in
accordance with generally accepted government auditing standards.
[End of section]
Appendix II: Two Recently Introduced Performance-Based Incentive Fees:
The Army's performance-based incentive fee to collaboratively improve
schedule and cost performance was established in 2005 because the use
of award fees had not been successful. Also, the Army plans to use a
recently authorized $165 million incentive fee to accelerate
destruction and closure activities to dates linked with the treaty
schedule.
Army's Performance-Based Incentive Fee Begun in 2005 Was Recently
Targeted to Objective Performance:
In 2005, CMA established the DPPBI fee to improve schedule and cost
performance because CMA's use of award fees had not been successful in
controlling cost and schedule at the incineration sites. This incentive
fee is 2 percent of the total 12 percent available fees, which includes
the 3 percent base fee that is not performance related. DPPBI was
structured so that the collective performance of the systems
contractors is evaluated and rewarded through improved collaboration
among the systems contractors that operate the four incineration sites.
That is, while each individual site is evaluated and scored separately,
the individual scores are averaged to determine a composite score that
results in all sites earning the same percentage of the available DPPBI
fee pool.
The evaluation factors for assessing the systems contractor performance
for DPPBI have changed over time and are divided between objective
criteria and a subjective assessment of systems contractor performance
during the fiscal year. Three criteria were established for the initial
DPPBI evaluation period of January 1 through September 30, 2005: risk
mitigation, production assessment, and lessons learned. In the initial
evaluation period, the contractors collectively were deemed to have
earned 51 percent of the 2005 DPPBI pool or $3,313,954 out of
$6,497,949. In fiscal year 2006, the DPPBI criteria were changed to put
more emphasis on performance outputs and less on processes and
procedures. The fiscal year 2006 DPPBI criteria were established to
encourage the contractors to reduce project schedules while maintaining
a high standard of surety, safety, and environmental compliance. The
criteria also were established to increase CMA's confidence levels that
the sites could achieve CWC destruction deadlines, although discrete
goals were not specified. For the fiscal year 2006 period of
performance, the contractors earned about 93 percent or $8,771,200 out
of the $9,411,159 in the DPPBI pool for that period.
Until fiscal year 2007, the DPPBI fee evaluation was not linked to
objective schedule performance criteria linked to the April 29, 2012,
CWC extended treaty deadline. In fiscal year 2007, schedule performance
attainment was added to the DPPBI objective criteria amounting to 70
percent of the total available DPPBI fee (or up to 1.4 percent for
average collective performance) and amounting to 30 percent for
subjective criteria related to the contractors' performance in the
application of lessons learned, risk management, programmatic
procurement, and other collaborative approaches to reduce cost and
accelerate destruction of the U.S. stockpile. The amount of the fee
that can be earned by the systems contractors depends not just on a
systems contractor's individual performance, but also on the
performance of the other systems contractors. For example, if the
systems contractor at Anniston is performing well in meeting its
schedule-related goals during a specific period, but the systems
contractor at Tooele is not, then the Anniston systems contractor would
be encouraged to offer assistance to the Tooele systems contractor. If
the Tooele systems contractor's schedule performance could not be
improved, then the Anniston systems contractor's potential to earn this
fee would be reduced as would the potential of systems contractors at
the other sites.
Recently Authorized Incentives Are Targeted to Completing Destruction
Operations around the Treaty Deadline and Accelerating Facility Closure
Activities:
CMA plans to offer incentive fees under the authority contained in
section 923 of the National Defense Authorization Act for Fiscal Year
2007,[Footnote 67] permitting systems contractors the opportunity to
earn incentive payments for the completion of destruction operations by
or near the CWC deadline and the acceleration of facility closure
activities, but the effectiveness of this approach is uncertain. The
purpose of section 923 is to provide the systems contractor for a
chemical demilitarization facility an incentive to accelerate the safe
elimination of the U.S. chemical weapons stockpile and to reduce the
total cost of the Chemical Demilitarization Program by providing
incentive payments for the early completion of destruction operations
and the closure of such a facility.[Footnote 68] Section 923 authorizes
the inclusion of incentives clauses in chemical demilitarization
contracts for the completion of both destruction operations and
facility closure activities within target incentive ranges. Target
incentive ranges are to be jointly agreed upon by the contracting
officer and the systems contractor concerned and are to be specified in
the contract's incentives clause. Under the statute, the amount of
incentive payment a systems contractor may earn through such incentive
clauses will be based on how early within the specified target
incentive range the systems contractor completes destruction operations
and facility closure activities. Section 923 sets limitations on the
amount of incentive payments a systems contractor may receive through
the incentives clauses. It establishes $110 million as the maximum
incentive payment for the completion of destruction operations within
the target incentive range specified under the contract's incentives
clause and $55 million for the completion of facility closure
activities within the target incentive range specified under the
contract's incentives clause.[Footnote 69]
Although it had not been implemented at the time of our review, CMA had
drafted a plan for implementation of the incentives authorized under
section 923.[Footnote 70] The plan articulates that the systems
contractors are to assume responsibility for identifying and mitigating
all site-specific and programmatic risks in order to successfully
complete program objectives within budgeted cost and schedule targets.
The plan further relates that in return, the systems contractors will
receive incentives measured against established schedule targets for
operations completion and facility closure.
Under CMA's plan, it would offer the additional incentive fee of up to
$150 million to each site--$100 million for destruction operations
completed within target ranges by or near the April 29, 2012, treaty
deadline, and $50 million for facility closure activities brought
successfully to completion within target ranges. CMA's plan establishes
the period from November 1, 2011, through October 31, 2013, as the
target incentive range for the completion of operations at all sites,
with the maximum amount available to be earned, exclusive of the bonus,
at $100 million. Ninety-five percent of the incentive can be earned for
completing operations by the CWC extended deadline of April 29, 2012.
After the CWC deadline, a percentage of the incentives can be earned
through October 31, 2013, on a reduced sliding scale. The CMA plan also
offers up to $50 million per site for the completion of facility
closure activities within a specified range of dates unique to each
site. One hundred percent of the $50 million would be earned by a
systems contractor for completion of facility closure activities on the
first date of the specified range. The systems contractor would earn
some lesser percentage of the $50 million for completion on any date
between the first and last dates of the specified range.
Under CMA's plan, systems contractors can earn up to $165 million at
their respective sites through bonus potential. In addition to the
potential $100 million fee for completing operations ahead of the
extended CWC deadline and $50 million for meeting specified closure
targets, bonuses can also be earned. A bonus of $10 million is offered
to each site if all the sites complete destruction operations prior to
the CWC deadline of April 29, 2012. A bonus of $5 million is also
offered to each site for accelerated completion of facility closure
activities.
CMA's plan, which would have to be implemented under each of the
contracts, includes language that insulates the government from having
to make incentive payments to the systems contractors if they did not
strictly perform under the terms CMA has set under the plan. In firmly
establishing the target incentive ranges, the plan states that the
target dates are not subject to extension except for catastrophic
causes beyond the control of and without the fault or negligence of the
systems contractor, its subcontractors, and suppliers. The plan
establishes that the systems contractors are not entitled to submit
claims for loss of opportunity to earn the incentives or to increase
incentive pools due to failure of the government to provide resources
over and above those specifically identified in the contract. It also
stipulates that payments of incentives will be subject to adjustment
based upon final audits of operation costs and closure costs. At the
same time, the plan recognizes, as does the statute, that the
enterprise of striving to capture these incentives is to be conducted
consistent with the existing obligation to provide for maximum
protection for the environment, the general public, and the personnel
who are involved in the destruction of the chemical agents and
munitions. The effectiveness of this planned incentive is uncertain
because it had not yet been negotiated or placed on contract at the
time of our review.
[End of section]
Appendix III: Comments from the Department of Defense:
Assistant To The Secretary Of Defense:
3050 Defense Pentagon:
Washington, DC 20301-3050:
Nuclear And Chemical And Biological Defense Programs:
November 16, 2007:
Ms. Davi M. D'Agostino:
Director, Defense Capabilities and Management:
United States Government Accountability Office:
441 G Street, N.W.:
Washington, DC 20548:
Dear Ms. D'Agostino:
This is the Department of Defense (DoD) response to the GAO draft
report, GAO-08-134, `Chemical Demilitarization: Additional Management
Actions Needed to Meet Key Performance Goals of DoD's Chemical
Demilitarization Program,' dated October 3, 2007 (GAO Code 350799).
The DoD concurs with most of the draft report's recommendations, except
those pertaining to when cost and schedule estimates should be revised.
The Department will continue to use acquisition management tools and
discipline to ensure the destruction of the United States' chemical
weapons stockpile in a safe and secure manner, while being economical
and meeting our commitments under the Chemical Weapons Convention.
The Department appreciates the opportunity to provide comments on the
draft report. These comments, including technical comments, are
enclosed. For further questions concerning this report, please contact
Barbara Burgess, Senior Program Analyst for the Chemical
Demilitarization Program, (703) 588-1983, extension 113.
Sincerely,
Signed by:
Jean D. Reed:
Special Assistant:
Chemical and Biological Defense and Chemical Demilitarization Programs:
Enclosures:
As stated:
cc:
Deputy Assistant Secretary of the Army (Elimination of Chemical
Weapons):
GAO Draft Report – Dated October 3, 2007 GAO Code 350799/GAO-08-134:
"Chemical Demilitarization: Additional Management Actions Needed to
Meet Key Performance Goals of DoD's Chemical Demilitarization Program"
Department Of Defense Comments To The Recommendations:
Recommendation 1: The GAO recommends the Secretary of Defense direct
the Director of U.S. Army Chemical Materials Agency to incorporate
baseline and trend data for past performance and multiyear performance
goals for the future in its annual performance plan.
DoD Response: Concur. The Department recognizes that U.S. Army Chemical
Materials Agency (CMA) is currently incorporating baseline and trend
data for past performance and multiyear performance measures in the CMA
Annual Performance Plan for 2008. This effort is expected to be
completed and staffed by end of the first quarter of Fiscal Year 2008.
Recommendation 2: The GAO recommends the Secretary of Defense direct
the Director of U.S. Army Chemical Materials Agency to develop interim
destruction goals, approaches, and milestones that are directly linked
to overall program goals for meeting Chemical Weapons Convention
deadlines.
DoD Response: Partially Concur. The Department recognizes that the U.S.
Army Chemical Materials Agency (CMA) annual destruction goals are
interim destruction goals for meeting the Chemical Weapons Convention
(CWC) destruction milestones. The Chemical Demilitarization Program
Fiscal Year 2007 Annual Performance Plan provided the annual
destruction goals for meeting the CWC 45 percent destruction milestone
of December 31, 2007. The Department will continue to ensure all future
CMA performance plans incorporate annual goals and approaches leading
toward the April 2012 destruction deadline through direction issued
annually in support of strategic governance. Therefore, no further
direction is required from the Secretary of Defense to develop interim
goals, approaches, and milestones.
Recommendation 3: The GAO recommends the Secretary of Defense direct
the Director of U.S. Army Chemical Materials Agency to establish a time
frame for completing its risk management approach, including
integration across sites and with DoD.
DoD Response: Concur. The U.S. Army Chemical Materials Agency (CMA) is
currently evaluating and updating the CMA risk management approach
including integration of risk management across sites and within CMA
Headquarters. The Department expects an approved risk management
approach to be completed by the third quarter of Fiscal Year 2008.
Recommendation 4: The GAO recommends the Secretary of Defense direct
the Director of U.S. Army Chemical Materials Agency to update the 2005
program schedule milestones for each of DoD's chemical demilitarization
sites to reflect recent site processing rates and then adjust the cost
estimate accordingly.
DoD Response: Partially Concur. The Department recognizes successes
within the Chemical Demilitarization Program (CDP) over the past two
years. However, there are still some areas of the program, (i.e.,
destruction of secondary hazardous waste and closure of the chemical
weapons destruction facilities), with limited information to adjust
cost and schedule estimates. As the GAO stated in their final report:
NSIAD-00-80, Chemical Weapons Disposal: Improvements Needed in Program
Accountability and Financial Management, May 2000: "The Army also needs
additional time to develop a reliable baseline to estimate the costs
for the closure and remediation of the chemical stockpile disposal
facilities.Furthermore, because no stockpile disposal facility has yet
to be remediated, the Army lacks real time experience on which to
estimate these costs." The CDP now has two sites to compare closure
activities, Johnston Atoll Chemical Agent Destruction System (JACADS)
and Aberdeen Chemical Agent Disposal Facility (ABCDF). JACADS completed
destruction operations in November 2000 and most closure activities in
June 2004, with the exception of final closure of the Resource
Conservation and Recovery Act permit, which is not expected until
November 2008. The ABCDF completed operations in February 2006 and
closure activities were completed in March 2007. With this limited
information for closure, the U.S. Army Chemical Materials Agency (CMA)
has recently completed contract performance baselines that include
operations, secondary waste processing, closure and known risk, for
sites at Anniston, Alabama; Pine Bluff, Arkansas; and Umatilla, Oregon.
In addition, the Department approved the revised Acquisition Program
Baselines for the Chemical Demilitarization Program – U.S. Army
Chemical Materials Agency (Chem Demil- CMA) and Chem Demil-CMA Newport
Major Defense Acquisition Programs in April 2006, with the cost and
schedule estimates based on the Cost Analysis Improvement Group's
(CAIG) estimates. The CAIG estimates were certified to Congress in May
2005 as part of the Nunn- McCurdy unit cost breach. The increase in the
unit cost was due primarily to more realistic schedule estimates.
Therefore, the Department does not believe it is prudent to adjust cost
and schedule estimates or rebaseline at this time due to limited
information gained during the past year associated with processing
rates, unknown risks, and the potential for a chemical incident. The
Department will evaluate the cost and schedule estimates for the entire
CDP during the upcoming Fiscal Year 2010-2015 Program Objectives
Memorandum build process and program review, and will determine whether
an update is warranted.
Recommendation 5: The GAO recommends the Secretary of Defense direct
the Chemical Analysis Improvement Group to establish a periodic
schedule to review the program's processing history to determine
whether the schedule milestones are still reasonable.
DoD RESPONSE: Concur. The Department conducts a biannual program review
of the Chemical Demilitarization Program in support of the budget
estimate submission. During this review, the program office schedule
and cost estimates are reviewed in comparison with the approved
Acquisition Program Baselines to determine whether there are any
significant differences. If there are significant differences, the
Department will request the Cost Analysis Improvement Group conduct an
independent assessment and possibly develop cost and schedule
estimates. The next review will occur during the Fiscal Year 2010-2015
Program Objectives Memorandum build process and program review in the
spring of calendar year 2008.
Recommendation 6: The GAO recommends the Secretary of Defense direct
the Under Secretary of Defense for Acquisition, Technology and
Logistics to include in the Chemical Demilitarization Program's
Acquisition Program Baseline and Selected Acquisition Report the
updated and reviewed scheduled milestones for completing destruction
operations and facility closure activities for each of the chemical
agent destruction sites.
DoD Response: Concur. The Department will request administrative
changes to the Chemical Demilitarization Program-U.S. Army Chemical
Materials Agency (Chem Demil – CMA) and Chem Demil-CMA Newport
Acquisition Program Baselines and upcoming Selected Acquisition Reports
to remove the "To Be Determined (TBDs)," which were used as a
placeholder for schedule milestones beyond April 2012, and replace with
the Cost Analysis Improvement Group's schedule estimates.
Recommendation 7: The GAO recommends the Secretary of Defense direct
the Director of the U.S. Army Chemical Materials Agency to coordinate
with the systems contractors in defining closure requirements and
developing accurate and realistic closure cost estimates for each of
DoD's chemical demilitarization sites.
DoD Response: Non-Concur. The Department recommends GAO delete this
recommendation since Recommendation #9 is more encompassing.
Recommendation 8: The GAO recommends the Secretary of Defense direct
the Cost Analysis Improvement Group to conduct an independent cost
review of the cost estimates.
DoD Response: Partially Concur. The Department conducts a biannual
program review of the Chemical Demilitarization Program in support of
the budget estimate submission. During this review, the program office
schedule and cost estimates are reviewed in comparison with the
approved Acquisition Program Baselines to determine whether there are
any significant differences. If there are significant differences, the
Department will request the Cost Analysis Improvement Group conduct an
independent assessment and possibly develop cost and schedule
estimates. Therefore, no direction to conduct a cost review will be
given at this time.
Recommendation 9: The GAO recommends the Secretary of Defense direct
the Director of the U.S. Army Chemical Materials Agency to define all
contract requirements, including those associated with facility closure
activities, and develop comprehensive cost estimates through closure at
each chemical destruction site.
DoD Response: Partially Concur. Since June 2007, three of the five U.S.
Army Chemical Materials Agency chemical weapons stockpile destruction
sites (Anniston, Alabama; Pine Bluff, Arkansas; and Umatilla, Oregon)
have completed contract performance baselines that include contract
requirements for operations, secondary waste processing, closure and
known risk. Negotiations are under way to complete the contract
performance baseline by the end of the third quarter of Fiscal Year
2008 for the Tooele, Utah, site. Negotiations are expected to be
completed for the contract requirements and cost estimates associated
with the closure phase for the Newport, Indiana, site by the end of the
third quarter of Fiscal Year 2008. The Department believes no further
direction is required.
Recommendation 10: The GAO recommends the Secretary of Defense direct
the Director of the U.S. Army Chemical Materials Agency to establish a
time frame for completing the negotiation for placing all authorized
work that has not been priced on contract and ensuring that the
negotiations take place.
DoD Response: Concur. The U.S. Army Chemical Materials Agency has
negotiated almost all work presently on contract (current and future)
resulting in low levels of authorized, unpriced work. The Department
expects negotiations to be completed by the end of the third quarter of
Fiscal Year 2008.
Recommendation 11: The GAO recommends the Secretary of the Defense
direct the Director of the U.S. Army Chemical Materials Agency to
determine for each of the four incineration site contracts whether
greater weight can be applied to performance measures, such as schedule
and cost, in award fee plans to further incentivize the systems
contractors to increase the destruction rate of chemical agents in a
safe and environmentally sound manner.
DoD Response: Concur. The Department will conduct a review of the award
fee plans of the U.S. Army Chemical Materials Agency incineration site
contracts to determine whether the weighting factors are appropriate to
incentivize the systems contractors to reduce the schedule while
operating in a safe, secure, and environmentally-sound manner. The
results will be available by second quarter of FY08.
Recommendation 12: The GAO recommends the Secretary of the Defense
direct the Director of the U.S. Army Chemical Materials Agency better
align its award fee policy with DoD's March 2006 guidance on award
fees.
DoD Response: Concur. This action was completed on September 7, 2007,
in accordance with the March 29, 2006, Memorandum: Subject: Award Fee
Contracts, from the Under Secretary of Defense for Acquisition,
Technology and Logistics. CMA revised Policy Statement 29, "Procedure
for Implementing Contractor Incentives to Include Award Fee (AF) and
Performance Based Incentive (PBI) Fee."
Recommendation 13: The GAO recommends the Secretary of the Defense
direct the Director of the U.S. Army Chemical Materials Agency to link
more of its award fee criteria to performance measures that focus on
identifiable interim outcomes, discrete events, or milestones.
DoD Response: Partially Concur. The Department will conduct a review of
the U.S. Army Chemical Materials Agency (CMA) award fee plans to
determine whether there are performance measures that focus on outcomes
or outputs. This review will be completed along with the review of the
award fee weighting factors by the end of the second quarter of Fiscal
Year 2008. Based on the findings of the review, the Department will
determine whether further direction and action is required. Currently,
the CMA Fiscal Year 2006 and Fiscal Year 2007 Award Fee plans included
objective schedule criteria that focused on U.S. tons of chemical agent
destroyed to complete destruction by April 2012.
[End of section]
Appendix IV: GAO Contact and Staff Acknowledgments:
GAO Contact:
Davi M. D'Agostino, (202) 512-5431 or dagostinod@gao.gov:
Acknowledgments:
Mark A. Pross, Assistant Director; Bonita P. Anderson; Rodell B.
Anderson; Susan C. Ditto; Jennifer K. Echard; Neil D. Feldman; James C.
Lawson; Lonnie J. McAllister; Brian L. Octeau; Charles W. Perdue; Karen
A. Richey; John W. Van Schaik; Cheryl A. Weissman; and Alex M. Winograd
made key contributions to this report.
[End of section]
Related GAO Products:
Cost Assessment Guide: Best Practices for Estimating and Managing
Program Costs. GAO-07-1134SP. Washington, D.C.: July 2007 (Exposure
Draft).
Chemical Demilitarization: Actions Needed to Improve the Reliability of
the Army's Cost Comparison Analysis for Treatment and Disposal Options
for Newport's VX Hydrolysate. GAO-07-240R. Washington, D.C.: January
26, 2007.
Chemical Weapons: Sustained Leadership, Along with Key Strategic
Management Tools, Is Needed to Guide DOD's Destruction Program. GAO-03-
1031. Washington, D.C.: September 5, 2003.
Chemical Weapons: Lessons Learned Program Generally Effective but Could
Be Improved and Expanded. GAO-02-890. Washington, D.C.: September 10,
2002.
Chemical Weapons: FEMA and Army Must Be Proactive in Preparing States
for Emergencies. GAO-01-850. Washington, D.C.: August 13, 2001.
Chemical Weapons Disposal: Improvements Needed in Program
Accountability and Financial Management. GAO/NSIAD-00-80. Washington,
D.C.: May 8, 2000.
Chemical Demilitarization: Funding Status of the Chemical
Demilitarization Program. GAO/NSIAD-99-232R. Washington, D.C.: July 29,
1999.
Chemical Weapons: DOD Does Not Have a Strategy to Address Low-Level
Exposures. GAO/NSIAD-98-228. Washington, D.C.: September 23, 1998.
Chemical Weapons Stockpile: Changes Needed in the Management of the
Emergency Preparedness Program. GAO/NSIAD-97-91. Washington, D.C.: June
11, 1997.
Chemical Weapons and Materiel: Key Factors Affecting Disposal Costs and
Schedule. GAO/T-NSIAD-97-118. Washington, D.C.: March 11, 1997.
Chemical Weapons Stockpile: Emergency Preparedness in Alabama Is
Hampered by Management Weaknesses. GAO/NSIAD-96-150. Washington, D.C.:
July 23, 1996.
Chemical Weapons Disposal: Issues Related to DOD's Management. GAO/T-
NSIAD-95-185. Washington, D.C.: July 13, 1995.
Chemical Weapons: Army's Emergency Preparedness Program Has Financial
Management Weaknesses. GAO/NSIAD-95-94. Washington, D.C.: March 15,
1995.
Chemical Stockpile Disposal Program Review. GAO/NSIAD-95-66R.
Washington, D.C.: January 12, 1995.
Chemical Weapons: Stability of the U.S. Stockpile. GAO/NSIAD-95-67.
Washington, D.C.: December 22, 1994.
Chemical Weapons Disposal: Plans for Nonstockpile Chemical Warfare
Materiel Can Be Improved. GAO/NSIAD-95-55. Washington, D.C.: December
20, 1994.
Chemical Weapons: Issues Involving Destruction Technologies. GAO/T-
NSIAD-94-159. Washington, D.C.: April 26, 1994.
Chemical Weapons Destruction: Advantages and Disadvantages of
Alternatives to Incineration. GAO/NSIAD-94-123. Washington, D.C.: March
18, 1994.
Arms Control: Status of U.S.-Russian Agreements and the Chemical
Weapons Convention. GAO/NSIAD-94-136. Washington, D.C.: March 15, 1994.
Chemical Weapon Stockpile: Army's Emergency Preparedness Program Has
Been Slow to Achieve Results. GAO/NSIAD-94-91. Washington, D.C.:
February 22, 1994.
Chemical Weapons Storage: Communities Are Not Prepared to Respond to
Emergencies. GAO/T-NSIAD-93-18. Washington, D.C.: July 16, 1993.
Chemical Weapons Destruction: Issues Affecting Program Cost, Schedule,
and Performance. GAO/NSIAD-93-50. Washington, D.C.: January 21, 1993.
Chemical Weapons Destruction: Issues Related to Environmental
Permitting and Testing Experience. GAO/T-NSIAD-92-43. Washington, D.C.:
June 16, 1992.
Chemical Weapons Disposal. GAO/NSIAD-92-219R. Washington, D.C.: May 14,
1992.
Chemical Weapons: Stockpile Destruction Cost Growth and Schedule
Slippages Are Likely to Continue. GAO/NSIAD-92-18. Washington, D.C.:
November 20, 1991.
Chemical Weapons: Physical Security for the U.S. Chemical Stockpile.
GAO/NSIAD-91-200. Washington, D.C.: May 15, 1991.
Chemical Warfare: DOD's Effort to Remove U.S. Chemical Weapons from
Germany. GAO/NSIAD-91-105. Washington, D.C.: February 13, 1991.
Chemical Weapons: Status of the Army's M687 Binary Program. GAO/NSIAD-
90-295. Washington, D.C.: September 28, 1990.
Chemical Weapons: Stockpile Destruction Delayed at the Army's Prototype
Disposal Facility. GAO/NSIAD-90-222. Washington, D.C.: July 30, 1990.
Chemical Weapons: Obstacles to the Army's Plan to Destroy Obsolete U.S.
Stockpile. GAO/NSIAD-90-155. Washington, D.C.: May 24, 1990.
[End of section]
Footnotes:
[1] The Convention on the Prohibition of the Development, Production,
Stockpiling and Use of Chemical Weapons and on Their Destruction is
commonly known as the Chemical Weapons Convention.
[2] In 1996, Congress directed DOD to evaluate the use of alternative
technologies and suspended incineration planning activities at two of
the nine storage sites with assembled weapons--Blue Grass Chemical
Agent-Destruction Pilot Plant, Blue Grass Chemical Activity, Kentucky,
and Pueblo Chemical Agent-Destruction Pilot Plant, Pueblo Chemical
Depot, Colorado. See Pub. L. No. 104-201, § 142 (1996) and Pub. L. No.
104-208, § 8065 (1996). In addition, Congress directed that those two
sites be managed in a program independent from the Army's Chemical
Demilitarization Program and report directly to DOD instead of to the
Army. See Pub. L. No. 105-261, § 142 (1998). The two sites are managed
under a separate program known as the Assembled Chemical Weapons
Alternatives program, and are not the focus of this report.
[3] The U.S. deadline for 100 percent destruction of the unitary
stockpile was extended from April 29, 2007, to April 29, 2012. The CWC
allows an extension of up to 5 years, and an extension was granted to
the United States at its request in December 2006. For purposes of this
report, we refer to the extended CWC deadline of April 29, 2012, as the
U.S. treaty deadline, unless otherwise indicated.
[4] A Nunn-McCurdy unit cost breach occurs when a major defense
acquisition program experiences an increase of at least 15 percent in
program acquisition unit cost or procurement unit cost above the unit
costs in the current APB. Unit cost reporting is required by 10 U.S.C.
§ 2433. For programs with unit cost increases of at least 25 percent, a
certification by the Secretary of Defense is required. Certification
responsibility has been delegated to the Under Secretary of Defense for
Acquisition, Technology, and Logistics, who, by law, must certify that
(1) such an acquisition program is essential to national security; (2)
there are no alternatives to such an acquisition program, which will
provide equal or greater military capability at less cost; (3) the new
estimates of the program acquisition unit cost or procurement unit cost
are reasonable; and (4) the management structure for the acquisition
program is adequate to manage and control program acquisition unit cost
or procurement unit cost. See 10 U.S.C. § 2433 (b)(2)(A) (2007).
[5] Total program cost estimates exceeded APBs in 2001 and again in
2005. On May 2, 2002, the Under Secretary of Defense for Acquisition,
Technology, and Logistics--the Defense Acquisition Executive--
submitted a Nunn-McCurdy breach certification to Congress for the
Chemical Demilitarization Program as required by 10 U.S.C. § 2433. The
unit cost breach was due primarily to (1) revised processing rates; (2)
schedule extensions; (3) new environmental regulations; (4) worse-than-
expected condition of the stockpile; (5) increase in equipment, labor
rates, and construction costs; and (6) higher emergency preparedness
costs. On May 6, 2005, the Under Secretary of Defense for Acquisition,
Technology, and Logistics again certified to Congress that the program
acquisition unit costs for the Chemical Demilitarization Program had
increased 41 percent (in then-year dollars) due primarily to more
realistic schedule estimates associated with operations and closure
phases, environmental regulatory compliance, and concept/design
maturation. Program acquisition unit costs for the Newport Chemical
Agent Disposal Facility, Newport Chemical Depot, Indiana, program
increased by 52 percent (in then-year dollars) due primarily to more
realistic schedule estimates associated with the nerve agent
neutralization process and secondary hazardous waste generation.
[6] The Chemical Stockpile Elimination Project includes the program's
four operating incineration sites and two bulk-agent-only
neutralization sites.
[7] These four incineration sites; the neutralization site at Newport
Chemical Agent Disposal Facility, Newport Chemical Depot, Indiana; and
the two closed sites that were at Johnston Atoll in the Pacific Ocean
and Aberdeen Chemical Agent Disposal Facility, Maryland, are managed
for the Army by the Chemical Materials Agency, which was formerly known
as the Program Manager for Chemical Demilitarization.
[8] GAO, Chemical Weapons: Sustained Leadership, Along with Key
Strategic Management Tools, Is Needed to Guide DOD's Destruction
Program, GAO-03-1031 (Washington, D.C.: Sept. 5, 2003).
[9] GAO, Defense Acquisitions: DOD Wastes Billions of Dollars through
Poorly Structured Incentives, GAO-06-409T (Washington, D.C.: Apr. 5,
2006).
[10] GAO, Defense Acquisitions: DOD Has Paid Billions in Award and
Incentive Fees Regardless of Acquisition Outcomes, GAO-06-66
(Washington, D.C.: Dec. 19, 2005).
[11] The Office of Management and Budget (OMB) defines EVM as a project
management tool that effectively integrates the project scope of work
with cost, schedule, and technical performance elements for optimal
project planning and control. Beginning in August 2005, OMB
specifically required federal agencies and contractors to use EVM
systems on all new capital assets. See Office of Management and Budget,
Capital Programming Guide (Washington, D.C.: 2006). Also, DOD requires
the use of EVM on all programs greater than $20 million.
[12] GAO's prior recommendations are found in GAO-03-1031, p. 26.
[13] GAO, Cost Assessment Guide: Best Practices for Estimating and
Managing Program Costs,GAO-07-1134SP (Washington, D.C.: July 2007,
Exposure Draft).
[14] The Department of Defense Authorization Act for Fiscal Year 1986,
Pub. L. No. 99-145, § 1412(a) (1985).
[15] Nerve agents are the most toxic and rapidly acting of known
chemical warfare agents. GB, also known as sarin, and VX are human-made
lethal chemical warfare nerve agents that affect the nervous system by
interfering with the signals sent from the brain to vital organs.
[16] Blister agents include mustard, which is a type of chemical
warfare agent and powerful irritant that causes blistering of the skin
and mucous membranes on contact.
[17] Pub. L. No. 104-201, § 142 (1996).
[18] Since the 1930s, the Army and the chemical industry have used ton
containers to store and ship bulk chemicals, including chemical agent.
Equivalent in length and diameter to two stacked 55-gallon drums, a ton
container weighs approximately 1,600 pounds and measures nearly 7 feet
in length.
[19] The stockpile at the Newport Chemical Depot, Indiana, is the first
U.S. stockpile containing VX that will be destroyed by using
neutralization--a process that mixes hot water and sodium hydroxide (a
caustic chemical) with VX to change the chemical composition to a less
toxic form. The resulting by-product is a liquid wastewater commonly
referred to as hydrolysate that consists mostly of water but also has a
caustic component and organic salts that need further treatment to meet
CWC requirements and to meet federal and state environmental
requirements for disposal. The hydrolysate is being stored in
intermodal storage containers on-site until a post-treatment plan can
be implemented. For additional information on the disposal of
hydrolysate, see GAO, Chemical Demilitarization: Actions Needed to
Improve the Reliability of the Army's Cost Comparison Analysis for
Treatment and Disposal Options for Newport VX Hydrolysate, GAO-07-240R
(Washington, D.C.: Jan. 26, 2007).
[20] Pub. L. No. 104-201, § 142 (1996).
[21] Pub. L. No. 105-261, § 142 (1998).
[22] The CWC implementing legislation provides the statutory authority
for domestic compliance with the convention's provisions. See Pub. L.
No. 105-277, Division I (1998).
[23] This report solely focuses on the weapons the convention defines
as category 1, which are the most dangerous chemicals in the stockpile.
[24] In 2004, DOD split the Chemical Demilitarization Program into
three major defense acquisition programs: (1) all CMA chemical agent
destruction sites, except for the Newport site; (2) the Newport site;
and (3) the ACWA program.
[25] GAO-03-1031.
[26] Pub. L. No. 108-375 § 931 (2004) directed the Under Secretary of
Defense for Acquisition, Technology, and Logistics and the Secretary of
the Army to jointly prepare a strategic plan for future activities for
the destruction of the U.S. stockpile of lethal chemical agents and
munitions. Section 931 further directed that the plan include, at a
minimum, realistic budgeting for stockpile destruction and related
support programs, contingency planning for foreseeable or anticipated
problems, and a management approach and associated actions that address
compliance with the obligations of the United States under the CWC
treaty and that take full advantage of opportunities to accelerate
destruction of the stockpile.
[27] The performance plan applies to both CMA and the ACWA portions of
the demilitarization program. It was the program's first plan and was
issued jointly by both organizations.
[28] GAO, Agency Performance Plans: Examples of Practices That Can
Improve Usefulness to Decisionmakers, GAO/GGD/AIMD-99-69 (Washington,
D.C.: Feb. 26, 1999).
[29] The Probability of Program Success Report is designed to improve
the Army's ability to accurately assess the program's probability of
success, and clearly and concisely represent that success probability
to leadership.
[30] See, for instance, GAO, Executive Guide: Effectively Implementing
the Government Performance Results Act, GAO/GGD-96-118 (Washington,
D.C.: June 1996).
[31] OMB's PART was developed to assess and improve program performance
so that the federal government can achieve better results. A review
using this tool helps identify a program's strengths and weaknesses to
inform funding and management decisions aimed at making the program
more effective. The tool, therefore, looks at all factors that affect
and reflect program performance, including program purpose and design;
performance measurement, evaluations, and strategic planning; program
management; and program results. Because the tool includes a consistent
series of analytical questions, it allows programs to show improvements
over time, and allows comparisons between similar programs. See Office
of Management and Budget, Program Assessment Rating Tool (PART),
[hyperlink, http://www.whitehouse.gov/omb/part/].
[32] GAO/GGD/AIMD-99-69.
[33] GAO-07-1134SP.
[34] A histogram is a graphical display of tabulated frequencies.
[35] The OSD CAIG has specific responsibility for providing independent
cost estimates when requested by the Under Secretary of Defense for
Acquisition, Technology, and Logistics in support of major milestone
decision points (Milestone B, Milestone C, or the full-rate production
decision review). Overall, CAIG serves as the principal advisory body
to the Milestone Decision Authority on all matters concerning an
acquisition program's life cycle cost.
[36] Defense Contract Management Agency, Department of Defense Earned
Value Management Implementation Guide (Washington, D.C.: Apr. 7, 2005).
Also see DOD Memorandum, Revision to DOD Earned Value Management Policy
(Washington, D.C.: Mar. 7, 2005), and Office of Management and Budget,
Capital Programming Guide (Washington, D.C.: 2006).
[37] EVM is a process required by OMB that can help program managers
track program progress by using the value of work done as a basis for
estimating the cost to compete the project.
[38] Defense Contract Management Agency, Department of Defense Earned
Value Management Implementation Guide.
[39] GAO-07-1134SP.
[40] Defense Contract Management Agency, Department of Defense Earned
Value Management Implementation Guide.
[41] In its technical comments on a draft of this report, DOD expressed
concern about our statement that the EVM system used at each site may
not be reliable. As a result, the department stated that it will
request the Defense Contract Management Agency to conduct an audit of
the CMA EVM system beginning no later than June 30, 2008.
[42] The systems contractor and government have identified work that
(authorized but unpriced) is negotiated and added to the contract price
annually.
[43] Defense Contract Management Agency, Department of Defense Earned
Value Management Implementation Guide. Also, see GAO-07-1134SP.
[44] In its technical comments on a draft of this report, DOD expressed
concern about our statement that shortcomings in the underlying (EVM)
data include invalid and unstable performance baselines. DOD stated
that the issues with the contract baselines are issues of program
management or risk, not issues of EVM data or process reliability. DOD
stated that this problem has been ameliorated by the recent completion
of life cycle contract negotiations at three of the five sites covered
by our report (Anniston, Pine Bluff, and Umatilla). The contract
negotiations for Newport and Toole are ongoing and should be completed
by June 30, 2008. The department stated that it will request the
Defense Contract Management Agency to conduct an audit of the CMA EVM
system beginning no later than June 30, 2008.
[45] "Comprehensive" means that the cost estimate is complete and
accounts for all possible costs. Comprehensive cost estimates
completely define the program, reflect the current schedule, and are
technically reasonable. See GAO-07-1134SP.
[46] Johnston Atoll was closed in December 2003 and Aberdeen was closed
in May 2007.
[47] The systems contractor and government have identified work that
(authorized but unpriced) is negotiated and added to the contract price
annually.
[48] The contracts use Operating and Maintenance funds, which are
available for obligation for 1 fiscal year. This creates the need to
negotiate the contracts on a yearly basis, since each fiscal year's
funds are appropriated.
[49] GAO-07-1134SP.
[50] Defense Contract Management Agency, Department of Defense Earned
Value Management Implementation Guide.
[51] GAO-07-1134SP.
[52] In its technical comments on a draft of this report, DOD stated
that the Army agrees that there is usually more work that is being
considered and not yet in the contract baseline. The Army did not agree
that this situation affects the EVM data. DOD stated that the Army uses
the data to determine current variance and trend information. The data
are also used by the systems contractors and CMA program office along
with estimates of additional future work to evaluate the life cycle
schedule and costs. The department stated that it will request the
Defense Contract Management Agency to conduct an audit of the CMA EVM
system beginning no later than June 30, 2008.
[53] CMA had previously incentivized schedule as a distinct performance
measure in fiscal year 2002 at all four incineration sites and
additionally in fiscal years 2000 and 2001 at Tooele.
[54] GAO-06-66.
[55] The initial actual percentages of the base and award fees for each
of these contracts were originally established through the full and
open competitions conducted for each of these contracts. Each
contractor's proposal included a percentage for both the base fee and
the award fee, which the government evaluated as part of the contract
selection process. With the exception of one systems contractor, all
the contractors proposed a total fee of 10 percent with up to 3 percent
proposed for the base fee and the remaining 7 percent allocated to
award fee. The one exception is Newport where the contractor proposed a
base fee of 3 percent and an award fee of 3 percent.
[56] However, these incentives did not keep the Newport contract on
schedule. The systems contractor missed the target completion date for
the third of its four performance incentive milestones and the program
was delayed by over a year. According to DOD, the failure to meet this
milestone was due to unforeseen technical difficulties and could not
have been ultimately influenced by any type of contractual language.
[57] Schedule was a distinct performance measure in the Tooele award
fee plan for fiscal years 2000 and 2001. In 2002, it was a distinct
performance measure in the award fee plans at each of the incineration
sites.
[58] CMA officials noted they were acting in accordance with Congress's
mandate that the program shall provide for the maximum protection of
the workforce, the public, and the environment. See 50 U.S.C. 1521
(c)(1)(A) (2007).
[59] See the Award Fee Review Board Report for the Anniston Chemical
Agent Disposal Facility, Evaluation Number 17, for the evaluation
period from April 1, 2005, to September 30, 2005.
[60] See the Award Fee Evaluation Board Report for the Umatilla
Chemical Agent Disposal Facility, First Period, Fiscal Year 2005, for
the period of performance from October 1, 2004, to March 31, 2005.
[61] Memorandum from the Under Secretary of Defense for Acquisition,
Technology, and Logistics on Chemical Demilitarization Program
Strategic Governance (Washington, D.C.: Jan. 6, 2006).
[62] GAO-06-66 and GAO, NASA Procurement: Use of Award Fees for
Achieving Program Outcomes Should Be Improved, GAO-07-58 (Washington,
D.C.: Jan. 17, 2007).
[63] Memorandum from the Under Secretary of Defense for Acquisition,
Technology, and Logistics on Award Fee Contracts (FAR 16, DFARS 215,
DFARS 216) (Washington, D.C.: Mar. 29, 2006). In addition, the National
Defense Authorization Act for Fiscal Year 2007 directed DOD to issue
guidance, with detailed implementation instructions, to ensure that all
new contracts using award fees link such fees to acquisition outcomes,
which are defined in terms of program cost, schedule, and performance.
See Pub. L. No. 109-364, § 814 (2006). Furthermore, in an April 2007
memorandum, DOD issued further award fee guidance stating that it is
the policy of the department that for new solicitations issued on or
after August 1, 2007, objective criteria be utilized, whenever
possible, to measure contract performance. If it is determined that
objective criteria do not exist, then the head of the contracting
activity must sign a determination and finding that "the work to be
performed is such that it is neither feasible nor effective to devise
predetermined objective incentive targets applicable to cost, technical
performance, or schedule."
[64] See GAO-06-66 and GAO-07-58.
[65] Pub. L. No. 109-364 (2006).
[66] Chemical Materials Agency, Incentive Agreement for Completion of
Destruction Operations and Facility Closure Activities at Chemical
Stockpile Disposal Facilities (Washington, D.C.: May 19, 2007).
[67] Pub. L. No. 109-364 (2006).
[68] Pub. L. No. 109-364, § 923 (a)(2) (2006).
[69] Pub. L. No. 109-364, § 923 (b)(2) (2006).
[70] Chemical Materials Agency, Incentive Agreement for Completion of
Destruction Operations and Facility Closure Activities at Chemical
Stockpile Disposal Facilities (Washington, D.C.: May 19, 2007).
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "Subscribe to Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Gloria Jarmon, Managing Director, JarmonG@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
