United States Coast Guard
Improvements Needed in Management and Oversight of Rescue System Acquisition Gao ID: GAO-06-623 May 31, 2006Search and rescue--one of the United States Coast Guard's (USCG) oldest missions and highest priorities--involves minimizing loss of life, injury, and property damage by aiding people and boats in distress. In September 2002, USCG contracted to replace its search and rescue communications system--installed in the 1970s--with a new system known as Rescue 21. However, the acquisition and initial implementation of Rescue 21 has resulted in significant cost overruns and schedule delays. GAO's objectives in reviewing the Rescue 21 program were to (1) assess the reasons for the significant cost overruns and implementation delays; (2) evaluate the viability of the revised cost and schedule estimates; and (3) evaluate the impact of the implementation delays.
Key factors that contributed to Rescue 21 cost overruns and schedule delays were inadequacies in requirements management, project monitoring, risk management, contractor cost and schedule estimation and delivery, and executive-level oversight. Accordingly, the estimated total acquisition cost for Rescue 21 has increased from $250 million in 1999 to $710.5 million in 2005, and the timeline for achieving full operating capability has been delayed from 2006 until 2011. USCG officials agreed that improvements need to be made to the management of the Rescue 21 project, and they are taking steps to address some of these issues. USCG's current acquisition cost estimate of $710.5 million is not viable. Our analysis of contractor performance trends indicates that additional overruns will likely bring the total acquisition cost to $872 million, unless critical changes are made. Additionally, USCG's schedule is uncertain due to on-going contract item renegotiations and pending decisions regarding vessel asset tracking functionality. Finally, further cost increases may result from the variability of costs for tower preparation and construction. To improve its current cost and schedule estimates, USCG plans to complete an integrated baseline review after contract items are renegotiated; however, a date for completing this review has not been established. The delays in deploying Rescue 21 could affect sites awaiting modernization. For example, key functionality, such as improved direction finding and improved coverage, will not be available for a significant period of time, and legacy equipment may fail, requiring costly repairs and upgrades to address coverage gaps and other operational issues.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-06-623, United States Coast Guard: Improvements Needed in Management and Oversight of Rescue System Acquisition
This is the accessible text file for GAO report number GAO-06-623
entitled 'United States Coast Guard: Improvements Needed in Management
and Oversight of Rescue System Acquisition' which was released on June
1, 2006.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Committees:
United States Government Accountability Office:
GAO:
May 2006:
United States Coast Guard:
Improvements Needed in Management and Oversight of Rescue System
Acquisition:
Coast Guard's Rescue System:
GAO-06-623:
GAO Highlights:
Highlights of GAO-06-623, a report to congressional committees.
Why GAO Did This Study:
Search and rescue”one of the United States Coast Guard‘s (USCG) oldest
missions and highest priorities”involves minimizing loss of life,
injury, and property damage by aiding people and boats in distress. In
September 2002, USCG contracted to replace its search and rescue
communications system”installed in the 1970s”with a new system known as
Rescue 21. However, the acquisition and initial implementation of
Rescue 21 has resulted in significant cost overruns and schedule
delays.
GAO‘s objectives in reviewing the Rescue 21 program were to (1) assess
the reasons for the significant cost overruns and implementation
delays; (2) evaluate the viability of the revised cost and schedule
estimates; and (3) evaluate the impact of the implementation delays.
What GAO Found:
Key factors that contributed to Rescue 21 cost overruns and schedule
delays were inadequacies in requirements management, project
monitoring, risk management, contractor cost and schedule estimation
and delivery, and executive-level oversight (see table). Accordingly,
the estimated total acquisition cost for Rescue 21 has increased from
$250 million in 1999 to $710.5 million in 2005, and the timeline for
achieving full operating capability has been delayed from 2006 until
2011. USCG officials agreed that improvements need to be made to the
management of the Rescue 21 project, and they are taking steps to
address some of these issues.
USCG‘s current acquisition cost estimate of $710.5 million is not
viable. Our analysis of contractor performance trends indicates that
additional overruns will likely bring the total acquisition cost to
$872 million, unless critical changes are made. Additionally, USCG‘s
schedule is uncertain due to on-going contract item renegotiations and
pending decisions regarding vessel asset tracking functionality.
Finally, further cost increases may result from the variability of
costs for tower preparation and construction. To improve its current
cost and schedule estimates, USCG plans to complete an integrated
baseline review after contract items are renegotiated; however, a date
for completing this review has not been established.
The delays in deploying Rescue 21 could affect sites awaiting
modernization. For example, key functionality, such as improved
direction finding and improved coverage, will not be available for a
significant period of time, and legacy equipment may fail, requiring
costly repairs and upgrades to address coverage gaps and other
operational issues.
Table: Key Factors in Rescue 21 Cost Overruns and Schedule Delays:
Requirements management:
USCG did not follow a rigorous requirements management process and
testing revealed incomplete and poorly defined requirements.
Project monitoring:
USCG did not effectively use earned value management data to measure
performance and take corrective action on negative trends.
Risk management:
USCG did not always effectively mitigate and communicate risks.
Contractor cost and schedule estimation and delivery: The contractor
created a schedule that underestimated the time required to complete
key tasks, and development took longer than planned, which led to
delays in testing.
Executive-level oversight:
USCG stated that it had an executive-level oversight process that
included semiannual and key decision point reviews. However, there is
no evidence that these reviews of Rescue 21 occurred before 2005.
[End of Table]
What GAO Recommends:
GAO recommends that the Commandant of USCG ensure that executive-level
management oversees Rescue 21‘s progress toward cost and schedule
milestones and manages risks; establishes milestones to complete an
integrated baseline review; and develops revised cost and schedule
estimates. The Department of Homeland Security agreed with the
recommendations and has begun to implement them; however, the
department expressed concerns with selected aspects of the report.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-623].
To view the full product, including the scope and methodology, click on
the link above. For more information, contact David A. Powner at (202)
512-9286 or pownerd@gao.gov or Stephen L. Caldwell at (202) 512-9610 or
caldwells@gao.gov.
[End of Section]
Contents:
Letter:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Briefing Slides:
Appendix II: Comments from the Department of Homeland Security:
Appendix III: GAO Contacts and Staff Acknowledgments:
Abbreviations:
DHS: Department of Homeland Security:
EVM: earned value management:
NDRS: National Distress and Response System:
OMB: Office of Management and Budget:
USCG: United States Coast Guard:
United States Government Accountability Office:
Washington, DC 20548:
May 31, 2006:
The Honorable Harold Rogers:
Chairman:
The Honorable Martin Sabo:
Ranking Member:
Subcommittee on Homeland Security,
Committee on Appropriations:
House of Representatives:
The Honorable Olympia Snowe:
Chairman:
The Honorable Maria Cantwell:
Ranking Member:
Subcommittee on Fisheries and Coast Guard,
Committee on Commerce, Science and Transportation:
United States Senate:
The United States Coast Guard (USCG) is currently replacing its 30-
year-old search and rescue communications systems with a new system
known as Rescue 21. Rescue 21 will be used not only for search and
rescue, but it will support other USCG missions, including those
involving homeland security. However, Rescue 21 has had significant
cost overruns and schedule delays. As a result of these delays,
completion of Rescue 21 deployment nationwide--originally scheduled for
fiscal year 2006--has been delayed until fiscal year 2011, and its
total acquisition cost has risen from $250 million in 1999 to its
current estimate of $710.5 million.[Footnote 1] In addition to the cost
overruns and schedule delays, Rescue 21 may not be deployed with all of
the functionality originally promised. For example, functionality for
vessel asset tracking was not deployed as part of the initial operating
capability due to the system's inability to meet original requirements,
and USCG is now considering alternative strategies for this
functionality. Also, there has been a reduction in promised
improvements to limit communications gaps; originally, Rescue 21 was
intended to limit communications gaps to 2 percent--now the target is
less than 10 percent.
This report responds to your request that we (1) assess the reasons for
the significant implementation delay and cost overruns against the
original Rescue 21 proposal, (2) evaluate the viability of the USCG's
revised cost and implementation schedule, and (3) evaluate the impact
of Rescue 21's implementation delay on USCG's field units that are
awaiting the modernization of antiquated communications equipment.
On March 27, 2006, we provided your staff with a briefing on the
results of our study. The slides from that briefing are included in
this report as appendix I. The purpose of this report is to formally
publish the briefing slides and officially transmit our recommendations
to the Commandant of USCG.
To assess the reasons for the implementation delay and cost overruns of
Rescue 21, we analyzed current and previous program documents,
including current and previous acquisition plans, cost estimates,
status reports, schedules, promised functionality, earned value
management (EVM)[Footnote 2] data, and cost analyses. We also
interviewed key officials and contractors. To evaluate the viability of
the USCG's revised cost estimate and implementation schedule, we
analyzed program cost and schedule documents, cost estimating
methodologies, and changes to development and deployment plans. To
identify factors that could affect the program baseline in the future,
we assessed the prime contractor's performance related to cost and
schedule. We applied established earned value analysis techniques to
data from contractor cost performance reports that spanned a 26-month
period from October 2003 to December 2005 to show trends in cost and
schedule performance and to estimate the likely costs at the completion
of the prime contract. To evaluate the impact of Rescue 21's
implementation delay, we reviewed reports on legacy system failures and
anticipated failures and interviewed field officials responsible for
overseeing legacy system operations at 11 field units, the Maintenance
and Logistic Commands for the Pacific and Atlantic Areas, and Rescue 21
users at the two initial operating regions.
We performed our work at USCG headquarters and at selected field units
and communications centers from December 2005 through March 2006, in
accordance with generally accepted auditing standards.
In summary, we made the following major points in our briefing:
* Key factors that contributed to Rescue 21 cost overruns and schedule
delays were shortcomings in requirements management, project
monitoring, risk management, contractor cost and schedule estimation
and delivery, and executive-level oversight. Among the points we made
were that USCG did not effectively use EVM data to measure performance
and take corrective actions; USCG, while identifying, analyzing, and
prioritizing risks, did not always mitigate and communicate risks; and
USCG stated that it had an executive oversight process that included
semiannual and key decision point reviews. However, there is no
evidence that these reviews of Rescue 21 occurred before 2005. USCG
agreed that improvements need to be made in the management of the
Rescue 21 project, and it is taking steps to address some of these
issues.
* The USCG's current acquisition cost estimate of $710.5 million is not
viable: based on our analysis of contractor performance trends and the
renegotiation of 91 contract items for site deployment, we forecast a
cost overrun of approximately $161 million, which will likely bring
total acquisition costs to $872 million, unless critical changes are
made. Also, the USCG's schedule is uncertain because of the continuing
contract item renegotiations and decisions regarding vessel asset
tracking functionality. Finally, these additional factors may increase
costs and delay schedules: (1) the variability of costs for tower
preparation and construction and (2) site reorganization.[Footnote 3]
USCG realizes that it needs to improve its cost and schedule management
and has plans to complete an integrated baseline review for the
contract items as they are renegotiated; however, a date for completing
this review has not been established.
* The delays in deploying Rescue 21 could have several effects: (1) key
functionality, such as improved direction finding and improved
coverage, will not be available for a significant period of time; (2)
legacy equipment may fail, requiring costly repairs; and (3) legacy
equipment may require upgrades to address coverage gaps and other
operational issues. For example, USCG instituted a moratorium on
upgrades to the legacy system because Rescue 21 was to replace the
equipment. However, despite the moratorium, according to USCG
officials, some field units have upgraded their legacy systems to
address operational challenges. Upgrades such as these would result in
higher costs, especially at field units scheduled for implementation at
a later date.
Recommendations for Executive Action:
To more effectively manage the remaining development and deployment of
Rescue 21, we recommend that the Commandant of USCG ensure that USCG
executive-level management implements the following three
recommendations:
* Oversee the project's progress toward cost and schedule milestones
and management of risks.
* Establish a milestone to complete Rescue 21's integrated baseline
review, to include all renegotiated contract items.
* Use the results of this baseline review to complete a revised cost
and schedule estimate.
Agency Comments and Our Evaluation:
In providing written comments on a draft to this report, the Department
of Homeland Security's (DHS) Director of the Departmental GAO/Office of
the Inspector General Liaison Office concurred with our recommendations
and stated that USCG would continue to aggressively pursue the
recommendations in the GAO report in order to ensure that Rescue 21
provides critical life-saving capabilities and that the acquisition is
effectively managed. DHS stated that it was currently addressing our
recommendations and provided examples of efforts to date. Specifically,
DHS and USCG increased oversight with quarterly flag officer/vice
president-level program reviews and periodic DHS Joint Requirements
Council and Investment Review Board reviews in October 2005 and plan to
initiate monthly progress reviews of the project schedule beginning in
June 2006.
Regarding the cost of Rescue 21, DHS stated that it had recently
revised its acquisition program baseline. The revised total acquisition
cost for Rescue 21 is $730.2 million, which was achieved by adjusting
several performance parameters and terminating the contractor's option
to provide vessel subsystem components. DHS plans to achieve this
capability through other means. However, we continue to stress the
importance of our recommendation that calls for the completion of an
integrated baseline review. If performed correctly, an integrated
baseline review will provide insight into the risks associated with the
$730.2 million estimate and the extent to which the estimate is
achievable.
DHS agreed that USCG has responsibilities for managing schedule delays
and cost overruns in major acquisition projects and with our findings
regarding the contractor's inability to manage cost and schedule. DHS
stated that the root cause for the systems deployment delays and costs
overruns was poor estimates and failed performance by the contractor.
The department, however, raised concerns with four of our findings
regarding USCG management and oversight and provided us with some
additional information regarding these findings that did not fully
address our concerns. Therefore, we continue to believe that
shortcomings in USCG's management and oversight of Rescue 21
contributed to cost overruns and schedule delays. USGC concerns and our
response associated with each of the four areas follows.
First, DHS stated that our findings on requirements management are not
consistent with our previously reported 2003 findings that USCG did
have a process for managing requirements and it cited various
requirements management documents. Our current finding states that USCG
did not follow rigorous requirements management processes for Rescue 21
not that the USCG did not have a process and is therefore consistent
with our previous report. DHS also stated that testing revealed that
requirements were clearly defined. However, as stated in our briefing,
a USCG usability and operability assessment of Rescue 21 stated that
most of the operational advancements envisioned for the project were
not achieved, and these problems could have been avoided by including
user requirements in the contract. DHS's response also cites software
and hardware defects that can be attributed to shortcomings in
requirements management, providing further support for our finding that
USCG did not follow a rigorous requirements management process.
Second, DHS disagreed with our finding that they did not effectively
use EVM data to measure performance and take corrective actions.
Evidence showing USCG actions to address schedule delays from EVM data
was not provided to us during our review. On May 8, 2006, 6 weeks after
the briefing on March 27, 2006, DHS provided documentation showing that
some actions were taken by USCG in 2004. However, this is nearly a year
after the EVM data began showing schedule slippages in March 2003 and
after the Rescue 21 project missed its initial operating capability
date of September 2003. Further, DHS stated that the contractor
undertook a significant reorganization of its management structure in
response to its concerns related to cost and schedule performance.
Because this information was not mentioned to us during our discussions
with either USCG or the contractor during our review, on May 15, 2006,
we contacted both the contractor's Rescue 21 project manager and
contract manager to clarify the reorganization statement. These
individuals told us that changes in contracting personnel were due to
internal movement between ongoing contracts and did not constitute a
reorganization.
Third, DHS disagreed with our finding that risks were not always
mitigated and communicated. DHS stated that there were various reports
and meetings where risks were reported and discussed. Documentation for
some of these reports and meetings was provided to us on May 8, 2006.
Furthermore, as stated in the briefing, there were examples of risks
that were not effectively mitigated or communicated. The Human System
Interface (HSI) work group referred to in the agency's response was
established only in 2005, after Rescue 21 had shown significant HSI
issues in its final operational testing and evaluation phase. Effective
risk management would have identified and addressed risks earlier in
the system life cycle.
Finally, DHS disagreed with our finding that there was no evidence to
support its statement that there was executive-level management
oversight prior to 2005. During our review, DHS cited a list of
semiannual and key decision point reviews, dating from 2002 to 2005;
however, as stated in our briefing, it did not provide evidence of the
pre-2005 briefings. On May 8, 2006, DHS provided some evidence that
these meetings occurred; however, based on our review of the
documentation, it was unclear which executives attended these meetings
and, therefore, whether the appropriate executives were in attendance.
Furthermore, the department did not provide evidence showing executive
direction to mitigate risks or address problems. Finally, the
department stated that it had increased oversight in October 2005 by
establishing quarterly flag officer/vice president-level program
reviews and periodic DHS Joint Requirements Council and Investment
Review Board reviews. This additional oversight provides further
support for our finding on the lack of executive level oversight prior
to 2005. DHS also challenged our finding that executive-level
monitoring of the risk mitigation activities that resulted from the
2005 meetings did not occur, citing various monitoring activities that
occurred. However, it did not provide evidence of the monitoring
activities.
DHS also provided technical comments that were incorporated in this
report where appropriate. DHS's written comments are reprinted in
appendix II.
We are sending copies of this report to the Secretary of Homeland
Security, the Commandant of the United States Coast Guard, and other
interested parties. We will make copies available to others on request.
In addition, the report will be available at no charge on the GAO Web
site at [Hyperlink, http://www.gao.gov].
If you or your staff have any questions concerning this report, please
contact David A. Powner at (202) 512-9286 or by e-mail at
pownerd@gao.gov or Stephen L. Caldwell at (202) 512-9610 or by e-mail
at caldwells@gao.gov.
Signed by:
David A. Powner:
Director, Information Technology Management Issues:
Signed by:
Stephen L. Caldwell:
Acting Director, Homeland Security and Justice Issue:
[End of section]
Appendix I: Briefing Slides:
Management and Oversight Improvements Needed for Coast Guard's Rescue
21 Acquisition:
Briefing for:
the House Committee on Appropriations, Subcommittee on Homeland
Security:
and:
the Senate Committee on Commerce, Science and Transportation,
Subcommittee on Fisheries and Coast Guard:
March 27, 2006:
Briefing Overview:
Introduction:
Objectives, Scope, and Methodology Results in Brief:
Background:
Objective 1: Reasons for Cost Overruns and Schedule Delays Objective 2:
Revised Cost and Schedule Estimates:
Objective 3: Delay in Implementation Conclusions:
Recommendations:
Agency Comments:
Introduction:
Search and rescue is one of the United States Coast Guard's (USCG)
oldest missions and highest priorities. The search and rescue mission
includes minimizing the loss of life, injury, and property damage by
aiding people and boats in distress.
The National Distress and Response System (NDRS) is the legacy
communications component of USCG's search and rescue program. However,
the 30-year-old system has several deficiencies and is difficult to
maintain, according to USCG officials. In September 2002, the USCG
contracted with General Dynamics Decision Systems (General Dynamics) to
modernize and replace the NDRS with a system called Rescue 21.
In October 2005, due to significant cost increases and schedule delays
associated with the Rescue 21 program, the Chairman and Ranking Member
of the House Appropriations Subcommittee on Homeland Security requested
that we evaluate the Rescue 21 program.
Objectives, Scope, and Methodology:
As agreed, our objectives were to:
* assess the reasons for the significant implementation delay and cost
overruns against the original Rescue 21 proposal;
* evaluate the viability of the USCG's revised cost and implementation
schedule; and:
* evaluate the impact of Rescue 21's implementation delay on the USCG's
field units that are awaiting the modernization of antiquated
communications equipment.
To assess the reasons for the implementation delay and cost overruns of
Rescue 21, we analyzed Rescue 21 documents, including current and
previous acquisition plans, cost estimates, status reports, schedules,
promised functionality, earned value management (EVM)[Footnote 4] data,
and cost analyses. We also interviewed key officials and contractors
involved in the development and deployment of Rescue 21.
To evaluate the viability of USCG's revised cost estimate and
implementation schedule, we analyzed current program cost and schedule
estimates, cost estimating methodologies, and changes to development
and deployment plans. To identify factors that could affect the program
baseline in the future, we assessed the prime contractor's performance
related to cost and schedule. We applied established earned value
analysis techniques to data from contractor cost performance reports
that spanned a 26-month period, from October 2003 to December 2005, to
show trends in cost and schedule performance and to estimate the likely
costs at the completion of the prime contract.
To evaluate the impact of Rescue 21's implementation delay, we reviewed
reports on legacy system failures and anticipated failures as well as
documentation from USCG sites regarding problems associated with the
system. We also interviewed field officials responsible for overseeing
legacy system operations at 11 field units. We visited five of these
field units (Humboldt, California; San Francisco, California; New
Orleans, Louisiana; Mobile, Alabama; and Miami, Florida) and
interviewed officials by telephone at six other locations (Honolulu,
Hawaii; Guam, Territory of Guam; Detroit, Michigan; Valdez, Alaska;
Kodiak, Alaska; and Juneau, Alaska). These locations were selected
based on coverage gaps previously identified and their significance to
search and rescue and homeland security operations.
We also interviewed Maintenance and Logistic Commands for the Pacific
and Atlantic Areas and Rescue 21 users at the initial operating sites
at the Atlantic City and Eastern Shore communications centers. We
compared planned Rescue 21 functionality with what was actually
delivered.
We performed our work at USCG headquarters and selected field units and
communications centers from December 2005 through March 2006, in
accordance with generally accepted auditing standards.
Results in Brief:
Key factors that contributed to Rescue 21 cost overruns and schedule
delays were shortcomings in requirements management, project
monitoring, risk management, contractor cost and schedule estimation
and delivery, and executive-level oversight. USCG agreed that
improvements need to be made in these areas and is taking steps to
address these issues.
The Coast Guard's current acquisition cost estimate of $710.5 million
is not viable: our analysis of contractor performance trends indicates
that additional cost overruns will likely bring total acquisition costs
to $872 million unless critical changes are made. This includes a
significant number of contract items that have not been completed as
planned and must now be renegotiated. Also, USCG's schedule, which now
shows full operating capability in 2011--5 years behind the originally
scheduled date of 2006 is not finalized because of the continuing
contract item renegotiations and decisions regarding vessel asset
tracking functionality. Finally, these additional factors may increase
costs: (1) the variability of costs for tower preparation and
construction and (2) site reorganization.[Footnote 5] USCG realizes
that it needs to be more effective in managing costs and schedule and
has plans to complete an integrated baseline review for the contract
items as these are renegotiated; however, a date for completing this
review has not been established. In addition, USCG plans to begin
monthly reviews of the project schedule and to incrementally deploy to
future Rescue 21 sites.
The delays in deploying Rescue 21 could have several effects:
* key functionality, such as improved direction finding and improved
coverage, will not be available for a significant period of time;
* legacy equipment may fail, requiring costly repairs; and:
* legacy equipment may require upgrades to address coverage gaps and
other operational issues.
We are recommending that the Secretary of Homeland Security direct the
Commandant of the United States Coast Guard to ensure that executive-
level management oversees the project's progress toward cost and
schedule milestones and manages risks. In addition, we recommend that
executive-level management establish a milestone to complete its
integrated baseline review and revise its cost and schedule estimates.
In providing oral comments on a draft of this briefing, the Coast Guard
Assistant Commandant for Acquisition stated that the briefing fairly
represented Rescue 21 risks and ongoing activities to address program
management and other issues. He also stated that this briefing would be
used as input to take corrective actions to improve program management.
Background:
Rescue 21 Description:
The NDRS is USCG's primary VHF communications system, important to the
USCG's search and rescue program as well as its homeland security
mission. However, the USCG has reported that NDRS-much of which was
installed in the 1970s--has several critical deficiencies:
* numerous gaps in communications coverage-in some areas the Coast
Guard cannot hear calls from mariners in distress or communicate with
other USCG vessels;
* limited direction-finding capabilities and no digital selective
calling capabilities;[Footnote 6] no effective way to track USCG assets
that conduct search and rescue;
* limited interoperability with other federal and state communications
systems; and:
* no means for protected communication.
Rescue 21's primary functions are to provide USCG with communication
for maritime search-and-rescue functions-such as monitoring distress
calls from boaters and coordinating the response or rescue. It is
intended to replace NDRS, and will be a short-range communications
system reaching about 20 nautical miles. The system will include VHF-FM
radios, communications towers, and hardware and software at USCG sites
as well as on Coast Guard vessels. This system also, in addition to
search and rescue, provides support to homeland security missions.
USCG's acquisition directorate, under the Assistant Commandant for
Acquisition, is responsible for managing the Rescue 21 acquisition.
In September 2002, USCG awarded an indefinite delivery, indefinite
quantity[Footnote 7] contract to General Dynamics for developing and
implementing Rescue 21. The contract and original schedule called for
the following:
* two initial Coast Guard regions to operate the new system with full
functionality-referred to as initial operating capability-scheduled for
September 2003 and:
* continued deployment to additional locations through 2006.
According to the Coast Guard contract with General Dynamics, the Rescue
21 system is to:
* allow continuous, uninterrupted communications on Channel
16;[Footnote 8]:
* limit the number of communication coverage gaps to less than 10
percent in the United States (originally this was to less than 2
percent);
* provide direction finding and digital selective calling to locate
boaters; * allow USCG to track its mobile assets, such as vessels;
* allow communication with other federal and state systems;
* allow protected communication of sensitive information; and:
* archive voice and data communications and make them instantly
accessible.
A table showing the capabilities of NDRS and Rescue 21 follows.
Table: Comparison of Rescue 21 Capabilities with Legacy System:
[See PDF for Image]
Source: U.S. Coast Guard.
[End of table]
Background:
Rescue 21 Components:
Key components of Rescue 21-some of which appear in the following
diagram-are:
* communications centers (46) that contain operator workstations and
electronic equipment;
* antenna towers (322);
* other equipment and/or infrastructure that could be used on vessels
or remote facilities, including communication lines and radio
equipment; and:
* disaster recovery assets (not pictured). A disaster recovery asset is
a smaller antenna tower and equipment similar to a remote facility. It
is portable so that it can be moved to locations as needed.
Group communications center processes distress signal and directs
rescue:
High level site towers determine direction of signal:
Vessel sends distress signal:
[See PDF for Image]
Source: GAO analysis of U.S. Coast Guard data.
[End of figure]
Background:
Previous GAO Work:
In September 2003, we reported that USCG's original schedule for
reaching initial operating capability by September 2003 had been
postponed and that it had yet to finalize a new schedule.[Footnote 9]
This postponement was due, in part, to the development of the system
taking longer than planned.
We also determined that USCG postponed key tests and the initial
operating capability because of the delays in software development, and
we highlighted risks associated with its plan to compress and overlap
key tests as a result of the delays. In addition, schedules for key
tests and deliverables, including test plans, were still not complete.
We recommended that the Secretary of the Department of Homeland
Security direct the Commandant of the United States Coast Guard to:
* establish a new schedule for critical testing phases and initial
operating capability and * ensure that milestones were established for
completing test plans for formal qualifications testing, system
integration testing, and operational testing and evaluation and that
test plans address all requirements for the system.
Following our report, key test schedules were revised and milestones
and test plans were developed.
Rescue 21 Schedule Delays:
However, further delays occurred during testing. [Footnote 10] For
example,
Formal qualification testing, originally expected to begin in February
2003 and end in March 2003, actually began in January 2004 and ended in
June 2004--15 months behind schedule. At that time, only 4 of the 235
allocated requirements were validated and performance issues in several
areas, including fault management, archive/restore, and channel
performance, were found that required additional work. Formal
qualification testing was repeated in June 2004, at which time the
remaining 231 allocated requirements were validated.
System integration testing, originally expected to begin in April 2003
and end in May 2003, actually began in August 2004 and ended in
December 2004--19 months behind schedule. Additional software defects
were found during this time and USCG decided to have the contractor
concurrently fix the system integration testing problems while going
into the operational test and evaluation phase.
Operational test and evaluation, originally intended to begin in July
2003 and to end in August 2003, actually began in January 2005 and
ended March 2005-19 months behind schedule. Further software defects
were found, and USCG also requested additional changes to the human
systems interface based on operator comments. The changes to the human
systems interface were prioritized into three builds, the first of
which was required by USCG before it would approve initial operating
capability. The remaining two builds are being negotiated with the
contractor and represent additional development. Finally, it was
determined that the vessel portion of Rescue 21 did not meet
performance requirements.
Because of the delays in development and testing, the date for initial
operating capability was delayed from September 2003 to December 2005.
Full operating capability, which was originally scheduled for fiscal
year 2006, was delayed to fiscal year 2011.
Figure: Development and Implementation Schedule:
[See PDF for Image]
[End of Figure]
Rescue 21 Cost Growth:
Rescue 21's total acquisition cost estimate has increased from $250
million to $710.5 million over a 6-year period, as shown in the
following chart.
[See PDF for image]
Source: U.S. Coast Guard Acquisition Project baseline data.
[End of figure]
Rescue 21 Appropriations:
Rescue 21 has been appropriated a total of $382.9 million for
development since award of the contract in fiscal year 2002.
[See PDF for image]
Source: U.S. Coast Guard data.
[End of figure]
Rescue 21 Current Status:
Rescue 21 was deployed on December 20, 2005, with an initial operating
capability[Footnote 11] and is being used at two regions (Atlantic
City, New Jersey, and Chincoteague, Virginia). The deployment included
all the station functionality and equipment that:
* allowed continuous, uninterrupted communications on Channel 16;
* limited the number of communication coverage gaps;
* provided direction finding and digital selective calling to locate
boaters;
* allowed radio communication with other federal and state officials;
* allowed protected communication of sensitive information; and:
* archived voice and data communications and made them accessible in
real time.
However, functionality for vessel asset tracking and data transfer plus
an expanded vessel communications suite was not delivered as planned.
This was deferred because the contractor could not meet the original
contract requirements for simultaneous communications. In January 2006,
a stop work order was issued on the vessel tracking and data transfer
functionality for 90 days. USCG and the Department of Homeland Security
(DHS) are currently determining whether Rescue 21 should complete
development of the vessel functionality or consider other sources for
that functionality.
The current schedule calls for the deployment of Rescue 21 to four low-
rate initial production sites during 2006.[Footnote 12] In addition,
according to USCG officials, they are currently revalidating
implementation plans for the first 15 of the remaining 40 sites. These
first 15 of the remaining 40 sites are approved for deployment and are
currently scheduled to be completed by fiscal year 2009. Completion of
the remaining 25 sites with full operating capability is currently
scheduled for fiscal year 2011.
However, some development activity remains. First, there are two
remaining builds of enhancements to the human systems interface, with
the second build scheduled for completion in October 2006 and the third
in March 2007. Second, a decision is pending on how to implement vessel
asset tracking and data transfer functionality.
Objective 1:
Reasons for Cost Overruns and Schedule Delays:
Key factors that contributed to Rescue 21 cost overruns and schedule
delays were shortcomings in requirements management, project
monitoring, risk management, contractor cost and schedule estimation
and delivery, and executive-level oversight. USCG agreed that
improvements need to be made in these areas and is taking steps to
address these issues.
To effectively manage major IT programs, organizations should use sound
acquisition and management practices to minimize risks and thereby
maximize a project's ability to be delivered on time, within budget,
and with promised functionality. Such practices have been identified by
leading organizations, such as the Software Engineering Institute, the
Chief Information Officer's Council, and in our prior work analyzing
best practices in industry and government.
Requirements management:
Requirements management includes processes for eliciting, documenting,
verifying and validating, and managing requirements through the system
life cycle.
The Coast Guard did not have a rigorous requirements management
process. Although the USCG developed high-level requirements for Rescue
21, it relied solely on the contractor to manage these requirements.
According to USCG acquisition officials, they took this approach
because of the performance-based contract vehicle. However, our
experience in reviewing major systems acquisitions has shown that it is
important for government organizations to exercise strong leadership in
managing requirements, regardless of the contracting vehicle.
In addition to not effectively managing requirements, testing revealed
numerous problems linked back to incomplete and poorly defined user
requirements. For example:
* A USCG usability and operability assessment of Rescue 21 stated that
most of the operational advancements envisioned for the system were not
achieved and concluded that these problems could have been avoided by
including user requirements in the contract.
Requirements management (continued):
* A key requirement stated, "Provide a consolidated regional geographic
display." The contractor provided a capability based on that statement,
but during testing the USCG operators felt the maps did not display
sufficient detail. Such discrepancies led to an additional statement of
work defining required enhancements to the system interface (e.g.,
screen displays).
After these initial problems, USCG recognized the need for an orderly
process to review changes that could have a significant impact to cost,
schedule, and performance. Specifically, in October 2005, USCG
established a Configuration Control Board to control changes to Rescue
21 requirements.
Project monitoring:
A key project monitoring activity uses earned value management (EVM)
techniques to track the progress of work completed against project cost
and schedule estimates.
USCG did not effectively use EVM data in order to measure performance
and take corrective action. The Coast Guard has received EVM cost
performance reports that identified problems on a regular basis from
Booz Allen Hamilton.[Footnote 13] However, USCG did not take action on
these reports. For example, the March 2003 report indicated that, given
the number of slipped tasks and small remaining time, it was unlikely
that the contractor would be able to recover the schedule deadlines;
the September and October 2003 reports described the contractor's
performance as rapidly worsening. Nevertheless, USCG did not take
action in response to these reports.
Risk management:
An effective risk management process identifies potential problems
before they occur, so that risk-handling activities may be planned and
invoked as needed across the life of the product and project in order
to mitigate adverse impacts on achieving objectives. Key activities
include identifying and analyzing risks, assigning resources,
developing risk mitigation plans and milestones for key mitigation
deliverables, briefing senior-level managers on high-priority risks,
and tracking risks to closure.
USCG‘s management process includes identifying, analyzing, and
prioritizing risks; establishing mitigation strategies; and assigning
responsibility for addressing risks. However, USCG did not always
mitigate and communicate risks. For example:
In 2003, USCG identified software development as a high risk to Rescue
21‘s schedule. However, despite early acknowledgment of this risk, it
was not mitigated and ultimately became one of the primary drivers for
the increase in costs.
USCG did not communicate risks to partners when applicable. During our
review, the contractor stated that it was unaware of the items in the
USCG risk list that related to the contractor‘s performance.
Contractor cost and schedule estimation and delivery:
The contractor underestimated the time for development and testing of
Rescue 21 and established an integrated project schedule that it could
not meet.
* Development took longer than planned, which led to delays in testing.
Subsequently, testing phases were not completed on schedule.
* Additionally, the initial estimate for completing a regional site
deployment was 12 months. The new estimate of 24 months is considered
to be more realistic for executing the same tasks.
The contractor did not always provide personnel in a timely manner or
with the knowledge and skills required to complete the project.
* According to USCG information, it took longer than expected to obtain
developers and subcontractors for Rescue 21.
* According to USCG documentation, support personnel involved in
initial site surveys were minimally qualified and had limited civil
engineering and environmental analysis skills.
Executive oversight and governance:
Office of Management and Budget and GAO guidance call for agencies to
establish IT investment management boards comprised of key executives
to regularly track the progress of major systems acquisitions. These
boards should have documented policies and procedures for management
oversight of IT projects and systems and should be able to adequately
oversee the project's progress toward cost and schedule milestones and
their risks. The board should also employ early warning systems that
enable it to take corrective actions at the first sign of cost,
schedule, and performance slippages.
USCG told us that it had an executive oversight process that included
semi-annual and key decision point reviews. However, there is no
evidence that these reviews of Rescue 21 occurred before 2005.
Rescue 21 oversight meetings commenced in 2005, but executive
management did not take action in response to the risks and problems
presented. For example, briefings given to the USCG Acquisition Review
Council and the Department of Homeland Security (DHS) Investment Review
Board for key decision point 3 approval identified numerous high-level
and moderate risks and DHS officials requested additional actions, but
executive-level monitoring to ensure the adequacy and completion of
risk mitigation activities did not occur.
Recent Efforts by USCG to Improve Rescue 21 Management:
USCG has recently begun to address several of the key factors that led
to cost overruns and schedule slippages.
According to the Assistant Commandant for Acquisition, the USCG plans
to improve the capabilities of the Acquisition Office by obtaining more
technical assistance to help manage requirements, monitor project
performance, and mitigate risks, among other things. For example, USCG
intends to begin monthly reviews of the project schedule in June 2006
to track actual against planned milestones and to use earned value
management data to improve its oversight of cost and schedule
performance.
Regarding contractor performance, USCG stated that it was difficult to
manage the project effectively from headquarters and that the distance
between USCG and the contractor delayed decisions. As a result, in
November 2005, USCG established a Project Resident Office adjacent to
the contractor's facility. This office's responsibilities include
overseeing nationwide regional site implementation; monitoring
contractor performance; managing risks; establishing closer
communication and coordination with the contractor to minimize contract
changes and control cost growth; and monitoring shipping and delivery
of Rescue 21 equipment.
USCG has also increased the visibility of Rescue 21 at the executive
level by establishing quarterly meetings between the USCG Assistant
Commandant for Acquisition and the vice president of the contractor.
These meetings may facilitate greater executive-level involvement in
Rescue 21.
Revised Cost and Schedule Estimates:
Coast Guard's current acquisition cost estimate of $710.5 million is
not viable. Our analysis of contractor performance trends indicates
that there will likely be additional cost overruns that will bring the
total acquisition cost to $872 million unless critical changes are
made. This includes a significant number of contract items that have
not been completed as planned and must now be renegotiated. Also,
USCG's schedule is uncertain and will likely change due to the
continuing contract item renegotiations and decisions regarding vessel
functionality. Negotiations for contract items for the first 15 full-
rate production regions continues and is expected to be completed by
June 2006. Subsequently, contract items to deploy Rescue 21 to the
remaining 25 full-rate production regions must also be renegotiated.
Finally, additional factors may increase costs, such as the variability
of costs for tower preparation and construction and site
reorganization.[Footnote 14]
As of June 2005, the current acquisition cost estimate to reach full
operational capability in 2011 for Rescue 21 is $710.5[Footnote 15]
million.
Based on our analysis of contractor performance trends (using
contractor EVM data [Footnote 16]), we forecast a cost overrun of
approximately $161 million. The total acquisition costs could be as
high as $872 million.
The current cost estimate does not take into account contract items
that are being renegotiated. There are 91 contract items for site
deployment that have expired and have to be renegotiated. According to
USCG, based on their experience with the cost overruns at the IOC
sites, costs for each of the contract item listed below will increase:
* regional implementation plans:
* regional infrastructure preparation:
* regional system installation:
Coast Guard officials agreed with our analysis of projected cost
overruns.
USCG's schedule is also uncertain and will likely change.
The current project schedule is uncertain because of the 91 contract
items that must be renegotiated. USCG estimates that this renegotiation
will be completed for its first 15 full-rate production regions in June
2006. Contract items for the remaining full-rate production regions-
intended to be addressed in two additional groups-are to be negotiated
at a later date.
There is also uncertainty regarding Rescue 21 vessel functionality.
Initial plans for Rescue 21 included functionality to allow USCG to
track its vessels and to allow data transfer between vessels and group
communications centers plus an expanded vessel communications suite.
This functionality is now being deferred while USCG and DHS decide
whether to complete development of its contractor-developed solution or
use a different solution such as the Automatic Identification
System[Footnote 17] for asset tracking and data transfer on vessels.
This decision is currently expected by the end of April 2006.
Additional issues may also contribute to further cost overruns and
schedule delays.
Variability in infrastructure preparation costs. According to USCG
officials, there is significant variation in total costs for antenna
towers. Additional costs can be incurred in building a tower and
significant schedule delays can occur. Further, environmental concerns
may further delay tower construction. For these reasons, there is
significant potential for cost overruns and schedule slippages in tower
construction.
Site reorganization. New site command centers are to be established
throughout USCG, based on a new standard organization model. There is
no guarantee that current sites will have the same functions that they
have now. For example, the Eastern Shore group will become a site with
the communication center at Hampton Roads. The equipment at the Eastern
Shore GCC will be moved to Hampton Roads and sites will be reconfigured
to the new arrangement. Therefore, rework may be needed when the site
centers are finalized.
Cost and Schedule Management:
USCG has plans to improve its cost and schedule management in the
future. For example:
USCG intends to conduct an integrated baseline review of the
contractor's cost and schedule estimates after it completes
renegotiation of expired contract items for the deployment to the first
15 full-rate production regions. This baseline review is currently
expected to begin in the June 2006 time frame and is supposed to
validate the accuracy of the Rescue 21 schedule. However, there is no
completion date for the baseline review. USCG also intends to complete
another baseline review of the schedule once the contracts for the
remaining 25 full-rate production sites are renegotiated.
USCG intends to conduct full-rate production site deployment
incrementally as it expects this gradual deployment to improve its
ability to manage cost and schedule. The contracts for the first group
of 15 regions are currently being negotiated; contracts for the second
set of 11 regions and third set of 14 regions will be addressed
afterwards.
Delay in Implementation:
The effects of the delays in deploying Rescue 21 include (1) key
functionality, such as improved direction finding and improved
coverage, will not be available; (2) potential failure of legacy
equipment requiring costly repairs; and (3) potential for further
upgrades to the legacy system and resulting costs, despite a moratorium
on upgrades.
Key functionality to production sites will be delayed. Originally,
Rescue 21 was to be deployed nationwide by fiscal year 2006. The
current schedule shows full deployment in fiscal year 2011, five years
behind schedule. This will result in a significant implementation delay
for many field units. Personnel at the two initial operating capability
stations acknowledged that Rescue 21 provides significantly improved
functionality compared to the legacy system. For example, according to
these personnel, Rescue 21's direction finding capability-which allows
USCG to locate boaters in distress-is significantly more accurate as
well as more reliable than the legacy system. As a result, until Rescue
21 is deployed, field units with the legacy system will continue to be
at risk of performing larger and potentially more costly searches due
to limited direction finding capabilities.
Until Rescue 21 is deployed, field units will likely continue to
experience communication coverage gaps, which limits their ability to
hear boaters in distress. However, the impact of coverage gaps varies
depending on the availability of local partners to conduct search and
rescue, or vessels to relay distress calls. Personnel at one field unit
explained that they rely on a marine patrol unit to conduct searches in
a coverage gap area, whereas at another site, the USCG field unit is
the only resource available to conduct search and rescue operations.
Nevertheless, until Rescue 21 is fully deployed, missed mayday calls-
and the potential for lives lost-creates uncertainty about the legacy
system's reliability and reduces operators' confidence in the system.
Potential for failure of legacy equipment requiring costly repairs. The
legacy system is about 30 years old and is difficult to maintain. Until
Rescue 21 is deployed, legacy equipment will continue to be at high
risk for failure; these failures could result in costly repairs. Some
USCG maintenance officials questioned whether legacy equipment will
continue to function until Rescue 21 is fully deployed, especially in
field units scheduled for later implementation. Field units could also
experience higher costs resulting from the need to replace equipment
and reengineer parts that are no longer manufactured. In addition,
maintenance expertise needed to repair the antiquated equipment is
limited and, as a result, repairs are not always timely.
Further upgrades to the legacy system to address operational issues
could occur, despite a moratorium on upgrades. The USCG instituted a
moratorium on upgrades to the legacy system because Rescue 21 was to
replace the equipment. However, with units facing delays of up to 5
years, USCG may decide that the risk to its operations is sufficiently
high that it should upgrade or install new legacy equipment rather than
wait to receive the new system. In certain cases, this has already
occurred, despite the moratorium. For example, USCG officials reported
that they upgraded a console at one unit to mitigate operational
challenges, and installed a new antenna at a second unit to address
coverage gaps. Additional upgrades such as these would result in higher
costs, especially at field units scheduled for later implementation.
Conclusions:
Although USCG plans to improve its capabilities in a number of areas to
better manage the remaining development and deployment of Rescue 21 by
establishing executive-level meetings with the prime contractor, it is
imperative that key executives aggressively monitor costs and schedule
performance and mitigate risks so the remaining development and
deployment of Rescue 21 is effectively managed and delivered.
USCG's current acquisition cost estimate is not viable and its schedule
is uncertain and likely to change. USCG has plans to complete an
integrated baseline review, but it has not established a target
completion date. Without the results of this review, the cost and
schedule estimate cannot be completed.
If deployment of Rescue 21 continues to be delayed, Coast Guard sites
and services will be impacted in several ways. First, key
functionality, such as improved direction finding and improved coverage
of coastal areas, will not be available as planned. Coast Guard
personnel at these sites will continue to use outdated legacy
communications systems for search and rescue operations and coverage of
coastal regions will remain limited. In addition, these delays could
result in costly upgrades to the legacy system to address
communications coverage gaps as well as other operational concerns.
Recommendations:
To more effectively manage the remaining development and deployment of
Rescue 21, we recommend that the Commandant of the United States Coast
Guard ensure that executive-level management implements the following
three recommendations:
* oversee the project's progress toward cost and schedule milestones
and manage risks,
* establish a milestone to complete Rescue 21's integrated baseline
review, to include all renegotiated contract items, and:
* use the results of this review to complete a revised cost and
schedule estimate.
Agency Comments:
In providing oral comments on a draft of this briefing, the Coast Guard
Assistant Commandant for Acquisition stated that the briefing fairly
represented Rescue 21 risks and ongoing activities to address program
management and other issues. He also stated that this briefing would be
used as input for them to take corrective actions to improve program
management.
Regarding the status of the vessel asset tracking and data transfer
functionality that was deferred, the Assistant Commandant stated that
he expects this functionality to be addressed with the Automatic
Identification System, rather than by Rescue 21. This decision is
pending Department of Homeland Security concurrence.
Coast Guard did not concur with our statements regarding executive
oversight. They stated that, prior to 2005, there were briefings to
executive management, such as the key decision point and semi-annual
meetings. However, as previously stated, there is no evidence that
these executive-level meetings occurred.
The Coast Guard also provided specific technical comments, which we
have incorporated into the briefing as appropriate.
[End of Slide Protection]
[End of section]
Appendix II: Comments from the Department of Homeland Security:
U.S. Department of Homeland Security:
Washington, DC 20528:
Homeland Security:
May 12, 2006:
Mr. David A. Powner:
Director:
Information Technology Management Issues:
U.S. Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Mr. Powner:
Thank you for the opportunity to review and comment on the Government
Accountability Office's (GAO's) draft report entitled UNITED STATES
COAST GUARD: Improvements Needed in Management and Oversight of Rescue
System Acquisition (GAO-06-632). This draft report includes briefing
slides presented on March 27, 2006 to the House Committee on
Appropriations, Subcommittee on Homeland Security and the Senate
Committee on Commerce, Science and Transportation, Subcommittee on
Fisheries and Coast Guard.
While we agree with GAO's three recommendations contained therein, and
are in the process of implementing each recommendation, we do have
substantial concerns with several parts of the report and the briefing.
We agree that the Coast Guard is responsible for managing schedule
delays and cost overruns in major acquisition projects. The following
program management improvements to control cost and schedule have been
initiated by the Department of Homeland Security (DHS) and the Coast
Guard:
* Established an on-site Coast Guard Project Resident Office for
increased contractor oversight. (Aug 2005).
* Ordered full rate production regions incrementally to improve control
(Sep 2005). * Implemented Alpha Contracting for improved communications
between the government and contractor to streamline the process (Sep
2005). * Implemented fixed price maintenance contracts after initial
year (Sep 2005).
* Increased oversight with quarterly Flag Officer/Vice President level
program reviews and periodic DHS Joint Requirements Council (JRC) and
Investment Review Board (IRB) reviews (Oct 2005).
* Issued a Terminate for Convenience of the Government letter to the
contractor on all vessel initiatives as a cost mitigating strategy
(April 2006):
* Will conduct Integrated Baseline Reviews (IBR) for Full Rate
Production to better identify and control cost beginning Aug 2006.
* Will conduct monthly reviews of project schedule with the contractor
beginning June 2006.
As reflected above, many of the cost and schedule concerns raised by
the report have since been addressed. Accordingly, the project's APB
(acquisition program baseline) for cost, schedule, and performance have
been revised. For instance, the total acquisition cost parameter is
$730.2M (vice $872M, as estimated by GAO) which has been achieved by
adjusting several performance parameters. Additionally, on April 21,
2006, the Coast Guard issued a notice of termination for convenience,
canceling General Dynamics' option to provide all vessel sub-system
(VSS) components. The Coast Guard anticipates that this work will be
handled organically or by another contractor or a combination of both.
The shortcomings cited in the GAO report, however, do not adequately
emphasize the root cause of the RESCUE 21 system deployment delays and
overruns. As confirmed by GAO's findings on page 27 of the briefing,
the contractor:
* underestimated the time for development and testing of RESCUE 21 and
could not meet its own established integrated project schedule, and:
* failed to consistently provide personnel in a timely manner or with
the knowledge and skills required to complete the project.
General Dynamics' inability to meet the technical performance within
its proposed schedule was due to challenges in software development.
The Coast Guard was fully aware of the problem the contractor was
experiencing and made a conscious decision during the software
development phase to offset cost and schedule to ensure technical
performance was not compromised for this critical life-saving system.
The GAO draft report highlights five key factors that contributed to
RESCUE 21 cost overruns and schedule delays. As noted above, the Coast
Guard agrees with GAO's fourth factor regarding "Contractor Cost and
Schedule Estimation and Delivery." However, the remaining key factors
are not accurate in important aspects and the Coast Guard provides the
following facts and rationale (the following citations reflect the
revisions GAO made to the draft report as a result of subsequent
discussions with the Coast Guard):
Requirements Management: GAO "Coast Guard did not effectively manage
requirements and testing revealed incomplete and poorly defined
requirements."
This draft report does not recognize well-established Coast Guard
processes and is not consistent with the GAO's previous acknowledgement
of these processes (Report GAO-03-1111). The Coast Guard's requirements
management process is outlined in the Major Systems Acquisition Manual
(MSAM COMDINST M4150.2) Chapter 3 "Requirements Management Process,"
which includes development of the:
* Mission Capability Gap Analysis documented in the Mission Analysis
Report (MAR) dated 19 May 1994.
* Broad Mission and Capability Need documented in the Mission Needs
Statement (MNS) dated 18 Apr 1995, updated 14 May 1998, and most
recently revalidated 12 Apr 2005.
* Mission Oriented Capabilities documented in the Preliminary
Operational Requirements Documents (PORD) dated 19 Nov 1997, and the
Operational Requirements Document (ORD) 1 dated 19 Jul 1999, updated 20
Aug 2001, and most recently revalidated 12 Apr 2005.
* System Specific Capabilities documented in the Performance
Specification (P-Spec) dated 8 Feb 2002.
The Coast Guard maintains that such requirements as, "receive a 1 watt
transmission from 2 meters above the water 20NM offshore" and
"Direction finding (DF) to within +/-2 degrees", demonstrate the
specificity of the RESCUE 21 system requirements, as documented in the
Performance Specification.
Moreover, field testing demonstrated that the requirements were clearly
defined and that such testing is the appropriate mechanism to verify
whether General Dynamics' design met the contractual requirements of
the Performance Specifications.
* General Dynamic's software design (and to a certain degree the
contractor's hardware configuration design) included over 200 defects
that required correction and re-testing for conformance to requirements
prior to acceptance by the Coast Guard. These defects were documented
in the Formal Qualification Testing (FQT) and System Integration
Testing (SIT) plans; both of which were provided to GAO.
* While General Dynamics ultimately met the system requirements as a
result of their efforts to correct defects and retest performance, the
Coast Guard determined that the functionality of the system could be
improved by incorporating human systems interface (HSI) enhancements
proposed by the users. Given the life-saving nature of the RESCUE 21
system, the Coast Guard believes that the decision to offset schedule
for greater HSI performance was prudent, and in the end ensures the
Coast Guard can adequately serve the American public.
Project Monitoring: GAO "Coast Guard did not effectively use earned
value management data to measure performance and take corrective action
on negative trends."
Since award of the Phase II contract to General Dynamics in September
2002, the Coast Guard established an Earned Value Management (EVM) work
group comprised of Coast Guard officials with support from Subject
Matter Experts from Booz Allen Hamilton. Together, this group was
responsible for reviewing the monthly Cost Performance Reports (CPR)
submitted by General Dynamics, conducting analyses to determine
performance trends and developing monthly Performance Measurement
Reports.
In accordance with the performance-based, performance-specification
nature of the Phase II contract, the RESCUE 21 Project Manager (PM)
employed the recommendations of the Coast Guard EVM work group to
highlight to General Dynamics' upper management their poor performance
(e.g., inability to maintain cost and schedule performance within the
variance thresholds). The PM further requested that General Dynamics
institute corrective actions to reverse the negative performance
trends. General Dynamics clearly acknowledged the challenges they were
facing in designing the system as documented in their Nov 2002 CPR:
From General Dynamics Source Document ID: PIA 16276/A008-002A (Nov
2002):
* "The schedule variance was primarily attributable to the Software
Development account"
* "The program's critical path continues to be driven by the software
development effort."
The following excerpts are from formal correspondence the Coast Guard
submitted to General Dynamics and this information was provided to GAO
as evidence of concrete actions taken by the RESCUE 21 PM in light of
the EVM data:
From Coast Guard Source Document ID: PIA 16276/AO08-017 (Feb 2004):
* "The CG is extremely concerned about the current and cumulative cost/
schedule performance trends and status related to the cost plus CLINs.
There are several key issues that must be addressed in order to provide
insight into the contractor's commitment to control cost overruns and
recover from schedule delays."
* "The variance reporting must be expanded to provide more insight into
ongoing and future risk mitigation and recovery plans related to cost
control and schedule recovery."
From Coast Guard Source Document ID PIA 16276/AO08-022 (Jul 2004):
* "The USCG is very concerned that the information reflected on the CPR
is not being used by GDDS as a management tool to make informed
decisions and manage risk."
In response to Coast Guard concerns, in June 2004 General Dynamics
undertook a significant re-organization of its management structure as
a corrective action in an attempt to better manage cost and schedule
performance.
Risk Management: GAO "Coast Guard did not effectively mitigate and
communicate risks."
In addition to the process for managing risks as documented in RESCUE
21's Risk Management Plan, risks are discussed and reported during the
Project Management Reviews (PMR), Semi-Annual briefs, Performance
Measurement Reports, Monthly Acquisition Reports, Coast Guard
Acquisition Review Council (CGARC) and IRB meetings, and various
weekly/monthly workgroup meetings. During each PMR, risks affecting
cost, schedule, logistics, and technical performance are communicated
to all General Dynamics and Coast Guard teams and follow on discussions
are held to validate the mitigation actions, ownership, and planned
dates for closure.
For example, the risk associated with software development was
communicated, and actions were taken to mitigate "TECHNICAL
PERFORMANCE" risk. This risk was assigned top priority, given the life-
saving aspect of RESCUE 21. Mitigation actions included (i)
establishment of an HSI workgroup comprised of Coast Guard and General
Dynamics personnel to discuss and resolve issues, (ii) assignment of a
member of the Coast Guard team to the General Dynamics' facility for 6
months to facilitate issue resolution, (iii) establishment of a
contract modification to include enhancements to the system proposed by
the Coast Guard operational community, (iv) development of a plan of
action and milestones to implement the HSI enhancements.
Executive-Level Oversight: GAO "Coast Guard stated that it had an
executive oversight process that included semi-annual and key decision
point reviews. However, there is no evidence that these reviews of
Rescue 21 occurred before 2005."
Early in the audit, the Coast Guard provided oral evidence as to the
existence of senior leader executive oversight, upon which GAO
requested documentary evidence. The Coast Guard later provided GAO
staff a CD containing the semi-annual and key decision reviews dating
from 2002 thru 2005. These reviews included:
Table:
Date: Jan 2002:
Executive Governance & Oversight Presentation: Key Decision Point 2
(milestone decision):
Date: Oct 2002:
Executive Governance & Oversight Presentation: Semi-Annual Presentation
to Coast Guard Acquisition Review Council (CGARC):
Date: Aug 2003:
Executive Governance & Oversight Presentation: Semi-Annual Presentation
to CGARC:
Date: Feb 2004:
Executive Governance & Oversight Presentation: Semi-Annual Presentation
to CGARC:
Date: July 2004:
Executive Governance & Oversight Presentation: Semi-Annual Presentation
to CGARC:
Date: Apr 2005:
Executive Governance & Oversight Presentation: Key Decision Point 3
CGARC:
Date: Jun 2005:
Executive Governance & Oversight Presentation: Key Decision Point 3 DHS
Investment Review Board:
[End of Table]
Additionally, throughout this period the RESCUE 21 Project Manager
provided status updates on cost, schedule, technical performance and
risks to executive management for oversight via established weekly
reports to the Acquisition Director; Monthly Acquisition Reports to the
Commandant; Quarterly Acquisition Reports to Congress (as approved by
DHS and OMB); annual reports to Congress and Business Case Exhibit 300
to the Office of Management and Budget (OMB) as well as periodic
milestone decision reviews to the CGARC and DHS Joint Requirements
Council (JRC)/Investment Review Board (IRB).
Page 28 of the GAO briefing states that "executive management did not
take action in response to the risks presented" at the DHS IRB in Jun
2005. To the contrary, the Top Ten project risks were presented to the
CGARC in Apr 2005 in preparation for the DHS IRB. The Vice Commandant
(in his role as the Coast Guard Acquisition Executive) led the
discussion regarding ongoing mitigation actions for the risks
identified. A number of these risks were successfully mitigated and
closed, and the remaining risks were highlighted in the DHS JRC/IRB
investment review process. The DHS Deputy Secretary specifically
addressed the cost, environmental, real estate and vessel subsystem
risks in his Acquisition Decision Memorandum of 23 Jun 2005.
The Department greatly appreciates the efforts of the GAO audit team
and acknowledges that the scope of the evaluation was broad, requiring
the review and analysis of a myriad of project plans and documentation
within an extremely compressed timeline to meet Congressional
deadlines. The Coast Guard will continue to aggressively pursue the
recommendations of the GAO report in order to ensure RESCUE 21 provides
critical life-saving capabilities, and that the acquisition itself is
effectively managed.
Thank you again for the opportunity to comment on this draft report and
we look forward to working with you on future homeland security issues.
Sincerely,
Signed by:
Steven J. Pecinovsky:
Director:
Departmental GAO/OIG Liaison Office:
[1] With respect to the ORD, the GAO draft report (Page 2) states,
"Also, originally, Rescue 21 was intended to limit communications gaps.
However, recent information indicates that while the Coast Guard has a
goal of achieving 2 percent gaps to the extent possible, constraints
may limit their ability to meet this goal. Coast Guard further stated
that limiting gaps to 2 percent was only a goal, its threshold gap
level was 10 percent or less." The Coast Guard has not made any
reduction in coverage requirement since the ORD (Operational
Requirements Document) was approved in 2001. The ORD specifies a
communications coverage goal of 98% and threshold of 90%. The goal of
the Coast Guard is to achieve 98% communications coverage to the
maximum extent possible. In discrete cases, where environmental,
historical preservation, geography or other external factors may
constrain communication coverage to something less than 98%, the
established threshold of 90% as documented in the ORD will be met. In
no case will the project deploy less than 90% communications coverage
unless a waiver is approved (only considered in rare situations where
the cost of covering a particular area is grossly disproportionate to
the potential benefits).
[End of section]
Appendix III GAO Contacts and Staff Acknowledgments:
GAO Contacts:
David A. Powner, (202) 512-9286 Stephen L. Caldwell, (202) 512-9610:
Staff Acknowledgments:
In addition to the contacts named above, Joel Aldape, Lisa Canini,
Carol Cha, Nancy Glover, Dawn Hoff, Tonia Johnson, George Kovachick,
Freda Paintsil, Karl Seifert, and Kim Zelonis made key contributions to
this report.
FOOTNOTES
[1] In April 2006, the Department of Homeland Security approved a new
acquisition program baseline for Rescue 21 with a total acquisition
cost of $730.2 million after decreasing certain functionality.
[2] EVM is a project management tool that integrates the investment
scope of work with schedule and cost elements for investment planning
and control. This method compares the value of work accomplished during
a given period with that of the work expected in the period.
Differences in expectations are measured in both cost and schedule
variances. The Office of Management and Budget (OMB) requires agencies
to use EVM as part of their performance-based management system for any
investment under development or with system improvements under way.
[3] USCG is currently reorganizing its field units into new,
integrated, and standardized organizations referred to as sectors.
[4]EVM is a project management tool that integrates the investment
scope of work with schedule and cost elements for investment planning
and control. This method compares the value of work accomplished during
a given period with that of the work expected in the period.
Differences in expectations are measured in both cost and schedule
variances. OMB requires agencies to use EVM as part of their
performance-based management system for any investment under
development or with system improvements under way.
[5] [5] USCG is in the process of reorganizing its field units into new
integrated and standardized organizations, referred to as sectors.
[6] A digital selective calling radio has the ability to send a mayday
signal that identifies the vessel and, when connected to a Global
Positioning System, sends the vessel's location. This occurs on channel
70.
[7] An indefinite-delivery, indefinite-quantity contract provides for
an indefinite quantity, within stated limits, of supplies or services
during a fixed period of time. The government schedules deliveries or
performance by placing orders with the contractor. 48 C.F.R. §16.504.
[8]According to USCG officials, Channel 16, which is analogous to 911,
is the channel designated for the use of distressed boaters to gain
assistance in an emergency.
[9] GAO, Coast Guard: New Communications System to Support Search and
Rescue Faces Challenges, GAO-03-1111 (Washington, D.C.: Sept. 30,
2003).
[10] Rescue 21 tests include: formal qualification testing, which
demonstrates that the system meets contract performance specifications;
system integration testing, which demonstrates that the system is fully
functional at USCG stations; and operational testing and evaluation,
which is conducted by the USCG and ensures that the system satisfies
contract performance requirements and is ready to be deployed at other
USCG locations.
[11]USCG also uses the term "ground sub-system" for the functionality
and equipment deployed to the stations in the initial operating
capability regions. The functionality and equipment that was to be
deployed to the vessels associated with these stations, which was not
delivered as scheduled, is referred to as the "vessel sub-system".
[12] USCG intends to use these low-rate initial production sites to
test Rescue 21 capabilities in various geographical climates.
[13] USCG contracted Booz Allen Hamilton to assist with program
management for Rescue 21.
[14] USCG is in the process of reorganizing its field units into new
integrated and standardized organizations, referred to as sectors.
[15] This estimate reflects only the acquisition component and does not
include operations & maintenance. The total life cycle cost estimate is
approximately $1 .6 billion.
[16] We used contractor-provided data that reflected work from October
2003 to December 2005.
[17] The Automatic Identification System is a Coast Guard project to
develop a new system that will be used to monitor and track vessels
traveling to and through US waters.
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
