Secure Border Initiative
DHS Needs to Strengthen Management and Oversight of Its Prime Contractor
Gao ID: GAO-11-6 October 18, 2010
The Department of Homeland Security's (DHS) Secure Border Initiative Network (SBInet) is to place surveillance systems along our nation's borders and provide Border Patrol command centers with the imagery and related tools and information needed to detect breaches and make agent deployment decisions. To deliver SBInet, DHS has relied heavily on its prime contractor. Because of the importance of effective contractor management and oversight to SBInet, GAO was asked to determine the extent to which DHS (1) defined and implemented effective controls for managing and overseeing the prime contractor and (2) effectively monitored the contractor's progress in meeting cost and schedule expectations. To do this, GAO analyzed key program documentation against relevant guidance and best practices, and interviewed program officials.
DHS has largely defined but has not adequately implemented the full range of controls that is reflected in relevant guidance and related best practices and is needed to effectively manage and oversee its SBInet prime contractor. To the department's credit, it has defined a number of key policies and procedures for verifying and accepting contract deliverables and conducting technical reviews, such as the criteria that need to be met before commencing and concluding a critical design review. Moreover, it has implemented some of these defined practices, such as those associated with training key contractor management and oversight officials. However, DHS has not effectively implemented other controls. For example, it has not adequately documented its review of contract deliverables and communicated to the prime contractor the basis for rejecting certain deliverables. Further, it has not ensured that key documentation satisfied relevant criteria before concluding key technical reviews. These weaknesses can be attributed in part to limitations in the defined verification and acceptance deliverable process, a program office decision to exclude certain deliverables from the process, and insufficient time to review technical review documentation. All told, DHS has not effectively managed and overseen its SBInet prime contractor, thus resulting in costly rework and contributing to SBInet's well-chronicled history of not delivering promised capabilities and benefits on time and within budget. DHS has not effectively monitored the SBInet prime contractor's progress in meeting cost and schedule expectations. While DHS has used earned value management (EVM), which is a proven management approach for understanding program status and identifying early warning signs of impending schedule delays and cost overruns, it has not ensured that its contractor has effectively implemented EVM. In particular, DHS has not ensured that validated performance baselines (the estimated value of planned work against which performance is measured) were timely, complete, and accurate. For example, the contractor was allowed to perform work on task orders for several months without a validated baseline in place. Further, not all work to be performed was included in the baselines that were eventually established, and the schedules for completing this work did not substantially comply with six of the eight key practices that relevant guidance states are important to having a reliable schedule. Also, DHS regularly received incomplete and anomalous EVM data from the prime contractor, which it had to rely on to measure progress and project the time and cost to complete the program. As a result, DHS has not been able to gain meaningful and proactive insight into potential cost and schedule performance shortfalls, and thus take corrective actions to avoid shortfalls in the future. Program officials attributed these weaknesses in part to the instability in the scope of the work to be performed, and the contractor's use of estimated, rather than actual, costs for subcontractor work and the subsequent adjustments that are made when actual costs are received. This inability has contributed to the program's failure to live up to expectations and to it costing more and taking longer than was necessary. GAO is making recommendations to DHS aimed at revising and implementing policies and procedures related to contractor deliverables and technical reviews, and improving EVM baselines and data. DHS agreed with GAO's recommendations and described actions to address them, but took exception with selected findings and conclusions regarding EVM implementation. GAO stands by these findings and conclusions for reasons discussed in the report.
Recommendations
Our recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director:
Randolph C. Hite
Team:
Government Accountability Office: Information Technology
Phone:
(202) 512-6256
GAO-11-6, Secure Border Initiative: DHS Needs to Strengthen Management and Oversight of Its Prime Contractor
This is the accessible text file for GAO report number GAO-11-6
entitled 'Secure Border Initiative: DHS Needs to Strengthen Management
and Oversight of Its Prime Contractor' which was released on October
18, 2010.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
United States Government Accountability Office:
GAO:
October 2010:
Secure Border Initiative:
DHS Needs to Strengthen Management and Oversight of Its Prime
Contractor:
GAO-11-6:
GAO Highlights:
Highlights of GAO-11-6, a report to congressional requesters.
Why GAO Did This Study:
The Department of Homeland Security‘s (DHS) Secure Border Initiative
Network (SBInet) is to place surveillance systems along our nation‘s
borders and provide Border Patrol command centers with the imagery and
related tools and information needed to detect breaches and make agent
deployment decisions. To deliver SBInet, DHS has relied heavily on its
prime contractor. Because of the importance of effective contractor
management and oversight to SBInet, GAO was asked to determine the
extent to which DHS (1) defined and implemented effective controls for
managing and overseeing the prime contractor and (2) effectively
monitored the contractor‘s progress in meeting cost and schedule
expectations. To do this, GAO analyzed key program documentation
against relevant guidance and best practices, and interviewed program
officials.
What GAO Found:
DHS has largely defined but has not adequately implemented the full
range of controls that is reflected in relevant guidance and related
best practices and is needed to effectively manage and oversee its
SBInet prime contractor. To the department‘s credit, it has defined a
number of key policies and procedures for verifying and accepting
contract deliverables and conducting technical reviews, such as the
criteria that need to be met before commencing and concluding a
critical design review. Moreover, it has implemented some of these
defined practices, such as those associated with training key
contractor management and oversight officials. However, DHS has not
effectively implemented other controls. For example, it has not
adequately documented its review of contract deliverables and
communicated to the prime contractor the basis for rejecting certain
deliverables. Further, it has not ensured that key documentation
satisfied relevant criteria before concluding key technical reviews.
These weaknesses can be attributed in part to limitations in the
defined verification and acceptance deliverable process, a program
office decision to exclude certain deliverables from the process, and
insufficient time to review technical review documentation. All told,
DHS has not effectively managed and overseen its SBInet prime
contractor, thus resulting in costly rework and contributing to SBInet‘
s well-chronicled history of not delivering promised capabilities and
benefits on time and within budget.
DHS has not effectively monitored the SBInet prime contractor‘s
progress in meeting cost and schedule expectations. While DHS has used
earned value management (EVM), which is a proven management approach
for understanding program status and identifying early warning signs
of impending schedule delays and cost overruns, it has not ensured
that its contractor has effectively implemented EVM. In particular,
DHS has not ensured that validated performance baselines (the
estimated value of planned work against which performance is measured)
were timely, complete, and accurate. For example, the contractor was
allowed to perform work on task orders for several months without a
validated baseline in place. Further, not all work to be performed was
included in the baselines that were eventually established, and the
schedules for completing this work did not substantially comply with
six of the eight key practices that relevant guidance states are
important to having a reliable schedule. Also, DHS regularly received
incomplete and anomalous EVM data from the prime contractor, which it
had to rely on to measure progress and project the time and cost to
complete the program. As a result, DHS has not been able to gain
meaningful and proactive insight into potential cost and schedule
performance shortfalls, and thus take corrective actions to avoid
shortfalls in the future. Program officials attributed these
weaknesses in part to the instability in the scope of the work to be
performed, and the contractor‘s use of estimated, rather than actual,
costs for subcontractor work and the subsequent adjustments that are
made when actual costs are received. This inability has contributed to
the program‘s failure to live up to expectations and to it costing
more and taking longer than was necessary.
What GAO Recommends:
GAO is making recommendations to DHS aimed at revising and
implementing policies and procedures related to contractor
deliverables and technical reviews, and improving EVM baselines and
data. DHS agreed with GAO‘s recommendations and described actions to
address them, but took exception with selected findings and
conclusions regarding EVM implementation. GAO stands by these findings
and conclusions for reasons discussed in the report.
View [hyperlink, http://www.gao.gov/products/GAO-11-6] or key
components. For more information, contact Randolph C. Hite at (202)
512-3439 or hiter@gao.gov.
[End of section]
Contents:
Letter:
Background:
DHS Has Largely Defined but Has Not Effectively Implemented the Full
Range of Needed Contractor Management and Oversight Controls:
DHS Has Not Effectively Monitored the Contractor's Progress in Meeting
Cost and Schedule Expectations:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Comments from the Department of Homeland Security:
Appendix III: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Summary of SBInet Task Orders as of May 2010:
Table 2: Percentage of Performance Measurement Baseline Work Tasks
Categorized as Level-of-Effort for Six Baselines:
Figures:
Figure 1: Partial High-Level Depiction of SBI Program Organization:
Figure 2: Partial High-Level Depiction of SBI Contracting Division
Organization:
Figure 3: Summary of Contract Deliverables That Did and Did Not Follow
the Defined Process:
Figure 4: Summary of Time Frames With and Without Validated
Performance Baselines from June 2008 through February 2010:
Figure 5: Summary of IBR Schedules' Satisfaction of Schedule
Estimating Practices:
Figure 6: Percentage of Work Breakdown Structure Elements with EVM
Data Anomalies by Month for Each of the Four Task Orders:
Figure 7: Total Number of Data Anomalies and Number of Explained
Anomalies in the Monthly EVM Reports across the Four Task Orders:
Abbreviations:
ADTO: Arizona Deployment Task Order:
AJO-1: Ajo Border Patrol Station:
ANSI: American National Standards Institute:
CBP: U.S. Customs and Border Protection:
CDR: Critical Design Review:
CMMIŽ: Capability Maturity Model IntegrationŽ:
COP: common operating picture:
COTR: contracting officer's technical representative:
C3I: command, control, communications, and intelligence:
DCMA: Defense Contract Management Agency:
DTO: Design Task Order:
DHS: Department of Homeland Security:
EVM: earned value management:
FAC-C: Federal Acquisition Certification in Contracting:
IBR: integrated baseline review:
IDIQ: Indefinite Delivery/Indefinite Quantity:
IV&V: independent verification and validation:
JPMR: Joint Program Management Review:
NOC/SOC: Network Operations Center/Security Operations Center:
SEP: Systems Engineering Plan:
SBI: Secure Border Initiative:
SBInet: Secure Border Initiative Network:
SEI: Software Engineering Institute:
SPO: SBInet System Program Office:
STO: System Task Order:
TUS-1: Tucson Border Patrol Station:
WBS: work breakdown structure:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
October 18, 2010:
The Honorable Bennie G. Thompson:
Chairman:
Committee on Homeland Security:
House of Representatives:
The Honorable Christopher P. Carney:
Chairman:
The Honorable Gus M. Bilirakis:
Ranking Member Subcommittee on Management, Investigations, and
Oversight:
Committee on Homeland Security:
House of Representatives:
The Honorable Mike Rogers:
Ranking Member:
Subcommittee on Emergency Communications, Preparedness, and Response:
Committee on Homeland Security:
House of Representatives:
To enhance border security and reduce illegal immigration, the
Department of Homeland Security (DHS) launched its multiyear,
multibillion dollar Secure Border Initiative (SBI) program in November
2005. Through SBI, DHS intends to enhance surveillance technologies,
increase staffing levels, enforce immigration laws, and improve the
physical infrastructure along our borders. Within SBI, the Secure
Border Initiative Network (SBInet) is a multibillion-dollar program
that involves the acquisition, development, integration, deployment,
and operation and maintenance of surveillance technologies to create a
"virtual fence" along the border as well as command, control,
communications, and intelligence (C3I) technologies to create a
picture of the border in command centers and vehicles.
DHS's strategy is to deliver SBInet capabilities incrementally. In
doing so, the department has relied heavily on its prime contractor--
the Boeing Company. Because of the importance of effective contractor
management and oversight to the successful deployment and operation of
SBInet capabilities, you asked us to determine to what extent DHS (1)
has defined and implemented effective controls for managing and
overseeing the SBInet prime contractor and (2) is effectively
monitoring the prime contractor's progress in meeting cost and
schedule expectations.
To accomplish our objectives, we focused on DHS's management and
oversight of four key task orders associated with designing,
developing, and deploying the first increment of SBInet, known as
Block 1, to its initial deployment sites in Arizona--the Tucson Border
Patrol Station (TUS-1) and the Ajo Border Patrol Station (AJO-1). In
doing so, we reviewed DHS and program guidance, policies, and plans
for managing and overseeing the contractor; task orders and associated
modifications; contract deliverable review comments; contractor
schedules; the work breakdown structure governing all task orders;
integrated baseline review documents for each task order; and task
order contract cost performance reports. We also analyzed a
nonprobability, random sample of 28 contract deliverables; a
nonprobability, random sample of 8 technical reviews conducted with
the contractor; and a nonprobability sample of 11 action items
identified during program management reviews.[Footnote 1] In addition,
we interviewed program officials about contractor management and
oversight activities, such as verifying and accepting contract
deliverables and conducting technical and management reviews. We also
interviewed program officials about the prime contractor's cost and
schedule performance. We then compared this information to relevant
guidance and leading industry practices on contractor management and
oversight, schedule estimating, and earned value management (EVM),
[Footnote 2] such as the Software Engineering Institute's (SEI)
Capability Maturity Model IntegrationŽ (CMMIŽ) for Acquisition, GAO's
Cost Estimating and Assessment Guide, and the American National
Standards Institute (ANSI) standard.[Footnote 3]
We conducted this performance audit from June 2009 to October 2010 in
accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives. Further
details of our objectives, scope, and methodology are in appendix I.
Background:
Managed by U.S. Customs and Border Protection (CBP), SBInet is to
strengthen DHS's ability to detect, identify, classify, track, and
respond to illegal breaches at and between ports of entry. It includes
the acquisition, development, integration, deployment, and operations
and maintenance of a mix of surveillance technologies, such as
cameras, radars, sensors, and C3I technologies. Surveillance
technologies include unattended ground sensors that can detect heat
and vibrations associated with foot traffic and metal associated with
vehicles, radar mounted on fixed towers that can detect movement, and
cameras mounted on fixed towers that can be used to identify and
classify items of interest detected and tracked by ground sensors and
radar. These technologies are generally commercial off-the-shelf
products. C3I technologies include customized software development to
produce a common operating picture (COP)--a uniform presentation of
activities within specific areas along the border--as well as the use
of CBP network capabilities. Together, the surveillance technologies
are to gather information along the border and transmit this
information to COP terminals located in command centers, which are to
assemble it to provide CBP agents with border situational awareness,
to, among other things, enhance agents' tactical decisionmaking
regarding potential apprehensions.
Since fiscal year 2006, DHS has received about $4.4 billion in
appropriations for SBI, including about $2.5 billion for physical
fencing and related infrastructure, about $1.5 billion for virtual
fencing (e.g., surveillance systems) and related infrastructure (e.g.,
towers), and about $300 million for program management.[Footnote 4] As
of May 2010, DHS had obligated about $1.02 billion for SBInet.
Program and Acquisition Organizational Structure:
The SBI Program Management Office, which is organizationally within
CBP, is responsible for managing key acquisition functions associated
with SBInet, including prime contractor management and oversight. It
is organized into four components: the SBInet System Program Office
(SPO), Operational Integration Division, Business Operations Division,
and Systems Engineering Division.[Footnote 5] The SPO is responsible
for such key contractor oversight activities as verifying and
accepting contract deliverables and conducting contractor management
and technical reviews. In addition, the Business Operations Division
has primary responsibility for monitoring the prime contractor's cost
and schedule performance. As of May 2010, the SBI Program Management
Office was staffed with 194 people--105 government employees, 78
contractor staff, and 11 detailees. (See figure 1 for a partial SBI
program organization chart.)
Figure 1: Partial High-Level Depiction of SBI Program Organization:
[Refer to PDF for image: organization chart]
Top level:
Program Management Office:
* Executive Director;
* Deputy Executive Director (reports to Executive Director).
Second level, reporting to Executive Director:
* Operational Integration Division;
* Business Operations Division;
* Systems Engineering Division.
Second level, reporting to Executive Director:
SBInet System Program Office:
* Executive Director;
* Deputy Director (reports to Executive Director).
Third level, reporting to Deputy Director:
* Systems Solution Division;
* Functional Engineering and Environmental Resources Division;
* Southwest Border Deployment Division;
* Northern Border Division.
Source: GAO analysis based on DHS data.
[End of figure]
In addition, CBP has engaged the Defense Contract Management Agency
to, among other things, perform surveillance of the prime contractor's
EVM, systems engineering, hardware and software quality assurance, and
risk management.
Further, the SBI Contracting Division, which is organizationally
within the CBP Office of Administration Procurement Directorate's
Enterprise Contracting Office, is responsible for performing contract
administration activities, such as maintaining the contract file and
notifying the contractor in writing of whether deliverables are
accepted or rejected. The SBI Contracting Division is organized into
three branches: SBI Contracting, SBInet System Contracting, and SBInet
Deployment Contracting. As of May 2010, the SBI Contracting Division
was staffed with 22 people--18 government employees and 4 contractor
staff. (See figure 2 for a partial SBI Contracting Division
organization chart.)
Figure 2: Partial High-Level Depiction of SBI Contracting Division
Organization:
[Refer to PDF for image: organization chart]
Top level:
* Office of Administration Procurement Directorate:
- Executive Director.
Second level, reporting to Office of Administration Procurement
Directorate:
* Enterprise Contracting Office:
- Deputy Executive Director.
Third level, reporting to Enterprise Contracting Office:
* SBI Contracting Division;
- Division Director.
Fourth level, reporting to SBI Contracting Division:
* SBI Contracting Branch;
* SBInet System Contracting Branch;
* SBInet Deployment Contracting Branch.
Source: GAO analysis based on DHS data.
[End of figure]
Overview of SBInet Acquisition Strategy and Program Status:
In September 2006, CBP awarded an Indefinite Delivery/Indefinite
Quantity (IDIQ)[Footnote 6] prime contract to the Boeing Company. The
contract was for 3 base years with three additional 1-year options for
designing, producing, testing, deploying, and sustaining SBI. In
September 2009, CBP exercised the first option year. Under the prime
contract, CBP has issued 11 task orders that relate to SBInet,
covering for example, COP design and development, system deployment,
and system maintenance and logistics support. As of May 2010, 5 of the
11 task orders were complete and 6 were ongoing. (See table 1 for a
summary of the SBInet task orders.)
Table 1: Summary of SBInet Task Orders as of May 2010:
Completed task orders:
Task order: Program Management;
Description: Completed task orders: Mission engineering, facilities
and infrastructure, systems engineering, test and evaluation, and
program management services;
Period of performance: Completed task orders: Sept. 2006-Apr. 2008;
Approximate contract value: $146.9 million;
Approximate contract obligation: $146.9 million;
Contract type: Cost-plus-fixed-fee[A].
Task order: Project 28;
Description: Completed task orders: Prototype along 28 miles of the
border in the Tucson sector;
Period of performance: Completed task orders: Oct. 2006-Feb. 2008;
Approximate contract value: $20.7 million;
Approximate contract obligation: $20.7 million;
Contract type: Firm-fixed-price.
Task order: Project 28 Contractor Maintenance Logistics and Support;
Description: Completed task orders: Project 28 operational maintenance
and logistics support;
Period of performance: Completed task orders: Dec. 2007-Dec. 2009;
Approximate contract value: $10.6 million;
Approximate contract obligation: $10.6 million;
Contract type: Cost-plus-fixed-fee.
Task order: Design for Buffalo Sector;
Description: Completed task orders: SBInet design of remote video
surveillance system capability for the Buffalo sector;
Period of performance: Completed task orders: Feb. 2009-July 2009;
Approximate contract value: $0.6 million;
Approximate contract obligation: $0.6 million;
Contract type: Firm-fixed-price[B].
Task order: System;
Description: Completed task orders: Program management and systems
engineering activities required to integrate all task orders;
Period of performance: Completed task orders: Apr. 2008-May 2010;
Approximate contract value: $222.1 million;
Approximate contract obligation: $215.9 million;
Contract type: Cost-plus-award-fee.
Ongoing task orders:
Task order: Command, Control, Communications, and Intelligence (C3I)
Common Operating Picture (COP);
Description: Completed task orders: Design, development,
demonstration, and operations and maintenance of a functional C3I/COP
system;
Period of performance: Completed task orders: Dec. 2007-June 2010;
Approximate contract value: $70.5 million[C];
Approximate contract obligation: $79.3 million;
Contract type: Cost-plus-award-fee/cost-plus-fixed-fee/firm-fixed-
price[D].
Task order: Design;
Description: Completed task orders: Design of deployment solution,
environmental clearance support, and other location-related work for
the Tucson sector;
Period of performance: Completed task orders: Aug. 2007-July 2010;
Approximate contract value: $115.0 million;
Approximate contract obligation: $115.0 million;
Contract type: Cost-plus-fixed-fee.
Task order: Arizona Deployment;
Description: Completed task orders: Deployment to two sites covering
approximately 53 miles of the southwest border in the Tucson sector;
Period of performance: Completed task orders: June 2008-May 2011;
Approximate contract value: 148.5;
Approximate contract obligation: 148.5;
Contract type: Cost-plus-incentive-fee/cost-plus-award-fee[E].
Task order: Integrated Logistics Support;
Description: Completed task orders: Maintenance logistics and
operational support;
Period of performance: Completed task orders: Aug. 2008-Sept. 2010;
Approximate contract value: $61.6 million;
Approximate contract obligation: $61.6 million;
Contract type: Cost-plus-fixed-fee[F].
Task order: Northern Border Project;
Description: Completed task orders: Design, installation, and
deployment of surveillance capabilities in the Detroit and Buffalo
sectors;
Period of performance: Completed task orders: Mar. 2009-Dec. 2010;
Approximate contract value: $22.7 million;
Approximate contract obligation: $22.7 million;
Contract type: Fixed-price[B].
Task order: Systems Support;
Description: Completed task orders: Program management and systems
engineering activities required to integrate all task orders;
C3I/COP maintenance;
Period of performance: Completed task orders: May 2010-Mar. 2011;
Approximate contract value: $28.6 million;
Approximate contract obligation: $28.6 million;
Contract type: Cost-plus-award-fee.
Total:
Approximate contract value: $847.80 million;
Approximate contract obligation: $850.40 million.
Source: GAO analysis of DHS data.
Note: "Fixed-price" types of contracts provide for a firm price or, in
appropriate cases, an adjustable price; "firm-fixed-price" contracts
provide for a price not subject to adjustment on the basis of the
contractor's experience in performing the contract; "cost-plus-
incentive-fee" contracts provide for the reimbursement of allowable
costs plus an initially negotiated fee, to be adjusted later based on
the relationship of total allowable costs to total target costs; "cost-
plus-award-fee" contracts provide for the reimbursement of allowable
costs plus a base fee fixed at the contract's inception (which may be
zero) and an award amount that the government determines to be
sufficient to motivate excellence in performance; and "cost-plus-fixed-
fee" contracts provide for the reimbursement of allowable costs plus a
negotiated fee fixed at the inception of the contract.
[A] The initial contract type of the task order was a cost-plus-award-
fee. A final award fee determination did not take place because of
scope and schedule changes. In lieu of the final award fee
determination, the contract type was changed to a cost-plus-fixed-fee.
[B] The travel component of the task order is cost reimbursable.
[C] According to SBI Contracting Division officials, the contract
value with the cost overruns is $79.3 million.
[D] The initial contract type of the task order was cost-plus-award-
fee. On July 31, 2009, additional development work was definitized as
a cost-plus-fixed-fee structure. Further, on September 30, 2009, the
software operations and maintenance component of the task order was
changed to a firm-fixed-price structure.
[E] The initial contract type of the task order was a cost-plus-
incentive-fee. On November 20, 2009, the performance and schedule
incentives component of the task order was changed to a cost-plus-
award-fee. The cost incentives component remains a cost-plus-incentive-
fee structure.
[F] The initial contract type of the task order was cost-plus-
incentive-fee. On November 6, 2009, future work under the task order
was changed to a cost-plus-fixed-fee structure.
[End of table]
Through the task orders, CBP's strategy is to deliver SBInet
capabilities incrementally. To accomplish this, the program office
adopted an evolutionary system life cycle management approach in which
system capabilities are to be delivered to designated locations in a
series of discrete subsets of system functional and performance
capabilities that are referred to as blocks. The first block (Block 1)
includes the purchase of commercially available surveillance systems,
development of customized COP systems and software, and use of
existing CBP communications and network capabilities. According to
program officials, as of July 2010, the Block 1 design was deployed to
TUS-1 and was being deployed to AJO-1, both of which are located in
CBP's Tucson Sector of the southwest border. Together, these two
deployments cover 53 miles of the 1,989-mile-long southern border.
Also, according to program officials, as of July 2010, TUS-1 and AJO-1
were to be accepted in September 2010 and December 2010, respectively.
In January 2010, the Secretary of Homeland Security ordered an
assessment of the SBI program. In addition, on March 16, 2010, the
Secretary froze fiscal year 2010 funding for any work on SBInet beyond
TUS-1 and AJO-1 until the assessment is completed. Also at that time,
the Secretary redirected $35 million that had been allocated to SBInet
Block 1 deployment to other tested and commercially-available
technologies, such as truck-mounted cameras and radar, called Mobile
Surveillance Systems, to meet near-term needs.[Footnote 7] According
to the SBI Executive Director, the department's assessment would
consist of a comprehensive and science-based assessment to determine
if there are alternatives to SBInet that may more efficiently,
effectively, and economically meet U.S. border security needs.
Further, the Executive Director stated that if the assessment suggests
that the SBInet capabilities are worth the cost, DHS will extend its
deployment to sites beyond TUS-1 and AJO-1. However, if the assessment
suggests that alternative technology options represent the best
balance of capability and cost-effectiveness, DHS will redirect
resources to these other options. According to program officials, the
initial phase of the assessment, which addresses the Arizona border,
was completed in July 2010, and the results are currently being
reviewed by senior DHS management. Officials were unable to provide a
date for completion of the review.
Use of Acquisition Management Disciplines Is Important to Programs
Like SBInet:
Our research and evaluations of information technology programs have
shown that delivering promised system capabilities and benefits on
time and within budget largely depends on the extent to which key
acquisition management disciplines are employed. These disciplines
include, among others, requirements management, risk management, and
test management. As is discussed in the following section, we have
previously reported on the extent to which these disciplines have been
employed on SBInet.
Contractor management and oversight, which is the focus of this
report, is another acquisition management discipline. Among other
things, this discipline helps to ensure that the contractor performs
the requirements of the contract and the government receives the
service or product intended. DHS acquisition policies and guidance,
along with other relevant guidance, recognize the importance of these
management and oversight disciplines. As we have previously reported,
not employing them increases the risk that system acquisitions will
not perform as intended and will require expensive and time-consuming
rework.[Footnote 8]
GAO Has Previously Reported on Numerous SBInet Acquisition Management
Weaknesses and Risks:
Since 2007, we have identified a range of management weaknesses and
risks facing SBInet, and we have made a number of recommendations to
address them that DHS has largely agreed with, and to varying degrees,
taken actions to address. For example, in February 2007, we reported
that DHS had not fully defined activities, milestones, and costs for
implementing the program or demonstrated how program activities would
further the strategic goals and objectives of SBI.[Footnote 9]
Further, we reported that the program's schedule contained a high
level of concurrency among related tasks and activities, which
introduced considerable risk. Accordingly, we recommended that DHS
define explicit and measurable commitments relative to, among other
things, program capabilities, schedules, and costs, and re-examine the
level of concurrency in the schedule and adjust the acquisition
strategy appropriately.
In September 2008, we reported that a number of program management
weaknesses put SBInet at risk of not performing as intended and taking
longer and costing more to deliver than was necessary.[Footnote 10]
Specifically, we reported the following:
* Important aspects of SBInet were ambiguous and in a continued state
of flux, making it unclear and uncertain what technology capabilities
were to be delivered when. For example, the scope and timing of
planned SBInet deployments and capabilities had continued to change
since the program began and remained unclear. Further, the SPO did not
have an approved integrated master schedule to guide the execution of
the program, and our assimilation of available information indicated
that key milestones continued to slip. This schedule-related risk was
exacerbated by the continuous change in and the absence of a clear
definition of the approach used to define, develop, acquire, test, and
deploy SBInet. Accordingly, we concluded that the absence of clarity
and stability in these key aspects of SBInet impaired the ability of
Congress to oversee the program and hold DHS accountable for results,
and it hampered DHS's ability to measure the program's progress.
* SBInet requirements had not been effectively defined and managed.
While the SPO had issued guidance that defined key practices
associated with effectively developing and managing requirements, the
guidance was developed after several key activities had been
completed. In the absence of this guidance, the SPO had not
effectively performed key requirements development and management
practices, such as ensuring that different levels of requirements were
properly aligned. As a result, we concluded that the risk of SBInet
not meeting mission needs and performing as intended was increased, as
were the chances of the system needing expensive and time-consuming
rework.
* SBInet testing was not being effectively managed. For example, the
program office had not tested the individual system components to be
deployed to initial locations, even though the contractor had
initiated integration testing of these components. Further, its test
management strategy did not contain, among other things, a clear
definition of testing roles and responsibilities or sufficient detail
to effectively guide planning for specific test events.
To address these issues, we recommended that DHS assess and disclose
the risks associated with its planned SBInet development, testing, and
deployment activities and that it address the system deployment,
requirements management, and testing weaknesses that we had
identified. DHS largely agreed to implement our recommendations.
In September 2009, we reported that SBInet had continued to experience
delays.[Footnote 11] For example, deployment to the entire southwest
border had slipped from early 2009 to 2016, and final acceptance of
TUS-1 and AJO-1 had slipped from November 2009 and March 2010 to
December 2009 and June 2010, respectively. We did not make additional
recommendations at that time.
In January 2010, we reported that SBInet testing was not being
effectively managed.[Footnote 12] Specifically, we reported the
following:
* Test plans, cases, and procedures for the most recent test events
were not defined in accordance with important elements of relevant
guidance. Further, a large percentage of the test cases[Footnote 13]
for these events were changed extemporaneously during execution. While
some of the changes were minor, others were more significant, such as
rewriting entire procedures and changing the mapping of requirements
to test cases. Moreover, these changes to procedures were not made in
accordance with documented quality assurance processes, but rather
were based on an undocumented understanding that program officials
said they established with the contractor. Compounding the number and
significance of changes were questions raised by the SPO and a support
contractor about the appropriateness of some changes, noting that some
changes made to system qualification test cases and procedures
appeared to be designed to pass the test instead of being designed to
qualify the system.
* About 1,300 SBInet defects had been found from March 2008 through
July 2009, with the number of new defects identified during this time
generally increasing faster than the number being fixed--a trend that
is not indicative of a system that is maturing and ready for
deployment. While the full magnitude of these unresolved defects was
unclear because the majority were not assigned a priority for
resolution, some of the defects that had been found were significant.
Although DHS reported that these defects had been resolved, they had
nevertheless caused program delays, and related problems had surfaced
that continued to impact the program's schedule.
In light of these weaknesses, we recommended that DHS (1) revise the
program's overall test plan; (2) ensure that test schedules, plans,
cases, and procedures are adequately reviewed and approved consistent
with the revised test plan; (3) ensure that sufficient time is
provided for reviewing and approving test documents prior to beginning
a given test event; and (4) triage the full inventory of unresolved
system problems, including identified user concerns, and periodically
report on their status to CBP and DHS leadership. DHS fully agreed
with the last three recommendations and partially agreed with the
first.
In May 2010, we raised further concerns about the program and called
for DHS to reconsider its planned investment in SBInet.[Footnote 14]
Specifically, we reported the following:
* DHS had defined the scope of the first incremental block of SBInet
capabilities that it intended to deploy and make operational; however,
these capabilities and the number of geographic locations to which
they were to be deployed continued to shrink. For example, the
geographic "footprint" of the initially deployed capability has been
reduced from three border sectors spanning about 655 miles to two
sectors spanning about 387 miles.
* DHS had not developed a reliable integrated master schedule for
delivering the first block of SBInet. Specifically, the schedule did
not sufficiently comply with seven of nine key practices that relevant
guidance states are important to having a reliable schedule. For
example, the schedule did not adequately capture all necessary
activities, assign resources to them, and reflect schedule risks. As a
result, it was unclear when the first block was to be completed, and
continued delays were likely.
* DHS had not demonstrated the cost-effectiveness of Block 1. In
particular, it had not reliably estimated the costs of this block over
its entire life cycle. Further, DHS had yet to identify expected
quantifiable and qualitative benefits from this block and analyze them
relative to costs. Without a meaningful understanding of SBInet costs
and benefits, DHS lacks an adequate basis for knowing whether the
initial system solution is cost effective.
* DHS had not acquired the initial SBInet block in accordance with key
life cycle management processes. While processes associated with,
among other things, requirements development and management and risk
management, were adequately defined they were not adequately
implemented. For example, key risks had not been captured in the risk
management repository and thus had not been proactively mitigated. As
a result, DHS is at increased risk of delivering a system that does
not perform as intended.
We concluded that it remains unclear whether the department's pursuit
of SBInet is a cost-effective course of action, and if it is, that it
will produce expected results on time and within budget. Accordingly,
we recommended that DHS (1) limit near-term investment in the first
incremental block of SBInet, (2) economically justify any longer-term
investment in SBInet, and (3) improve key program management
disciplines. DHS largely agreed with our recommendations, and noted
ongoing and planned actions to address each of them.
In June 2010, we reported several technical, cost, and schedule
challenges facing SBInet, such as problems with radar functionality,
and noted that the program office was staffed substantially below
planned levels for government positions.[Footnote 15] We did not make
any additional recommendations at that time.
DHS Has Largely Defined but Has Not Effectively Implemented the Full
Range of Needed Contractor Management and Oversight Controls:
Federal regulations and relevant guidance recognize effective
contractor management and oversight as a key acquisition management
discipline. In addition, they describe a number of practices
associated with it, including training persons who are responsible for
contractor management and oversight, verifying and deciding whether to
accept contract deliverables, conducting technical reviews with the
contractor to ensure that products and services will satisfy user
needs, and holding management reviews with the contractor to monitor
contractor performance.[Footnote 16]
To DHS's credit, it has largely defined key practices aimed at
employing each of these contractor management and oversight controls.
Moreover, it has implemented some of them, such as training key
contractor management and oversight officials and holding management
reviews with the prime contractor. However, it has not effectively
implemented others, such as documenting contract deliverable reviews
and using entry and exit criteria when conducting technical reviews.
Reasons for these weaknesses include limitations in the defined
verification and acceptance deliverable process and a SPO decision to
exclude some deliverables from the process, and insufficient time to
review technical review documentation. Without employing the full
range of practices needed to effectively manage and oversee the prime
contractor, DHS is limited in its ability to know whether the
contractor is meeting performance and product expectations. Moreover,
it increases the chances that SBInet will not function as intended and
will take more time and resources than necessary to deliver, which we
have previously reported is the case for Block 1.
SBInet Contractor Management and Oversight Officials Have Been
Properly Trained:
Training supports the successful performance of relevant activities
and tasks by helping to ensure that the responsible people have the
necessary skills and expertise to perform the tasks. According to
relevant guidance, organizations should define training expectations
and should ensure that these expectations are met for individuals
responsible for contractor management and oversight.[Footnote 17]
DHS's acquisition management directives define training requirements
for, among others, program managers, contracting officers, and
contracting officer's technical representatives (COTR).[Footnote 18]
Specifically:
* Program managers must be certified at a level commensurate with
their acquisition responsibilities. For a Level 1 information
technology program,[Footnote 19] like SBInet, the designated program
manager must be certified at a program management Level 3.[Footnote 20]
* Contracting officers must be certified at the appropriate level to
support their respective warrant authority.[Footnote 21]
* COTRs must be trained and certified within 60 days of appointment to
a contract or task order. The minimum mandatory requirements are the
completion of 40 hours of COTR training and a 1-hour procurement
ethics training course.
For SBInet, CBP has ensured that people in each of these key positions
have been trained in accordance with DHS requirements. Specifically,
the two program managers associated with the development and
deployment of SBInet Block 1--the SBI Executive Director and the SPO
Executive Director--were issued training certifications that qualify
each as an Acquisition Professional. Further, each contracting officer
responsible for executing actions on the Arizona Deployment Task Order
(ADTO), the Design Task Order (DTO), the C3I/COP task order, and the
System Task Order (STO) between June 2008 and February 2010 was
certified commensurate with his or her respective warrant authority.
In addition, for the same time period, each COTR assigned to each of
the four task orders received DHS-issued certificates indicating that
each had met the minimum training requirements before being assigned
to a task order.
According to CBP officials, DHS leadership has made contractor
management and oversight training a high priority, which helped to
ensure that key officials were trained. By doing so, CBP has
established one of the key controls to help ensure that prime
contractor products and services meet expectations.
DHS Has Not Effectively Verified and Accepted All Contract
Deliverables:
Effectively implementing a well-defined process for verifying and
accepting prime contractor deliverables is vital to SBInet's success.
DHS has a defined process for verifying and accepting contract
deliverables, but this process does not ensure that deliverable
reviews are sufficiently documented. Further, while the SPO has
followed key aspects of this process, it has not effectively
documented its review of certain deliverables and has not effectively
communicated to the prime contractor the basis for rejecting all
deliverables. Reasons for not doing so include limitations in the
defined verification and acceptance process and a SPO decision to
exclude some deliverables from the process. Without documenting all
its reviews and effectively communicating the review results to the
contractor, the SPO has increased the chances of accepting
deliverables that do not meet requirements and having the contractor
repeat work to correct deliverable problems.
CBP Has a Defined Process for Verifying and Accepting SBInet Contract
Deliverables:
The purpose of contractor deliverable verification and acceptance is
to ensure that contractor-provided products and services meet
specified requirements and otherwise satisfy the terms of the
contract. According to relevant guidance, organizations should have
written policies and procedures for verifying and accepting
deliverables that, among other things, (1) assign roles and
responsibilities for performing verification and acceptance tasks and
(2) provide for conducting and documenting deliverable reviews and for
effectively communicating to the contractor deliverable acceptance and
rejection decisions.[Footnote 22]
To its credit, CBP has defined policies and procedures for verifying
and accepting SBInet deliverables. Specifically, it issued its
Deliverable Review and Approval Process in July 2007, which specifies
how the SPO is to receive, review, and respond to all contract
deliverables. Among other things, this guide assigns verification and
acceptance roles and responsibilities to key program management
positions. For example, it assigns the project manager[Footnote 23]
responsibility for overseeing the review process and determining the
acceptability of the deliverables, designates reviewers[Footnote 24]
responsibilities for examining the deliverable, and assigns the
contracting officer responsibility for communicating the decision to
accept or reject the deliverable to the contractor. In addition, it
provides for conducting deliverable reviews, which according to
program officials, involves comparing the deliverable to the
requirements enumerated in the applicable task order statement of work
(typically within the Data Item Description). The process further
specifies that the decision to accept or reject the deliverable is to
be communicated in writing to the contractor.
However, the process does not state that the results of all reviews
are to be documented. Instead, its states that a deliverable review
comment form is to be prepared only when deficiencies or problems
exist. If the deliverable is acceptable, the form does not need to be
prepared. Program officials could not explain why review documentation
was not required for acceptable deliverables. As a result, and as
discussed in the next section of this report, the SPO cannot
demonstrate its basis for accepting a number of deliverables, which in
turn has increased the risk of, and actually resulted in, deliverables
being accepted that do not meet requirements.
CBP Has Not Fully Followed its Defined Process for Verifying and
Accepting SBInet Deliverables:
The SPO followed its process for verifying and accepting SBInet-
related deliverables about 62 percent of the time, based on the 29
deliverables that we reviewed.[Footnote 25] Specifically, the process
was fully followed for 18 of the deliverables: (1) 6 that were
accepted without documented review comments, (2) 5 that were accepted
with documented review comments, and (3) 7 that were rejected with
documented review comments. In addition, the acceptance or rejection
of all 18 deliverables was communicated in writing to the contractor.
For example, the ADTO Security Plan Addendum was delivered to the SPO
in August 2008. The SPO reviewed the plan and documented its review
comments, which included a determination that the plan did not address
all required items specified in the task order's Data Item
Description. The CBP contracting officer subsequently notified the
contractor in writing that the plan was rejected for this and other
reasons. In February 2009, the contractor resubmitted the plan, the
SPO reviewed the plan and documented its comments, and the contracting
officer again notified the contractor in writing that the plan was
rejected. The contractor resubmitted the plan in May 2009, and program
officials documented their review of the deliverable. The contracting
officer subsequently communicated the deliverable's acceptance in
writing to the contractor. (Figure 3 summarizes the number of contract
deliverables that did and did not follow the defined process.)
Figure 3: Summary of Contract Deliverables That Did and Did Not Follow
the Defined Process:
[Refer to PDF for image: 3 pie-charts]
Deliverables following process: 18;
Accepted with comments and with letter: 5;
Accepted without comments and with letter: 6;
Rejected with comments and with letter: 7.
Deliverables not following process: 11;
Accepted without comments and without letter: 5;
Rejected without comments and without letter: 3;
Decision not made, no comments, and no letter: 3.
Source: GAO analysis of DHS data.
[End of figure]
The remaining 11 deliverables, however, did not fully adhere to the
defined process. Of these, five were accepted without any documented
review comments and without communicating the acceptance in writing to
the contractor. The following are examples of these five and reasons
for not adhering to the process.
* Three of the five deliverables related to the C3I/COP task order did
not require government approval. However, the Deliverable Review and
Approval Process document does not provide for such treatment of these
deliverables, and thus this practice is not in accordance with the
process. Program officials told us that they have since recognized
that this was a poor practice, and they have modified the task order
to now require approval of all C3I/COP task order deliverables.
* One of the five deliverables relates to the STO and is for the C2
Component Qualification Test package, which includes, among other
things, the test plan and test cases and procedures for conducting the
C2 qualification test event. In this case, the SPO accepted the test
plan because, according to program officials, several days had passed
since the deliverable was received, and they had not received any
comments from the reviewers. They said that they therefore accepted
the deliverable on the basis of not receiving any review comments to
the contrary, but did not notify the contractor in writing of the
acceptance.
The 11 deliverables also include 3 that were rejected without
documented review comments and without the rejection being
communicated to the contractor in writing. The following are examples
of these three and reasons for not adhering to the process are
discussed below.
* One of the three deliverables relates to the C3I/COP task order and
is for the Network Operations Center/Security Operations Center (NOC/
SOC)[Footnote 26] Test Plan/Procedures/Description. According to
program officials, the contractor did not submit the plan on time,
thus requiring them to review it during the readiness review.[Footnote
27] Based on this review, the plan was rejected, which was
communicated verbally to the contractor during the review. Despite
rejecting the plan, the program office began testing the NOC/SOC
component on the day of the review, without a revised plan being
submitted, reviewed, and approved. According to program officials,
this occurred in part because of insufficient time and resources to
review contractor test-related deliverables.
* Another one of the three deliverables also relates to the C3I/COP
task order, and is for the Release 0.5 Software Test Plan/Procedures/
Description. According to program officials, the contractor submitted
the plan late. The program office rejected the plan and provided oral
comments during a teleconference prior to the review. Nevertheless,
the test event again occurred without a revised plan being submitted,
reviewed, and accepted. According to program officials, this was also
due to insufficient time and resources to review the test plan. In
this case, the plan was approved in late April 2009, which was 5
months after the test event was conducted.
The 11 deliverables also include 3 for which a decision to accept or
reject the deliverable was not made. See the following examples:
* One of the three relates to the C3I/COP task order, and is for the
NOC/SOC Interface Control Document. For this deliverable, review
comments were not documented and no written communication with the
contractor occurred. The deliverable was subsequently submitted three
times and ultimately accepted. However, program officials could not
explain whether the initial deliverable was accepted or rejected, or
why the deliverable was submitted multiple times.
* Another one of these relates to STO, and is for the Dynamic Object-
Oriented Requirements System.[Footnote 28] For this deliverable,
review comments were not documented, but CBP communicated in writing
to the contractor that it was withholding comment on this submission
of the deliverable and was to provide a consolidated set of comments
on the subsequent submission. Subsequently, the contractor resubmitted
the deliverable, and because it was accepted, the review was not
documented. The contracting officer communicated the deliverable's
acceptance in writing to the contractor.
By not effectively verifying and accepting contractor deliverables,
the SPO cannot ensure that the deliverables will satisfy stated
requirements, thus increasing the risk of costly and time-consuming
rework. For example, we recently reported that contractor-delivered
test plans were poorly defined and resulted in problems during
testing. In particular, NOC/SOC testing was hampered by requirements
incorrectly mapped to test cases, did not provide for testing all
requirements, and required significant extemporaneous changes to test
cases during the test events.[Footnote 29] As a result of the testing
problems, the SPO had to conduct multiple test events.
Key Contract Technical Reviews Have Not Been Adequately Conducted:
Technical reviews are performed throughout the project life cycle to
confirm that products and services being produced by the contractor
provide the desired capability and ultimately satisfy user needs. To
its credit, DHS has defined a process for conducting technical
reviews, but it has not effectively implemented it. In particular, the
SPO did not ensure that all key documentation was reviewed and
relevant criteria were satisfied before concluding key technical
reviews. Program officials attributed these limitations to the
program's aggressive schedule, which resulted in insufficient time to
review relevant documentation. Concluding technical reviews without
adequate justification has resulted in schedule delays and costly
rework.
CBP Has a Defined Process for Conducting Technical Reviews:
According to relevant guidance, organizations should have written
policies and procedures for conducting technical reviews that, among
other things, (1) assign roles and responsibilities for performing the
specific technical review tasks and (2) establish entry and exit
criteria to determine the readiness of the technical solution to
proceed to the technical review and to demonstrate and confirm
completion of required accomplishments.[Footnote 30]
To its credit, DHS has policies and guidance for conducting technical
reviews. Specifically, DHS's Systems Engineering Life Cycle outlines
the key reviews to be performed as well as how these reviews are
aligned with the department's governance process.[Footnote 31] In
addition, DHS guidance defines expectations for technical review exit
criteria, stating that compliance with exit criteria is based upon the
satisfaction of the content of the criteria, and not upon only the
delivery of specified documents.
To augment DHS policy and guidance, the SBInet Systems Engineering
Plan (SEP), dated November 2008, identifies and describes the
technical reviews to be conducted. They include, for example:
* Requirements Review, which is to ensure that requirements have been
completely and properly identified and are understood by the SPO and
the contractor. Documentation associated with this review is to
include, among other things, a requirements traceability matrix (i.e.,
a tool for demonstrating that component-level requirements are
traceable to higher-level system level requirements).
* Critical Design Review (CDR), which is to (1) demonstrate that the
designs are complete and baselined and (2) ensure that the solution is
ready for fabrication, coding, assembly, and integration.
Documentation for this review is to include, among other things, (1)
baselined requirements, (2) interface descriptions, and (3) identified
risks and mitigation plans.
* Test Readiness Review, which is to assess the readiness of the
system solution to begin formal testing. Documentation for this review
is to include, among other things, (1) test plans that include test
cases and procedures and (2) a traceability matrix that maps each
requirement to be tested to a corresponding test case.
In addition, the SBInet SEP describes high-level roles and
responsibilities for performing these reviews, and establishes entry
and exit criteria for each. For example, it states that the SPO
program manager and the chief engineer are responsible for leading the
reviews. Further, the SEP defines entry and exit criteria for the CDR.
For example:
* Entry. System-level requirements should be traceable to component-
level requirements; system internal and external interfaces should be
defined.
* Exit. Design baseline should be established and balanced across
cost, schedule, performance, and risk considerations over the
investment's life cycle; system risks should be identified and
mitigation plans should be in place.
Contractor Technical Reviews Have Not Been Performed Adequately:
The SPO did not follow the defined process for conducting technical
reviews. Instead, program officials told us that they used the
requirements defined in the respective task orders to guide each
review. However, the task orders do not define entry and exit
criteria. Rather, they list a set of documents that the contractor is
to provide and the SPO is to review. For example, for the Block 1 CDR,
the relevant task order requires that the contractor deliver, among
other documents, (1) baselined component and system requirements, (2)
interface descriptions (i.e., descriptions of the data to be exchanged
and the protocols used to exchange the data), and (3) all identified
risks and mitigation plans for those risks. However, the task orders
do not associate these documents with either entry or exit criteria,
and they do not specify characteristics or qualities that the
documents are to satisfy.
Without explicit entry and exit criteria, the basis for beginning and
ending the technical reviews is unclear, thus increasing the risk that
a program will be allowed to proceed and begin the next phase of
development before it is ready to do so. In fact, this risk was
realized for SBInet. Technical reviews were concluded without adequate
justification, which ultimately resulted in problems that required
additional time and resources to fix. For example:
* NOC/SOC Requirements Review. At this review, the contractor did not
deliver a requirements traceability matrix, as required by the
relevant task order, until almost a month after the review was
completed. Nonetheless, program officials stated that they concluded
the review in June 2008, without knowing whether the applicable higher-
level system requirements were fully satisfied.
* Block 1 CDR. For this review, the contractor delivered (1) the
baselined component and system requirements, (2) the interface
descriptions, and (3) all identified risks and mitigation plans for
those risks.
However, these deliverables did not demonstrate that all component-
level requirements were baselined and interface descriptions were
understood. As we previously reported, baselined requirements
associated with the NOC/SOC were not adequately defined at the time of
the CDR, as evidenced by the fact that they were significantly changed
2 months later.[Footnote 32] Program officials stated that while they
knew that requirements were not adequately baselined at this review,
they believed that the interface requirements were understood well
enough to begin system development. However, this was also not the
case. Specifically, 39 of 90 NOC/SOC interface requirements were
removed from the baseline, and 2 new interface requirements were added
after CDR.
Further, all relevant risks were not identified, and not all
identified risks had mitigation plans. Specifically, 7 of 31
identified risks did not have mitigation plans, including risks
associated with poorly established requirements traceability and
inadequately defined requirements for integration suppliers. Moreover,
the risks identified were as of May 2008, prior to the beginning of
CDR, and did not include four risks identified between June and
October 2008, when CDR was concluded. For example, a risk associated
with the instability of the C3I/COP software was not addressed during
CDR.
Without properly baselined requirements (including interfaces) and
proactive mitigation of known risks, system performance shortfalls are
likely. To illustrate, we previously reported that ambiguities in
requirements actually forced testers to rewrite test steps during
execution based on interpretations of what they thought the
requirements meant, and they required the SPO to incur the time and
expense of conducting multiple events to test NOC/SOC requirements.
[Footnote 33]
* NOC/SOC Component Qualification Test Readiness Review. The SPO did
not ensure that a well-defined test plan was in place, to include,
among other things, test cases and procedures and a traceability
matrix that maps each requirement to be tested to a corresponding test
case. Specifically, the contractor delivered the test plan on the day
of the review, rather than 10 days prior to the review, as required by
the relevant task order. Nevertheless, the SPO concluded the review
based on its review of the plan during the test readiness review. In
this regard, we previously reported problems with the NOC/SOC test
plan, noting that the plan mapped 28 out of 100 requirements to
incorrect test cases. Program officials attributed the test plan
limitations to, among other things, insufficient time and resources to
review the deliverables.
The SBInet independent verification and validation (IV&V) contractor
[Footnote 34] also identified weaknesses within technical reviews.
Specifically, the IV&V contractor reported that the SPO was not
provided with documentation, including the test plan, early enough for
the NOC/SOC test readiness review to allow sufficient time for review.
Moreover, in December 2009, the program identified technical oversight
of technical review milestones as a major risk to the cost, schedule,
and performance goals of the program. According to program officials,
they are developing a technical review manual that is to supplement
the SEP and provide detailed guidance for conducting technical
reviews. In commenting on a draft of this report, DHS stated that it
plans to complete and implement its technical review guide by December
2010.
Program officials attributed the limitations with the technical
reviews, in large part, to the program's aggressive schedule, which
resulted in insufficient time to review relevant documentation. As
discussed above, these limitations have resulted in schedule delays
and costly rework.
Contractor Management Reviews Were Conducted as Planned, but Were
Limited by Unreliable Contractor Performance Data:
Management reviews help to ensure that the contractor's interpretation
and implementation of the requirements are consistent with those of
the program office. According to relevant guidance, organizations
should have written policies and procedures for conducting management
reviews that, among other things, (1) involve relevant stakeholders;
(2) assign roles and responsibilities for performing management review
tasks; (3) communicate project status information, including cost and
schedule information, and risks; and (4) identify, document, and track
action items to closure.[Footnote 35]
CBP policy also recognizes the importance of these reviews by
requiring the conduct of management reviews.[Footnote 36] Further, the
SBInet Program Management Plan identifies the types of management
reviews that are to be conducted with the contractor. For SBInet, the
primary management review is known as the Joint Program Management
Review (JPMR). The plan also identifies, for example, the stakeholders
that are to participate in the reviews, including the program manager,
project managers, program control staff, and the risk management team;
and it specifies the topics that are to be discussed at the reviews,
such as project status, cost and schedule performance, and risks.
To its credit, the program office has held monthly JPMRs, and these
reviews include the prime contractor, the SBI Executive Director, the
SBInet Program Manager, program control staff, risk management team,
border patrol agents, DHS representatives, and the Defense Contract
Management Agency (DCMA). Further, review documentation shows that
JPMRs addressed cost and schedule performance to include EVM
performance data, project milestones status and potential delays, and
risks. Review documentation also shows action items were identified,
documented, and tracked to closure, as part of these reviews. For
example, during the May 2009 JPMR, an action to review the risk
management process--including roles, responsibilities, and decision
authority--was identified, and this action was documented in Boeing's
Management Emphasis Tracking database.[Footnote 37] To address and
close the action item, the program subsequently completed a review of
the SBInet risk management process and tool, including reviewing
lessons learned from other programs. The results of the review were
presented during a February 2010 briefing, and the action item was
closed.
Effectively conducting management reviews has helped to provide
program leadership with an understanding of the contractor's progress
and the program's exposure to risks so that appropriate corrective
action can be taken and the chances of delivering a system solution
that meets mission needs within budget are enhanced. However, as
discussed in the next section, the EVM performance data presented at
these management reviews were not reliable, thus rendering those
reviews, at best, limited in the extent to which they disclosed the
true status of the program.
DHS Has Not Effectively Monitored the Contractor's Progress in Meeting
Cost and Schedule Expectations:
Measuring and reporting progress against cost and schedule
expectations (i.e., baselines) is a vital element of effective
contractor management and oversight. As noted earlier, EVM provides a
proven means for measuring progress against cost and schedule
commitments and thereby identifying potential cost overruns and
schedule delays early, when the impact can be minimized.
However, DHS has not ensured that its prime contractor's EVM system,
which was certified as meeting relevant standards, has been
effectively implemented on SBInet. In particular, it has not ensured
that performance measurement baselines were validated in a timely
manner, that established baselines were complete and realistic, and
that contractor-provided cost and schedule data were reliable. Reasons
cited by program officials for these weaknesses include the
instability in the scope of the work to be performed, an unexpected
temporary stop in Block 1 design and deployment work when SBInet
funding was redirected, and the contractor's use of estimated, rather
than actual, costs for subcontractor work, which are subsequently
adjusted when actual costs are received. Without effectively
implementing EVM, DHS has not been positioned to identify potential
cost and schedule problems early, and thus has not been able to take
timely actions to correct problems and avoid program schedule delays
and cost increases.
Contractor EVM System Has Been Certified:
In August 2005, the Office of Management and Budget issued guidance
[Footnote 38] that, among other things, directs agencies to ensure
that EVM systems are compliant with the American National Standards
Institute (ANSI) standard.[Footnote 39] The ANSI standard consists of
32 guidelines associated with a sound EVM system that are intended to
ensure that data are reliable and can be used for informed decision-
making.
The program office relies on the prime contractor's EVM system to
provide cost and schedule performance data. This system was certified
in April 2005 by DCMA as being compliant with the ANSI standard. DCMA
certified the contractor's EVM system again in February 2009.
Notwithstanding these certifications, DCMA identified a number of
issues with the contractor's implementation of its EVM system.
[Footnote 40] In particular, in January 2010, DCMA reported that the
SBInet prime contractor's implementation of EVM was not consistent
with all of the 32 ANSI guidelines. Specifically, DCMA identified
concerns with the quality of scheduling and reporting, and the
identification of significant differences between planned and actual
cost and schedule performance, as well as reasons for those
differences.
Program Office Has Not Established Timely Performance Baselines:
According to relevant guidance, the performance measurement baseline,
which is the foundation of an EVM system and the estimated cumulative
value of planned work, serves as the value against which performance
is measured for the life of the program or task order.[Footnote 41] As
such, it should be established as early as possible after contract or
task order award, or whenever a major contract modification or
baseline change occurs. DHS guidance further states that a baseline
should be validated within 90 days of the contract or task order
award.[Footnote 42]
However, the program office validated a performance measurement
baseline within 90 days for only two of the six baselines that we
reviewed (see figure 4). For the other four, the length of time to
establish a validated baseline ranged from 5 to 10 months. For
example, the program office issued the ADTO in June 2008, and it did
not establish a validated baseline until 10 months later in April
2009. Similarly, in February 2009, the program office modified the
scope of the STO and extended the period of performance, but it did
not validate the revised baseline to include the additional scope and
time until 7 months later in September 2009. Figure 4 summarizes the
periods of time during which earned value was, and was not, measured
against a validated baseline.
Figure 4: Summary of Time Frames With and Without Validated
Performance Baselines from June 2008 through February 2010:
[Refer to PDF for image: illustration]
Design Task Order:
June 2008 and earlier through March, 2009: Earned value data was not
measured against a government-validated baseline.
March 2009 through August 2009: (IBR 3/09) Earned value data was
measured against a government-validated baseline.
August 2009 through December 2009: Earned value data was not measured
against a government-validated baseline.
December 2009 and beyond: Earned value data was measured against a
government-validated baseline.
C3I/COP:
IBR (2/08): through March 2009: Earned value data was measured against
a government-validated baseline.
System Task Order:
IBR (5/08): through March 2009: Earned value data was measured against
a government-validated baseline.
March 2009 through September 2009: Earned value data was not measured
against a government-validated baseline.
IBR (9/09): September 2009 and beyond: Earned value data was measured
against a government-validated baseline.
Arizona Deployment Task Order:
Contract award: June 2008 through April 2009: Earned value data was
not measured against a government-validated baseline.
IBR (4/09): April 2009 and beyond: Earned value data was measured
against a government-validated baseline.
Source: GAO analysis of SBInet data.
Notes: Generally, the government established baselines through the
period of performance for the task order at the time of the Integrated
Baseline Review (IBR). The government had not completed all work for
the DTO and STO within the original period of performance, and as
such, extended the period of performance, and in some cases,
established a new baseline.
According to program officials, they held an IBR for the C3I/COP task
order in March 2009. However, they did not provide sufficient
documentation to support this statement.
[End of figure]
According to program officials, the delays in validating performance
baselines were due to instability in the work to be performed, and the
need to temporarily stop Block 1 design and deployment work between
September 2008 and January 2009 because of DHS's decision to redirect
funds from SBInet to the physical infrastructure.[Footnote 43] Without
validated baselines, DHS was not positioned to identify potential cost
and schedule problems early and to take timely corrective actions to
mitigate those problems.
DHS Has Not Established Reliable Performance Measurement Baselines:
An integrated baseline review (IBR) is used to validate the
performance measurement baseline. This review is intended to verify
that the baseline is realistic and ensure that the contractor and the
government mutually understand scope, schedule, and risks for a given
task order before a substantial amount of work is performed. According
to relevant guidance, establishing a complete and realistic
performance measurement baseline includes (1) assigning responsibility
for managing, tracking, and reporting earned value data for work
performed; (2) estimating needed resources (i.e., budgets and staff),
including management reserve, for performing assigned tasks; (3)
defining a product-oriented description of all work to be performed;
[Footnote 44] (4) scheduling all work in a time-phased sequence that
reflects the duration of the program's activities; and (5)
establishing objective performance measures for each task.[Footnote 45]
In validating the performance measurement baselines for the four task
orders that we reviewed, the program office implemented two of the
above elements, but it did not implement the other three.
Specifically, for each of the six baselines associated with the task
orders, the program office (1) assigned responsibility for managing,
tracking, and reporting earned value data associated with each work
breakdown structure element and (2) estimated a time-phased budget,
including the anticipated staff needed, for each work breakdown
structure element, and established a management reserve. However, as
discussed in the following section, the program office did not (1)
define a product-oriented description of all work to be performed, (2)
reliably estimate schedule baselines, and (3) adequately measure
earned value performance.
Program officials attribute these limitations in establishing
comprehensive baselines to instability in the nature of the work to be
performed and the prime contractor's method for determining
subcontractor performance. Nevertheless, without complete and
realistic baselines, the SPO has been hampered in its ability to
conduct meaningful measurement and oversight of the prime contractor's
status and progress, as well as holding the contractor accountable for
results. More importantly, the lack of meaningful measurement and
oversight has contributed to program cost overruns and schedule delays.
Work Breakdown Structure Was Not Product Oriented and Did Not Include
All Work:
According to relevant guidance, a work breakdown structure
deconstructs a program's end product into successively smaller levels
until the work is subdivided to a level suitable for management
control.[Footnote 46] Further, a work breakdown structure should be
product oriented and include all work to be performed.
The work breakdown structure that was used to define each of the task
order baselines was not product oriented. Instead, it was defined in
terms of functions that span multiple system products, such as systems
engineering, system test and evaluation, and program management.
Additionally, the work breakdown structure did not reflect all work to
be performed. Specifically, for four of the six performance
measurement baselines, the work breakdown structure did not include
all work described in the corresponding task order's statement of
work. For example, the work breakdown structure used to define the May
2008 STO baseline did not include the work associated with identifying
and selecting components that meet system requirements and program
security. Similarly, DCMA reported in June 2008 that the work
breakdown structure included in this baseline did not account for all
work identified in the system task order.
Estimated Baseline Schedules Were Not Reliable:
A reliable schedule provides a road map for systematic execution of a
program and the means by which to gauge progress, identify and address
potential problems, and promote accountability. Our research has
identified nine best practices associated with developing and
maintaining a reliable schedule: (1) capturing all activities, (2)
sequencing all activities, (3) assigning resources to all activities,
(4) establishing the duration of all activities, (5) integrating
activities horizontally and vertically, (6) establishing the critical
path[Footnote 47] for all activities, (7) identifying reasonable
"float"[Footnote 48] between activities, (8) conducting a schedule
risk analysis, and (9) updating the schedule using logic and
durations.[Footnote 49]
The six task order baselines were not reliable because they
substantially complied with only two of the eight key schedule
estimating practices, and they did not comply with, or only partially
or minimally complied with, the remaining six practices.[Footnote 50]
(See figure 5 for a summary of the extent to which each of the
baseline schedules met each of the eight practices.)
Figure 5: Summary of IBR Schedules' Satisfaction of Schedule
Estimating Practices:
[Refer to PDF for image: illustrated table]
Schedule practice: Capturing all activities:
Task order schedule:
C3I/COP February 2008: Minimally met;
STO May 2008: Minimally met;
DTO March 2009: Partially met;
ADTO April 2009: Minimally met;
STO September 2009: Minimally met;
DTO December 2009: Minimally met.
Schedule practice: Sequencing all activities:
Task order schedule:
C3I/COP February 2008: Substantially met;
STO May 2008: Substantially met;
DTO March 2009: Substantially met;
ADTO April 2009: Substantially met;
STO September 2009: Substantially met;
DTO December 2009: Substantially met.
Schedule practice: Assigning resources to all activities:
Task order schedule:
C3I/COP February 2008: Minimally met;
STO May 2008: Minimally met;
DTO March 2009: Minimally met;
ADTO April 2009: Partially met;
STO September 2009: Minimally met;
DTO December 2009: Partially met.
Schedule practice: Establishing the duration of all activities:
Task order schedule:
C3I/COP February 2008: Substantially met;
STO May 2008: Substantially met;
DTO March 2009: Substantially met;
ADTO April 2009: Substantially met;
STO September 2009: Substantially met;
DTO December 2009: Substantially met.
Schedule practice: Integrating activities horizontally and vertically:
Task order schedule:
C3I/COP February 2008: Partially met;
STO May 2008: Partially met;
DTO March 2009: Partially met;
ADTO April 2009: Partially met;
STO September 2009: Partially met;
DTO December 2009: Partially met.
Schedule practice: Establishing the critical path for all activities:
Task order schedule:
C3I/COP February 2008: Partially met;
STO May 2008: Partially met;
DTO March 2009: Partially met;
ADTO April 2009: Partially met;
STO September 2009: Partially met;
DTO December 2009: Partially met.
Schedule practice: Identifying reasonable float between activities:
Task order schedule:
C3I/COP February 2008:
STO May 2008: Partially met;
DTO March 2009: Minimally met;
ADTO April 2009: Partially met;
STO September 2009: Partially met;
DTO December 2009: Partially met.
Schedule practice: Conducting a schedule risk analysis:
Task order schedule:
C3I/COP February 2008: Not met;
STO May 2008: Not met;
DTO March 2009: Not met;
ADTO April 2009: Not met;
STO September 2009: Not met;
DTO December 2009: Not met.
Met = DHS provided complete evidence that satisfied the criteria.
Substantially Met = DHS provided evidence that satisfied more than one-
half of the criteria.
Partially Met = DHS provided evidence that satisfied about one-half of
the criteria.
Minimally Met = DHS provided evidence that satisfied less than one-
half of the criteria.
Not Met = DHS provided no evidence that satisfied any of the criteria.
Source: GAO analysis of DHS data.
[End of figure]
* Capturing all activities. The six schedules did not capture all
activities defined in the task order baseline. Specifically, five of
the six schedules did not reflect the work to be performed across the
four task orders (i.e., integrated master schedule). Further, as
previously mentioned, four of six work breakdown structures were
missing elements defined in the respective task order statements of
work. Moreover, two of the six schedules did not reflect all work that
was defined in the work breakdown structure. For example, the December
2009 DTO schedule omitted efforts associated with design work for TUS-
1 and AJO-1.
* Sequencing all activities. The six schedules substantially met this
practice. Each of the schedules identified almost all of the
predecessor and successor activities. However, each contained improper
predecessor and successor relationships. For example, the May 2008 STO
baseline included 52 of 538 activities (about 10 percent) with
improper predecessor and successor relationships. Additionally, many
activities in four of the schedules were constrained by "start no
earlier than" dates. For example, as previously reported, the
September 2009 baseline schedule contained 403 of 1,512 activities
(about 27 percent) with "start no earlier than" constraints, which
means that these activities are not allowed to start earlier than
their assigned dates, even if their respective predecessor activities
have been completed.
* Assigning resources to all activities. Two of the six schedules
partially met this practice. Specifically, two schedules included
resources; however, those resources were allocated to less than 15
percent of the activities identified in each schedule. Moreover, the
remaining four schedules did not include estimated resources. Instead,
resources for all six schedules were maintained separately as part of
the contractor's earned value system and only available to DHS upon
request.
* Establishing the duration of all activities. Each of the six
baseline schedules substantially met this practice. Specifically, each
schedule established the duration of key activities and included
baseline start and end dates for most of the activities. Further,
reasonable durations were established for the majority of the
activities in the schedules, meaning that the durations established
were less than 44 days. Nevertheless, each of the schedules included
activities that were not of short duration, that is, more than 44
days. For example, the April 2009 ADTO baseline included 29 of 1,009
activities with durations ranging from 45 days to 352 days.
* Integrating activities horizontally and vertically. Each of the
schedules partially met this practice. As mentioned previously, the
six schedules did not capture all activities defined in the task order
baseline. Further, four of six work breakdown structures were missing
elements defined in respective task order statements of work.
Additionally, five of six schedules did not reflect the work performed
across the four task orders (i.e., integrated master schedule), and
each had improper predecessor and successor relationships.
* Establishing the critical path for all activities. Each of the six
schedules partially met this practice. Specifically, four of six work
breakdown structures were missing elements defined in the respective
task order statements of work. Additionally, four of the six schedules
were missing predecessor and successor activities, and each of the
schedules included improper predecessor and successor relationships.
Further, five of the six schedules did not reflect the work to be
performed across the four task orders (i.e., integrated master
schedule). Unless all activities are included and properly linked, it
is not possible to generate a true critical path.
* Identifying reasonable float between activities. Each of the
schedules identified float; however, the amount of float was
excessive. For example, the February 2008 C3I/COP task order baseline
included 259 of 294 activities (about 88 percent) with float greater
than 100 days and 189 of the 259 (about 73 percent) with float in
excess of 200 days.
* Conducting a schedule risk analysis. DHS did not conduct a risk
analysis of any of the schedules.
Earned Value Performance Was Not Adequately Measured:
According to the ANSI standard for EVM systems,[Footnote 51] only work
for which measurement is impractical may be classified as "level-of-
effort."[Footnote 52] Our research shows that if more than 15 percent
of a program's budget is measured using level-of-effort, then that
amount should be scrutinized because it does not allow schedule
performance to be measured (i.e., performance equals planned work).
[Footnote 53]
However, the six baselines had between 34 and 85 percent of the
baseline dollar value categorized as level-of-effort, including four
with more than 50 percent (see table 2). Moreover, for five of the six
baselines, program documentation showed that the program office did
not identify any action items during the respective IBRs related to
the high use of level-of-effort.
According to program officials, the STO, which categorized between 70
and 85 percent of the baseline dollar value as level-of-effort,
includes many program management activities (e.g., cost, schedule, and
subcontractor management). Nevertheless, they recognized that the
level-of-effort for this task order was high, and in November 2009,
they directed the contractor to minimize the use of level-of-effort
for STO. According to program officials, the high level-of-effort was
due, in part, to the prime contractor's use of this measurement for
subcontractor work. In November 2009, DCMA stated that the SPO's use
of level-of-effort activities was high, noting that this could be
masking true contractor performance.
Table 2: Percentage of Performance Measurement Baseline Work Tasks
Categorized as Level-of-Effort for Six Baselines:
Task order IBR: C3I/COP task order February 2008;
Level-of-effort percentage: 46 percent.
Task order IBR: STO May 2008;
Level-of-effort percentage: 85 percent.
Task order IBR: DTO March 2009;
Level-of-effort percentage: 53 percent.
Task order IBR: ADTO April 2009;
Level-of-effort percentage: 56 percent.
Task order IBR: STO September 2009;
Level-of-effort percentage: 70 percent.
Task order IBR: DTO December 2009;
Level-of-effort percentage: 34 percent.
Source: GAO analysis of DHS data.
[End of table]
Unreliable EVM Data Limit DHS's Ability to Measure Contractor
Performance:
If performed properly, EVM can provide an objective means for
measuring program status and forecasting potential program cost
overruns and schedule slippages so that timely action can be taken to
minimize their impact. To do so, however, the underlying EVM data must
be reliable, meaning that they are complete and accurate and all data
anomalies are explained.
In the case of SBInet, the EVM data provided by the prime contractor
for the 21-month period ending in February 2010 have not been
reliable, as evidenced by numerous and unexplained anomalies in
monthly EVM reports. Reasons for the anomalies include the
contractor's use of estimated, rather than actual, costs for
subcontractor work, which are subsequently adjusted when actual costs
are received. Without reliable performance data, the true status of
the SBInet program is unclear, thus limiting the SPO's ability to
identify potential cost and schedule shortfalls.
EVM Data Can Provide Objective Insight into Program Cost and Schedule
Performance:
EVM is a proven program measurement approach that, if implemented
appropriately, can create a meaningful and coherent understanding of a
program's true health and status. As a result, the use of EVM can
alert decision makers to potential program problems sooner than is
possible by using actual versus planned expenditure alone, and thereby
reduce the chance and magnitude of program cost overruns and schedule
slippages.
Simply stated, EVM measures the value of completed work in a given
period (i.e., earned value) against (1) the actual cost of work
completed for that period (i.e., actual cost) and (2) the value of the
work that is expected to be completed for that period (i.e., planned
value). Differences in these values are referred to as cost and
schedule variances, respectively.
* Cost variances compare the value of the work completed with the
actual cost of the work performed. For example, if a contractor
completed $5 million worth of work and the work actually cost $6.7
million, there would be a negative $1.7 million cost variance.
* Schedule variances are also measured in dollars, but they compare
the value of the work completed with the value of the work that was
expected to be completed. For example, if a contractor completed $5
million worth of work at the end of the month but was expected to
complete $10 million worth of work, there would be a negative $5
million schedule variance.
Positive variances indicate that activities are costing less or are
being completed ahead of schedule. Negative variances indicate
activities are costing more or are falling behind schedule. To
determine both cost and schedule variances, all three values are
necessary.
SBInet EVM Data Have Not Been Reliable:
According to relevant guidance, EVM data should be valid and free from
unexplained anomalies (e.g., missing or negative values) because they
can limit program management's ability to identify potential cost and
schedule shortfalls.[Footnote 54] Therefore, anomalies should be
minimized for each of the three values--earned value, planned value,
and actual cost. Moreover, all anomalies should be identified, and the
reason for each should be fully explained in the monthly EVM reports.
To do less limits the completeness and accuracy of these values, and
thus makes the resulting variance determinations unreliable. While an
industry standard for what constitutes an acceptable volume of
anomalies does not exist, EVM experts in the public and private
sectors that we interviewed stated that the occurrence of EVM data
anomalies should be rare. Of these experts, some agreed that an
anomaly should occur in no more than 5 percent of the work breakdown
structure elements for a given contract or task order, while some of
these advocated an occurrence percentage of no more than 1-2 percent.
However, the EVM data that the prime contractor delivered to the SPO
from June 2008 through February 2010 (21 months) contained numerous,
unexplained anomalies. Specifically, the monthly EVM reports for all
four task orders that we reviewed showed one or more anomalies (e.g.,
missing or negative values for earned value, planned value, and actual
cost) in each of the months that had a validated performance
measurement baseline. More specifically, the average number of work
breakdown structure elements across the four task orders that had data
anomalies during this 21-month period ranged from 11 percent to 41
percent. For the C3I/COP task order in particular, the monthly
percentage of work breakdown structure elements with anomalies ranged
between 25 and 67 percent over the 21 months. (See figure 6 for the
percentage of work breakdown structure elements with anomalies by
month for each of the four task orders.)
Figure 6: Percentage of Work Breakdown Structure Elements with EVM
Data Anomalies by Month for Each of the Four Task Orders:
[Refer to PDF for image: vertical bar graph]
2008:
June:
Arizona Deployment: 0;
Design: 0;
C3I/COP: 33%;
System: 7%.
July:
Arizona Deployment: 0;
Design: 0;
C3I/COP: 33%;
System: 4%.
August:
Arizona Deployment: 0;
Design: 0;
C3I/COP: 33%;
System: 13%.
September:
Arizona Deployment: 0;
Design: 0;
C3I/COP: 42%;
System: 22%.
October:
Arizona Deployment: 0;
Design: 0;
C3I/COP: 42%;
System: 18%.
November:
Arizona Deployment: 0;
Design: 0;
C3I/COP: 58%;
System: 9%.
December:
Arizona Deployment: 0;
Design: 0;
C3I/COP: 67%;
System: 16%.
2009:
January:
Arizona Deployment: 0;
Design: 0;
C3I/COP: 33%;
System: 13%;
February:
Arizona Deployment: 0;
Design: 0;
C3I/COP: 25%;
System: 2.
March:
Arizona Deployment: 0;
Design: 0;
C3I/COP: 0;
System: 0.
April:
Arizona Deployment: 0;
Design: 14%;
C3I/COP: 0;
System: 0.
May:
Arizona Deployment: 11
Design: 10
C3I/COP: 0;
System: 0.
June:
Arizona Deployment: 7%;
Design: 10%;
C3I/COP: 0;
System: 0.
July:
Arizona Deployment: 11%;
Design: 10%;
C3I/COP: 0;
System: 0.
August:
Arizona Deployment: 15%;
Design: 0;
C3I/COP: 0;
System: 0.
September:
Arizona Deployment: 22%;
Design: 0;
C3I/COP: 0;
System: 0.
October:
Arizona Deployment: 11%;
Design: 0;
C3I/COP: 0;
System: 24%.
November:
Arizona Deployment: 19%;
Design: 0;
C3I/COP: 0;
System: 29%.
December:
Arizona Deployment: 41%;
Design: 0;
C3I/COP: 0;
System: 64%.
2010:
January:
Arizona Deployment: 41%;
Design: 10%;
C3I/COP: 0;
System: 22%.
February:
Arizona Deployment: 56%;
Design: 14%;
C3I/COP: 0;
System: 20%.
Source: GAO analysis of DHS data.
Note: We only analyzed EVM reports for those months with a validated
baseline for any of the four task orders. There were no validated
baselines for any of the four task orders in March 2009.
[End of figure]
The October 2009 STO monthly EVM report illustrates how the anomalies
can distort the contractor's performance. According to this report,
about $13,000 worth of work was planned to be completed on integration
and management, approximately $13,000 worth of work was performed, and
actual costs were about negative $550,000. Thus, the report
erroneously suggests that the contractor performed $13,000 worth of
work, and actually saved about $550,000 in doing so. Similarly, the
September 2009 ADTO monthly report showed that about $200,000 worth of
work was planned to be completed on tower sites and infrastructure,
and $25,000 worth of work was performed, but that no costs were
incurred, suggesting that the work was performed for free.
Exacerbating the large percentage of monthly data anomalies across the
four task orders is the fact that in most cases the reasons for the
anomalies were not explained in the monthly EVM variance analysis
reports. Specifically, about 79 percent of all anomalies across all
four task orders during the 21-month period were not explained. In
particular, 82 of 119 (or about 69 percent) of all data anomalies for
the STO task order were not explained in the monthly reports, and none
of the anomalies were explained for DTO. (See figure 7 for the total
number of data anomalies and the number that were explained in the
monthly reports across the four task orders.)
Figure 7: Total Number of Data Anomalies and Number of Explained
Anomalies in the Monthly EVM Reports across the Four Task Orders:
[Refer to PDF for image: vertical bar graph]
2008:
June:
Total EVM data anomalies: 7;
Total EVM data anomalies with explanations: 0.
July:
Total EVM data anomalies: 6;
Total EVM data anomalies with explanations: 0.
August:
Total EVM data anomalies: 10;
Total EVM data anomalies with explanations: 3.
September:
Total EVM data anomalies: 15;
Total EVM data anomalies with explanations: 4.
October:
Total EVM data anomalies: 13;
Total EVM data anomalies with explanations: 2.
November:
Total EVM data anomalies: 11;
Total EVM data anomalies with explanations: 2.
December:
Total EVM data anomalies: 15;
Total EVM data anomalies with explanations: 1.
2009:
January:
Total EVM data anomalies: 10;
Total EVM data anomalies with explanations: 2.
February:
Total EVM data anomalies: 4;
Total EVM data anomalies with explanations: 2.
March:
Total EVM data anomalies: 0;
Total EVM data anomalies with explanations: 0.
April:
Total EVM data anomalies: 3;
Total EVM data anomalies with explanations: 0.
May:
Total EVM data anomalies: 5;
Total EVM data anomalies with explanations: 1.
June:
Total EVM data anomalies: 4;
Total EVM data anomalies with explanations: 1.
July:
Total EVM data anomalies: 5;
Total EVM data anomalies with explanations: 1.
August:
Total EVM data anomalies: 4;
Total EVM data anomalies with explanations: 1.
September:
Total EVM data anomalies: 6;
Total EVM data anomalies with explanations: 1.
October:
Total EVM data anomalies: 14;
Total EVM data anomalies with explanations: 2.
November:
Total EVM data anomalies: 18;
Total EVM data anomalies with explanations: 0.
December:
Total EVM data anomalies: 40;
Total EVM data anomalies with explanations: 26.
2010:
January:
Total EVM data anomalies: 23;
Total EVM data anomalies with explanations: 1.
February:
Total EVM data anomalies: 27;
Total EVM data anomalies with explanations: 1.
Source: GAO analysis of DHS data.
Note: As mentioned previously, there were no validated baselines for
any of the four task orders in March 2009.
[End of figure]
Program officials acknowledged problems with the EVM data and stated
that they meet with the prime contractor each month to discuss the EVM
reports, including the reliability of the data. According to program
officials, limitations in the EVM data are due, in part, to the
contractor's use of estimated, rather than actual, costs for
subcontractor work, which are subsequently adjusted when actual costs
are received. Officials further stated that they have been working
with the contractor to reduce the volume of unexplained anomalies, and
they believe that the reliability of the data has improved since
February 2010. However, program officials did not provide any
documentation to support this statement. Without reliable EVM data,
the program office is unable to identify actual cost and schedule
shortfalls, which along with the other contractor tracking and
oversight weaknesses discussed in this report, has limited its ability
to effectively minimize program cost increases and schedule delays.
Conclusions:
Effective management and oversight of a program's prime contractor is
essential to successfully acquiring and deploying a system like
SBInet. Integral to accomplishing this is defining and implementing a
range of contractor management and oversight controls (e.g., processes
and practices) that reflect relevant federal guidance and best
practices. To do less increases the chances that contractor-delivered
products and services will not satisfy stated requirements and will
not meet customer expectations. The result is incurring the additional
time and expense to redo or rework contractor deliverables, accepting
products and services that do not perform as intended and do not meet
mission needs, or both.
Overall, DHS has not done an effective job of managing and overseeing
its prime contractor, including monitoring the contractor's
performance. DHS has largely defined key management and oversight
processes and practices that it should have followed, and it
implemented a number of these processes and practices. However,
several key management and oversight controls were not adequately
defined, and essential controls were not implemented. Most
significantly, DHS did not adequately document deliverable reviews and
communicate the basis for rejecting certain deliverables in writing to
the contractor, which contributed to deliverables that did not live up
to expectations and necessitated rework and caused later problems.
Further, technical reviews were not grounded in explicit criteria for
determining when reviews should begin and conclude, which also
contributed to contract deliverables requiring costly and time-
consuming rework. In addition, the cost and schedule baselines for
measuring the contractor's performance were frequently validated too
late and without sufficient accuracy and completeness to provide a
meaningful basis for understanding performance, which precluded DHS
from taking timely action to correct unfavorable results and trends.
Compounding these serious baseline limitations was contractor-provided
data about actual performance that were replete with unexplained
anomalies, thus rendering the data unfit for effective contractor
management and oversight. Notwithstanding of a number of contractor
management and oversight definition and implementation efforts that
DHS executed well, such as defining key processes and practices and
training key staff, these above-cited weaknesses collectively mean
that DHS's management and oversight of its prime contractor has been a
major contributor to the SBInet program's well-chronicled history of
not delivering promised system capabilities on time and on budget.
These limitations can be attributed to a number of factors, including
gaps in how certain processes and practices were defined, as well as
not enforcing other processes and practices that were defined and
applicable and not taking sufficient time to review deliverables that
were submitted late. The limitations can be further attributed to the
fact that SBInet has from its outset lacked clear definition and
stability, and thus experienced continuous change in scope and
direction--an issue that we have previously reported and made
recommendations to address. Collectively, these factors have helped to
create a contractor management and oversight environment, which, when
combined with the many other acquisition management weaknesses that we
have previously reported about and made recommendations to address,
have produced a program that to date has not been successful, and if
not corrected, can become worse.
Recommendations for Executive Action:
To improve DHS management and oversight of the SBInet prime
contractor, we recommend that the Secretary of Homeland Security
direct the Commissioner of the U.S. Customs and Border Protection to
have the SBI Executive Director, in collaboration with the SBInet
Program Director, take the following four actions:
* Revise and implement, as applicable, contractor deliverable review
processes and practices to ensure that (1) contractor deliverables are
thoroughly reviewed and are not constrained by late contractor
deliverables and imposed milestones, (2) the reviews are sufficiently
documented, and (3) the acceptance or the rejection of each contractor
deliverable is communicated in writing to the contractor, to include
explicit explanations of the basis for any rejections.
* Ensure that applicable entry and exit criteria for each technical
review are used and satisfied before initiating and concluding,
respectively, a given review.
* Establish and validate timely, complete, and accurate performance
measurement baselines for each new task order or major modification of
an existing task order, as appropriate, to include, but not be limited
to, ensuring that (1) the work breakdown structure includes all work
to be performed, (2) baseline schedules reflect the key schedule
estimating practices discussed in this report, and (3) level-of-effort
performance measurement in excess of 15 percent is scrutinized,
justified, and minimized.
* Ensure that all anomalies in contractor-delivered monthly earned
value management reports are identified and explained, and report
periodically to DHS acquisition leadership on relevant trends in the
number of unexplained anomalies.
Because we have already made recommendations in prior reports to
address the other management and oversight weaknesses discussed in
this report, such as those related to requirements management, risk
management, and Systems Engineering Plan implementation, we are not
making any additional recommendations at this time.
Agency Comments and Our Evaluation:
In written comments on a draft of this report, signed by the Director,
Departmental GAO/OIG Liaison Office and reprinted in appendix II, DHS
agreed with our four recommendations and described actions under way
or planned, which we summarize below, to address them.
* With respect to our recommendation to revise and implement the
contractor deliverable review process, DHS stated that it is updating
the process to require written documentation of each review and the
communication to the contractor of review results.
* With respect to our recommendation to ensure that entry and exit
criteria are used to initiate and conclude each technical review, DHS
stated that it has established an SBI Systems Engineering Directorate
to focus on technical oversight of the acquisition process, adding
that the Directorate is developing a technical review guide that
describes in detail the review process and the relevant entry and exit
criteria for each technical review.
* With respect to our recommendation to establish and validate timely,
complete, and accurate performance measurement baselines, DHS stated
that it is mindful of the need to establish and maintain current
performance baselines, and to plan and implement baseline updates as
completely and promptly as practicable, which it indicated is done
through IBRs. DHS also noted that while scheduling practices remain a
challenge, it continues to make improvements to its process, including
implementing scheduling tools and templates.
* With respect to our recommendation to identify and explain all
anomalies in monthly EVM reports, and to report periodically relevant
trends to DHS acquisition leadership, DHS acknowledged the need to
correctly document anomalies in the monthly EVM reports, and stated
that it is working with DCMA to improve contractor quality control
issues and the content of the monthly EVM reports. It also stated that
it is augmenting the reports with routine conversations between
contractor and project management staff. The department also committed
to advising the appropriate acquisition leaders through established
reporting and oversight opportunities as issues arise with contractor
performance or reporting.
Notwithstanding its agreement with our recommendations, the department
also commented that it took exception to selected findings and
conclusions regarding the program's implementation of EVM. A summary
of DHS's comments and our responses are provided below.
* The department stated that it took exception to our finding that it
did not ensure performance measurement baselines were validated in a
timely manner, and said that it was not accurate to conclude that the
lack of validated baselines precluded the program office from
identifying cost and schedule problems and taking corrective action.
In support of these positions, the department made the following three
points, which our response addresses.
First, the department stated that the SBInet program office delayed
formal IBRs until it had finalized negotiated modifications to the
task orders, and in doing so, was able to complete an IBR within 90
days of each major task order modification. We do not question whether
the program office held IBRs within 90 days of final negotiation of
major task order modifications. Our point is that DHS did not validate
task order performance measurement baselines (i.e., hold IBRs) within
90 days of task order award, which is what DHS guidance states should
occur. As our report states, the program office only met this 90-day
threshold for two of the six baselines that we reviewed. Further, the
length of time to validate the performance baselines for the four task
orders far exceeded 90 days (5 to 10 months), during which time DHS
reports show that significant work was performed and millions of
dollars were expended. In fact, the DHS reports show that most of the
planned work for some of these task orders had already been performed
by the time the IBR was held and the baseline was validated. As we
state in our report, and DHS acknowledged in its comments, the purpose
of an IBR is to verify that the performance baseline is realistic and
that the scope, schedule, and risks are mutually understood by the
contractor and the government before a substantial amount of work is
performed.
Second, DHS commented that the program office maintained what it
referred to as "interim" performance measurement baselines during the
period of major program scope, schedule, and budget changes. We
acknowledge that in some cases the program office had these "interim"
baselines. However, these baselines are the contractor-provided
baselines, meaning that the program office and the contractor had not
mutually agreed to the scope, schedule, and risks associated with the
work to be performed. Moreover, for two of the task orders, the
program office did not have an "interim" baseline, even though the
contractor performed significant work under these task orders.
[Footnote 55]
Third, the department stated that program leadership reviewed the
contractor's technical and financial performance information relative
to performance measurement baselines and implemented management
actions as needed. We do not question whether program leadership
reviewed contractor-provided performance information or whether
actions to address problems may have been taken. However, our report
does conclude, as is discussed later in this section, that the
combination of the EVM weaknesses that our report cites, to include
unreliable performance baselines and contractor-provided performance
data, did not allow the program office to identify performance
problems early and to take timely actions to avoid the well-documented
schedule delays and cost increases that the program has experienced.
* The department expressed two concerns with how it said our report
characterized and quantified EVM anomalies.
First, the department stated that our report failed to distinguish
between factual errors and legitimate monthly accounting adjustments.
We agree that our report does not distinguish between the two types of
anomalies, and would add that this was intentional because making the
distinction was not relevant to our finding. Specifically, our finding
is that the reasons for the numerous anomalies were not explained in
the monthly EVM variance analysis reports, therefore making the true
status of the program unclear.
Second, DHS stated that we incorrectly concluded that both errors and
adjustments are problematic, distort cost performance, and limit
management insight. In response, we did not conclude that all errors
and adjustments have these impacts, but rather that the lack of
explanation associated with such a large volume of anomalies made the
true status of the program unclear, thus limiting the program office's
ability to identify actual cost and schedule shortfalls, which is
certainly problematic. Further, our report cites examples of cost
performance data that provide a distorted picture of actual
performance vis-ŕ-vis expectations. Accordingly, the correct
characterization of the report's conclusion concerning the reliability
of EVM data is that the lack of explanation of the numerous anomalies
in monthly reports is problematic, provides a distorted picture of
cost performance, and limits management insight. To this very point,
DHS acknowledged in its comments the importance of explaining the
reason for anomalies in the monthly variance reports, regardless of
whether they are due to factual errors or accounting adjustments.
* The department stated that it took exception to our conclusion that
the program office's lack of validated baselines in particular, and
EVM shortcomings in general, contributed to cost and schedule growth
and made it unable to identify cost and schedule problems early and
take corrective actions to avoid them. In response, we did not
conclude that the lack of validated baselines alone had either of
these impacts. However, we did conclude that the collection of EVM
weaknesses discussed in our report, to include untimely validated
baselines, incomplete and unreliable baselines, and unreliable
performance data, together precluded the program office from
identifying problems early and taking corrective action needed to
avoid the program's well-chronicled history of schedule delays and
cost increases. In support of this conclusion, we state in the report,
for example, that the performance measurement baselines that we
reviewed understated the cost and time necessary to complete the work
because they did not capture all work in the task orders' statements
of work and because they were not grounded in a range of scheduling
best practices. Given that cost and schedule growth is a function of
the baseline against which actual cost and schedule performance is
measured, it follows logically that an understated baseline would
produce actual cost overruns and schedule delays. In addition, we
would note that beyond these EVM shortcomings, our report also
recognizes other contract tracking and oversight, test management, and
requirements management weaknesses that have collectively contributed
to the program's cost, schedule, and performance shortfalls.
In addition to the above points, DHS provided technical comments,
which we have incorporated in the report as appropriate.
We are sending copies of this report to the Chairmen and Ranking
Members of the Senate and House Appropriations Committees and other
Senate and House committees and subcommittees that have authorization
and oversight responsibilities for homeland security. We will also
send copies to the Secretary of Homeland Security, the Commissioner of
U.S. Customs and Border Protection, and the Director of the Office of
Management and Budget. In addition, the report will be available at no
cost on the GAO Web site at [hyperlink, http://www.gao.gov].
Should you or your offices have any questions on matters discussed in
this report, please contact me at (202) 512-3439 or at hiter@gao.gov.
Contact points for our Offices of Congressional Relations and Public
Affairs may be found on the last page of this report. Key contributors
to this report are listed in appendix III.
Signed by:
Randolph C. Hite:
Director, Information Technology Architecture and Systems Issues:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
Our objectives were to determine the extent to which the Department of
Homeland Security (DHS) (1) has defined and implemented effective
controls for managing and overseeing the Secure Border Initiative
Network (SBInet) prime contractor and (2) is effectively monitoring
the prime contractor's progress in meeting cost and schedule
expectations. To accomplish our objectives, we focused on four key
task orders--the Arizona Deployment Task Order, the Design Task Order,
the System Task Order, and the Command, Control, Communications, and
Intelligence/Common Operating Picture Task Order--that are integral to
the design, development, and deployment of the first increment of
SBInet, known as Block 1. We also focused on the SBInet System Program
Office's (SPO) contracting and oversight activities that occurred from
June 2008 through February 2010.
To determine the extent to which DHS has defined and implemented
effective controls for managing and overseeing the SBInet prime
contractor, we focused on training, verifying and accepting contract
deliverables, conducting technical reviews, and conducting management
reviews. We chose these four because each is important to tracking and
overseeing contractor performance for the four task orders.
* Training. We compared relevant DHS contractor management and
oversight training requirements to the program managers', contracting
officers', and contracting officer's technical representatives'
training certifications.
* Verifying and accepting contract deliverables. We compared U.S.
Customs and Border Protection's (CBP) process for verifying and
accepting contract deliverables to leading industry practices[Footnote
56] to identify any variances. We then assessed a nonprobability,
random sample of 28 contract deliverables that the SPO identified as
being delivered between June 1, 2008, and August 31, 2009.[Footnote
57] We also judgmentally selected one additional review that was
conducted between September 1, 2009, and February 28, 2010.[Footnote
58] For each of the 29 deliverables, we reviewed relevant
documentation, such as the contract deliverables, review comment
forms, and documented communications with the prime contractor
indicating acceptance or rejection of the deliverable, and compared
them to the CBP process and leading industry practices to determine
what, if any, deviations existed.
* Technical reviews. We compared relevant DHS and CBP guidance and
entry and exit criteria in the task order Data Item Descriptions to
leading industry practices to identify any variances. We assessed a
nonprobability, random sample of technical reviews. Specifically, we
assessed a technical review from each of the eight unique combinations
of task orders and review types held between June 1, 2008, and August
31, 2009. We also judgmentally selected one additional review that was
conducted between September 1, 2009, and February 28, 2010.[Footnote
59] For each of the nine reviews, we compared the package of
documentation prepared for and used during these reviews to the
criteria defined in the relevant task orders to determine the extent
to which the reviews satisfied the criteria.
* Management reviews. We compared relevant CBP guidance to leading
industry practices to identify any variances. We then compared
relevant documentation prepared for and used during monthly joint
program management reviews to determine the extent to which the
reviews addressed cost, schedule, and program risks. We also assessed
a nonprobability sample of 11 action items that were identified during
the reviews held from October 2008 through October 2009,[Footnote 60]
and assessed relevant documentation to determine the extent to which
they were tracked to closure.
To determine the extent to which DHS is effectively monitoring the
prime contractor's progress in meeting cost and schedule expectations,
we focused on the program's implementation of earned value management
(EVM) because it was the tool used to monitor the contractor's cost
and schedule performance. Specifically, we analyzed the six
performance measurement baselines, and the associated integrated
baseline review documentation, such as briefings, the work breakdown
structure (WBS) governing all task orders, task order statements of
work, schedules, monthly contract performance reports, control account
plans, and responsibility assignment matrixes. In doing so, we
compared this documentation to EVM and scheduling best practices as
identified in our Cost Estimating and Assessment Guide.[Footnote 61]
Specifically, for each of the six baselines:
* We reviewed control account plans and responsibility assignment
matrixes to determine the period of performance and scope of work for
each baseline, compared the work described in the respective task
order statements of work to the work described in the responsibility
assignment matrix, and reviewed the control account plans to determine
the extent to which the level-of-effort measurement method was to
measure contractor performance.
* We analyzed the schedule presented at each baseline review against
eight key schedule estimating practices in our Cost Estimating and
Assessment Guide.[Footnote 62] In doing so, we used commercially
available software tools to determine whether each schedule, for
example, included all critical activities, a logical sequence of
activities, and reasonable activity durations. Further, we
characterized the extent to which the schedule met each of the
practices as either not met, minimally met, partially met,
substantially met, or met.
* We analyzed the contract performance reports for each of the four
task orders for each month that there was a validated baseline.
Specifically, we identified instances of the following: (1) negative
planned value, earned value, or actual cost; (2) planned value and
earned value without actual cost; (3) earned value and actual cost
without planned value; (4) actual cost without planned value or earned
value; (5) earned value without planned value and actual cost; (6)
inconsistencies between the estimated cost at completion and the
planned cost at completion; (7) actual cost exceeding estimated cost
at completion; and (8) planned or earned values exceeding planned cost
at completion. To determine the number of anomalies, we identified
each WBS element that had one or more of the above anomalies. Then, we
identified the number of WBS elements at the beginning and the end of
the baseline period of performance, and calculated the average number
of WBS elements. We used this to determine the percentage of WBS
elements with anomalies for each task order and for each month for
which there was a validated baseline.
* To support our work across this objective, we interviewed officials
from the Department of Defense's Defense Contract Management Agency
(DCMA), which provides contractor oversight services to the SPO,
including oversight of EVM implementation, and prime contractor
officials. We also reviewed DCMA monthly status reports and corrective
action reports.
For both objectives, we interviewed program officials to obtain
clarification on the practices, and to determine the reasons for any
deviations.
To assess the reliability of the data that we used to support the
findings in this report, we reviewed relevant program documentation to
substantiate evidence obtained through interviews with knowledgeable
agency officials, where available. We determined that the data used in
this report are sufficiently reliable. We have also made appropriate
attribution indicating the sources of the data.
We performed our work at the CBP headquarters and prime contractor
facilities in the Washington, D.C., metropolitan area and with DCMA
officials from Huntsville, Alabama. We conducted this performance
audit from June 2009 to October 2010 in accordance with generally
accepted government auditing standards. Those standards require that
we plan and perform the audit to obtain sufficient, appropriate
evidence to provide a reasonable basis for our findings and
conclusions based on our audit objectives. We believe that the
evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.
[End of section]
Appendix II: Comments from the Department of Homeland Security:
Department of Homeland Security:
Washington, DC 25528:
September 23, 2010:
Mr. Randolph C. Hite:
Director, Information Technology Architecture and Systems Issues:
U.S. Government Accountability Office:
Washington, D.C. 20548:
Dear Mr. Hite:
Thank you for the opportunity to review and offer comment on the
Government Accountability Office (GAO) draft report entitled, "Secure
Border Initiative: DHS Needs to Strengthen Management and Oversight of
its Prime Contractor," GAO-11-6, dated October 2010. GAO was asked to
determine the extent to which the Department of Homeland Security
(OHS) (I) defined and implemented effective controls for managing and
overseeing the SBInet prime contractor and (2) effectively monitored
the prime contractor's progress in meeting cost and schedule
expectations, To do this, GAO analyzed key performance documentation
against relevant guidance and industry best practices, and interviewed
program officials.
GAO acknowledges that OHS has largely defined key management and
oversight processes and procedures for verifying and accepting
contract deliverables and conducting technical reviews. However, the
draft report identifies measures that the U.S. Customs and Border
Protection (CBP) SBInet can take to further strengthen contractor
management and oversight.
GAO made four recommendations to improve DHS management and oversight
of the SBInet prime contractor. CBP concurs with the four
recommendations. However, not withstanding our concurrence, we take
exception to several assertions regarding aspects of the reliability
of the Secure Border Initiative (SB1) program's Earned Value
Management (EVM) practices, and government oversight of contractor
activities. The comments below address our specific concerns.
First, GAO asserts the program office did not ensure EVM performance
measurement baselines (PMB) were "validated" in a timely manner, and
concludes that the program office's EVM shortcomings led to cost and
schedule growth in the program. We firmly take exception to these
assertions.
As background to many of the EVM findings and recommendations, GAO
acknowledges throughout the report the significant amount of program
change occurring with the SBInet program in late 2008-2009. These
positive changes followed a major program re-baselining (Acquisition
Program Baseline) with the Department as well as SBInet final system
design updates after Critical Design Review. The program baseline
change drove major scope, schedule, and budget changes across all of
the Boeing task orders and associated performance measurement
baselines.
GAO concluded that without a "validated" baseline through the conduct
of an Integrated Baseline Review (IBR), the program office was unable
to identify cost and schedule problems early and to take timely
corrective actions to mitigate those problems. The report also
concluded that the program's lack of a "validated" baseline
contributed to cost overruns and schedule delays. We believe these
conclusions are inaccurate, and the causal assertion is not supported
by any analysis presented in the report. Of significance is the fact
that the program office maintained interim performance measurement
baselines throughout the transition period with the best available
task plan information, as required by EVM best practices. Program
leadership also reviewed in various forum, the contractor's technical
and financial performance information relative to the established PMBs
and implemented rigorous management actions as needed. The program
office delayed formal IBR(s) until completing final, negotiated task
order modifications, and in each case the IBR was completed within the
timeframe specified in DEIS directives (less than 90 days after the
major contract modification). Certainly during a period of significant
program changes. earned value management is challenged, but the
program office believes prudent and compliant EVM measures were
executed.
GAO documented other EVM implementation concerns reported by the
program office, specifically the prime contractor's over-reliance on
the "level of effort" (LoE) EVM measure, and recurring quality control
lapses, or "anomalies," contained in the contractor's monthly reports.
For the LoE concern, the program office and the prime contractor
initiated several actions to improve task order work planning by
creating discreet work packages, enabling improved performance
management and issue identification. However, given the stage of the
program today, we will continue to see high levels of effort in the
task order baselines due to maintaining sizeable contractor program
management "overhead" (typically LoE), while transitioning from
construction and testing completion into a sustaining system and
software engineering mode (typically LoE), as well as performing a
variety of system maintenance functions (also typically LoE).
Secondly. we are concerned with GAO's characterization and
quantification of "anomalies." The SBI program office demands accurate
contractor performance reports, and is continuously engaged with the
contractor to fix underlying problems. The draft report fails to
distinguish between factual errors versus legitimate monthly
accounting adjustments. For example:
* Errors, and the resultant corrections, occur when the contractor
collects and reports costs for one activity when the costs actually
apply to another activity, and might include incorrect contract charge
numbers issued to subcontractors or other simple administrative
mistakes. Errors distort cost performance status however the SBI
program office and the contractor promptly identify and take
corrective actions to correct identified problems and improve quality
controls for final monthly reports.
* Adjustments routinely occur when the prime contractor reports
estimated costs for a subcontractor (while awaiting a subcontractor to
submit an invoice), and then in a subsequent report, the prime
reconciles previously reported estimated costs with the final,
invoiced actual costs. Adjustments may also be necessary to reflect
contractor rate changes. These adjustments improve the accuracy and
reliability of the cumulative cost performance data, yet
may appear anomalous for the current month. The prime contractor's
adjustments are completed in accordance with the contractor's approved
system guide and established best practices.
The GAO report combines both errors and adjustments when quantifying
"anomalies" and incorrectly concludes that both errors and adjustments
are problematic, distorting cost performance, and limits management
insight.
Regardless of error corrections or adjustments, the contractor is
required to document such actions in the monthly cost performance
report (Format 5). Over the past eight months, the contractor
implemented quality control measures, and has significantly improved
the process for monthly reporting of both errors and anomalies as well
as significant accounting adjustments.
The recommendations and CBP's corrective actions to address the
recommendation are described below.
Recommendation 1: Revise and implement, as applicable, contractor
deliverable review processes and practices to ensure that (I)
contactor deliverables are thoroughly reviewed and are not constrained
by late contractor deliverables and imposed milestones, (2) the
reviews are sufficiently documented, and (3) the acceptance or the
rejection of each contractor deliverable is communicated in writing to
the contractor, to include explicit explanations of the basis for any
rejections.
CBP Response: CBP concurs with the recommendation. Since the draft
report was issued, we have taken several major steps to improve the
SBInet program management structure, capabilities, procedures, and
processes. This includes significant improvement in SBInet execution
of the Contract Deliverable Requirements List (CDRL) review process.
The program has commenced reviewing and updating the current CDRL,
review policy. The updated CDRL policy will document the process and
procedures for program technical reviews, and written documentation to
the contractor regarding results of the review to include reviewer
comment forms and formal communication of the acceptance, conditional
acceptance, or rejection of the contract deliverable. In implementing
the updated CDRL policy, contract deliverable review participants will
be trained to ensure that reviewers are technically qualified and
possess the procedural knowledge to effectively review and provide
concise and thorough, review comments and that the process is fully
aligned with the approved CDRL review policy.
Due Date: Policy update expected to be completed by December 2010.
Recommendation 2: Ensure that applicable entry and exit criteria for
each technical review are used and satisfied before initiating and
concluding, respectively, a given review.
CLIP Response: CBP concurs with the recommendation. We have taken
several major steps to improve the SBInet program management
structure, capabilities, procedures, and processes. This has led to a
more rigorous application of review entry and exit criteria. In
addition, the SBI Systems Engineering (SE) Directorate was established
to ensure that systems engineering capabilities are focused on
providing the technical oversight required to support knowledge-based
decision making throughout the acquisition process. The SE Directorate
is incorporating the DIN Engineering Life Cycle best practices into a
Technical Review Guide (TRG) that describes in detail the SE technical
review process and the relevant entry and exit criteria for each
technical review. The draft TRG is currently under review and when
implemented will serve as the roadmap used by the SBI System Program
Office to ensure that all relevant entry and exit criteria are
understood and satisfied in a manner consistent with programmatic
cost, schedule, and performance risk considerations.
Due Date: Technical Review Guide expected to be completed and
implemented by December 2010.
Recommendation 3: Establish and validate timely, complete, and
accurate performance measurement baselines for each new task order or
major modification of an existing task order, as appropriate, to
include, but not be limited to, ensuring that (1) the work breakdown
structure includes all work to be performed; (2) baseline schedules
reflect the key schedule estimating practices discussed in this
report; and (3) level-of-effort performance measurement in excess of
15 percent is scrutinized, justified, and minimized.
CRP Response: CBP concurs with the recommendation. The SBI program
office is mindful of the need to establish and maintain current
Performance Measurement Baselines (PMB), and to plan and implement
baseline updates as completely and promptly as practicable. For new
task orders and significant modifications. this is done through
Integrated Baseline Reviews (IBRs). The purpose of the IBR is to
confirm that the PMB covers the scope of work; the work is
realistically and accurately scheduled; the proper mix of resources
and budgets are assigned; the appropriate management processes are in
place; and the risks associated with the PMB are identified,
documented, and managed. Prior to an IBR, the program office routinely
evaluates the control accounts, and the integrated PMB, for
consistency, traceability, and completeness relative to the work
breakdown structure (WBS), to include a cross-check of the WBS back to
the control accounts. Also, the program office documents any adverse
findings and resolves such with the prime contractor as part of the
1BR proceedings. As described above, the program office scrutinizes,
and with the prime contractor, minimizes level of effort tasks in the
PMB.
Scheduling practices remain a challenge, and as reported in our
response to prior GAO reviews, we continue to upgrade our processes.
The program office is revising our scheduling standards to reflect
procedural improvements for schedule development, management and
surveillance. We have created and put into practice the use of 26
diagnostic schedule filters to analyze the health of our schedules.
The filters fully leverage GAO's scheduling best practices. We have
also implemented improvements to our practices for developing
Integrated Master Plan/Integrated Master Schedule that utilizes newly
revised planning and schedule templates based on the OHS Systems
Engineering Lifecycle.
Due Date: Ongoing corrective actions are in place, and the program
office continues to assess and manage contractor progress and
improvements.
Recommendation 4: Ensure that all anomalies in contractor-delivered
monthly earned value management reports are identified and explained,
and report periodically to DHS acquisition leadership on relevant
trends in the number of unexplained anomalies.
CRP Response: CBP concurs with the recommendation. The monthly
contract performance report "anomalies" (which include a variety of
legitimate monthly updates to actual costs, rate changes, etc.) need
to be documented correctly each month. The SBI program office shared
with the GAO significant, recent activities and progress with
associated corrective actions:
* Fixing prime contractor systemic and quality control issues, with
the assistance of the Defense Contract Management Agency, through
active participation in program EVM surveillance; and;
* Improving discipline and content of Format 5 reporting narrative,
augmented with routine management discussions with contractor and
government project management staff.
Our focus remains to ensure we are able to determine program status
relative to well-defined performance baselines. As issues arise with
either contractor performance or reporting, we will advise the
appropriate acquisition leaders through established reporting and
oversight opportunities.
Due Date: Ongoing; corrective actions are in place, and the program
office continues to assess and manage contractor progress and
improvements.
Sincerely,
Signed by:
Jerald E. Levine:
Director:
Departmental GAO/OIG Liaison Office:
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
Randolph C. Hite, (202) 512-3439 or hiter@gao.gov:
Staff Acknowledgments:
In addition to the contact named above, Deborah A. Davis (Assistant
Director), Tisha Derricotte, Neil Doherty, Kaelin Kuhn, Lee McCracken,
Jamelyn Payan, Karen Richey, Matt Snyder, Sushmita Srikanth, Stacey
Steele, and Matthew Strain made key contributions to this report.
[End of section]
Footnotes:
[1] We also selected one additional contract deliverable and one
additional technical review for review.
[2] EVM is a project management approach that, if implemented
appropriately, provides objective reports of project status, produces
early warning signs of impending schedule delays and cost overruns,
and provides unbiased estimates of a program's total costs.
[3] SEI, CMMIŽ for Acquisition, Version 1.2 (Pittsburgh, Pa., November
2007); and GAO, GAO Cost Estimating and Assessment Guide: Best
Practices for Developing and Managing Capital Program Costs,
[hyperlink, http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.:
March 2009). Also, recognizing the importance of ensuring quality
earned value data, ANSI and the Electronic Industries Alliance (EIA)
jointly established a national standard for EVM systems in May 1998
(ANSI/EIA-748-A-1998). This standard, commonly called the ANSI
standard, is comprised of guidelines on how to establish a sound EVM
system. This document was updated in July 2007 and is referred to as
ANSI/EIA-748-B.
[4] The remaining $126 million was for upgrading CBP
telecommunications links.
[5] The physical infrastructure (e.g., physical fencing) portion of
SBI is managed on a day-to-day basis by CBP's Office of Finance
Facilities Management and Engineering division.
[6] IDIQ contracts are used when the government cannot predetermine,
above a specified minimum, the precise quantities of supplies or
services that it will require during the contract period. Under IDIQ
contracts, the government can issue delivery orders (for supplies) and
task orders (for services).
[7] The $35 million was part of American Recovery and Reinvestment Act
of 2009 funding. Pub. L. No. 111-5, 123 Stat. 115, 162 (Feb. 17, 2009).
[8] See for example, GAO, Secure Border Initiative: DHS Needs to
Reconsider Its Proposed Investment in Key Technology Program,
[hyperlink, http://www.gao.gov/products/GAO-10-340] (Washington, D.C.:
May 5, 2010); Secure Border Initiative: DHS Needs to Address Testing
and Performance Limitations That Place Key Technology Program at Risk,
[hyperlink, http://www.gao.gov/products/GAO-10-158] (Washington, D.C.:
Jan. 29, 2010); and DOD Business Systems Modernization: Important
Management Controls Being Implemented on Major Navy Program, but
Improvements Needed in Key Areas, [hyperlink,
http://www.gao.gov/products/GAO-08-896] (Washington, D.C.: Sept. 8,
2008).
[9] GAO, Secure Border Initiative: SBInet Expenditure Plan Needs to
Better Support Oversight and Accountability, [hyperlink,
http://www.gao.gov/products/GAO-07-309] (Washington, D.C.: Feb. 15,
2007).
[10] GAO, Secure Border Initiative: DHS Needs to Address Significant
Risks in Delivering Key Technology Investment, [hyperlink,
http://www.gao.gov/products/GAO-08-1086] (Washington, D.C.: Sept. 22,
2008).
[11] GAO, Secure Border Initiative: Technology Deployment Delays
Persist and the Impact of Border Fencing Has Not Been Addressed,
[hyperlink, http://www.gao.gov/products/GAO-09-896] (Washington, D.C.:
Sept. 9, 2009).
[12] [hyperlink, http://www.gao.gov/products/GAO-10-158].
[13] A test case is a set of test inputs, execution conditions, and
expected results developed for a particular objective, such as to
verify compliance with a specific requirement.
[14] [hyperlink, http://www.gao.gov/products/GAO-10-340].
[15] GAO, Department of Homeland Security: Assessments of Selected
Complex Acquisitions, [hyperlink,
http://www.gao.gov/products/GAO-10-588SP] (Washington, D.C.: June 30,
2010).
[16] See, for example, Federal Acquisition Regulation, part 46,
quality assurance; and SEI, CMMIŽ for Acquisition.
[17] SEI, CMMIŽ for Acquisition.
[18] Training requirements for key officials are defined in three DHS
policies: (1) Acquisition Certification Requirements for Program
Manager, Management Directive (MD) 0782 (May 26, 2004); (2)
Contracting Officer Warrant Program (Apr. 16, 2007); and (3) COTR
Certification, Appointment & Responsibilities, MD 0780.1 (Dec. 20,
2004).
[19] DHS has categorized major information technology investments as
Levels 1, 2, and 3. A Level 1 major investment is defined in DHS
Acquisition Directive 102-01 as program at or more than $1 billion in
life cycle costs.
[20] DHS's Acquisition Certification Requirements for Program Manager,
MD 0782 (May 26, 2004) establishes three levels of certification:
Level 1 establishes the foundation for managing increasingly complex
investments or portfolios, Level 2 includes knowledge of strategic and
capital planning, and Level 3 represents mastery of the knowledge and
skills associated with the complexities of managing DHS's most
strategic and sophisticated acquisitions.
[21] For warrant authority up to $100,000, the contracting officer
must have a Federal Acquisition Certification in Contracting (FAC-C)
Program Level I certification; for warrant authority up to $25
million, the contracting officer must have a FAC-C Level II
certification; and for warrant authority more than $25 million,
including unlimited warrant authority, the contracting officer must
have a FAC-C Level III certification.
[22] SEI, CMMIŽ for Acquisition.
[23] A project manager is responsible for each of the task orders.
These project managers have overall responsibility and accountability
for achieving the task order's scope, cost, schedule, and technical
objectives.
[24] According to the SPO, subject matter experts review the
deliverables based on the contractual requirements that are outlined
in the Data Item Descriptions, as well as their own expertise and
knowledge about the deliverables.
[25] These 29 deliverables actually resulted in 46 separate
submissions because, for example, an initial submission was rejected
and had to be resubmitted.
[26] The NOC/SOC monitors networked equipment, provides alerts for
network problems, protects equipment from network-based attacks, and
provides user authentication.
[27] According to the C3I/COP task order statement of work, the plan
was to be submitted 10 days before the test readiness review. However,
the contractor submitted the plan the day of this review.
[28] This system is the program office's requirements management
database.
[29] [hyperlink, http://www.gao.gov/products/GAO-10-158].
[30] SEI, CMMIŽ for Acquisition.
[31] DHS, Acquisition Instruction/Guidebook, 102-01-001, Appendix B,
Systems Engineering Life Cycle, Interim Version 1.9 (Nov. 7, 2008).
[32] [hyperlink, http://www.gao.gov/products/GAO-10-340].
[33] [hyperlink, http://www.gao.gov/products/GAO-10-158].
[34] In 2008, the SPO contracted with an IV&V agent to, among other
things, review the program's test documentation, execution, and
related processes. Generally, the purpose of IV&V is to independently
determine the extent to which program processes and products meet
quality standards. The use of IV&V is recognized as an effective
practice for large and complex system development and acquisition
programs, like SBInet.
[35] SEI, CMMIŽ for Acquisition.
[36] CBP, System Life Cycle Handbook, Version 1.2 (Sept. 30, 2008).
[37] This database is Boeing-managed and tracks SBInet action items
throughout their lifecycle. The database captures such information as
individual action assignments, status and updates, and relevant
comments.
[38] Office of Management and Budget Memorandum, M-05-23 (Aug. 4,
2005).
[39] ANSI/EIA-748-B.
[40] As mentioned previously, DCMA conducts periodic surveillance
reviews of the prime contractor's EVM system implementation.
[41] [hyperlink, http://www.gao.gov/products/GAO-09-3SP], 227-229, and
Department of Defense, Earned Value Management Implementation Guide
(October 2006).
[42] DHS, Department of Homeland Security Acquisition Manual, Section
3034.202 Integrated Baseline Reviews (October 2009). Integrated
baseline reviews are intended to verify that the baseline is realistic
and the scope, schedule, and risks are mutually understood by the
contractor and the government.
[43] Physical infrastructure includes physical fencing and roads.
[44] The scope of the work is generally defined in a work breakdown
structure, which is a hierarchical structure that shows how work tasks
relate to one another as well as to the overall end product.
[45] [hyperlink, http://www.gao.gov/products/GAO-09-3SP], 215-231.
[46] [hyperlink, http://www.gao.gov/products/GAO-09-3SP], 215.
[47] The critical path represents the chain of dependent activities
with the longest total duration in the schedule.
[48] Float is the amount of time a task can slip before affecting the
critical path.
[49] [hyperlink, http://www.gao.gov/products/GAO-09-3SP], 218-224.
[50] We did not assess the ninth practice because the schedules that
were reviewed at the IBRs were the initial schedules for each task
order, and as such, were not updated.
[51] ANSI/EIA-748-B.
[52] Level-of-effort is unmeasured effort of a general or supportive
nature which does not produce definitive end products (e.g., program
administration).
[53] [hyperlink, http://www.gao.gov/products/GAO-09-3SP], 226.
[54] [hyperlink, http://www.gao.gov/products/GAO-09-3SP], 226.
[55] The program office did not have a baseline for ADTO and DTO, from
July 2008 through October 2008 and from June 2008 through January
2009, respectively.
[56] Software Engineering Institute, Capability Maturity Model
IntegrationŽ for Acquisition, Version 1.2 (Pittsburgh, Pa., November
2007).
[57] We conducted a nonprobability sample because we determined that
the SPO's contract deliverable log was not sufficiently reliable for
purposes of identifying all required deliverables during our time
frames, and thus we did not select a sample that could be projected to
the total population.
[58] We selected one additional deliverable that was associated with
the technical review noted below to provide a more current assessment
of the program's adherence to its process.
[59] We selected one additional technical review based on the
combinations of task orders and review types in our existing sample to
provide a more current assessment of the program's implementation of
established criteria.
[60] We selected one action item for each month in which a joint
program management review was held and action items were documented.
[61] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for
Developing and Managing Capital Program Costs, [hyperlink,
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March
2009), 215 steps 1-6.
[62] We did not assess the ninth practice--updating the schedule using
logic and durations--because the schedules that were reviewed at the
integrated baseline reviews were the initial schedules for each task
order, and as such, were not updated.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548: