Secure Border Initiative
Testing and Problem Resolution Challenges Put Delivery of Technology Program at Risk Gao ID: GAO-10-511T March 18, 2010This testimony is based on our report "Secure Border Initiative: DHS Needs to Address Testing and Performance Limitations That Place Key Technology Program at Risk", which is being released at this hearing. In September 2008, we reported to you that important aspects of SBInet were ambiguous and in a continuous state of flux, making it unclear and uncertain what technology capabilities were to be delivered when. In addition, the program did not have an approved integrated master schedule to guide the program's execution, and key milestones continued to slip. This schedule-related risk was exacerbated by the continuous change in and the absence of a clear definition of the approach used to define, develop, acquire, test, and deploy SBInet. Furthermore, different levels of SBInet requirements were not properly aligned, and all requirements had not been properly defined and validated. Also, the program office had not tested the individual system components to be deployed to initial locations, even though the contractor had initiated integration testing of these components with other system components and subsystems, and its test management strategy did not contain, among other things, a clear definition of testing roles and responsibilities; or sufficient detail to effectively guide planning for specific test events, such as milestones and metrics. Accordingly, we made recommendations to address these weaknesses which DHS largely agreed to implement. In light of SBInet's important mission, high cost, and risks, you asked us to conduct a series of four SBInet reviews. This statement and report being released today provide the results for the first of these reviews.3 Specifically, they address (1) the extent to which SBInet testing has been effectively managed, including identifying the types of tests performed and whether they were well planned and executed; (2) what the results of testing show; and (3) what processes are being used to test and incorporate maturing technologies into SBInet.
SBInet testing has not been adequately managed, as illustrated by poorly defined test plans and numerous and extensive last-minute changes to test procedures. Further, testing that has been performed identified a growing number of system performance and quality problems--a trend that is not indicative of a maturing system that is ready for deployment anytime soon. Further, while some of these problems have been significant, the collective magnitude of the problems is not clear because they have not been prioritized, user reactions to the system continue to raise concerns, and key test events remain to be conducted. Collectively, these limitations increase the risk that the system will ultimately not perform as expected and will take longer and cost more than necessary to implement. For DHS to increase its chances of delivering a version of SBInet for operational use, we are recommending that DHS improve the planning and execution of future test events and the resolution and disclosure of system problems. DHS agreed with our recommendations.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-10-511T, Secure Border Initiative: Testing and Problem Resolution Challenges Put Delivery of Technology Program at Risk
This is the accessible text file for GAO report number GAO-10-511T
entitled 'Secure Border Initiative: Testing and Problem Resolution
Challenges Put Delivery of Technology Program at Risk' which was
released on March 18, 2010.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
United States Government Accountability Office:
GAO:
Testimony:
Before the Committee on Homeland Security's Subcommittees on
Management, Investigations, and Oversight; and Border, Maritime, and
Global Counterterrorism:
For Release on Delivery:
Expected at 10:00 a.m. EDT:
Thursday, March 18, 2010:
Secure Border Initiative:
Testing and Problem Resolution Challenges Put Delivery of Technology
Program at Risk:
Statement of Randolph C. Hite, Director:
Information Technology Architecture and System Issues:
GAO-10-511T:
March 18, 2010:
Mr. Chairman and Members of the Subcommittees:
Thank you for the opportunity to participate in today's hearing on the
technology component of the Department of Homeland Security's (DHS)
Secure Border Initiative (SBI). My statement today is based on our
report Secure Border Initiative: DHS Needs to Address Testing and
Performance Limitations That Place Key Technology Program at Risk,
which is being released at this hearing.[Footnote 1]
As you know, SBI is intended to help secure the 6,000 miles of
international borders that the contiguous United States shares with
Canada and Mexico. The program, which began in November 2005, seeks to
enhance border security and reduce illegal immigration by improving
surveillance technologies, raising staffing levels, increasing
domestic enforcement of immigration laws, and improving physical
infrastructure along the nation's borders. Within SBI, the Secure
Border Initiative Network (SBInet) is a multibillion dollar program
that includes the acquisition, development, integration, deployment,
and operation of surveillance technologies--such as unattended ground
sensors and radar and cameras mounted on fixed and mobile towers--to
create a "virtual fence" along the border. In addition, command,
control, communications, and intelligence software and hardware are to
use the information gathered by the surveillance technologies to
create a common operating picture (COP) of activities within specific
areas along the border and transmit the information to command centers
and vehicles.
In September 2008, we reported to you that important aspects of SBInet
were ambiguous and in a continuous state of flux, making it unclear
and uncertain what technology capabilities were to be delivered when.
In addition, the program did not have an approved integrated master
schedule to guide the program's execution, and key milestones
continued to slip. This schedule-related risk was exacerbated by the
continuous change in and the absence of a clear definition of the
approach used to define, develop, acquire, test, and deploy SBInet.
Furthermore, different levels of SBInet requirements were not properly
aligned, and all requirements had not been properly defined and
validated. Also, the program office had not tested the individual
system components to be deployed to initial locations, even though the
contractor had initiated integration testing of these components with
other system components and subsystems, and its test management
strategy did not contain, among other things, a clear definition of
testing roles and responsibilities; or sufficient detail to
effectively guide planning for specific test events, such as
milestones and metrics. Accordingly, we made recommendations to
address these weaknesses which DHS largely agreed to implement.
[Footnote 2]
In light of SBInet's important mission, high cost, and risks, you
asked us to conduct a series of four SBInet reviews. This statement
and report being released today provide the results for the first of
these reviews.[Footnote 3] Specifically, they address (1) the extent
to which SBInet testing has been effectively managed, including
identifying the types of tests performed and whether they were well
planned and executed; (2) what the results of testing show; and (3)
what processes are being used to test and incorporate maturing
technologies into SBInet.
In summary, SBInet testing has not been adequately managed, as
illustrated by poorly defined test plans and numerous and extensive
last-minute changes to test procedures. Further, testing that has been
performed identified a growing number of system performance and
quality problems--a trend that is not indicative of a maturing system
that is ready for deployment anytime soon. Further, while some of
these problems have been significant, the collective magnitude of the
problems is not clear because they have not been prioritized, user
reactions to the system continue to raise concerns, and key test
events remain to be conducted. Collectively, these limitations
increase the risk that the system will ultimately not perform as
expected and will take longer and cost more than necessary to
implement. For DHS to increase its chances of delivering a version of
SBInet for operational use, we are recommending that DHS improve the
planning and execution of future test events and the resolution and
disclosure of system problems. DHS agreed with our recommendations.
Background:
Managed by DHS's Customs and Border Protection (CBP), SBInet is to
strengthen CBP's ability to detect, identify, classify, track, and
respond to illegal breaches at and between ports of entry. CBP's SBI
Program Office is responsible for managing key acquisition functions
associated with SBInet, including tracking and overseeing the prime
contractor.
In September 2006, CBP awarded a 3-year contract to the Boeing Company
for SBInet, with three additional 1-year options. As the prime
contractor, Boeing is responsible for designing, producing, testing,
deploying, and sustaining the system. In September 2009, CBP extended
its contract with Boeing for the first option year. CBP is acquiring
SBInet incrementally in a series of discrete units of capabilities,
referred to as "blocks." Each block is to deliver one or more system
capabilities from a subset of the total system requirements.
In August 2008, the DHS Acquisition Review Board decided to delay the
initial deployment of Block 1 of SBInet so that fiscal year 2008
funding could be reallocated to complete physical infrastructure
projects. In addition, the board directed the SBInet System Program
Office (SPO) to deliver a range of program documentation, including an
updated Test and Evaluation Master Plan (TEMP),[Footnote 4] detailed
test plans, and a detailed schedule for deploying Block 1 to two
initial sites in the Tucson Sector of the southwest border. This
resulted in a revised timeline for deploying Block 1, first to the
Tucson Border Patrol Station (TUS-1) in April 2009, and then to the
Ajo Border Patrol Station (AJO-1) in June 2009. Together, these two
deployments are to cover 53 miles of the 1,989-mile-long southern
border. However, the SBI Executive Director told us in December 2009
that these and other SBInet scheduled milestones were being
reevaluated. As of January 2010, the TUS-1 system is scheduled for
government acceptance in September 2010, with AJO-1 acceptance in
November 2010. However, this schedule has yet to be approved by CBP.
DHS Has Not Effectively Managed SBInet Testing:
Testing is essential to knowing whether the system meets defined
requirements and performs as intended. Effective test management
involves, among other things, developing well-defined test plans and
procedures to guide test execution. It is intended to identify and
resolve system quality and performance problems as early as possible
in the system development life cycle.
DHS has not effectively managed key aspects of SBInet testing, which
has in turn increased the risk that the system will not perform as
expected and will take longer and cost more than necessary. While the
department's testing approach appropriately consists of a series of
progressively expansive test events, some of which have yet to be
completed, test plans and test cases for recently executed test events
were not defined in accordance with relevant guidance. For example,
none of the plans for tests of system components addressed testing
risks and mitigation strategies.
Further, SBInet test procedures were generally not executed as
written. Specifically, about 70 percent of the procedures for key test
events were rewritten extemporaneously during execution because
persons conducting the tests determined that the approved procedures
were not sufficient or accurate. Moreover, changes to these procedures
were not made according to a documented quality assurance process but
were instead made based on an undocumented understanding that program
officials said they established with the contractor. While some of
these changes were relatively minor, others were significant, such as
adding requirements or completely rewriting verification steps. The
volume and nature of the changes made to the test procedures, in
conjunction with the lack of a documented quality assurance process,
increases the risk that system problems may not be discovered until
later in the sequence of testing. This concern is underscored by a
program office letter to the prime contractor stating that changes
made to system qualification test procedures appeared to be designed
to pass the test instead of being designed to qualify the system.
These limitations are due, among other things, to a lack of detailed
guidance in the TEMP, the program's aggressive milestones, schedule,
and ambiguities in requirements. Collectively, these limitations
increase the likelihood that testing will not discover system issues
or demonstrate the system's ability to perform as intended.
SBInet Testing Results Have Identified a Growing Number of System
Performance and Quality Problems:
The number of new SBInet defects that have been discovered during
testing has increased faster than the number that has been fixed. (See
figure 1 for the trend in the number of open defects from March 2008
to July 2009.) As we previously reported[Footnote 5] such an upward
trend is indicative of an immature system.
Figure 1: SBInet Open Defects from March 2008 to July 2009:
[Refer to PDF for image: illustrated line graph]
Date: March 2008;
Open defects: 24.
Date: April 2008;
Open defects: 63.
Date: May 2008;
Open defects: 69.
Date: June 2008 (begin Integration testing);
Open defects: 106.
Date: July 2008;
Open defects: 149.
Date: August 2008;
Open defects: 168.
Date: September 2008;
Open defects: 139.
Date: October 2008 (end Integration testing, begin CQT);
Open defects: 103.
Date: November 2008;
Open defects: 105.
Date: December 2008 (end CQT, begin SQT);
Open defects: 233.
Date: January 2009 (end SQT);
Open defects: 278.
Date: February 2009 (begin SQT regression [on-going]; begin CQT
regression [on-going]);
Open defects: 247.
Date: March 2009;
Open defects: 330.
Date: April 2009;
Open defects: 253.
Date: May 2009;
Open defects: 219.
Date: June 2009;
Open defects: 302.
Date: July 2009;
Open defects: 240.
Source: GAO analysis of DHS data.
[End of figure]
Some of the defects found during testing have been significant,
prompting the DHS Acquisition Review Board in February 2009 to
postpone deployment of Block 1 capabilities to TUS-1 and AJO-1. These
defects included the radar circuit breaker frequently tripping when
the radar dish rotated beyond its intended limits, COP workstations
crashing, and blurry camera images, among others.
While program officials have characterized the defects and problems
found during development and testing as not being "show stoppers,"
they have nevertheless caused delays, extended testing, and required
time and effort to fix. Moreover, the SPO and its contractor have
continued to find problems that further impact the program's schedule.
For example, the radar problems mentioned previously were addressed by
installing a workaround that included a remote ability to reactivate
the circuit breaker via software, which alleviated the need to send
maintenance workers out to the tower to manually reset the circuit.
However, this workaround did not fully resolve the problem, and
program officials said that root cause analysis continues on related
radar power spikes and unintended acceleration of the radar dish that
occasionally render the system inoperable. One factor that has
contributed to the time and resources needed to resolve this radar
problem, and potentially other problems, is the ability of the prime
contractor to effectively determine root causes for defects. According
to program officials, including the SBI Executive Director, the
contractor's initial efforts to isolate the cause of the radar
problems were flawed and inadequate. Program officials added, however,
that they have seen improvements in the contractor's efforts to
resolve technical issues.
Along with defects revealed by system testing, Border Patrol operators
participating in an April 2009 user assessment identified a number of
concerns. During the assessment, operators compared the performance of
Block 1 capabilities to those of existing technologies. While Border
Patrol agents noted that Block 1 offered functionality above existing
technologies, it was not adequate for optimal effectiveness in
detecting items of interest along the border. Users also raised
concerns about the accuracy of Block 1's radar, the range of its
cameras, and the quality of its video. Officials attributed some of
the identified problems to users' insufficient familiarity with Block
1; however, Border Patrol officials reported that the participating
agents had experience with the existing technologies and had received
2 days of training prior to the assessment. The Border Patrol thus
maintained that the concerns generated should be considered
operationally relevant.
Effectively managing identified defects requires a defined process
for, among other things, assigning priorities to each defect and
ensuring that more severe ones are given priority attention. However,
the SPO does not have such a documented approach but instead relies on
the prime contractor for doing so. Under this approach, defects were
not consistently assigned priorities. Specifically, about 60 percent
(or 801 of 1,333) of Block 1 defects identified from March 2008 to
July 2009 were not assigned a priority. This is partly attributable to
the SPO's lack of a defined process for prioritizing and managing
defects. Officials acknowledge this and stated that they intend to
have the contractor prioritize all defects in advance of future test
readiness reviews. Until defects are managed on a priority basis, the
program office cannot fully understand Block 1's maturity or its
exposure to related risks, nor can it make informed decisions about
allocating limited resources to address defects.
DHS Science and Technology Directorate Testing Process Is Being Used
to Leverage Maturing Technologies for SBInet:
The SPO does not have its own process for testing the relevance to
SBInet of technologies that are maturing or otherwise available from
industry or other government entities. Instead, it relies on DHS's
Science and Technology Directorate (S&T), whose mission is to provide
technology solutions that assist DHS programs in achieving their
missions. To leverage S&T, CBP signed a multiyear Interagency
Agreement with the directorate in August 2007. According to this
agreement, S&T is to research, develop, assess, test, and report on
available and emerging technologies that could be incorporated into
the SBInet system. To date, S&T has focused on potential technologies
to fill known performance gaps or improve upon already-made technology
choices, such as gaps in the radar system's ability to distinguish
true radar hits from false alarms. S&T officials told us that they
interact with Department of Defense (DOD) components and research
entities to identify DOD systems for SBInet to leverage. In this
regard, SPO officials stated that the current SBInet system makes use
of DOD technologies, such as common operating picture software and
radar systems. Nevertheless, S&T officials added that defense-related
technologies are not always a good fit with SBInet, due to operational
differences.
GAO Is Making Recommendations to Improve SBInet Test Management and
Problem Resolution:
To improve the planning and execution of future test events and the
resolution and disclosure of system problems, we are making the
following four recommendations to DHS:
* Revise the SBInet Test and Evaluation Master Plan to include
explicit criteria for assessing the quality of test documentation and
for analyzing, prioritizing, and resolving defects.
* Ensure that test schedules, plans, cases, and procedures are
adequately reviewed and approved consistent with the Test and
Evaluation Master Plan.
* Ensure that sufficient time is provided for reviewing and approving
test documentation prior to beginning a given test event.
* Triage the full inventory of unresolved problems, including
identified user concerns, and periodically report the status of the
highest priority defects to Customs and Border Protection and
Department of Homeland Security leadership.
In written comments on a draft of our report, DHS stated that the
report was factually sound, and it agreed with our last three
recommendations and agreed with all but one aspect of the first one.
DHS also described actions under way or planned to address the
recommendations.
In closing, I would like to stress how integral effective testing and
problem resolution are to successfully acquiring and deploying a large-
scale, complex system, like SBInet Block 1. As such, it is important
that each phase of Block 1 testing be managed with rigor and
discipline. To do less increases the risk that a deployed version of
the system will not perform as intended, and will ultimately require
costly and time-consuming rework to fix problems found later rather
than sooner. Compounding this risk is the unfavorable trend in the
number of unresolved system problems, and the lack of visibility into
the true magnitude of these problems' severity. Given that major test
events remain to be planned and conducted, which in turn are likely to
identify additional system problems, it is important to correct these
testing and problem resolution weaknesses.
This concludes my prepared statement. I would be pleased to respond to
any questions that you or other Members of the Subcommittees may have.
Contacts and Staff Acknowledgments:
For questions about this statement, please contact Randolph C. Hite at
(202) 512-3439 or hiter@gao.gov. Individuals making key contributions
to this testimony include Deborah Davis, Assistant Director; Carl
Barden, James Crimmer, Neil Doherty, Lauren Giroux, Nancy Glover, Dan
Gordon, Lee McCracken, Sushmita Srikanth, and Jennifer Stavros-Turner.
[End of section]
Attachment 1: Summary of GAO's Ongoing SBInet Work for the Committee
on Homeland Security:
SBInet's Commitment, Progress, and Acquisition Management. Our
objectives are to determine the extent to which DHS has (1) defined
the scope of its proposed system solution, (2) developed a reliable
schedule for delivering this solution, (3) demonstrated the cost
effectiveness of this solution, (4) acquired this solution in
accordance with key life cycle management processes, and (5) addressed
our recent recommendations. We plan to report our results in April
2010.
SBInet's Contractor Management and Oversight. Our objectives are to
determine the extent to which DHS (1) has established and implemented
effective controls for managing and overseeing the SBInet prime
contractor and (2) is effectively monitoring the prime contractor's
progress in meeting cost and schedule expectations. We plan to report
our results during the summer of 2010.
Security Border Initiative Financial Management Controls Over
Contractor Oversight. Our objectives are to determine the extent to
which DHS has (1) developed internal control procedures over SBInet
contractor invoice processing and contractor compliance with selected
key contract terms and conditions and (2) implemented internal control
procedures to ensure payments to SBInet's prime contractor are proper
and in compliance with selected key contract terms and conditions. We
plan to report our results during the summer of 2010.
[End of section]
Footnotes:
[1] GAO-10-158 (Washington, D.C.: Jan. 29, 2010). Both the report and
this statement are based on work performed in accordance with
generally accepted government standards. Those standards require that
we plan and perform the audit to obtain sufficient, appropriate
evidence to provide a reasonable basis for our findings and
conclusions based on our audit objectives. We believe that the
evidence obtained during the course of this review does provide a
reasonable basis for our findings and conclusions based on our audit
objectives.
[2] GAO, Secure Border Initiative: DHS Needs to Address Significant
Risks in Delivering Key Technology Investment, [hyperlink,
http://www.gao.gov/products/GAO-08-1086] (Washington, D.C.: Sept. 22,
2008).
[3] See attachment 1 for the objectives and status of the other three
reviews.
[4] The TEMP defines the program's integrated test and evaluation
approach, including the scope of testing and the staff, resources
(equipment and facilities), and funding requirements associated with
testing.
[5] GAO, Office of Personnel Management: Improvements Needed to Ensure
Successful Retirement Systems Modernization, [hyperlink,
http://www.gao.gov/products/GAO-08-345] (Washington, D.C.: Jan. 31,
2008).
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
