Combating Nuclear Smuggling
DHS Needs to Consider the Full Costs and Complete All Tests Prior to Making a Decision on Whether to Purchase Advanced Portal Monitors Gao ID: GAO-08-1178T September 25, 2008The Department of Homeland Security's (DHS) Domestic Nuclear Detection Office (DNDO) is responsible for addressing the threat of nuclear smuggling. Radiation detection portal monitors are key elements in our national defenses against such threats. DHS has sponsored testing to develop new monitors, known as advanced spectroscopic portal (ASP) monitors, to replace radiation detection equipment currently being used at ports of entry. ASPs may offer improvements over current generation portal monitors, particularly the potential to identify as well as detect radioactive material and thereby minimize both missed threats and false alarms. However, ASPs cost significantly more than current generation portal monitors, and testing of ASPs' capabilities needs to be more objective and rigorous. Due to concerns about ASPs' cost and performance, Congress has required that the Secretary of DHS certify that ASPs will provide a significant increase in operational effectiveness before obligating funds for full-scale ASP procurement. DHS is currently testing ASPs and anticipates a decision on certification in November 2008. This testimony addresses (1) the highlights of GAO's September 2008 report on the life cycle cost estimate to deploy ASPs (GAO-08-1108R), and (2) preliminary observations from ongoing work reviewing the current program of ASP testing.
GAO's independent cost estimate suggested that from 2007 through 2017 the cost of DNDO's program to equip U.S. ports of entry with radiation detection equipment will likely be about $3.1 billion, but could range from $2.6 billion to $3.8 billion. GAO's estimate was based on the anticipated costs of DNDO implementing its 2006 project execution plan, the most recent official documentation of the program. DNDO's cost estimate of $2.1 billion to implement its project execution plan is unreliable because it omits major project costs, such as maintenance, and relies on a flawed methodology. For example, although the normal life expectancy of the standard cargo ASP is about 10 years, DNDO's estimate considers only 8 years. According to DNDO officials, the agency is now following a scaled-back ASP deployment strategy rather than the 2006 project execution plan, and a senior DNDO official told GAO the ASP deployment strategy could change dramatically depending on the outcome of ongoing testing. GAO's analysis indicated the cost to implement the scaled-back plans over the period 2008 through 2017 will be about $2 billion, but could range from $1.7 billion to $2.3 billion. However, frequent changes in DNDO's deployment strategy make it difficult to assess ASP program costs. GAO's recent report recommended that the Secretary of Homeland Security direct DNDO to update the project execution plan, revise its cost estimate, and communicate the revised estimate to the Congress so that it is fully apprised of the program's scope and funding requirements. DHS agreed with the recommendations. DNDO has made progress in addressing a number of problems GAO identified in previous rounds of ASP testing. However, GAO's ongoing review of the 2008 ASP testing program identified several potential areas of concern. First, the DHS criteria for "significant increase in operational effectiveness" appear to set a low bar for improvement--for example, by requiring ASPs to perform at least as well as current generation equipment when nuclear material is present in cargo but not specifying an actual improvement. GAO recently requested additional information from DNDO about the rationale behind these criteria, particularly in light of seemingly stricter criteria found in other documents. Second, the ASP certification schedule does not allow for completion of computer simulations that could provide additional data on ASP performance. While these computer simulations may have limitations, they also could provide useful data on ASP capabilities prior to the Secretary's decision on certification. Finally, the test schedule is highly compressed and is running at least 8 weeks behind, leaving limited time for analysis and review of test results. Assuming that DHS addresses these concerns, the 2008 round of testing could provide an objective basis for comparing ASPs with current generation equipment. However, GAO recommended in March 2006 that DHS analyze the benefits and costs of deploying ASPs to determine whether any additional detection capability provided by ASPs is worth the cost, and would still question the replacement of current generation equipment with ASPs until DNDO demonstrates that any additional increase in security would be worth the ASPs' much higher cost.
GAO-08-1178T, Combating Nuclear Smuggling: DHS Needs to Consider the Full Costs and Complete All Tests Prior to Making a Decision on Whether to Purchase Advanced Portal Monitors
This is the accessible text file for GAO report number GAO-08-1178T
entitled 'Combating Nuclear Smuggling: DHS Needs to Consider the Full
Costs and Complete All Tests Prior to Making a Decision on Whether to
Purchase Advanced Portal Monitors' which was released on September 25,
2008.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Testimony:
Before the Committee on Homeland Security and Governmental Affairs
United States Senate:
United States Government Accountability Office:
GAO:
For Release on Delivery:
Expected at 9:30 a.m. EDT:
Thursday, September 25, 2008:
Combating Nuclear Smuggling:
DHS Needs to Consider the Full Costs and Complete All Tests Prior to
Making a Decision on Whether to Purchase Advanced Portal Monitors:
Statement of Gene Aloise, Director:
Natural Resources and Environment:
GAO-08-1178T:
GAO Highlights:
Highlights of GAO-08-1178T, a testimony before the Committee on
Homeland Security and Governmental Affairs, United States Senate.
Why GAO Did This Study:
The Department of Homeland Security‘s (DHS) Domestic Nuclear Detection
Office (DNDO) is responsible for addressing the threat of nuclear
smuggling. Radiation detection portal monitors are key elements in our
national defenses against such threats. DHS has sponsored testing to
develop new monitors, known as advanced spectroscopic portal (ASP)
monitors, to replace radiation detection equipment currently being used
at ports of entry. ASPs may offer improvements over current generation
portal monitors, particularly the potential to identify as well as
detect radioactive material and thereby minimize both missed threats
and false alarms. However, ASPs cost significantly more than current
generation portal monitors, and testing of ASPs‘ capabilities needs to
be more objective and rigorous. Due to concerns about ASPs‘ cost and
performance, Congress has required that the Secretary of DHS certify
that ASPs will provide a significant increase in operational
effectiveness before obligating funds for full-scale ASP procurement.
DHS is currently testing ASPs and anticipates a decision on
certification in November 2008.
This testimony addresses (1) the highlights of GAO‘s September 2008
report on the life cycle cost estimate to deploy ASPs (GAO-08-1108R),
and (2) preliminary observations from ongoing work reviewing the
current program of ASP testing.
What GAO Found:
GAO‘s independent cost estimate suggested that from 2007 through 2017
the cost of DNDO‘s program to equip U.S. ports of entry with radiation
detection equipment will likely be about $3.1 billion, but could range
from $2.6 billion to $3.8 billion. GAO‘s estimate was based on the
anticipated costs of DNDO implementing its 2006 project execution plan,
the most recent official documentation of the program. DNDO‘s cost
estimate of $2.1 billion to implement its project execution plan is
unreliable because it omits major project costs, such as maintenance,
and relies on a flawed methodology. For example, although the normal
life expectancy of the standard cargo ASP is about 10 years, DNDO‘s
estimate considers only 8 years. According to DNDO officials, the
agency is now following a scaled-back ASP deployment strategy rather
than the 2006 project execution plan, and a senior DNDO official told
GAO the ASP deployment strategy could change dramatically depending on
the outcome of ongoing testing. GAO‘s analysis indicated the cost to
implement the scaled-back plans over the period 2008 through 2017 will
be about $2 billion, but could range from $1.7 billion to $2.3 billion.
However, frequent changes in DNDO‘s deployment strategy make it
difficult to assess ASP program costs. GAO‘s recent report recommended
that the Secretary of Homeland Security direct DNDO to update the
project execution plan, revise its cost estimate, and communicate the
revised estimate to the Congress so that it is fully apprised of the
program‘s scope and funding requirements. DHS agreed with the
recommendations.
DNDO has made progress in addressing a number of problems GAO
identified in previous rounds of ASP testing. However, GAO‘s ongoing
review of the 2008 ASP testing program identified several potential
areas of concern. First, the DHS criteria for ’significant increase in
operational effectiveness“ appear to set a low bar for improvement”for
example, by requiring ASPs to perform at least as well as current
generation equipment when nuclear material is present in cargo but not
specifying an actual improvement. GAO recently requested additional
information from DNDO about the rationale behind these criteria,
particularly in light of seemingly stricter criteria found in other
documents. Second, the ASP certification schedule does not allow for
completion of computer simulations that could provide additional data
on ASP performance. While these computer simulations may have
limitations, they also could provide useful data on ASP capabilities
prior to the Secretary‘s decision on certification. Finally, the test
schedule is highly compressed and is running at least 8 weeks behind,
leaving limited time for analysis and review of test results. Assuming
that DHS addresses these concerns, the 2008 round of testing could
provide an objective basis for comparing ASPs with current generation
equipment. However, GAO recommended in March 2006 that DHS analyze the
benefits and costs of deploying ASPs to determine whether any
additional detection capability provided by ASPs is worth the cost, and
would still question the replacement of current generation equipment
with ASPs until DNDO demonstrates that any additional increase in
security would be worth the ASPs‘ much higher cost.
To view the full product, including the scope and methodology, click on
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-1178T]. For more
information, contact Gene Aloise at (202) 512-3841 or aloisee@gao.gov.
[End of section]
Mr. Chairman and Members of the Committee:
I am pleased to be here today to discuss our work on the plans of the
Department of Homeland Security's (DHS) Domestic Nuclear Detection
Office (DNDO) to develop and deploy advanced spectroscopic portal (ASP)
radiation detection monitors to the nation's ports of entry.[Footnote
1] Preventing radioactive material from being smuggled into the United
States--perhaps to be used by terrorists in a nuclear weapon or in a
radiological dispersal device (a "dirty bomb")--is a key national
security objective. Today I will discuss our September report, which
focuses on developing an independent life cycle cost estimate for
replacing radiation detection equipment already deployed at U.S. ports
of entry with ASPs,[Footnote 2] and our ongoing work reviewing DNDO's
current program of ASP testing activities, which started in April 2008
and are scheduled to be completed in November 2008 under DNDO's latest
plan. These tests are critically important because they will serve as
the primary support for a congressionally mandated DHS Secretarial
certification of the effectiveness of ASPs, also scheduled for November
2008.
Radiation portal monitors--large stationary detectors through which
cargo containers and trucks pass as they enter the United States--are
an important component of the radiation detection system. The portal
monitors in use today, known as polyvinyl toluene (PVT) monitors, are
capable of detecting radiation but not identifying specific radioactive
materials. To address this limitation, U.S. Customs and Border
Protection (CBP) relies on handheld detection technologies,
particularly radioactive isotope identification devices (RIID), to help
CBP officers distinguish between dangerous and benign materials.
[Footnote 3] CBP's standard operating procedures for use of radiation
detection equipment include conducting primary inspections with PVTs to
detect the presence of radioactivity, and secondary inspections with
PVTs and RIIDs to confirm and identify the source and determine whether
it constitutes a threat. Any vehicle triggering a PVT alarm is referred
to a secondary screening area, where it is sent through a second PVT to
confirm the original alarm. Whether the second PVT confirms the alarm
or not, the vehicle, driver, and any passengers or cargo are scanned by
a CBP officer with a RIID, which can detect radiation and also identify
many of the most commonly used radioactive materials by name. All PVT
alarms must be resolved--that is, CBP officers must investigate each
alarm until they are convinced that the vehicle, occupants, and any
cargo pose no threat and, if radioactive materials are found, that the
vehicle occupants appear to have a legitimate reason to possess and
transport them--before the vehicle, driver, and any passengers can be
allowed to enter the United States.
In contrast to PVTs, ASPs offer the potential to detect radiation and
identify the source, reducing the need for secondary screenings of
cargo containing benign radioactive materials. According to DNDO, the
inability of the current generation PVT systems to identify the type of
material causing an alarm results in the need to balance system
sensitivity with the false and innocent alarm burden to the flow of
commerce. DNDO also believes that CBP may use an inordinate amount of
inspection resources for radiation detection at the expense of other
missions, such as drug interdiction. To address these limitations, DNDO
is sponsoring the development of ASPs with the expectation that they
will minimize both missed threats and false alarms when deployed for
primary screening and improve upon the RIID's identification
performance and time required to correctly resolve primary alarms when
deployed for secondary screening.
ASPs cost significantly more than PVTs. DNDO's latest unit cost
estimate (including deployment costs) is about $800,000 for the
standard cargo version of the ASP and about $425,000 for the PVT
standard cargo portal. Due to concerns about the performance and higher
cost of ASPs relative to PVTs, the Congress has required that the
Secretary of Homeland Security submit a report certifying that ASPs
will provide a significant increase in operational effectiveness.
[Footnote 4] Prior to primary and secondary deployment of the ASPs, the
Secretary must submit separate and distinct certifications that address
the unique requirements for operational effectiveness of each type of
deployment.
Our prior work on DNDO's efforts to develop and procure ASPs raised a
number of serious concerns. In March 2006, we reported on, among other
things, DNDO's efforts to develop ASPs; we recommended that DHS analyze
the benefits and costs of deploying ASPs to determine whether any
additional detection capability provided by ASPs is worth the
additional cost.[Footnote 5] In October 2006, we concluded that DNDO's
cost-benefit analysis did not provide a sound basis for its decision to
purchase and deploy ASPs at an estimated cost at that time of $1.2
billion--for example, because DNDO relied on assumptions of the
anticipated performance level of ASPs instead of actual test data.
[Footnote 6] We also reported that DNDO did not assess the likelihood
that ASPs would either misidentify or fail to detect nuclear or
radiological material; rather, it focused its analysis on reducing the
time necessary to screen traffic at ports of entry and the impact of
any delays on commerce. We recommended, in part, that DNDO conduct
further testing before spending additional funds to purchase ASPs.
Similarly, in September 2007, we testified that DNDO's testing of ASPs
at the Department of Energy's (DOE) Nevada Test Site did not represent
an objective or rigorous assessment because DNDO used biased test
methods that enhanced the apparent performance of the ASPs and did not
test the limitations of the ASPs' detection capabilities.[Footnote 7]
We recommended that DHS delay ASP certification and full-scale
production decisions until all relevant tests and studies have been
completed, limitations to these tests and studies have been identified
and addressed, and results of the tests and analyses have been reported
to the appropriate congressional committees. For reasons I am about to
discuss, these recommendations are as important today as when we made
them last year.
My testimony today addresses (1) the highlights of our September 2008
report on our independent life cycle cost estimate for deploying ASPs
at ports of entry, and (2) preliminary observations from our ongoing
work reviewing the current program of ASP testing. For our September
2008 report, we contracted with a company with expertise in estimating
the life cycle costs of major federal acquisitions. Together we
developed an independent cost estimate of DHS's 2006 strategy to deploy
radiation detection portal monitors, covering fiscal years 2007 through
2017, to ensure that the Congress has authoritative information on all
the life-cycle costs associated with a full-scale acquisition of
radiation portal monitors. To review ASP testing, we analyzed ASP test
plans and interviewed senior DNDO officials responsible for managing
the ASP program, and we observed testing conducted at the DOE's Nevada
Test Site and Pacific Northwest National Laboratory. We also
interviewed DOE, national laboratory, National Institute of Standards
and Technology, and DHS officials. We conducted this work, including
our work to date on ASP testing, in accordance with generally accepted
government auditing standards. Those standards require that we plan and
perform the audit to obtain sufficient, appropriate evidence to provide
a reasonable basis for our findings and conclusions based on our audit
objectives. We believe that the evidence obtained provides a reasonable
basis for our findings and conclusions based on our audit objectives.
Summary:
Our independent cost estimate suggests that from 2007 through 2017 the
cost of DNDO's program to equip U.S. ports of entry with radiation
detection equipment will likely be about $3.1 billion, but could range
from $2.6 billion to $3.8 billion. We based our estimate on the
anticipated costs of DNDO implementing its 2006 project execution plan,
the most recent official documentation of the radiation portal monitor
project's objectives, scope, schedule, costs, and funding requirements.
According to the project execution plan, DNDO will buy and deploy
multiple types of portal monitors, including ASPs designed to screen
rail cars, airport and seaport cargo, as well as mobile ASPs to provide
greater flexibility in screening commerce. In March 2008, DNDO
estimated the total cost of deploying radiation portal monitors to be
about $2.1 billion, but we believe the agency's estimate is unreliable
because it omits major project costs, such as maintenance, and relies
on a flawed methodology. For example, although the normal life
expectancy of the standard cargo ASP is about 10 years, DNDO's estimate
considers only 8 years. Furthermore, during the course of our review,
DNDO officials told us the only ASP that the agency now plans to deploy
is the standard cargo portal monitor. Although DNDO could not fully
document this change, the agency provided a one-page summary of its
current deployment plans. We performed a limited analysis of these
summary data and determined that the cost to deploy standard ASP cargo
portals over the period 2008 through 2017 will likely be about $2
billion, but could range from $1.7 billion to $2.3 billion. However, a
DNDO official responsible for overseeing the agency's operations told
us that even the deployments listed on the one-page summary could
change dramatically depending on the outcome of ongoing testing. In our
view, it is difficult to assess the costs of the ASP program because of
the frequent changes in DNDO's deployment strategy. Our report
recommended that the Secretary of Homeland Security direct DNDO to
update the project execution plan, revise its cost estimate, and
communicate the revised estimate to the Congress so that it is fully
apprised of the program's scope and funding requirements. DHS agreed
with our recommendations.
While DNDO has made progress in addressing a number of problems we
identified in previous rounds of ASP testing, we have identified
potential areas of concern based on our ongoing review of the 2008 ASP
testing program. First, the DHS criteria for "significant increase in
operational effectiveness" appear to set a low bar for improvement.
Most notably, the criteria for primary screening require ASPs to
perform at least as well as current generation equipment when special
nuclear material or medical or industrial isotopes are present in
cargo, but they do not specify an actual improvement. We recently
requested additional information from DNDO about the rationale behind
these criteria, particularly in light of seemingly stricter criteria
found in other documents. Second, the schedule leading up to ASP
certification does not allow for completion of "injection studies"--a
type of computer simulation for testing the response of the ASP threat
identification algorithms to data on threat objects combined with
stream-of-commerce data from a port of entry. While we recognize that
injection studies have limitations, they could provide useful data on
ASP capabilities, particularly for primary screening. Finally, the test
schedule leading up to certification is highly compressed and is
running at least 8 weeks behind, leaving limited time for analysis and
review of test results. Assuming that DHS addresses these concerns, the
2008 round of testing could provide an objective basis for comparing
ASPs with current generation equipment. However, consistent with our
March 2006 recommendation that DHS analyze the benefits and costs of
deploying ASPs, we would still question the replacement of current
generation equipment with ASPs until DNDO demonstrates that any
additional detection capability and increase in security provided by
ASPs would be worth the ASPs' much higher cost.
DNDO's Program to Deploy Radiation Detection Portal Monitors at U.S.
Ports of Entry Is Likely to Cost About $3 Billion:
As we discuss in our report, our independent cost estimate suggested
the total cost of DNDO's program to equip U.S. ports of entry with
radiation detection equipment will likely be about $3.1 billion, but
could range between $2.6 billion and $3.8 billion. We based our
estimate on the anticipated costs of DNDO implementing its 2006 project
execution plan, the most recent official documentation of the radiation
portal monitor project's objectives, scope, schedule, costs, and
funding requirements. According to this plan, DNDO plans to buy and
deploy several types of ASPs, including those designed to screen rail
cars, airport cargo, and seaport cargo, as well as mobile ASPs that
provide greater flexibility in screening commerce. The plan also
provides for the deployment of several types of PVTs. Clearly, the
numbers and types of portal monitors deployed will significantly affect
the total cost of the radiation portal monitor project. In all, the
project execution plan calls for the purchase of 2,754 portal monitors,
with a total of 2,582 scheduled for deployment--approximately 1,034
ASPs and 1,548 PVTs--and 172 held in excess at the project's
completion.
In developing our estimate, we categorized radiation portal monitor
project costs according to program phases--design and development,
procurement, deployment, maintenance, and operational sustainment. We
did not incorporate operational costs, in particular the cost of CBP
officers operating the radiation detection equipment. DNDO and CBP
believe that deploying ASPs will reduce the use of CBP staff resources
for radiation detection tasks, but currently there are no usable
estimates of how CBP's staffing would change with the deployment of
ASPs. Our analysis included a period of 11 years: actual life-cycle
expenses from fiscal year 2007, and estimated life cycle costs from
fiscal year 2008 through 2017.
DNDO's estimate of $2.1 billion to deploy radiation detection equipment
(submitted as part of its budget request to OMB in March 2008) is
unreliable because it omits major project costs and relies on flawed
methodology. As a result, DNDO's cost estimates and budget requests for
the radiation portal monitor project are too low, which could lead to
significant cost overruns later in the project. DNDO's estimate
contains the following major deficiencies:
* It does not appear to include the costs of all variations of ASPs
contained in DNDO's project execution plan. DNDO's current baseline
considers only the standard ASP cargo portal and ignores the costs of
other types of ASPs, such as those designed to screen rail cars,
airport and seaport cargo, and mobile detectors. According to senior
DNDO officials, the current approved project execution plan no longer
reflects the agency's procurement and deployment plans for ASPs. In
acknowledging that the project execution plan should be revised and
updated, these officials told us that DNDO currently plans to field
only the standard cargo ASP portal, mainly at high-volume ports of
entry. (We used the 2006 project execution plan because it is the
agency's only official plan for ASP deployment.)
* DNDO's estimate considers only 8 years rather than 10 years, the
operational life expected by the manufacturer of sodium iodide
crystals, a key ASP component. DHS cost guidance maintains that a
project's life cycle can be estimated over that period of time during
which equipment will remain available before it is exhausted, that is,
decayed or deteriorated. DNDO officials agreed that a 10-year life
cycle cost estimate would have been more appropriate and said that they
would have used a 10-year estimate had they not been constrained by OMB
budget submission software, which limits the number of years of costs
that can be included.
* DNDO's cost estimate does not include all of the elements of the
ASPs' life cycle, such as maintenance or operational sustainment. These
costs are approximately $999.2 million and $364.9 million,
respectively, under our cost estimate.
* DNDO did not have detailed documentation of the costs to support its
estimates. According to OMB, DHS, and GAO guidelines, such
documentation is necessary to establish the basis of the estimates and
to provide assurances that the estimates are credible.
Assuming DNDO attempts to implement its authorized 2006 project
execution plan rather than its scaled-back plan, our report estimated a
$753 million budget shortfall for the radiation portal monitor project
for fiscal years 2008 through 2012 (the years included in both our
analysis and the DNDO analysis). Additionally, we estimated that DNDO
will require another $833 million from fiscal year 2013 through fiscal
year 2017 to complete the entire life cycle of the program.
According to DNDO officials, our cost estimate will result in estimates
higher than the program's current true cost because it is based on the
outdated project execution plan. However, in the absence of more recent
documentation, we believe our independent cost estimate must be based
on the agency's most recent approved plan (the 2006 project execution
plan). Furthermore, several official DNDO documents specify that
multiple versions of ASPs will be deployed. For example, the agency's
February 2006 expenditure plan submitted to the Congress foresees
"several variants" of ASP systems being deployed, with standard cargo,
rail, and automobile versions mentioned specifically. And DNDO's
submissions to OMB for budget years 2008 and 2009 describe a program
that includes land crossings, seaports, rail lines, airports, and other
ports of entry. Finally, agency officials acknowledged the program
requirements that would have been fulfilled by the discontinued ASPs
remain valid, including screening rail cars, airport cargo, and cargo
at seaport terminals, but the agency has no current plans for how such
screening will be accomplished. These officials told us the technology
to accomplish these requirements likely will not be ASP monitors and
could be a totally new technology. We believe a comprehensive estimate
of the cost to provide radiation detection equipment for U.S. ports of
entry should account for meeting these objectives, even if DNDO decides
that ASP technology is not suited to them.
DNDO agreed in June 2008 to update its project execution plan so that
we could better estimate the costs of the agency's current plans. DNDO
also agreed to provide an updated estimate of the standard portal's
life cycle costs and to meet with us to reconcile our two estimates.
However, DNDO did not provide a revised project execution plan or cost
estimate and instead provided in July 2008 a one-page spreadsheet of
summary information outlining its plans to buy and deploy portal
monitors for the 7-year period 2008 through 2014. DNDO's summary data
indicate that during this time period the agency plans to deploy 717
ASPs and 1,005 PVTs. The summary data do not provide the breadth and
depth of information needed to generate detailed and fully documented
cost estimates. Furthermore, according to subsequent discussions with a
senior DNDO official, if ongoing tests indicate the ASPs' performance
warrants it, the agency may speed its deployment of ASPs over the next
few months. In our view, the frequent changes in deployment plans and
the lack of available cost documentation raise concerns about the
overall management of the radiation portal monitor project and whether
it is guided by a sound and stable strategy. Nonetheless, we used the
agency's summary data to perform a more limited cost estimate for only
the standard cargo portal. We found that from 2008 to 2017 the total
program cost for buying and deploying standard cargo portals would
likely be about $2 billion, but could range from about $1.7 billion to
$2.3 billion.
Our report recommended that the Secretary of Homeland Security direct
the Director of DNDO to (1) work with the Commissioner of CBP to update
the project execution plan to guide the entire radiation detection
program at U.S. ports of entry, (2) revise DNDO's estimate of the
program's cost and ensure that the estimate considers all the costs
associated with its project execution plan, and (3) communicate this
revised estimate to the Congress so that it is fully apprised of the
program's scope and funding requirements. DHS agreed with our
recommendations in its comments responding to our draft report.
Preliminary Observations of Ongoing Testing of ASPs:
We are currently reviewing the ongoing 2008 ASP testing program and
process leading to a decision by the Secretary of Homeland Security on
certification of ASPs, and we plan to issue a final report in November
2008, prior to the Secretary's decision on certification. Our work to
date shows that DNDO has made progress in addressing a number of
problems we identified in previous rounds of ASP testing. A particular
area of improvement is in the performance testing at the Nevada Test
Site, where DNDO conducts trials of the ability of ASPs to detect and
identify radioactive materials, particularly those that could be used
in a nuclear weapon. In 2007, we reported that DNDO had allowed ASP
system contractors to adjust their systems after preliminary runs using
the same radiological materials that would be used in the formal tests,
potentially biasing the test results. In contrast, in our more recent
work, we found that the plan for the 2008 round of performance testing
stipulated that there would be no system contractor involvement in test
execution, and no ASP system contractors were present at the test
location on the day we observed performance testing. Furthermore, DNDO
officials told us, and we observed, that they did not conduct
preliminary runs with threat objects that were used in the formal
tests. A further area of improvement in the performance testing
concerns the use of handheld RIIDs. Specifically, DNDO used actual CBP
officers who adhered to nearly all of the standard operating procedures
in testing the RIIDs. This improvement addresses our concern from last
year that DNDO did not objectively test the performance of the RIIDs
because it did not use a critical CBP standard operating procedure that
is fundamental to the equipment's performance in the field.
Despite these improvements, we have identified potential areas of
concern based on our ongoing review. Our concerns fall into three
areas:
* DHS criteria for "significant increase in operational effectiveness."
DNDO, CBP and the DHS management directorate agreed on six criteria for
the degree of improvement ASPs must provide over current generation
equipment--four criteria for primary screening and two for secondary
screening. These criteria appear to set a low bar for improvement. Most
notably, the criteria for primary screening require ASPs to perform at
least as well as current generation equipment when special nuclear
material or medical or industrial isotopes are present in cargo, but
they do not specify an actual improvement. Similarly, one of the
criteria for secondary screening requires ASPs to reduce the average
time to release conveyances but does not indicate a specific level of
reduction. As such, the criteria leave open the possibility of a
Secretarial decision in favor of certification even if ASPs do not
provide a significantly higher probability of detection compared with
current generation PVTs when deployed for primary screening and only a
small reduction in the time required for secondary screening. We
recently requested additional information from DNDO about the rationale
behind these criteria, particularly in light of seemingly stricter
criteria found in other documents such as the ASP performance
specification, which requires an 80 percent probability of detection,
at a 95 percent confidence, of government-designated representative
threat objects. We are awaiting DNDO's response.
* The extent to which the 2008 testing program provides a sound basis
for determining ASPs' operational effectiveness. While the performance
testing at Nevada Test Site could provide a useful comparison of the
performance of the ASPs and current generation equipment for both
primary and secondary screening given the improvements described above,
DNDO does not plan to complete "injection studies" that could provide
additional data on ASP performance prior to ASP certification.
(Injection studies are a type of computer simulation for testing the
response of the ASP threat identification algorithms to data on threat
objects combined with stream-of-commerce data from a port of entry.)
Rather, DNDO officials told us that testing using actual threat objects
is sufficient for certification. While we recognize the limitations of
injection studies such as the inability to exactly model threat objects
hidden in commerce, reliance on performance testing but not injection
studies for the Secretary's decision on certification will result in
less data with which to evaluate the capabilities of ASPs, particularly
for primary screening.[Footnote 8] According to officials from Pacific
Northwest National Laboratory, which has a lead role in injection
studies, computer simulations allow for an increased statistical
confidence in the measured probability of detection compared with
testing at Nevada Test Site. The ASP test and evaluation master plan
similarly emphasizes the importance of injection studies for a complete
test of radiation detection system performance, and a February 2008
DNDO presentation to us includes among its fiscal year 2008 objectives
the use of preliminary results from injection studies to evaluate
performance of ASPs for primary screening. However, injection studies
have taken longer than projected due to a number of issues such as
deficiencies in the stream-of-commerce data.
* The schedule leading up to a decision on ASP certification by the
Secretary of Homeland Security. The test schedule leading up to
certification is highly compressed and is running at least 8 weeks
behind the schedule DNDO provided to us in May 2008, leaving limited
time for analysis and review of test results. Key phases of the test
schedule include system qualification, followed by testing at Pacific
Northwest National Laboratory to demonstrate that ASPs are ready to be
integrated into the interdiction systems at U.S. ports of entry and
concurrent testing at Nevada Test Site to compare the ASP performance
in detecting and identifying threats with that of current generation
equipment. According to DNDO, the systems qualification and performance
tests have been completed, but only after being delayed due to problems
with system qualification. The final two phases, field validation at
four ports of entry operated by CBP and operational testing at one port
of entry, have not yet started. Field validation was to have started
yesterday--September 24--but as with integration and performance
testing this phase has also been delayed, and CBP has not yet indicated
when it will begin. Even before this latest delay, the time between the
scheduled end of operational testing in early November and a
Secretarial decision on certification on November 28 allowed less than
a month for review and analysis of results. Furthermore, DHS officials
have indicated that the Secretary may make a decision on certification
on the basis of "quick look" (preliminary) rather than final reports if
the quick look reports are favorable to ASPs. The limited time between
completion of testing and a decision on certification, combined with
the potential reliance on quick look reports, increases the risk of a
decision in favor of certification being called into question later by
a more thorough analysis and review of results.
Assuming that DHS addresses these concerns--for example, by clarifying
the criteria for significant increase in operational effectiveness,
performing the injection studies, and delaying a Secretarial decision
on certification--the 2008 round of testing could provide an objective
basis for comparing ASPs with current generation equipment. However,
consistent with our March 2006 recommendation that DHS analyze the
benefits and costs of deploying ASPs to determine whether any
additional detection capability provided by ASPs is worth the cost, we
would still question the replacement of current generation equipment
with ASPs, particularly considering the gaps identified by DNDO in the
global nuclear detection architecture--essentially, an integrated
system of radiation detection equipment and interdiction activities to
combat nuclear smuggling in foreign countries, at the U.S. border, and
inside the United States. As we testified before this committee earlier
this year, such gaps include land border crossings into the United
States between formal points of entry, small maritime vessels, and
international general aviation.[Footnote 9] An updated cost-benefit
analysis, based on the latest information on ASP costs and test
performance, could help policymakers address any trade-offs in
addressing these gaps versus replacing current generation radiation
detection equipment already deployed at ports of entry. The current
generation equipment has known limitations, particularly with regard to
the limited ability to detect certain nuclear materials and the
inability to distinguish between types of radiological material.
However, the existing equipment provides at least some detection
capability and may act as a deterrent, and there is no evidence we are
aware of that it is impeding the flow of commerce.
Mr. Chairman, this concludes my prepared statement. I would be pleased
to respond to any questions that you or other members of the Committee
may have.
GAO Contact and Staff Acknowledgments:
For further information on this testimony, please contact Gene Aloise
at (202) 512-3841 or by e-mail at aloisee@gao.gov. Individuals making
key contributions to this testimony include Joseph Cook, Jennifer
Echard, Brian Octeau, Alison O'Neill, Tim Persons (GAO's Chief
Scientist), Karen Richey, Benjamin Shouse, Kevin Tarmann, Eugene
Wisnoski, and Ned Woodward.
[End of section]
Footnotes:
[1] DNDO was established within DHS in 2005; its mission includes
developing, testing, acquiring, and supporting the deployment of
radiation detection equipment at U.S. ports of entry.
[2] GAO, Combating Nuclear Smuggling: DHS's Program to Procure and
Deploy Advanced Radiation Detection Portal Monitors Is Likely to Exceed
the Department's Previous Cost Estimates, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-08-1108R] (Washington, D.C.:
Sept. 22, 2008).
[3] CBP, also part of DHS, began deploying radiation detection
equipment in 2002, prior to DNDO's creation, under the radiation portal
monitor project. CBP remains responsible for operating the equipment.
[4] Public Law 110-161 (121 Stat. 1844, 2069).
[5] GAO, Combating Nuclear Smuggling: DHS Has Made Progress Deploying
Radiation Detection Equipment at U.S. Ports of Entry, but Concerns
Remain, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-389]
(Washington, D.C.: Mar. 22, 2006).
[6] GAO, Combating Nuclear Smuggling: DHS's Cost-Benefit Analysis to
Support the Purchase of New Radiation Detection Portal Monitors Was Not
Based on Available Performance Data and Did Not Fully Evaluate All the
Monitors' Costs and Benefits, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-07-133R] (Washington, D.C.: Oct. 17, 2006).
[7] GAO, Combating Nuclear Smuggling: Additional Actions Needed to
Ensure Adequate Testing of Next Generation Radiation Detection
Equipment, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1247T]
(Washington, D.C.: Sept. 18, 2007).
[8] Injection studies as currently designed have less relevance to
evaluating the use of ASPs for secondary screening because the studies
will not compare ASP performance with that of the handheld RIIDs, a key
component of the current generation equipment used for secondary
screening.
[9] GAO, Nuclear Detection: Preliminary Observations on the Domestic
Nuclear Detection Office's Efforts to Develop a Global Nuclear
Detection Architecture, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-08-999T] (Washington, D.C.: July 16, 2008).
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
