Combating Nuclear Smuggling
Additional Actions Needed to Ensure Adequate Testing of Next Generation Radiation Detection Equipment Gao ID: GAO-07-1247T September 18, 2007The Department of Homeland Security's (DHS) Domestic Nuclear Detection Office (DNDO) is responsible for addressing the threat of nuclear smuggling. Radiation detection portal monitors are key elements in our national defenses against such threats. DHS has sponsored testing to develop new monitors, known as advanced spectroscopic portal (ASP) monitors. In March 2006, GAO recommended that DNDO conduct a cost-benefit analysis to determine whether the new portal monitors were worth the additional cost. In June 2006, DNDO issued its analysis. In October 2006, GAO concluded that DNDO did not provide a sound analytical basis for its decision to purchase and deploy ASP technology and recommended further testing of ASPs. DNDO conducted this ASP testing at the Nevada Test Site (NTS) between February and March 2007. GAO's statement addresses the test methods DNDO used to demonstrate the performance capabilities of the ASPs and whether the NTS test results should be relied upon to make a full-scale production decision.
Based on our analysis of DNDO's test plan, the test results, and discussions with experts from four national laboratories, we are concerned that DNDO's tests were not an objective and rigorous assessment of the ASPs' capabilities. Our concerns with the DNDO's test methods include the following: (1) DNDO used biased test methods that enhanced the performance of the ASPs. Specifically, DNDO conducted numerous preliminary runs of almost all of the materials, and combinations of materials, that were used in the formal tests and then allowed ASP contractors to collect test data and adjust their systems to identify these materials. It is highly unlikely that such favorable circumstances would present themselves under real world conditions. (2) DNDO's NTS tests were not designed to test the limitations of the ASPs' detection capabilities--a critical oversight in DNDO's original test plan. DNDO did not use a sufficient amount of the type of materials that would mask or hide dangerous sources and that ASPs would likely encounter at ports of entry. DOE and national laboratory officials raised these concerns to DNDO in November 2006. However, DNDO officials rejected their suggestion of including additional and more challenging masking materials because, according to DNDO, there would not be sufficient time to obtain them based on the deadline imposed by obtaining Secretarial Certification by June 26. 2007. By not collaborating with DOE until late in the test planning process, DNDO missed an important opportunity to procure a broader, more representative set of well-vetted and characterized masking materials. (3) DNDO did not objectively test the performance of handheld detectors because they did not use a critical CBP standard operating procedure that is fundamental to this equipment's performance in the field. Because of concerns raised that DNDO did not sufficiently test the limitations of ASPs, DNDO is attempting to compensate for weaknesses in the original test plan by conducting additional studies--essentially computer simulations. While DNDO, CBP, and DOE have now reached an agreement to wait and see whether the results of these studies will provide useful data regarding the ASPs' capabilities, in our view and those of other experts, computer simulations are not as good as actual testing with nuclear and masking materials.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-07-1247T, Combating Nuclear Smuggling: Additional Actions Needed to Ensure Adequate Testing of Next Generation Radiation Detection Equipment
This is the accessible text file for GAO report number GAO-07-1247T
entitled 'Combating Nuclear Smuggling: Additional Actions Needed to
Ensure Adequate Testing of Next Generation Radiation Detection
Equipment' which was released on September 18, 2007.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Testimony:
Before the Subcommittee on Oversight and Investigations, Committee on
Energy and Commerce, House of Representatives:
United States Government Accountability Office:
GAO:
For Release on Delivery Expected at 10:00 a.m. EDT:
Tuesday, September 18, 2007:
Combating Nuclear Smuggling:
Additional Actions Needed to Ensure Adequate Testing of Next Generation
Radiation Detection Equipment:
Statement of Gene Aloise, Director:
Natural Resources and Environment:
Keith Rhodes, Chief Technologist:
Combating Nuclear Smuggling:
GAO-07-1247T:
GAO Highlights:
Highlights of GAO-07-1247T, a testimony before the Subcommittee on
Oversight and Investigations, Committee on Energy and Commerce, House
of Representatives.
Why GAO Did This Study:
The Department of Homeland Security‘s (DHS) Domestic Nuclear Detection
Office (DNDO) is responsible for addressing the threat of nuclear
smuggling. Radiation detection portal monitors are key elements in our
national defenses against such threats. DHS has sponsored testing to
develop new monitors, known as advanced spectroscopic portal (ASP)
monitors. In March 2006, GAO recommended that DNDO conduct a cost-
benefit analysis to determine whether the new portal monitors were
worth the additional cost. In June 2006, DNDO issued its analysis. In
October 2006, GAO concluded that DNDO did not provide a sound
analytical basis for its decision to purchase and deploy ASP technology
and recommended further testing of ASPs. DNDO conducted this ASP
testing at the Nevada Test Site (NTS) between February and March 2007.
GAO‘s statement addresses the test methods DNDO used to demonstrate the
performance capabilities of the ASPs and whether the NTS test results
should be relied upon to make a full-scale production decision.
What GAO Found:
Based on our analysis of DNDO‘s test plan, the test results, and
discussions with experts from four national laboratories, we are
concerned that DNDO‘s tests were not an objective and rigorous
assessment of the ASPs‘ capabilities. Our concerns with the DNDO‘s test
methods include the following:
* DNDO used biased test methods that enhanced the performance of the
ASPs. Specifically, DNDO conducted numerous preliminary runs of almost
all of the materials, and combinations of materials, that were used in
the formal tests and then allowed ASP contractors to collect test data
and adjust their systems to identify these materials. It is highly
unlikely that such favorable circumstances would present themselves
under real world conditions.
* DNDO‘s NTS tests were not designed to test the limitations of the
ASPs‘ detection capabilities”a critical oversight in DNDO‘s original
test plan. DNDO did not use a sufficient amount of the type of
materials that would mask or hide dangerous sources and that ASPs would
likely encounter at ports of entry. DOE and national laboratory
officials raised these concerns to DNDO in November 2006. However, DNDO
officials rejected their suggestion of including additional and more
challenging masking materials because, according to DNDO, there would
not be sufficient time to obtain them based on the deadline imposed by
obtaining Secretarial Certification by June 26. 2007. By not
collaborating with DOE until late in the test planning process, DNDO
missed an important opportunity to procure a broader, more
representative set of well-vetted and characterized masking materials.
* DNDO did not objectively test the performance of handheld detectors
because they did not use a critical CBP standard operating procedure
that is fundamental to this equipment‘s performance in the field.
Because of concerns raised that DNDO did not sufficiently test the
limitations of ASPs, DNDO is attempting to compensate for weaknesses in
the original test plan by conducting additional studies”essentially
computer simulations. While DNDO, CBP, and DOE have now reached an
agreement to wait and see whether the results of these studies will
provide useful data regarding the ASPs‘ capabilities, in our view and
those of other experts, computer simulations are not as good as actual
testing with nuclear and masking materials.
What GAO Recommends:
GAO recommends that, among other things, the Secretary of Homeland
Security delay a full-scale production decision of ASPs until all
relevant studies and tests have been completed, and determine in
cooperation with U.S. Customs and Border Protection(CBP), the
Department of Energy (DOE), and independent reviewers, whether
additional testing is needed.
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1247T].
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Gene Aloise, at (202) 512-
3841 or aloisee@gao.gov.
[End of section]
Mr. Chairman and Members of the Subcommittee:
I am pleased to be here today to discuss our assessment of the testing
of advanced spectroscopic portal (ASP) monitors conducted by the
Domestic Nuclear Detection Office (DNDO). The results of these tests,
including the methods by which they were conducted, are critically
important because they will serve as the primary support for a required
Department of Homeland Security (DHS) Secretarial Certification of the
performance of this equipment and, in turn, authorization to spend up
to $1.2 billion for the full-scale production of the next generation of
radiation detection technology to be deployed to U.S. ports of entry.
The radiation portal monitors in use today can detect the presence of
radiation, but they cannot distinguish between types of radiological
material. For example, they cannot tell the difference between harmless
products that emit radiation, such as ceramic tile, and dangerous
materials, such as highly enriched uranium that could be used to
construct a nuclear weapon. DNDO is primarily responsible for
preventing unauthorized nuclear or radiological materials from entering
the United States. U.S. Customs and Border Protection (CBP) is
responsible for screening cargo as it enters the nation at our borders,
which includes operating radiation detection equipment to intercept
dangerous nuclear and radiological materials. The Department of Energy
(DOE) is the primary agency responsible for the international
deployment of radiation detection equipment.
CBP's standard procedures require incoming cargo to pass through a
radiation portal monitor to screen for the presence of radiation. This
"primary inspection" alerts CBP officers when a radioactive threat
might be present. If there is a potential threat, CBP procedures
require a "secondary inspection." To confirm the presence of radiation,
this secondary inspection usually includes a second screening by a
radiation portal monitor as well as a screening by CBP officers using
handheld radioactive isotope identification devices (RIIDs). These
devices are used to differentiate between types of radioactive material
to determine if the radiation being detected is dangerous. Both the
radiation portal monitors and handheld devices are limited in their
abilities to detect and identify nuclear material.
DNDO asserts that false alarms, or "nuisance alerts," result in large
numbers of secondary inspections--especially at high-volume ports of
entry. CBP officials believe that the number of secondary inspections
required by the currently deployed system are resource intensive and
could be reduced, allowing CBP officers to focus on other border
enforcement responsibilities such as illegal immigration and drug
interdiction. Importantly, however, these officials acknowledged that
the current system provides the best possible radiological and nuclear
screening coverage available and that it does not have a significant
impact on commerce.
DHS would like to improve the capabilities of its portal monitors so
that they can perform the dual roles of detecting radiation and
identifying radiological materials. In this regard, DHS has sponsored
research, development, and testing activities designed to create ASP
portal monitors capable of performing both functions. In July 2006, DHS
awarded contracts to three vendors to develop the ASPs' capabilities.
These awards were based mainly on performance tests conducted at DHS's
Nevada Test Site in 2005, where ten competing ASP vendors' monitors
were evaluated. At the same time, three currently deployed portal
monitors that use polyvinyl toluene plastic scintillators, known as
PVTs, were also tested.
To ensure that DHS's substantial investment in radiation detection
technology yields the greatest possible level of detection capability
at the lowest possible cost, in a March 2006 GAO report,[Footnote 1] we
recommended that once the costs and capabilities of ASPs were well
understood, and before any of the new equipment was purchased for
deployment, the Secretary of DHS work with the Director of DNDO to
analyze the costs and benefits of deploying ASPs. Further, we
recommended that this analysis focus on determining whether any
additional detection capability provided by the ASPs was worth the
considerable additional costs. In response to our recommendation, DNDO
issued its cost-benefit analysis in May 2006[Footnote 2] and an
updated, revised version in June 2006. According to senior agency
officials, DNDO believes that the basic conclusions of its cost-benefit
analysis showed that the new ASP monitors are a sound investment for
the U.S. government.
In an October 2006 GAO report[Footnote 3], we concluded that DNDO's
cost benefit analysis did not provide a sound basis for DNDO's decision
to purchase and deploy ASP technology because it relied on assumptions
of the anticipated performance level of ASPs instead of actual test
data, and that it did not justify DHS's plan to spend $1.2 billion to
purchase and deploy ASPs. We also reported that DNDO did not assess the
likelihood that ASPs would either misidentify or fail to detect nuclear
or radiological material. Rather, it focused its analysis on reducing
the time necessary to screen traffic at border check points and reduce
the impact of any delays on commerce. We recommended that DNDO conduct
further testing of ASPs and the currently deployed PVTs before spending
additional funds to purchase ASPs.
Mr. Chairman, my remarks today focus on the tests conducted by DNDO at
the Nevada Test Site between February and March of this year and the
test methods DNDO used to demonstrate the performance capabilities of
the ASPs. Specifically, I will discuss how the tests were conducted at
the Nevada Test Site, and whether these test results should be relied
upon to support Secretarial Certification or to make a full-scale
production decision. I will also discuss current DNDO testing efforts
and how they may impact future decision making.
In conducting our review, we analyzed DNDO's test plans and procedures
and interviewed senior DNDO officials responsible for managing the ASP
program, including the development and testing of ASP monitors. We
observed DNDO's testing conducted at the Nevada Test Site and the New
York Container Terminal. We obtained information on DNDO's test methods
from National Institute of Standards and Technology (NIST) officials
and discussed the efficacy of DNDO's test methods with experts from
NIST, DOE, the private sector, and four national laboratories. We also
met with senior CBP and DOE officials as the main end users of portal
monitor equipment. We conducted our review in Washington, D.C. from
March to September 2007 in accordance with generally accepted
government auditing standards.
In Summary:
Based on our analysis of DNDO's test plan, the test results, and
discussions with experts from four national laboratories, we are
concerned that DNDO's tests were not an objective and rigorous
assessment of the ASPs capabilities. Our concerns with DNDO's test
methods include the following:
* DNDO used biased test methods that enhanced the performance of the
ASPs. Specifically, DNDO conducted numerous preliminary runs of almost
all of the materials, and combinations of materials, that were used in
the formal tests and then allowed ASP contractors to collect test data
and adjust their systems to identify these materials. It is highly
unlikely that such favorable circumstances would present themselves
under real world conditions.
* DNDO's tests conducted at the Nevada Test Site were not designed to
test the limitations of the ASPs' detection capabilities--a critical
oversight in DNDO's original test plan. DNDO did not use a sufficient
amount of the type of materials that would mask or hide dangerous
sources and that ASPs would likely encounter at ports of entry. DOE and
national laboratory[Footnote 4] officials raised these concerns to DNDO
in November 2006. However, DNDO officials rejected their suggestion of
including additional and more challenging masking materials because,
according to DNDO, it would not be able to obtain such materials in
time to meet the Secretarial Certification deadline. By not
collaborating with DOE until late in the test planning process, DNDO
missed an important opportunity to procure a broader, more
representative set of well-vetted and characterized masking materials.
* DNDO did not objectively test the performance of handheld detectors
because it did not use a critical CBP standard operating procedure that
is fundamental to this equipment's performance in the field.
As a result of concerns raised that DNDO's NTS tests did not
sufficiently test the limitations of ASPs, DNDO is now attempting to
compensate for weaknesses in the original test plan by conducting
additional testing known as injection studies--essentially computer
simulations. While DNDO, CBP, and DOE have now reached an agreement to
wait and see whether the results of these studies will provide useful
data regarding the ASPs' capabilities, in our view and those of other
experts, computer simulations are not as good as actual testing with
nuclear and masking materials.
We are recommending that the Secretary of DHS delay certification until
all tests and studies have been completed and validated, and all test
results have been provided to relevant parties, including CBP and DOE.
If DNDO, CBP and DOE determine that additional testing is needed, then
an independent group within DHS should be formed to conduct this
testing. In addition, the results of the tests and analyses should be
reported to the appropriate congressional committees before large scale
purchases are made.
Background:
The Department of Homeland Security Appropriations Act for Fiscal Year
2007 states that "none of the funds appropriated—shall be obligated for
full scale procurement of [ASP] monitors until the Secretary of
Homeland Security has certified—that a significant increase in
operational effectiveness will be achieved."[Footnote 5] DNDO noted
that certification would meet DHS guidelines for the review and
approval of complex acquisitions. Specifically, DNDO stated that the
Secretary's decision would be made in the context of DHS "Key Decision
Point 3," which details the review and approval necessary for DHS
acquisition programs to move from the "Capability Development and
Demonstration" phase to the "Production and Deployment Phase."
To meet the statutory requirement to certify the ASPs will provide a
"significant increase in operational effectiveness," and requirements
outlined in DHS Management Directive 1400, DNDO, with input from
subject matter experts, developed a series of tests intended to
demonstrate, among other things, ASP performance and deployment
readiness. The tests were conducted at several venues, including the
Nevada Test Site, the New York Container Terminal, the Pacific
Northwest National Laboratory, and five ports of entry. DNDO stated
that its request for full-scale production approval would be based upon
completed and documented results of these tests. To meet the
Secretary's goal of deploying 225 ASPs by the end of calendar year
2008, Secretarial Certification was scheduled for June 26, 2007.
To guide the test operations, DNDO defined a set of Critical
Operational Issues that outlined the tests' technical objectives and
provided the baseline to measure demonstrated effectiveness. The
purpose of the Critical Operational Issue 1 is to "verify operational
effectiveness" of ASPs and determine whether "ASP systems significantly
increase operational effectiveness relative to the current generation
detection and identification system." DNDO conducted a series of tests
at the Nevada Test Site, the single focus of which, according to DNDO,
was to resolve Critical Operational Issue 1. According to DNDO, these
tests began in February 2007 and concluded in March 2007. DNDO's Nevada
Test Site test plan, dated January 12, 2007, identified three primary
test objectives comparing the operational effectiveness of the ASP
systems with existing detection and identification systems at current
high-volume operational thresholds. Specifically, DNDO sought to
determine the ASPs' probability to (1) detect and identify nuclear and
radiological threats (2) discriminate threat and non-threat
radionuclides in primary [screening positions], and (3) detect and
identify threat radionuclides in the presence of non-threat
radionuclides.
How the Tests at the Nevada Test Site Were Conducted:
The Nevada Test Site test plan had two key components. First, DNDO
developed guidelines for basic test operations and procedures,
including test goals and expectations, test tasks and requirements, and
roles and responsibilities of personnel involved in the testing,
including the ASP contractors. The second component involved the
National Institute of Standards and Technology developing test
protocols that defined, among other things, how many times a container
carrying test materials would need to be driven through portal monitors
in order to obtain statistically relevant results.
DNDO's tests at the Nevada Test Site were designed to compare the
current system--using PVTs in primary inspections and a PVT and RIID
combination in secondary inspections--to other configurations including
PVTs in primary and ASPs in secondary, and ASPs in both primary and
secondary inspection positions. DNDO tested three ASPs and four PVTs.
The ASP vendors included Thermo, Raytheon, and Canberra. The PVT
vendors included SAIC, TSA, and Ludlum. According to the test plan, to
the greatest extent possible, PVT, ASP, and RIID handheld devices would
be operated consistent with approved CBP standard operating procedures.
Prior to "formal" collection of the data that would be used to support
the resolution of Critical Operational Issue 1, DNDO conducted a series
of tests it referred to as "dry runs" and "dress rehearsals." The
purpose of the dry runs was to, among other things, verify ASP systems'
software performance against representative test materials and allow
test teams and system contractors to identify and implement software
and hardware improvements to ASP systems. The purpose of the dress
rehearsals was to observe the ASPs in operation against representative
test scenarios and allow the test team to, among other things:
* develop confidence in the reliability of the ASP system so that
operators and data analysts would know what to expect and what data to
collect during the formal test,
* collect sample test data, and:
* determine what errors were likely to occur in the data collection
process and eliminate opportunities for error.
In addition to improving ASP performance through dry runs and dress
rehearsals conducted prior to formal data collection, ASP contractors
were also significantly involved in the Nevada Test Site test
processes. Specifically, the test plan stated that "[ASP] contractor
involvement was an integral part of the NTS test events to ensure the
systems performed as designed for the duration of the test."
Furthermore, ASP contractors were available on site to repair their
system at the request of the test director and to provide quality
control support of the test data through real time monitoring of
available data. DNDO stated that Pacific Northwest National Laboratory
representatives were also on site to provide the same services for the
PVT systems.
DNDO conducted its formal tests in two phases. The first, called Phase
1, was designed to support resolution of Critical Operational Issue 1
with high statistical confidence. DNDO told us on multiple occasions
and in a written response that only data collected during Phase 1 would
be included in the final report presented to the Secretary to request
ASP certification. According to DNDO, the second, called Phase 3,
provided data for algorithm development which targeted specific and
known areas in need of work and data to aid in the development of
secondary screening operations and procedures. According to DNDO
documentation, Phase 3 testing was not in support of the full-scale
production decision. Further, DNDO stated that Phase 3 testing
consisted of relatively small sample sizes since the data would not
support estimating the probability of detection with a high confidence
level.
On May 30, 2007, following the formal tests and the scoring of their
results, DNDO told GAO that it had conducted additional tests that DNDO
termed "Special Testing." The details of these tests were not outlined
in the Nevada Test Site test plan. On June 20, 2007, DNDO provided GAO
with a test plan document entitled "ASP Special Testing" which
described the test sources used to conduct the tests but did not say
when the tests took place. According to DNDO, special testing was
conducted throughout the formal Phase 1 testing process and included 12
combinations of threat, masking, and shielding materials that differed
from "dry run," "dress rehearsal," and formal tests. DNDO also stated
that the tests were "blind," meaning that neither DNDO testing
officials nor the ASP vendors knew what sources would be included in
the tests. According to DNDO, these special tests were recommended by
subject matter experts outside the ASP program to address the
limitations of the original NTS test plan, including:
* available time and funding resources,
* special nuclear material sources, and:
* the number of test configurations that could be incorporated in the
test plan, including source isotope and activity, shielding materials
and thicknesses, masking materials, vehicle types, and measurement
conditions.
Unlike the formal tests, National Institute of Standards and Technology
officials were not involved in determining the number of test runs
necessary to obtain statistically relevant results for the special
tests.
DNDO's Test Methods Raise Concerns Regarding the Reliability of Test
Results:
Based on our analysis of DNDO's test plan, the test results, and
discussions with experts from four national laboratories, we are
concerned that DNDO used biased test methods that enhanced the
performance of the ASPs. In the dry runs and dress rehearsals, DNDO
conducted many preliminary runs of radiological, nuclear, masking, and
shielding materials so that ASP contractors could collect data on the
radiation being emitted, and modify their software accordingly.
Specifically, we are concerned because almost all of the materials, and
most combinations of materials, DNDO used in the formal tests were
identical to those that the ASP contractors had specifically set their
ASPs to identify during the dry runs and dress rehearsals. It is highly
unlikely that such favorable circumstances would present themselves
under real world conditions.
A key component of the NTS tests was to test the ASPs' ability to
detect and identify dangerous materials, specifically when that
material was masked or "hidden" by benign radioactive materials. Based
on our analysis, the masking materials DNDO used at NTS did not
sufficiently test the performance limits of the ASPs. DOE national
laboratory officials raised similar concerns to DNDO after reviewing a
draft of the test plan in November 2006. These officials stated that
the masking materials DNDO planned to use in its tests did not emit
enough radiation to mask the presence of nuclear materials in a
shipping container and noted that many of the materials that DOE
program officials regularly observe passing through international ports
emit significantly higher levels of radiation than the masking
materials DNDO used for its tests.
DNDO officials told us that the masking materials used at the Nevada
Test Site represented the average emissions seen in the stream of
commerce at the New York Container Terminal. However, according to data
accumulated as part of DOE's program to secure international ports (the
Megaports program), a significant percentage of cargo passing through
one European port potentially on its way to the United States has
emission levels greater than the average radiation level for cargo that
typically sets off radiation detection alarms. Importantly, DNDO
officials told us that the masking materials used at the Nevada Test
Site were not intended to provide insight into the limits of ASP
detection capabilities. Yet, DNDO's own test plan for "ASP Special
Testing" states, "The DNDO ASP NTS Test Plan was designed to— measure
capabilities and limitations in current ASP systems."
In addition, the NTS tests did not objectively test the ASPs against
the currently deployed radiation detection system. DNDO's test plan
stated that, to the greatest extent possible, PVT, ASP, and RIID
handheld devices would be operated consistent with approved CBP
standard operating procedures. However, after analyzing test results
and procedures used at the Nevada Test Site, CBP officials determined
that DNDO had, in fact, not followed a key CBP procedure. In
particular, if a threat is identified during a secondary screening, or
if the result of the RIID screening isn't definitive, CBP procedures
require officers to send the data to CBP's Laboratories and Scientific
Services for further guidance. DNDO did not include this critical step
in its formal tests. CBP officials also expressed concern with DNDO's
preliminary test results when we met with them in May 2007.
In regards to the special tests DNDO conducted, based on what DNDO has
told us and our own evaluation of the special test plan, we note that:
* because DNDO did not consult NIST on the design of the blind tests,
we do not know the statistical significance of the results, and:
* the tests were not entirely blind because some of the nuclear
materials used in the blind tests were also used to calibrate the ASPs
on a daily basis.
During the course of our work, CBP, DOE, and national laboratory
officials we spoke to voiced concern about their lack of involvement in
the planning and execution of the Nevada Test Site tests. We raised our
concerns about this issue and those of DOE and CBP to DNDO's attention
on multiple occasions. In response to these concerns, specifically
those posed by DOE, DNDO convened a conference on June 27, 2007, of
technical experts to discuss the Nevada test results and the methods
DNDO used to test the effects of masking materials on what the ASPs are
able to detect. As a result of discussions held during that meeting,
subject matter experts agreed that computer-simulated injection studies
could help determine the ASPs' ability to detect threats in the
presence of highly radioactive masking material.
According to a Pacific Northwest National Laboratory report submitted
to DNDO in December 2006[Footnote 6], injection studies are
particularly useful for measuring the relative performance of
algorithms, but their results should not be construed as a measure of
(system) vulnerability. To assess the limits of portal monitors'
capabilities, the Pacific Northwest National Laboratory report states
that actual testing should be conducted using threat objects immersed
in containers with various masking agents, shielding, and cargo. DNDO
officials stated at the meeting that further testing could be
scheduled, if necessary, to fully satisfy DOE concerns.
On July 20, 2007, DHS Secretary Chertoff notified certain members of
the Congress that he planned to convene an independent expert panel to
review DNDO's test procedures, test results, associated technology
assessments, and cost-benefit analyses to support the final decision to
deploy ASPs. In making this announcement, Secretary Chertoff noted the
national importance of developing highly effective radiation detection
and identification capabilities as one of the main reasons for seeking
an independent review of DNDO's actions. On August 30, 2007, the DHS
Undersecretary for Management recommended that the Secretary of
Homeland Security delay Secretarial Certification of ASPs for an
additional two months. According to DHS, the current delay is in order
to provide CBP more time to field ASP systems, a concern CBP had raised
early in our review.
Conclusions:
Effectively detecting and identifying radiological or nuclear threats
at U.S. borders and ports of entry is a vital matter of national
security, and developing new and advanced technology is critical to
U.S. efforts to prevent a potential attack. However, it is also
critical to fully understand the strengths and weaknesses of any next
generation radiation detection technology before it is deployed in the
field and to know, to the greatest extent possible, when or how that
equipment may fail.
In our view, the tests conducted by DNDO at the Nevada Test Site
between February and March 2007 used biased test methods and were not
an objective assessment of the ASPs' performance capabilities. We
believe that DNDO's test methods--specifically, conducting dry runs and
dress rehearsals with contractors prior to formal testing--enhanced the
performance of the ASPs beyond what they are likely to achieve in
actual use. Furthermore, the tests were not a rigorous evaluation of
the ASPs' capabilities, but rather a developmental demonstration of ASP
performance under controlled conditions which did not test the
limitations of the ASP systems.
As a result of DNDO's test methods and the limits of the tests--
including a need to meet a secretarial certification deadline and the
limited configurations of special nuclear material sources, masking,
and shielding materials used--we believe that the results of the tests
conducted at the Nevada Test Site do not demonstrate a "significant
increase in operational effectiveness" relative to the current
detection system, and cannot be relied upon to make a full-scale
production decision.
Recommendations:
We recommend that the Secretary of Homeland Security take the following
actions:
* Delay Secretarial Certification and full-scale production decisions
of the ASPs until all relevant tests and studies have been completed
and limitations to these tests and studies have been identified and
addressed. Furthermore, results of these tests and studies should be
validated and made fully transparent to DOE, CBP, and other relevant
parties.
* Once the tests and studies have been completed, evaluated, and
validated, DHS should determine in cooperation with CBP, DOE, and other
stakeholders including independent reviewers, if additional testing is
needed.
* If additional testing is needed, the Secretary should appoint an
independent group within DHS, not aligned with the ASP acquisition
process, to conduct objective, comprehensive, and transparent testing
that realistically demonstrates the capabilities and limitations of the
ASP system. This independent group would be separate from the recently
appointed independent review panel.
* Finally, the results of the tests and analyses should be reported to
the appropriate congressional committees before large scale purchases
of ASP's are made.
Mr. Chairman, this concludes our prepared statement. We would be happy
to respond to any questions you or other members of the subcommittee
may have.
GAO Contact and Staff Acknowledgements:
For further information about this testimony, please contact me, Gene
Aloise, at (202) 512-3841 or at aloisee@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this statement. Erika D. Carter, Alison O'Neill, Jim
Shafer, Daren Sweeney, and Eugene Wisnoski made key contributions to
this statement.
Related GAO Products:
Combating Nuclear Smuggling: DHS's Decision to Procure and Deploy the
Next Generation of Radiation Detection Equipment Is Not Supported by
Its Cost-Benefit Analysis. GAO-07-581T. Washington, D.C.: March.14,
2007.
Nuclear Nonproliferation: Focusing on the highest Priority Radiological
Sources Could Improve DOE's Efforts to Secure Sources in Foreign
Countries. GAO-07-580T. Washington, D.C.: March. 13, 2007.
Combating Nuclear Smuggling: DNDO Has Not Yet Collected Most of the
National Laboratories' Test Results on Radiation Portal Monitors in
Support of DNDO's Testing and Development Program. GAO-07-347R.
Washington, D.C.: March 9, 2007.
Technology Assessment: Securing the Transport of Cargo Containers. GAO-
06-68SU. Washington, D.C.: January 25, 2006.
Combating Nuclear Smuggling: DHS's Cost-Benefit Analysis to Support the
Purchase of New Radiation Detection Portal Monitors Was Not Based on
Available Performance Data and Did Not Fully Evaluate All the Monitors'
Costs and Benefits. GAO-07-133R. Washington, D.C.: October 17, 2006.
Combating Nuclear Terrorism: Federal Efforts to Respond to Nuclear and
Radiological Threats and to Protect Emergency Response Capabilities
Could Be Strengthened. GAO-06-1015. Washington, D.C.: September 21,
2006.
Border Security: Investigators Transported Radioactive Sources Across
Our Nation's Borders at Two Locations. GAO-06-940T. Washington, D.C.:
July 7, 2006.
Combating Nuclear Smuggling: Challenges Facing U.S. Efforts to Deploy
Radiation Detection Equipment in Other Countries and in the United
States. GAO-06-558T. Washington, D.C.: March 28, 2006.
Combating Nuclear Smuggling: DHS Has Made Progress Deploying Radiation
Detection Equipment at U.S. Ports-of-Entry, but Concerns Remain. GAO-
06-389. Washington, D.C.: March 22, 2006.
Combating Nuclear Smuggling: Corruption, Maintenance, and Coordination
Problems Challenge U.S. Efforts to Provide Radiation Detection
Equipment to Other Countries. GAO-06-311. Washington, D.C.: March 14,
2006.
Combating Nuclear Smuggling: Efforts to Deploy Radiation Detection
Equipment in the United States and in Other Countries. GAO-05-840T.
Washington, D.C.: June 21, 2005.
Preventing Nuclear Smuggling: DOE Has Made Limited Progress in
Installing Radiation Detection Equipment at Highest Priority Foreign
Seaports. GAO-05-375. Washington, D.C.: March 31, 2005.
Homeland Security: DHS Needs a Strategy to Use DOE's Laboratories for
Research on Nuclear, Biological, and Chemical Detection and Response
Technologies. GAO-04-653. Washington, D.C.: May 24, 2004.
Homeland Security: Summary of Challenges Faced in Targeting Oceangoing
Cargo Containers for Inspection. GAO-04-557T. Washington, D.C.: March
31, 2004).
Homeland Security: Preliminary Observations on Efforts to Target
Security Inspections of Cargo Containers. GAO-04-325T. Washington,
D.C.: December 16, 2003.
Homeland Security: Radiation Detection Equipment at U.S. Ports of
Entry. GAO-03-1153TNI. Washington, D.C.: September 30, 2003.
Homeland Security: Limited Progress in Deploying Radiation Detection
Equipment at U.S. Ports of Entry. GAO-03-963. Washington, D.C.:
September 4, 2003).
Container Security: Current Efforts to Detect Nuclear Materials, New
Initiatives, and Challenges. GAO-03-297T. Washington, D.C.: November
18, 2002.
Customs Service: Acquisition and Deployment of Radiation Detection
Equipment. GAO-03-235T. Washington, D.C.: October 17, 2002.
Nuclear Nonproliferation: U.S. Efforts to Combat Nuclear Smuggling. GAO-
02-989T. Washington, D.C.: July 30, 2002.
Nuclear Nonproliferation: U.S. Efforts to Help Other Countries Combat
Nuclear Smuggling Need Strengthened Coordination and Planning. GAO-02-
426. Washington, D.C.: May 16, 2002.
Footnotes:
[1] GAO, Combating Nuclear Smuggling: DHS Has Made Progress Deploying
Radiation Detection Equipment at U.S. Ports of Entry, but Concerns
Remain, GAO-06-389 (Washington, D.C.: Mar.22, 2006).
[2] DNDO, Cost Benefit Analysis for Next Generation Passive Radiation
Detection of Cargo at the Nation's Border Crossings, May 30, 2006.
[3] GAO, Combating Nuclear Smuggling: DHS's Cost-Benefit Analysis to
Support the Purchase of New Radiation Detection Portal Monitors Was Not
Based on Available Performance Data and Did Not Fully Evaluate All the
Monitors' Cost and Benefits, GAO-07-133R (Washington, D.C.: Oct. 17,
2006).
[4] DOE manages the largest laboratory system of its kind in the world.
The mission of DOE's 22 laboratories has evolved. Originally created to
design and build atomic weapons, these laboratories have since expanded
to conduct research in many disciplines--from high-energy physics to
advanced computing.
[5] Department of Homeland Security Appropriations Act for Fiscal Year
2007, Pub. L. No. 109-295, tit. IV, 120 Stat. 1355, 1376 (October 4,
2006).
[6] PNNL, Energy Window Ratio Algorithms For Plastic Scintillator
Portal Monitors: Development, Deployment and Performance, PNNL-16283
(Richland, WA: December, 2006).
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "Subscribe to Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Susan Becker, Acting Manager, Beckers@gao.gov (202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
