Combating Nuclear Smuggling
DHS's Phase 3 Test Report on Advanced Portal Monitors Does Not Fully Disclose the Limitations of the Test Results Gao ID: GAO-08-979 September 30, 2008The Department of Homeland Security's (DHS) Domestic Nuclear Detection Office (DNDO) is responsible for addressing the threat of nuclear smuggling. Radiation detection portal monitors are part of the U.S. defense against such threats. In 2007, Congress required that funds for new advanced spectroscopic portal (ASP) monitors could not be spent until the Secretary of DHS certified that these machines represented a significant increase in operational effectiveness over currently deployed portal monitors. In addition to other tests, DNDO conducted the Phase 3 tests on ASPs to identify areas in which the ASPs needed improvement. GAO was asked to assess (1) the degree to which the Phase 3 test report accurately depicts the test results and (2) the appropriateness of using the Phase 3 test results to determine whether ASPs represent a significant improvement over current radiation detection equipment. GAO also agreed to provide its observations on special tests conducted by Sandia National Laboratories (SNL).
Because the limitations of the Phase 3 test results are not appropriately stated in the Phase 3 test report, the report does not accurately depict the results from the tests and could potentially be misleading. In the Phase 3 tests, DNDO performed a limited number of test runs. Because of this, the test results provide little information about the actual performance capabilities of the ASPs. The report often presents each test result as a single value; but considering the limited number of test runs, the results would be more appropriately stated as a range of potential values. For example, the report narrative states in one instance that an ASP could identify a source material during a test 50 percent of the time. However, the narrative does not disclose that, given the limited number of test runs, DNDO can only estimate that the ASP would correctly identify the source from about 15 percent to about 85 percent of the time--a result that lacks the precision implied by DNDO's narrative. DNDO's reporting of the test results in this manner makes them appear more conclusive and precise than they really are. The purpose of the Phase 3 tests was to conduct a limited number of test runs in order to identify areas in which the ASP software needed improvement. While aspects of the Phase 3 report address this purpose, the preponderance of the report goes beyond the test's original purpose and makes comparisons of the performance of the ASPs with one another or with currently deployed portal monitors. In GAO's view, it is not appropriate to use the Phase 3 test report in determining whether the ASPs represent a significant improvement over currently deployed radiation equipment because the limited number of test runs do not support many of the comparisons of ASP performance made in the Phase 3 report. As the report shows, if an ASP can identify a source material every time during a test, but the test is run only five times, the only thing that can be inferred with a high level of statistical confidence is that the probability of identification is no less than about 60 percent. Although DNDO states in the Phase 3 test report that the results will be relevant to the Secretary's certification that the ASPs represent a significant increase in operational effectiveness, it does not clarify in what ways the results will be relevant. Furthermore, DNDO offers no explanation as to why it changed its view from the Phase 3 test plan, which states that these tests will not be used to support a certification decision. The goal of SNL's special tests was, among other things, to identify potential vulnerabilities in the ASPs by using different test scenarios from those that DNDO planned to use in other ASP tests. SNL concluded in its test report that the ASPs' software and hardware can be improved and that rigor could be added to DNDO's testing methods. Furthermore, the report acknowledges that (1) a specific objective of the testing at the Nevada Test Site was to refine and improve the ASP's performance and (2) the special tests were never intended to demonstrate conformity of the ASPs with specific performance requirements. In GAO's view, these statements appear to accurately describe the purpose, limitations, and results of the special tests.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-08-979, Combating Nuclear Smuggling: DHS's Phase 3 Test Report on Advanced Portal Monitors Does Not Fully Disclose the Limitations of the Test Results
This is the accessible text file for GAO report number GAO-08-979
entitled 'Combating Nuclear Smuggling: DHS's Phase 3 Test Report on
Advanced Portal Monitors Does Not Fully Disclose the Limitations of the
Test Results' which was released on October 30, 2008.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Committees:
United States Government Accountability Office:
GAO:
September 2008:
Combating Nuclear Smuggling:
DHS's Phase 3 Test Report on Advanced Portal Monitors Does Not Fully
Disclose the Limitations of the Test Results:
GAO-08-979:
GAO Highlights:
Highlights of GAO-08-979, a report to congressional committees.
Why GAO Did This Study:
The Department of Homeland Security‘s (DHS) Domestic Nuclear Detection
Office (DNDO) is responsible for addressing the threat of nuclear
smuggling. Radiation detection portal monitors are part of the U.S.
defense against such threats. In 2007, Congress required that funds for
new advanced spectroscopic portal (ASP) monitors could not be spent
until the Secretary of DHS certified that these machines represented a
significant increase in operational effectiveness over currently
deployed portal monitors. In addition to other tests, DNDO conducted
the Phase 3 tests on ASPs to identify areas in which the ASPs needed
improvement. GAO was asked to assess (1) the degree to which the Phase
3 test report accurately depicts the test results and (2) the
appropriateness of using the Phase 3 test results to determine whether
ASPs represent a significant improvement over current radiation
detection equipment. GAO also agreed to provide its observations on
special tests conducted by Sandia National Laboratories (SNL).
What GAO Found:
Because the limitations of the Phase 3 test results are not
appropriately stated in the Phase 3 test report, the report does not
accurately depict the results from the tests and could potentially be
misleading. In the Phase 3 tests, DNDO performed a limited number of
test runs. Because of this, the test results provide little information
about the actual performance capabilities of the ASPs. The report often
presents each test result as a single value; but considering the
limited number of test runs, the results would be more appropriately
stated as a range of potential values. For example, the report
narrative states in one instance that an ASP could identify a source
material during a test 50 percent of the time. However, the narrative
does not disclose that, given the limited number of test runs, DNDO can
only estimate that the ASP would correctly identify the source from
about 15 percent to about 85 percent of the time”a result that lacks
the precision implied by DNDO‘s narrative. DNDO‘s reporting of the test
results in this manner makes them appear more conclusive and precise
than they really are. The purpose of the Phase 3 tests was to conduct a
limited number of test runs in order to identify areas in which the ASP
software needed improvement. While aspects of the Phase 3 report
address this purpose, the preponderance of the report goes beyond the
test‘s original purpose and makes comparisons of the performance of the
ASPs with one another or with currently deployed portal monitors.
In GAO‘s view, it is not appropriate to use the Phase 3 test report in
determining whether the ASPs represent a significant improvement over
currently deployed radiation equipment because the limited number of
test runs do not support many of the comparisons of ASP performance
made in the Phase 3 report. As the report shows, if an ASP can identify
a source material every time during a test, but the test is run only
five times, the only thing that can be inferred with a high level of
statistical confidence is that the probability of identification is no
less than about 60 percent. Although DNDO states in the Phase 3 test
report that the results will be relevant to the Secretary‘s
certification that the ASPs represent a significant increase in
operational effectiveness, it does not clarify in what ways the results
will be relevant. Furthermore, DNDO offers no explanation as to why it
changed its view from the Phase 3 test plan, which states that these
tests will not be used to support a certification decision.
The goal of SNL‘s special tests was, among other things, to identify
potential vulnerabilities in the ASPs by using different test scenarios
from those that DNDO planned to use in other ASP tests. SNL concluded
in its test report that the ASPs‘ software and hardware can be improved
and that rigor could be added to DNDO‘s testing methods. Furthermore,
the report acknowledges that (1) a specific objective of the testing at
the Nevada Test Site was to refine and improve the ASP‘s performance
and (2) the special tests were never intended to demonstrate conformity
of the ASPs with specific performance requirements. In GAO‘s view,
these statements appear to accurately describe the purpose,
limitations, and results of the special tests.
What GAO Recommends:
GAO‘s recommendations include proposing that the Secretary of DHS
revise the Phase 3 report to better disclose test results and
limitations if it is to be used in any certification decision for ASP
acquisition. DHS disagreed with two of GAO‘s recommendations but agreed
to take action on a third. GAO continues to believe that all of its
recommendations need to be implemented.
To view the full product, including the scope and methodology, click on
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-979]. For more
information, contact Gene Aloise at 202-512-3841 or aloisee@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
DNDO's Phase 3 Test Report Frequently Overlooks the Limitations
Associated with the Tests' Small Sample Sizes:
Phase 3 Test Results Provide Little Evidence as to whether ASPs
Represent an Improvement over Currently Deployed Technology:
SNL's Special Tests were Designed to Improve ASP Performance:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Comments from the Department of Homeland Security:
Appendix II: GAO Contact and Staff Acknowledgments:
Abbreviations:
ASP: advanced spectroscopic portal:
CBP: Customs and Border Protection:
DHS: Department of Homeland Security:
DNDO: Domestic Nuclear Detection Office:
PVT: polyvinyl toluene:
RIID: radioactive isotope identification device:
SNL: Sandia National Laboratories:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
September 30, 2008:
The Honorable John D. Dingell:
Chairman:
The Honorable Joe Barton:
Ranking Member:
Committee on Energy and Commerce:
House of Representatives:
The Honorable Bart Stupak:
Chairman:
The Honorable John Shimkus:
Ranking Member:
Subcommittee on Oversight and Investigations:
Committee on Energy and Commerce:
House of Representatives:
Preventing a nuclear weapon or radiological dispersal device (a "dirty
bomb") from being smuggled into the United States is a key national
security priority. The Department of Homeland Security (DHS), through
its Domestic Nuclear Detection Office (DNDO), has lead responsibility
for conducting the research, development, testing, and evaluation of
equipment that can be used to detect smuggled nuclear or radiological
materials. U.S. Customs and Border Protection (CBP) is responsible for
screening cargo as it enters the nation at our borders, including
operating radiation detection equipment to intercept dangerous nuclear
and radiological materials.
Much of DNDO's work on radiation detection equipment has focused on the
development and use of radiation detection portal monitors that can
screen vehicles, people, and cargo entering the United States. In the
case of cargo, these portal monitors typically include two detector
panels that form sidewalls of a portal through which trailer trucks
carrying cargo containers must pass. Currently, CBP employs portal
monitors made of polyvinyl toluene (a plastic), known as PVTs, which
can detect the presence of radiation but cannot identify the specific
radiological material generating the radiation. As a result, PVTs
cannot distinguish between benign, naturally occurring radiological
materials such as ceramic tile, and dangerous materials such as highly
enriched uranium. Therefore, if a PVT detects the presence of radiation
in a shipping container during a primary inspection, CBP conducts a
second inspection with another PVT and uses a handheld radioactive
isotope identification device (RIID) to identify the type of source
material emitting radiation. However, RIIDs use detectors that are
relatively small and, as a result, are limited in their ability to
correctly identify radiological and nuclear source materials, so CBP
officials sometimes must consult with scientists at CBP's Laboratories
and Scientific Services or physically search a container to identify
the radiation source. Nonetheless, CBP officials have stated that the
current system of radiation detection equipment and standard operating
procedures provide the best possible radiological and nuclear screening
coverage available with current technology and that it does not have a
significant adverse impact on the flow of commerce.
In 2005, in an effort to overcome the technical limitations of PVTs and
RIIDs, DNDO sponsored research, development, and testing intended to
produce Advanced Spectroscopic Portal (ASP) monitors, which are
designed to both detect the radiological or nuclear source material and
identify the specific type of source material. According to DNDO, ASPs
will reduce the number of false positives or nuisance alarms--instances
in which a portal monitor detects the presence of radiation and signals
an alarm, but the source material turns out to be benign.
In July 2006, DNDO awarded contracts worth potentially $1.2 billion to
three vendors to further develop and produce ASPs over five years.
Shortly thereafter, Congress required in DHS's fiscal year 2007
appropriation that the Secretary of Homeland Security certify that ASPs
represent "a significant increase in operational effectiveness" before
DNDO could order full-scale production. Congress enacted a similar
requirement in DHS's fiscal year 2008 appropriation.
In early 2007, DNDO conducted formal tests on ASPs in two phases at the
Nevada Test Site. The first, called Phase 1, was designed to assess the
ASP's performance capabilities in order to support a full-scale
production decision with a high degree of statistical confidence. DNDO
told us on multiple occasions and in a written response that only data
collected during Phase 1 would be included in the final report
presented to the Secretary to request ASP certification. According to
DNDO, the second group of tests, called Phase 3, provided data for
algorithm (software) improvement that targeted specific and known areas
in need of work and data to aid in the development of secondary
screening operations and procedures.[Footnote 1] For example, the Phase
3 tests attempt to determine, among other things, how ASP performance
is affected by the presence of various substances--known as shielding
materials--that partially absorb or alter the radiation emitted by the
source material in a container. During the Phase 3 tests, DNDO also
tested the ASPs to determine how their performance changes when the
container moves through the portal monitor at different speeds.
The size of the samples used in the Phase 1 and Phase 3 tests are
important in determining the confidence one can place in the test
results.[Footnote 2] Larger sample sizes, such as the 15 to 60 test
runs performed during the Phase 1 tests, usually allow a more precise
interpretation of results, i.e., estimates of ASP performance may fall
within a fairly narrow range of values. Conversely, estimates drawn
from small sample sizes, such as the 1 to 10 test runs done for Phase
3, normally have much less precision associated with them--thus the
range of potential values may be quite wide. According to DNDO's Phase
3 test plan, the Phase 3 testing consisted of small sample sizes to
allow more time to test a wide range of source materials in order to
make improvements to the ASPs. However, the small samples associated
with the Phase 3 tests would make it difficult to use the test results
as a reliable indicator of the ASPs' performance capabilities or for
comparisons of performance among various detection systems. In
contrast, the Phase 1 tests involved larger sample sizes so that DNDO
could assess the performance capabilities of the ASPs with a higher
degree of statistical precision. Because of the small sample sizes, the
Phase 3 test plan stated that Phase 3 testing would not be used to
support a full-scale production decision.
In September 2007, we testified that, in our view, DNDO's Phase 1 tests
did not represent an objective or rigorous assessment of the ASPs. More
specifically, we stated that DNDO used biased test methods that
enhanced the apparent performance of the ASPs and did not identify the
limitations of the ASPs' detection capabilities.[Footnote 3] During
that hearing, DNDO's Director changed DNDO's position and stated that
the Secretary could use the Phase 3 tests in combination with other
test results when deciding whether to certify the ASPs.
In its report on the Phase 3 tests, DNDO states that:
* The ASPs were as good as or better than the PVTs at detecting the
presence of radiological source materials at low levels of radiological
activity.
* The performance of the ASPs from each of the three vendors was
statistically indistinguishable with few exceptions, for each category
of source material (i.e., medical sources, industrial sources, and
special nuclear material).
* The performance of the RIIDs was poor compared to the performance of
the ASPs in identifying the specific radiological or nuclear source
material inside a container.
At the same time the Phase 1 and Phase 3 tests were ongoing, Sandia
National Laboratories (SNL), at the request of DNDO, conducted a series
of tests that would go beyond those covered in the Phase 1 and Phase 3
tests. The goal of these tests, called special tests, was, among other
things, to identify potential vulnerabilities in the ASPs by using
source materials and test scenarios different from those that DNDO
planned to use in either the Phase 1 or the Phase 3 tests. The tests
were "blind" in that neither the ASP vendors nor the ASP test operators
knew what was in the containers being tested. The review was also to
focus on vulnerabilities in the test processes DNDO described in its
Phase 1 and Phase 3 test plans.
Given DNDO's change in how it believes the Phase 3 test results may be
applied, the significant costs of the ASPs, and the importance of
protecting our borders from nuclear smuggling you asked us to assess
(1) the degree to which the Phase 3 test report accurately depicts the
test results and (2) the appropriateness of using the Phase 3 test
results to determine whether ASPs represent a significant improvement
over current radiation detection equipment. We also agreed to provide
our observations on the special tests conducted by SNL.
To perform our work, we reviewed the Phase 3 test report and SNL's
special tests report. We met with key officials from the National
Institutes of Standards and Technology who were responsible for
designing part of the Phase 3 tests and analyzing their results. We
also relied on documents and other evidence gathered during our
previous review of ASP testing at the Nevada Test Site. We conducted
this performance audit from February 2008 to August 2008 in accordance
with generally accepted government auditing standards. Those standards
require that we plan and perform the audit to obtain sufficient,
appropriate evidence to provide a reasonable basis for our findings and
conclusions based on our audit objectives. We believe that the evidence
obtained provides a reasonable basis for our findings and conclusions
based on our audit objectives.
Results in Brief:
Because the limitations of the Phase 3 test results are not
appropriately stated in the Phase 3 test report, the report does not
accurately depict the results from the tests and could be misleading.
In the Phase 3 tests, DNDO performed a limited number of test runs.
Because of this, the test results provide little information about the
actual performance capabilities of the ASPs. The report narrative often
presents test results using a single value or percentage. Considering
the limited number of test runs, the results would be more
appropriately stated as a range of potential values. This important
limitation is apparent only by reviewing more technical data elsewhere
in the report. For example, the report narrative states in one instance
that an ASP could identify a source material during a specific test 50
percent of the time. However, the narrative does not disclose that,
given the limited number of test runs, DNDO can only estimate that the
ASP would be able to correctly identify the source from about 15
percent to about 85 percent of the time--a result that lacks the
precision implied by DNDO's narrative. DNDO's reporting of the test
results in this manner makes the results appear more conclusive and
precise than they really are. For example, the executive summary states
that the ASPs "demonstrated detection limits equal to or better than
those of any of the PVT systems," but fails to mention that the test
used only a single source material and that the results would not
necessarily be the same for other radiological sources. In fact, the
Phase 3 results showed that the PVTs could detect some source materials
much better than the ASPs. The purpose of the Phase 3 tests was to
identify areas in which the ASP software needed improvement. While
aspects of the Phase 3 report address this purpose, the preponderance
of the report goes beyond the test's original purpose and makes
comparisons of the performance of the ASPs with each other or with PVTs
and RIIDs.
In our view, it is not appropriate to use the Phase 3 test report in
determining whether the ASPs represent a significant improvement over
currently deployed radiation equipment because the limited number of
test runs does not support many of the comparisons of ASP performance
made in the Phase 3 report. As the report shows, if an ASP can identify
a source material every time during a test, but the test is run only
five times, the only thing that can be inferred at the 9 percent
confidence level is that the true probability of identification is no
less than approximately 60 percent. Although DNDO states in the Phase 3
test report that the test results will be relevant to the Secretary's
certification that the ASPs represent a significant increase in
operational effectiveness, the report does not clarify in what ways the
results will be relevant. Furthermore, DNDO offers no explanation as to
why it changed its view from the Phase 3 test plan which states that
the Phase 3 tests will not be used to support a certification decision.
Regarding SNL's special tests, SNL concluded in its test report that
the ASPs' software and hardware can be improved in some areas and that
rigor could be added to DNDO's testing methods. Furthermore, the report
acknowledges that (1) a specific objective of the testing at the Nevada
Test Site was to refine and improve the ASP software performance and
(2) the special tests "were never intended to demonstrate conformity of
the [ASP] systems against specific performance requirements." In our
view, these statements appear to accurately describe the purpose,
limitations, and results of the special tests. In addition, the report
concluded that upon reviewing data from DNDO's previous ASP tests, the
reported results were consistent with the data that DNDO collected and
that as a result, "DNDO's ASP system assessment was not biased." It is
important to note, however, SNL's report also concluded that the "ASP
system assessment [in 2007] was not biased" and that SNL "observed no
data suggesting that the ASP system performance was inappropriately
manipulated." In making this statement, SNL is referring to the data
derived from ASP tests. In contrast, when we stated in September 2007
that DNDO's Phase 1 tests were biased, we were referring to the test
methods DNDO used, such as (1) using the same test sources and
shielding materials during preliminary runs as were used during the
actual tests, and (2) not using standard CBP operating procedures in
testing the RIIDs.
We are recommending that the Secretary of DHS use the results of the
Phase 3 tests solely for the purposes for which they were intended--to
identify areas needing improvement, not as a justification for
certifying whether the ASPs represent a significant increase in
operational effectiveness. However, if the Secretary of DHS intends to
consider the results of the Phase 3 tests, along with other test data,
in making a certification decision regarding ASPs, then we also
recommend that the Secretary (1) direct the Director of DNDO to revise
and clarify the Phase 3 test report to more fully disclose and
articulate the limitations present in the Phase 3 tests--particularly
the limitations associated with making comparisons between detection
systems from a small number of test runs---and (2) clearly state which
"relevant insights into important aspects of system performance" from
the Phase 3 report are factored into any decision regarding the
certification that ASPs demonstrate a significant increase in
operational effectiveness. Finally, we further recommend that since
there are several phases of additional ASP testing currently ongoing,
the Secretary should direct the Director of DNDO take steps to ensure
that any limitations associated with ongoing testing are properly
disclosed when the results of the current testing are reported.
We provided DHS with a draft of this report for its review and comment.
The department stated that it strongly disagreed with our draft report
and two of our recommendations, it agreed to take some action on a
third recommendation, and offered no comments on a fourth
recommendation. In its comments, DHS cites narrative from the Phase 3
report explaining that the Phase 3 tests employed fewer test runs per
test so as "to allow for more substantial variation among test cases"
rather than "running sufficient number of repetitions— to provide high
statistical significance results." Thus, in DHS's view, our assertion
that the report does not "fully disclose" the Phase 3 test's
limitations concerning the statistical significance of the results is
incorrect. Our draft report recognizes DHS's description of how the
Phase 3 tests were conducted. Our concern is that although DNDO cited
the limited statistical significance of the test results at the outset
of the Phase 3 report, it does not clearly state this limitation in
expressing the test's findings. For example, as we note in our draft
report, the Phase 3 report repeatedly states that the performances of
the various ASPs were "statistically indistinguishable" even though
DNDO did not perform enough test runs to estimate with a high degree of
confidence whether the performances were actually similar. DNDO
presents many of its findings as conclusive statements about ASP
performance despite the fact that the Phase 3 test design cannot
support these findings. DHS had additional comments, which are
discussed at the end of this letter.
Background:
In the summer of 2005, DNDO tested ASPs from 10 vendors to evaluate
their performance capabilities and to select the ASPs that warranted
further development and possible procurement. In July 2006, DNDO
awarded contracts totaling $1.2 billion over five years to three
vendors--Raytheon, Canberra, and Thermo.
The Department of Homeland Security Appropriations Act for Fiscal Year
2007 states that "none of the funds appropriated — shall be obligated
for full scale procurement of [ASP] monitors until the Secretary of
Homeland Security has certified — that a significant increase in
operational effectiveness will be achieved." Congress enacted a similar
requirement in DHS's fiscal year 2008 appropriation. In hopes of
obtaining secretarial certification by June 2007, DNDO tested ASPs at
several sites, including the Nevada Test Site, the New York Container
Terminal, the Pacific Northwest National Laboratory, and five ports of
entry. DNDO conducted the tests at NTS in two phases. DNDO stated that
the Phase 1 tests, performed in February-March 2007, attempted to
estimate the performance capabilities of the ASPs with a high degree of
statistical confidence. DNDO intended these tests to support the
Secretary's decision on whether to certify the ASPs for the purposes of
a full-scale production decision, while the Phase 3 tests were intended
to help improve the computer algorithms that the ASPs use to identify
the specific radiological or nuclear source inside a container.
On September 18, 2007, we testified that DNDO's Phase 1 tests did not
constitute an objective and rigorous assessment of the ASPs'
capabilities because, among other things, DNDO conducted preliminary
test runs on source materials to be used in the tests, and then allowed
the vendors to adjust their ASPs to specifically identify the source
materials to be tested. We testified that in our view, DNDO's approach
biased the tests in ways that enhanced the apparent performance of the
ASPs. We also noted that the tests did not attempt to estimate the
limits of ASPs' detection abilities--an important concern to those who
will use them such as CBP officers. During that hearing, DNDO's
Director stated that, contrary to statements DNDO made in its final
Phase 3 test plan, DNDO would use the Phase 3 test results to help
support the Secretary's decision on whether to certify the ASPs for
full-scale production. Subsequently, DNDO delayed its anticipated date
for secretarial certification to the fall of 2008 in order to conduct
additional performance tests and field tests during fiscal year 2008.
DNDO's Phase 3 Test Report Frequently Overlooks the Limitations
Associated with the Tests' Small Sample Sizes:
Because the limitations of the Phase 3 test results are not properly
discussed in the Phase 3 test report, the report does not accurately
portray the results from the Phase 3 tests and could be misleading. The
purpose of the Phase 3 tests was to identify areas in which the ASPs
needed improvement. While some of the Phase 3 report addresses this
purpose, much of the report compares the performance of the ASPs with
each other or with PVTs and RIIDs during the tests. However, because
DNDO performed each test a limited number of times, the data it used to
make some of these comparisons provide little information about the
actual capabilities of the ASPs. The narrative of the report often
presents each test result as a single value, although, because of the
limited number of test runs, the results would be more thoroughly and
appropriately stated as a range of potential values. In addition, the
report's narrative sometimes omits key facts that conflict with DNDO's
analysis of the results.
The Phase 3 Test Report Largely Overlooks the Limiting Effects of
Performing a Small Number of Tests:
The purpose of the Phase 3 tests was to provide data that would help
further develop and improve the ASPs' identification software and data
to aid in the development of secondary screening operations and
procedures. DNDO acknowledged early in the test report that the Phase 3
tests did not involve enough test runs to assess the performance of the
ASPs with a high degree of statistical confidence:
"The primary goals of the testing were to provide information by giving
the ASP systems an opportunity to perform against a wider range of
radionuclides, shielding, and cargo configurations. To allow for more
substantial variation among test cases, the test plan called for a
smaller number of trials over a larger range of objects and
configurations rather than running sufficient number of repetitions for
each individual test case to provide higher statistically significan[t]
results." (p. 2):
DNDO also acknowledged early in the Phase 3 report that, given the
small number of test runs, it would be difficult to compare the
performance of the ASPs with each other:
In these comparisons [of the performances of different ASP systems],
results are often indistinguishable [i.e., not statistically
significantly different] because the small sample sizes induce large
uncertainties in the estimates of the probabilities being compared [for
example: n 5]." [Footnote 4](p.9):
Nonetheless, while some of the Phase 3 report addresses the stated
purpose of the tests, the preponderance of the report compares the
performance of the ASPs with each other or with PVTs or RIIDs during
the tests, as shown in the following examples:
"For [category of source material] at 2 mph, the ASP system
performances are statistically indistinguishable." (p.13):
"For shielded [category of source material], performance for all three
systems is statistically indistinguishable with probabilities of
correct alarm varying approximately between 0.84 and .0.92." (p.11):
The statements imply that the performances of the ASPs were similar
because the results were "statistically indistinguishable." However,
given the small number of test runs, it is impossible to determine with
a high degree of confidence whether or not the performances were
actually similar. Yet the report's text describing specific results
rarely qualifies the results by stating that the test was run only a
few times or that the results should not be considered conclusive of
the ASPs' capabilities.
Similarly, the report's executive summary presents the test results as
conclusions about the performance capabilities of the ASPs, PVTs, and
RIIDs:
"For the source configurations tested, the ASP systems have equal
performance down to the lowest source activity tested." (p. iii):
"The PVT systems display lower performance than the ASP systems for
[category] and [category] sources." (p. iv):
"When comparing the ASP systems _ mph identification metric with the _
RIID measurements—, it is observed that the RIID performance is poor
compared to the ASP systems." (p. iv)[Footnote 5]
Report Text Often Omits the Range of Values Surrounding Each Test
Result:
The Phase 3 test report makes some of its performance comparisons by
citing the percentage of correct detections or identifications that an
ASP made on a test. For example:
"For bare [category of source material] only, — [T]he probability of
correct identification [for the 3 ASPs] varied between 0.34 and 0.5."
(p.14)[Footnote 6]
However, because each test involved a small number of test runs, these
percentages provide little information about the performance
capabilities of the ASPs. In fact, because of the small number of test
runs, DNDO can only estimate that each ASP can correctly identify the
type of source material within a range of values.[Footnote 7] The fewer
the number of test runs, the larger the range. For example, for the ASP
that correctly identified the source material 34 percent of the time
during the tests, the report text omits the fact that, as shown on an
accompanying graph, DNDO can only estimate that the ASP would be able
to correctly identify the source between about 10 percent and 65
percent of the time. By stating that the ASP identified the source 34
percent of the time without clarifying that the results came from only
a few test runs, the report's text makes the test results seem more
precise than they really are. Similarly, for the ASP that correctly
identified the source material 50 percent of the time during the tests,
the small number of test runs means that DNDO can only estimate that
the ASP would be able to correctly identify the source material between
about 15 percent and 85 percent of the time. This range is too wide to
have much value in determining how well the ASP may perform in the real
world. Although these ranges are clearly shown on the report's graphs,
they are omitted in the report's descriptions and interpretations of
the test results.
Similarly, DNDO's analysis comparing the performances of ASPs and RIIDs
fails to consider the uncertainties created by the tests' small sample
sizes. The report states that the RIIDs "performance is poor compared
to the ASP systems." For example, during the tests, one vendor's ASP
correctly identified one type of source material about 50 percent of
the time, while the RIIDs correctly identified the same type of source
material about 10 percent of the time. However, given the small number
of test runs, DNDO cannot be confident that these percentages precisely
indicate the performance capabilities of the ASPs and RIIDs. On the
basis of the tests, DNDO can only infer that the ASPs' and RIIDs'
performance capabilities lie somewhere within a relatively large range
of values. As these ranges are illustrated in the report's graphs, it
appears that the difference in the performance of the ASPs and RIIDs
may not be statistically different for three of the five types of
source materials DNDO tested. This does not necessarily mean that the
ASPs and RIIDs performed equally well; rather, DNDO did not conduct
each test enough times to determine that the superior performance of
the ASPs over the RIIDs reflects the capabilities of the ASPs rather
than mere chance.
DNDO's Phase 3 Test Report Omits Important Details that Affect the
Interpretation of the Test Results:
In a few instances, the report omits important details concerning
DNDO's interpretation of the results. For example, DNDO seems to assert
in the report's executive summary that the ASPs are as good as the PVTs
at detecting radiological or nuclear source materials:
"The ASP systems demonstrated detection limits equal to or better than
those of any of the PVT systems as configured during testing." (p.iii):
However, the report's executive summary fails to note that because DNDO
used only one type of source material, the results are largely specific
to that particular source material and would not necessarily apply to
other specific source materials. In fact, for other types of source
material, the report shows several instances in which the PVTs were
apparently able to detect other types of source materials better than
the ASPs. Moreover, other Phase 3 tests showed that simply moving the
source material referred to in the above quote to another place in the
container affected the relative performances of the ASPs and PVTs.
Similarly, in reporting how well the ASPs performed when the radiation
from the source material was partially blocked by a shielding material,
DNDO stated:
"the ASP systems have the ability to identify sources when placed
inside almost all but the thickest shielding configuration tested."
(p.iv):
Again, however, DNDO fails to note in its report that, as it explained
in its Phase 3 test plan, all the shielding used in the Phase 3 tests
represented "light shielding." The report also fails to state how many
specific sources the ASPs could correctly identify or how frequently
the ASPs could identify them.
Phase 3 Test Results Provide Little Evidence as To Whether ASPs
Represent an Improvement Over Currently Deployed Technology:
In our view, it is not appropriate to use the Phase 3 test report in
determining whether the ASPs represent a significant improvement over
currently deployed radiation equipment because the limited number of
test runs does not support many of the comparisons of ASP performance
made in the Phase 3 report. As noted, DNDO's use of a small number of
runs for each test means that DNDO can only be certain that the ASP can
correctly identify or detect a source material over a broad range of
possible values rather than at a specific rate. This is true even if
the ASP was successful every time a test was conducted. For example, as
noted in the Phase 3 test report, if the ASP correctly identified a
source material 100 percent of the time, but the test was run only five
times, the most DNDO can estimate is that the ASP should be able to
correctly identify the source no worse than about 60 percent of the
time.
The Phase 3 test results do not help to determine an ASP's "true" level
of performance because DNDO did not design the tests to assess ASP
performance with a high degree of statistical confidence. In the Phase
3 test plan, DNDO was very clear that it had intended the tests to help
develop a conduct of operations for secondary screenings and to cover a
larger array of source materials and test scenarios than were conducted
in the Phase 1 tests.
DNDO also originally stated that the Phase 3 tests would not be used
for secretarial certification that the ASPs represented a "significant
operational improvement" over currently deployed radiation detection
equipment. DNDO stated that it had designed the Phase 1 tests to
"evaluate the current state of performance of the ASP—systems."
However, prior to releasing the Phase 3 report, DNDO changed its
position, stating in the final Phase 3 test report that the test
results are relevant to secretarial certification:
"The Phase 3 test campaign was not originally intended to support the
Secretarial Certification of the ASP systems. However, the test results
provide relevant insights into important aspects of system performance
and should be taken into consideration by the Secretary of Homeland
Security in making his (ASP procurement) decision." (p.iii):
It is important to note that DNDO does not elaborate in the test report
as to what the "relevant insights" are or how they relate to
Secretarial certification. DNDO also does not explain why those
insights would be relevant considering that, as stated in the Phase 3
test plan, the results from the tests lack a high degree of statistical
significance. Finally, it should be noted that when the Director of
DNDO testified in September 2007 that the Phase 3 test results would
help inform the Secretary's recommendation, he also acknowledged that
the Phase 3 test report had not yet been prepared.
SNL's Special Tests Were Designed to Improve ASP Performance:
The special tests were performed by experts from Sandia National
Laboratories who were not part of the Phase 1 or Phase 3 tests. The
special tests were designed to examine potential vulnerabilities
associated with either the ASPs or the Phase 1 or Phase 3 test plan and
vulnerabilities in DNDO's test processes. Conducting this type of test
would allow the ASP vendors the opportunity to make improvements to
their systems in order to address weaknesses revealed during the
special tests. Like the Phase 3 tests, the special tests used a small
number of runs for each testing scenario. Because of the small number
of runs, the test results do not support estimating the probability of
detection or identification with a high confidence level making it
difficult to use the results of the special tests to support a
certification decision by the Secretary of DHS. On this point, the
special test report acknowledges that "the special tests were never
intended to demonstrate conformity of the [ASP] systems against
specific performance requirements."
From the special tests, SNL concluded:
1. "Areas for software and hardware improvement have been identified
based on system performance issues observed for the versions of the ASP
hardware and software tested at the NTS during Winter 2007."
2. "For the data made available to us, the reported results — are
consistent with the underlying collected data--indicating that the DNDO
ASP system assessment was not biased."
3. "Recommendations to improve the testing rigor have been made—(noting
that) their implementation must be balanced against other test campaign
impacts (such as) cost, schedule, availability of resources, etc.,"
and:
4. "Based on our limited tests we observed no data suggesting that the
ASP system performance was inappropriately manipulated by either the
vendors or the test team."
Overall, the special test report appears to accurately describe the
purpose, limitations, and results of the special tests. In our view,
DNDO should consider SNL's views as it proceeds with additional ASP
testing in 2008. It is important to note, however, in Sandia's
conclusions that the "ASP system assessment [in 2007] was not biased"
and that it "observed no data suggesting that the ASP system
performance was inappropriately manipulated," Sandia is referring to
the data derived from ASP tests. However, SNL does not comment on the
biased testing methods we identified during the Phase 1 ASP tests at
the Nevada Test Site in 2007. Specifically, when we stated in September
2007 that DNDO's Phase 1 tests were biased, we were referring to DNDO's
test methods which (1) used the same test sources and shielding
materials during preliminary runs as were used during the actual tests
and (2) did not use standard CBP operating procedures in testing the
RIIDs.
Conclusions:
Preventing the material for a nuclear weapon or a radiological
dispersal device from being smuggled into the United States remains a
key national security priority. Testing radiation detection equipment
to understand its capabilities and limitations is an important part of
preventing nuclear smuggling. The Phase 3 and special tests were part
of DNDO's 2007 effort to test ASPs in order to identify areas for
further development to these devices. The Phase 3 test results are
relevant to DNDO's original objective for the Phase 3 tests--to
identify areas in which the ASPs needed improvement. However, because
of the limitations of the tests, DNDO should not be using the test
results as indicators of the overall performance capabilities of the
ASPs. Moreover, in the Phase 3 report, DNDO presented and analyzed the
test results without fully disclosing key limitations of the tests,
which is not consistent with basic principles of statistics and data
analysis. Because of this, many of the report's presentations and
comparisons of performance among ASPs and between ASPs and PVTs are not
well supported and are potentially misleading. Regarding the special
tests, SNL notes in its test report that it designed the tests to
identify areas where the ASPs need to improve--not to measure the ASPs
performance against requirements. Overall, because of the limitations
discussed in this report, it is our view that neither the Phase 3 tests
nor the special tests should serve as a basis for the Secretary of DHS
whether the ASPs represent "a significant increase in operational
effectiveness" over current radiation detection equipment.
Recommendations for Executive Action:
To ensure that the limitations of the Phase 3 test results, and future
ASP test results, are clearly understood, we are making the following
four recommendations.
We recommend that the Secretary of DHS use the results of the Phase 3
tests solely for the purposes for which they were intended--to identify
areas needing improvement, not as a justification for certifying
whether the ASPs warrant full-scale production.
However, if the Secretary of DHS intends to consider the results of the
Phase 3 tests, along with other test data information, in making a
certification decision regarding ASPs, then we recommend that the
Secretary take the following actions:
* Direct the Director of DNDO to revise and clarify the Phase 3 test
report to more fully disclose and articulate the limitations present in
the Phase 3 tests--particularly the limitations associated with making
comparisons between detection systems from a small number of test runs.
* Clearly state which "relevant insights into important aspects of
system performance" from the Phase 3 report are factored into any
decision regarding the certification that ASPs demonstrate a
significant increase in operational effectiveness.
Finally, we further recommend that since there are several phases of
additional ASP testing currently ongoing, the Secretary should direct
the Director of DNDO take steps to ensure that any limitations
associated with ongoing testing are properly disclosed when the results
of the current testing are reported.
Comments from the Department of Homeland Security and Our Evaluation:
We provided DHS with a draft of this report for its review and comment.
Its written comments are presented in appendix I. The department stated
that it strongly disagreed with our draft report and two of our
report's recommendations. DHS agreed to take some action on a third
recommendation and offered no comments on a fourth recommendation. The
department stated several reasons for its disagreement. First, DHS
cites narrative from the Phase 3 report explaining that the Phase 3
tests employed fewer test runs per test so as "to allow for more
substantial variation among test cases" rather than "running sufficient
number of repetitions — to provide high statistical significance
results." Thus, in DHS's view, our assertion that the report does not
"fully disclose" the Phase 3 tests' limitations concerning the
statistical significance of the results is incorrect. Our draft report
recognizes DHS's description of how the Phase 3 tests were conducted.
Our concern is that although DNDO cited the limited statistical
significance of the test results at the outset of the Phase 3 report,
DNDO's findings do not reflect this limitation. For example, as we note
in our draft report, the Phase 3 report repeatedly states that the
performances of the various ASPs were "statistically indistinguishable"
even though DNDO did not perform enough test runs to estimate with a
high degree of confidence whether the performances were actually
similar. DNDO presents many of its findings as conclusive statements
about ASP performance despite the fact that the Phase 3 test design
cannot support these findings.
Second, the department commented that the Phase 3 test report clearly
and succinctly stated another limitation of the test methodology--
specifically, that the tests were not designed to be a precise
indicator of ASP performance. In the department's view, noting this
limitation throughout the Phase 3 report would have been unwieldy. We
did not expect DNDO to repeat this limitation throughout the report.
However, as suggested in our report, the Phase 3 report should
accurately reflect the test results without portraying the results as
being more precise than they really are. Using an example from the
Phase 3 report, if DNDO notes that an ASP successfully identified a
specific source material 34 percent of the time during the tests, it
should also indicate that, given the small number of test runs, DNDO
can only estimate that the ASP would be able to correctly identify the
specific source material between 10 and 65 percent of the time.
However, no such discussion of the wide range of potential results is
included in the report's narrative. In our view, presenting the test
results without sufficient narrative about the tests' limitations is
potentially misleading.
Third, the department stated that although the Phase 3 tests were not
intended to support the DHS Secretary's certification decision, DHS
decided that it needed to consider all available test results in making
this decision. DHS further commented that not doing so would subject it
to criticism of "cherry-picking" the results. In response, although we
acknowledge the need to consider all available test results, we believe
they should be considered in their appropriate context, and that test
results do not all carry the same weight. In our view, test results
with a high degree of statistical significance (i.e., unlikely to be
the result of chance) should be considered a better indicator of ASP
performance than those with a lower level of statistical significance.
Because the Phase 3 tests involved only 1-10 runs per test, very few of
the results can be generalized as reliable estimates of how the ASPs
perform and thus potentially provide questionable evidence for the
certification process. We also note that, in its comments, DHS did not
address what Phase 3 results or important insights it considered to be
relevant to Secretarial certification.
Fourth, DHS comments that our draft report failed "to acknowledge the
depth and breadth of the ASP test campaign, which is by far the most
comprehensive test campaign ever conducted on radiation detection
equipment." However, our report describes previous ASP testing and some
of our prior findings about that testing, and notes that ASP testing
continues in 2008. More importantly, the extent of testing is not the
issue at hand. In our view, regardless of how many tests are performed,
the tests must employ sound, unbiased methodologies and DNDO should
draw and present conclusions from the test results in ways that
accurately and fully reflect the data and disclose their limitations.
DHS stated that it disagreed with our recommendations to (1) use the
Phase 3 test's to identify areas needing improvement and not as a basis
for certification and (2) revise and clarify the Phase 3 report to
reflect the limitations in the tests' methodology and results. It did
not offer comments on our recommendation that the Secretary clearly
state what relevant insights from the Phase 3 report are factored into
any certification decision. We continue to believe that the Phase 3
tests should be used only for the intended purpose stated in its test
plan--to improve the software of ASPs. We would also note that our
draft report recommends that DNDO revise and clarify the Phase 3 test
report only if it includes Phase 3 test results among the data that
will be presented to the Secretary prior to his decision on
certification. If DNDO chooses to use the Phase 3 test results for
certification, we believe it is important that DNDO explain what test
results are relevant to certification and why the value of those
results are not mitigated by the limitations associated with the Phase
3 tests' small sample sizes.
In response to our last recommendation, the department stated that it
has taken and will continue to take steps to ensure that it properly
discloses any limitations associated with ongoing testing as it moves
toward secretarial certification of the ASPs.
As agreed with your offices, unless you publicly announce the contents
of this report, we plan no further distribution until 30 days from the
report date. At that time, we will send copies of this report to the
Secretary of DHS and interested congressional committees. We will also
make copies available to others upon request. In addition, this report
will be available at no charge on the GAO Web site at [hyperlink,
http://www.gao.gov].
If you or your staffs have any questions about this report, please
contact me at (202) 512-3841 or aloisee@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. GAO staff that made major contributions
to this report are listed in appendix II.
Gene Aloise Director, Natural Resources and Environment:
[End of section]
Appendix I: Comments from the Department of Homeland Security:
U.S. Department of Homeland Security:
Washington, DC 20528:
August 29, 2008:
Mr. Gene Aloise:
Director, Natural Resources and Environment:
Government Accountability Office:
441 G Street NW:
Washington, DC 20548:
Dear Mr. Aloise:
Re: Draft Report GAO-08-979, Combating Nuclear Smuggling: DHS's Phase 3
Test Report on Advanced Portal Monitors Does Not Fully Disclose the
Limitations of the Test Results.
The Department of Homeland Security strongly disagrees with the GAO
Draft Report. The title of the GAO report "DHS's Phase 3 Test Report on
Advanced Portal Monitors Does Not Fully Disclose the Limitations of the
Test Results" is misleading and not substantiated by the body of the
GAO report. First, the Test Purpose section of the test report clearly
states that "to allow for more substantial variation among test cases,
the test plan called for a smaller number of trials over a larger range
of objects and configurations rather than running sufficient number of
repetitions for each individual test case to provide high statistical
significance results."[Footnote 8] This statement applies to the entire
test report; therefore to say that DHS did not "fully disclose"
limitations that could be "potentially misleading" is false. Repeating
the same statement throughout the test report would be unnecessary and
tedious.
Second, DHS clearly acknowledges that the main purpose of the Phase 3
test was to provide information to improve the system algorithms.
[Footnote 9] The ASP Phase 3 test was designed to identify areas for
ASP development and improvement. The test was not intended to be a
precise indicator of ASP performance, nor does the test report ever
claim to draw such conclusions. The underlying test design purposely
involved test cases that were different from those against which ASP
performance is to be measured. The test report clearly and succinctly
stated the limitations associated with the test methodology and analysis
approach. Again, given the amount of material and density of the test
report, it would have been unwieldy to repeat caveats throughout the
report.
Third, even though the Phase 3 report was not designed to support the
certification decision or to substantiate performance of the systems
against the defined threat basis, DHS quickly recognized that it needed
to consider all available test results in making a certification
decision. For that reason, the Phase 3 Test report clearly states
"Although the Phase 3 test campaign was not originally intended to
support the secretarial certification of the ASP systems, the test
results provide relevant insights into important aspects of the system
performance and should be taken into consideration by the Secretary of
Homeland Security."[Footnote 10] Indeed, had we not done so, DHS would
now be accused of "cherry-picking" test results, or worse, ignoring
data.
Once again, by reporting on ASP testing in a piece-meal fashion, GAO
fails to acknowledge the depth and breadth of the ASP test campaign,
which is by far the most comprehensive test campaign ever conducted on
radiation detection equipment. The ASP Phase 3 test is but one in a
series of carefully-designed tests conducted over a period of years in
the path to secretarial certification and full-rate production. To
date, these campaigns have included a developmental set of testing for
the engineering developmental models in Winter 2007 and performance
testing at the Nevada. Test Site (NTS), deployment readiness testing at
the Pacific Northwest National Laboratory (PNNL), operational testing
at the New York Container Terminal (NYCT), and field validations at
multiple POEs conducted in Summer 2008.
In conjunction with the Department of Energy (DOE), DNDO is also
conducting Threat Injection Studies for ASP systems. These threat
injection studies will examine the limits of performance of ASP in
order to guide the setting of thresholds and concept of operations
(CONOPS) and also compare ASP and polyvinyl toluene (PVT) performance.
To date, DNDO has developed, integrated, and validated a set of tools
to perform the injection studies using a standard data format. Data
collection has also been underway to create threat-representative
signatures that can be injected into stream-of-commerce data. In
preparation for the studies, DNDO has worked to collect data to
validate injection methodology and prepare data set of approximately
8000 validated stream-of-commerce data files into which threat
signatures will be injected and also create threat-representative
signatures based on the collected data.
Additional 2008 ASP testing is currently underway, and includes: system
qualification testing (SQT) to verify compliance with the ASP
performance specification; integration testing at PNNL to verify that
ASP performance remains sound when it is integrated into the POE
architecture; performance testing at NTS to validate the detection and
identification capabilities of ASP systems against special nuclear
materials (SNM) and materials for radiological dispersal devices (RDD);
and operational test and evaluation activities to validate operational
performance of the system at POEs. The successful completion of these
steps will provide data for the Secretary's Certification decision.
DNDO will use a combination of cost-benefit analyses as well as
demonstrated performance metrics to assist in the Secretary's
certification decision. Part of the certification process will involve
working with the National Academy of Sciences, to review DNDO test
plans and procedures, as required in the FY 2008 Consolidated
Appropriations Act. DHS will also use the test results, along with
other information, to seek approval of the DHS Investment Review Board
(IRB) prior to proceeding to full-scale production and deployment of
ASP systems at POEs.
Based on the above, DHS believes that the Phase 3 Test Report more than
adequately discloses the purpose and limitations associated with the
Phase 3 test results, and therefore, disagrees with the GAO's
recommendation that the Phase 3 report needs to be revised and
clarified. DHS also believes that all data needs to be considered for
the secretarial certification decision, and therefore, disagrees with
the GAO's recommendation that the Phase 3 data not be considered. It is
neither prudent nor scientifically justifiable to pick the data one
chooses to use for making a decision. While some data may carry more
weight than others (particularly the most recent data for the newest
software version), no data should be ignored. DHS has taken, and will
continue to take, steps to ensure that any limitations associated with
ongoing testing are properly disclosed in the path to certification.
Thank you for the opportunity to review and provide comments to the
draft report.
Sincerely,
Signed by:
E. Levine:
Director, Departmental GAO/OIG Liaison Office:
[End of section]
Appendix II: GAO Contact and Staff Acknowledgments:
GAO Contact:
Gene Aloise, (202) 512-8051 or aloisee@gao.gov:
Staff Acknowledgements:
In addition to the contact named above, Ned Woodward, Assistant
Director; James Ashley, Nabajyoti Barkakati, Carol Kolarik, Omari
Norman, Alison O'Neill, Anna Maria Ortiz, Daren Sweeney, Michelle
Treistman, and Gene Wisnoski made significant contributions to this
report.
[End of section]
Footnotes:
[1] According to DNDO, Phase 2 was its completion of the test report
for the Phase 1 tests.
[2] Equipment testing involves repeating a single test multiple times
to estimate how often a device performs its function correctly. The set
of tests is referred to as a sample. Each test in the sample is
considered a random trial and therefore the estimates derived from the
sample are subject to random variations; in other words, if the series
of tests were repeated then each sample could yield different
estimates. Because of these random variations, estimates from samples
are often presented as a range of possible values called the 95 percent
confidence interval. This is the range of values that will contain the
true probability of performance in 95 percent of the samples that we
might select. In general, when the sample size (number of tests) is
larger, the range of possible values is smaller, allowing more precise
estimates of the likely performance of the machine.
[3] GAO, Combating Nuclear Smuggling, Additional Actions Needed to
Ensure Adequate Testing of Next Generation Radiation Detection
Equipment, GAO-07-1247T (Washington, D.C.: Sept. 18, 2007).
[4] The report does not present significance-of-differences tests with
its analyses.
[5] DNDO's Phase 3 report is classified. Because of this, some of the
quotes from the report are missing information in order to protect
sensitive information.
[6] DNDO analyzed this series of tests by source category (medical,
industrial, or special nuclear material) rather than by specific source
material or isotope. For ease of discussion, we refer to each category
as a source material.
[7] Unless stated otherwise, the range of values represents the
confidence intervals surrounding the point estimate at the 95 percent
level.
[8] "Test Report in Support of Advanced Spectroscopic Portal (ASP)
Systems Development at the Nevada Test Site (NTS)", Section 1.2 Test
Purpose, Page 2.
[9] "Test Report in Support of Advanced Spectroscopic Portal (ASP)
Systems Development at the Nevada Test Site (NTS)", Section 1.2 Test
Purpose, Page 2 and Section 2 Test Results, Page 4.
[10] "Test Report in Support of Advanced Spectroscopic Portal (ASP)
Systems Development at the Nevada Test Site (NTS)", Section 1.2 Test
Purpose, Page 2 and Executive Summary, Page iii.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office: 441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400:
U.S. Government Accountability Office: 441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800:
U.S. Government Accountability Office: 441 G Street NW, Room 7149:
Washington, D.C. 20548:
