Combating Nuclear Smuggling

Lessons Learned from DHS Testing of Advanced Radiation Detection Portal Monitors Gao ID: GAO-09-804T June 25, 2009

The Department of Homeland Security's (DHS) Domestic Nuclear Detection Office (DNDO) is responsible for addressing the threat of nuclear smuggling. Radiation detection portal monitors are key elements in the nation's defenses against such threats. DHS has sponsored testing to develop new monitors, known as advanced spectroscopic portal (ASP) monitors, to replace radiation detection equipment being used at ports of entry. DNDO expects that ASPs may offer improvements over current-generation portal monitors, particularly the potential to identify as well as detect radioactive material and thereby to reduce both the risk of missed threats and the rate of innocent alarms, which DNDO considers to be key limitations of radiation detection equipment currently used by Customs and Border Protection (CBP) at U.S. ports of entry. However, ASPs cost significantly more than current generation portal monitors. Due to concerns about ASPs' cost and performance, Congress has required that the Secretary of Homeland Security certify that ASPs provide a significant increase in operational effectiveness before obligating funds for full-scale ASP procurement. This testimony addresses (1) GAO findings on DNDO's latest round of ASP testing, and (2) lessons from ASP testing that can be applied to other DHS technology investments. These findings are based on GAO's May 2009 report GAO-09-655 and other related reports.

GAO's report on the latest round of ASP testing found that DHS increased the rigor in comparison with previous tests and thereby added credibility to the test results. However, GAO's report also questioned whether the benefits of the ASPs justify the high cost. In particular, the DHS criteria for a significant increase in operational effectiveness require only a marginal improvement in the detection of certain weapons-usable nuclear materials, which DNDO considers a key limitation of current-generation portal monitors. The marginal improvement required of ASPs is particularly notable given that DNDO has not completed efforts to fine-tune current-generation equipment to provide greater sensitivity. Moreover, the preliminary test results show that ASPs performed better than current-generation portal monitors in detection of such materials concealed by light shielding approximating the threat guidance for setting detection thresholds, but that differences in sensitivity were less notable when shielding was slightly below or above that level. Finally, DNDO has not yet updated its cost-benefit analysis to take into account the results of the latest round of ASP testing and does not plan to complete computer simulations that could provide additional insight into ASP capabilities and limitations prior to certification even though test delays have allowed more time to conduct the simulations. DNDO officials believe the other tests are sufficient for ASPs to demonstrate a significant increase in operational effectiveness. GAO recommended that DHS assess ASPs against the full potential of current-generation equipment and revise the program schedule to allow time to conduct computer simulations and to uncover and resolve problems with ASPs before full-scale deployment. DHS agreed to a phased deployment that should allow time to uncover ASP problems but disagreed with the other recommendations, which GAO believes remain valid. The challenges DNDO has faced in developing and testing ASPs illustrate the importance of following best practices for investments in complex homeland security acquisitions and for testing of new technologies. GAO recently found that many major DHS investments, including DNDO's ASP program, had not met the department's requirements for basic acquisition documents necessary to inform the investment review process, which has adopted many acquisition best practices. As a result, DHS had not consistently provided the oversight needed to identify and address cost, schedule, and performance problems in its major investments. A primary lesson to be learned regarding testing is that the push to replace existing equipment with the new portal monitors led to an ASP testing program that until recently lacked the necessary rigor. Even for the most recent round of testing, DNDO's schedule consistently underestimated the time required to conduct tests and resolve problems uncovered during testing. In contrast, GAO has previously found that testing programs designed to validate a product's performance against increasing standards for different stages in product development are a best practice for acquisition strategies for new technologies. Aspects that improved the latest round of ASP testing could also, if properly implemented, provide rigor to DHS's testing of other advanced technologies.

GAO-09-804T, Combating Nuclear Smuggling: Lessons Learned from DHS Testing of Advanced Radiation Detection Portal Monitors This is the accessible text file for GAO report number GAO-09-804T entitled 'Combating Nuclear Smuggling: Lessons Learned from DHS Testing of Advanced Radiation Detection Portal Monitors' which was released on June 25, 2009. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Testimony: Before the Subcommittee on Investigations and Oversight, Committee on Science and Technology, House of Representatives: United States Government Accountability Office: GAO: Combating Nuclear Smuggling: Lessons Learned from DHS Testing of Advanced Radiation Detection Portal Monitors: Statement of Gene Aloise, Director: Natural Resources and Environment: GAO-09-804T: GAO Highlights: Highlights of GAO-09-804T, a testimony before the Subcommittee on Investigations and Oversight, Committee on Science and Technology, House of Representatives. Why GAO Did This Study: The Department of Homeland Security‘s (DHS) Domestic Nuclear Detection Office (DNDO) is responsible for addressing the threat of nuclear smuggling. Radiation detection portal monitors are key elements in the nation‘s defenses against such threats. DHS has sponsored testing to develop new monitors, known as advanced spectroscopic portal (ASP) monitors, to replace radiation detection equipment being used at ports of entry. DNDO expects that ASPs may offer improvements over current- generation portal monitors, particularly the potential to identify as well as detect radioactive material and thereby to reduce both the risk of missed threats and the rate of innocent alarms, which DNDO considers to be key limitations of radiation detection equipment currently used by Customs and Border Protection (CBP) at U.S. ports of entry. However, ASPs cost significantly more than current generation portal monitors. Due to concerns about ASPs‘ cost and performance, Congress has required that the Secretary of Homeland Security certify that ASPs provide a significant increase in operational effectiveness before obligating funds for full-scale ASP procurement. This testimony addresses (1) GAO findings on DNDO‘s latest round of ASP testing, and (2) lessons from ASP testing that can be applied to other DHS technology investments. These findings are based on GAO‘s May 2009 report GAO-09-655 and other related reports. What GAO Found: GAO‘s report on the latest round of ASP testing found that DHS increased the rigor in comparison with previous tests and thereby added credibility to the test results. However, GAO‘s report also questioned whether the benefits of the ASPs justify the high cost. In particular, the DHS criteria for a significant increase in operational effectiveness require only a marginal improvement in the detection of certain weapons-usable nuclear materials, which DNDO considers a key limitation of current-generation portal monitors. The marginal improvement required of ASPs is particularly notable given that DNDO has not completed efforts to fine-tune current-generation equipment to provide greater sensitivity. Moreover, the preliminary test results show that ASPs performed better than current-generation portal monitors in detection of such materials concealed by light shielding approximating the threat guidance for setting detection thresholds, but that differences in sensitivity were less notable when shielding was slightly below or above that level. Finally, DNDO has not yet updated its cost-benefit analysis to take into account the results of the latest round of ASP testing and does not plan to complete computer simulations that could provide additional insight into ASP capabilities and limitations prior to certification even though test delays have allowed more time to conduct the simulations. DNDO officials believe the other tests are sufficient for ASPs to demonstrate a significant increase in operational effectiveness. GAO recommended that DHS assess ASPs against the full potential of current-generation equipment and revise the program schedule to allow time to conduct computer simulations and to uncover and resolve problems with ASPs before full- scale deployment. DHS agreed to a phased deployment that should allow time to uncover ASP problems but disagreed with the other recommendations, which GAO believes remain valid. The challenges DNDO has faced in developing and testing ASPs illustrate the importance of following best practices for investments in complex homeland security acquisitions and for testing of new technologies. GAO recently found that many major DHS investments, including DNDO‘s ASP program, had not met the department‘s requirements for basic acquisition documents necessary to inform the investment review process, which has adopted many acquisition best practices. As a result, DHS had not consistently provided the oversight needed to identify and address cost, schedule, and performance problems in its major investments. A primary lesson to be learned regarding testing is that the push to replace existing equipment with the new portal monitors led to an ASP testing program that until recently lacked the necessary rigor. Even for the most recent round of testing, DNDO‘s schedule consistently underestimated the time required to conduct tests and resolve problems uncovered during testing. In contrast, GAO has previously found that testing programs designed to validate a product‘s performance against increasing standards for different stages in product development are a best practice for acquisition strategies for new technologies. Aspects that improved the latest round of ASP testing could also, if properly implemented, provide rigor to DHS‘s testing of other advanced technologies. View [hyperlink, http://www.gao.gov/products/GAO-09-804T] or key components. For more information, contact Gene Aloise at (202) 512-3841 or aloisee@gao.gov. [End of section] Mr. Chairman and Members of the Subcommittee: I am pleased to be here today to discuss GAO's work on the Department of Homeland Security's (DHS) testing of advanced spectroscopic portal (ASP) radiation detection monitors. As you are aware, the national security mission of U.S. Customs and Border Protection (CBP), an agency within DHS, includes screening for smuggled nuclear or radiological material that could be used in a nuclear weapon or radiological dispersal device (a "dirty bomb"). To screen cargo at ports of entry, CBP conducts primary inspections with radiation detection equipment called portal monitors--large stationary detectors through which cargo containers and vehicles pass as they enter the United States. When radiation is detected, CBP conducts secondary inspections using a second portal monitor to confirm the original alarm and a handheld radioactive isotope identification device to identify the radiation's source and determine whether it constitutes a threat. The polyvinyl toluene (PVT) portal monitors currently in use can detect radiation but cannot identify the type of material causing an alarm. As a result, the monitors' radiation alarms can be set off even by benign, naturally occurring radioactive material. One way to reduce the rate of such innocent alarms--and thereby minimize unnecessary secondary inspections and enhance the flow of commerce--is to adjust the operational thresholds (i.e., operate the PVTs at a reduced level of sensitivity). However, reducing the sensitivity may make it more difficult to detect certain nuclear materials. To address the limitations of current-generation portal monitors, DHS's Domestic Nuclear Detection Office (DNDO) in 2005 began to develop and test ASPs, which are designed to both detect radiation and identify the source.[Footnote 1] DNDO hopes to use the new portal monitors to replace at least some PVTs currently used for primary screening, as well as PVTs and handheld identification devices currently used for secondary screening. However, we estimated in September 2008 that the lifecycle cost of each standard cargo version of the ASP (including deployment costs) to be about $822,000, compared with about $308,000 for the PVT standard cargo portal, and that the total program cost for DNDO's latest plan for deploying radiation portal monitors--which relies on a combination of ASPs and PVTs and does not deploy radiation portal monitors at all border crossings--would be about $2 billion. [Footnote 2] Concerned about the performance and cost of the ASP monitors, Congress required the Secretary of Homeland Security to certify that the monitors will provide a "significant increase in operational effectiveness" before DNDO obligates funds for full-scale ASP procurement.[Footnote 3] The Secretary must submit separate certifications for primary and secondary inspection. In response, CBP, DNDO, and the DHS management directorate jointly issued criteria in July 2008 for determining whether the new technology provides a significant increase in operational effectiveness. The primary screening criteria require that the new portal monitors detect potential threats as well as or better than PVTs, show improved performance in detection of highly enriched uranium (HEU), and reduce innocent alarms. To meet the secondary screening criteria, the new portal monitors must reduce the probability of misidentifying special nuclear material (e.g., HEU and plutonium) and the average time to conduct secondary screenings. DNDO designed and coordinated a new series of tests, originally scheduled to run from April 2008 through September 2008, to determine whether the new portal monitors meet the certification criteria for primary and secondary screening and are ready for deployment. Key phases of this testing program include concurrent testing led by DNDO of the new and current equipment's ability to detect and identify threats and of ASPs' readiness to be integrated into operations for both primary and secondary screening at ports of entry; field validation led by CBP at four northern and southern border crossings and two seaports; and an independent evaluation, led by the DHS Science and Technology Directorate at one of the seaports, of the new portal monitors' effectiveness and suitability. Since 2006, we have issued six reports and four testimonies on development of radiation detection portal monitors, including today's testimony, and have made 19 recommendations for improving DNDO's efforts to develop and test portal monitors. Our concerns have focused on key areas in which DNDO's efforts have lacked the necessary rigor given ASPs' high cost and the importance of the radiation portal monitor project to our national security. These areas include objective and realistic testing of ASPs' performance in comparison with the performance of current-generation equipment; full disclosure and reporting of the limitations of tests used to support a decision by the Secretary of Homeland Security on ASP certification; development of a cost estimate that considers the full costs of the plan for deploying radiation detection portal monitors; and development of a cost-benefit analysis based on ASPs' demonstrated performance and a complete accounting of the portal monitor project's costs. (App. I presents a summary of our key findings and recommendations related to ASPs.) As I will discuss today, DNDO has improved the rigor of testing but has not yet updated the cost-benefit analysis that is critical to a decision on whether to replace radiation detection equipment already deployed at ports of entry with the significantly more expensive ASPs. Specifically, my testimony discusses (1) our key findings on the most recent round of ASP testing and (2) lessons from ASP testing that can be applied to other DHS technology investments. These findings are based on our report released this week and other related GAO reports. [Footnote 4] We conducted this performance audit work in June 2009 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to produce a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our statement today. The Latest Round of Testing Highlights the Limitations of ASPs: Our report on the latest round of ASP testing found that DHS increased the rigor of ASP testing in comparison with previous tests and that a particular area of improvement was in the performance testing at the Nevada Test Site, where DNDO compared the capability of ASP and current- generation equipment to detect and identify nuclear and radiological materials. For example, unlike in prior tests, the plan for the 2008 performance test stipulated that there would be no system contractor involvement in test execution. Such improvements addressed concerns we previously raised about the potential for bias and provided credibility to the results. Nevertheless, based on the following factors, we continue to question whether the benefits of the new portal monitors justify the high cost: * The DHS criteria for a significant increase in operational effectiveness. Our chief concern with the criteria is that they require a marginal improvement over current-generation portal monitors in the detection of certain weapons-usable nuclear materials when ASPs are deployed for primary screening. DNDO considers detection of such materials to be a key limitation of current-generation portal monitors. We are particularly concerned about the marginal improvement required of ASPs because the detection threshold for the current-generation portal monitors does not specify a level of radiation shielding that smugglers could realistically use. DOE and national laboratory officials told us that DOE's threat guidance used to set the current detection threshold is based not on an analysis of the capabilities of potential smugglers to take effective shielding measures but rather on the limited sensitivity of PVTs to detect anything more than certain lightly shielded nuclear materials. DNDO officials acknowledge that both the new and current-generation portal monitors are capable of detecting certain nuclear materials only when unshielded or lightly shielded. The marginal improvement in detection of such materials required of ASPs is particularly notable given that DNDO has not completed efforts to fine-tune PVTs' software and thereby improve sensitivity to nuclear materials. DNDO officials expect they can achieve small improvements in sensitivity, but DNDO has not yet funded efforts to fine-tune PVTs' software. In contrast to the marginal improvement required in detection of certain nuclear materials, the primary screening requirement to reduce the rate of innocent alarms could result in hundreds of fewer secondary screenings per day, thereby reducing CBP's workload and delays to commerce. In addition, the secondary screening criteria, which require ASPs to reduce the probability of misidentifying special nuclear material by one-half, address the inability of relatively small handheld devices to consistently locate and identify potential threats in large cargo containers. * Preliminary results of performance testing and field validation. The preliminary results presented to us by DNDO are mixed, particularly in the capability of ASPs used for primary screening to detect certain shielded nuclear materials. Preliminary results show that the new portal monitors detected certain nuclear materials better than PVTs when shielding approximated DOE threat guidance, which is based on light shielding. In contrast, differences in system performance were less notable when shielding was slightly increased or decreased: Both the PVTs and ASPs were frequently able to detect certain nuclear materials when shielding was below threat guidance, and both systems had difficulty detecting such materials when shielding was somewhat greater than threat guidance. With regard to secondary screening, ASPs performed better than handheld devices in identification of threats when masked by naturally occurring radioactive material. However, differences in the ability to identify certain shielded nuclear materials depended on the level of shielding, with increasing levels appearing to reduce any ASP advantages over the handheld identification devices. Other phases of testing uncovered multiple problems in meeting requirements for successfully integrating the new technology into operations at ports of entry. Of the two ASP vendors participating in the 2008 round of testing, one has fallen behind due to severe problems encountered during testing of ASPs' readiness to be integrated into operations at ports of entry ("integration testing"); the problems may require that the vendor redo previous test phases to be considered for certification. The other vendor's system completed integration testing, but CBP suspended field validation after 2 weeks because of serious performance problems resulting in an overall increase in the number of referrals for secondary screening compared with existing equipment. * DNDO's plans for computer simulations. DNDO does not plan to complete injection studies--computer simulations for testing the response of ASPs and PVTs to simulated threat objects concealed in cargo containers--prior to the Secretary of Homeland Security's decision on certification even though delays to the ASP test schedule have allowed more time to conduct the studies. According to DNDO officials, injection studies address the inability of performance testing to replicate the wide variety of cargo coming into the United States and the inability to place special nuclear material and other threat objects in cargo during field validation. DNDO had earlier indicated that injection studies could provide information comparing the performance of the two systems as part of the certification process for both primary and secondary screening. However, DNDO subsequently decided that performance testing would provide sufficient information to support a decision on ASP certification. DNDO officials said they would instead use injection studies to support effective deployment of the new portal monitors. * Lack of an updated cost-benefit analysis. DNDO has not yet updated its cost-benefit analysis to take into account the results of the latest round of ASP testing. An updated analysis that takes into account the results from the latest round of testing, including injection studies, might show that DNDO's plan to replace existing equipment with ASPs is not justified, particularly given the marginal improvement in detection of certain nuclear materials required of ASPs and the potential to improve the current-generation portal monitors' sensitivity to nuclear materials, most likely at a lower cost. Our report recommended that the Secretary of Homeland Security direct DNDO to (1) assess whether ASPs meet the criteria for a significant increase in operational effectiveness based on a valid comparison with PVTs' full performance potential and (2) revise the schedule for ASP testing and certification to allow sufficient time for review and analysis of results from the final phases of testing and completion of all tests, including injection studies. We further recommended that, if ASPs are certified, the Secretary direct DNDO to develop an initial deployment plan that allows CBP to uncover and resolve any additional problems not identified through testing before proceeding to full-scale deployment. DHS agreed to a phased deployment that should allow time to uncover ASP problems but disagreed with GAO's other recommendations, which we continue to believe remain valid. Procurement Decisions for New Technologies Require Rigorous Testing and Thorough Analysis of Results: The challenges DNDO has faced in developing and testing ASPs illustrate the importance of following existing DHS policies as well as best practices for investments in complex homeland security acquisitions and for testing of new technologies. The DHS investment review process calls for executive decision making at key points in an investment's life cycle and includes many acquisition best practices that, if applied consistently, could help increase the chances for successful outcomes. However, we reported in November 2008 that, for the period from fiscal year 2004 through the second quarter of fiscal year 2008, DHS had not effectively implemented or adhered to its investment review process due to a lack of senior management officials' involvement as well as limited monitoring and resources.[Footnote 5] In particular, of DHS's 48 major investments requiring milestone and annual reviews under the department's investment review policy, 45 were not assessed in accordance with this policy. In addition, many major investments, including DNDO's ASP program, had not met the department's requirements for basic acquisition documents necessary to inform the investment review process. As a result, DHS had not consistently provided the oversight needed to identify and address cost, schedule, and performance problems in its major investments. Among other things, our November 2008 report recommended that the Secretary of Homeland Security direct component heads, such as the Director of DNDO, to ensure that the components have established processes to manage major investments consistent with departmental policies. DHS generally concurred with our recommendations, and we noted that DHS had begun several efforts to address shortcomings in the investment review process identified in our report, including issuing an interim directive requiring DHS components to align their internal policies and procedures by the end of the third quarter of fiscal year 2009. In January 2009, DHS issued a memorandum instructing component heads to create acquisition executives in their organizations to be responsible for management and oversight of component acquisition processes. If fully implemented, these steps should help ensure that DHS components have established processes to manage major investments. Based on our body of work on ASP testing, one of the primary lessons to be learned is to avoid the pitfalls in testing that stem from a rush to procure new technologies. GAO has previously reported on the negative consequences of pressures imposed by closely linking testing and development programs with decisions to procure and deploy new technologies, including the creation of incentives to postpone difficult tests and limit open communication about test results. [Footnote 6] We found that testing programs designed to validate a product's performance against increasing standards for different stages in product development are a best practice for acquisition strategies for new technologies. In the case of ASPs, the push to replace existing equipment with the new portal monitors led to a testing program that until recently lacked the necessary rigor. Even for the most recent round of testing, DNDO's schedule consistently underestimated the time required to conduct tests, resolve problems uncovered during testing, and complete key documents, including final test reports. In addition, DNDO's schedule did not anticipate the time required to update its cost- benefit analysis to take into account the latest test results. DNDO's original working schedule anticipated completion of testing in mid- September 2008 and the DHS Secretary's decision on ASP certification between September and November 2008. However, testing is still not completed, and DNDO took months longer than anticipated to complete the final report on performance testing. As previously mentioned, a number of aspects of the latest round of ASP testing increased the rigor in comparison with earlier rounds and, if properly implemented, could improve the rigor in DHS's testing of other advanced technologies. Key aspects included the following: * Criteria for ensuring test requirements are met. The test and evaluation master plan established criteria requiring that the ASPs meet certain requirements before starting or completing any test phase. For example, the plan required that ASPs have no critical or severe issues rendering them completely unusable or impairing their function. The criteria provided a formal means to ensure that ASPs met certain basic requirements prior to the start of each phase of testing. DNDO and CBP adhered to the criteria even though doing so resulted in integration testing taking longer than anticipated and delaying the start of field validation. * Participation of the technology end user. The participation of CBP (the end user of the new portal monitors) provided an independent check, within DHS, of DNDO's efforts to develop and test the new portal monitors. For example, CBP added a final requirement to integration testing before proceeding to field validation to demonstrate ASPs' ability to operate for 40 hours without additional problems and thereby provide for a productive field validation. In addition, the participation of CBP officers in the 2008 round of performance testing allowed DNDO to adhere more closely than in previous tests to CBP's standard operating procedure for conducting a secondary inspection using the handheld identification devices, thereby providing for an objective test. * Participation of an independent test authority. The DHS Science and Technology Directorate, which is responsible for developing and implementing the department's test and evaluation policies and standards, will have the lead role in the final phase of ASP testing and thereby provide an additional independent check on testing efforts. The Science and Technology Directorate identified two critical questions, related to ASPs' operational effectiveness (i.e., detection and identification of threats) and suitability (e.g., reliability, maintainability, and supportability), and drafted its own test plan to address those questions. Mr. Chairman, this completes my prepared statement. I would be happy to respond to any questions that you or other Members of the Subcommittee may have at this time. GAO Contact and Staff Acknowledgments: For further information about this testimony, please contact me at (202) 512-3841 or aloisee@gao.gov. Ned Woodward (Assistant Director), Joseph Cook, and Kevin Tarmann made key contributions to this testimony. Dr. Timothy Persons (Chief Scientist), James Ashley, Steve Caldwell, John Hutton, Omari Norman, Alison O'Neill, Amelia Shachoy, and Rebecca Shea also made important contributions. [End of section] Appendix I: Key Findings and Recommendations from Related GAO Products on Testing and Development of ASPs: Combating Nuclear Smuggling: DHS Has Made Progress Deploying Radiation Detection Equipment at U.S. Ports-of-Entry, but Concerns Remain. [hyperlink, http://www.gao.gov/products/GAO-06-389]. Washington, D.C.: March 22, 2006. * Key findings. Prototypes of advanced spectroscopic portals (ASP) were expected to be significantly more expensive than current-generation portal monitors but had not been shown to be more effective. For example, Domestic Nuclear Detection Office (DNDO) officials' preliminary analysis of 10 ASPs tested at the Nevada Test Site found that the new portal monitors outperformed current-generation equipment in detecting numerous small, medium-size, and threatlike radioactive objects and were able to identify and dismiss most naturally occurring radioactive material. However, the detection capabilities of both types of portal monitors converged as the amount of source material decreased. * Recommendations. We recommended that the Secretary of Homeland Security work with the Director of DNDO to analyze the benefits and costs of deploying ASPs before any of the new equipment is purchased to determine whether any additional detection capability is worth the additional cost. We also recommended that the total program cost estimate for the radiation portal monitor project be revised after completion of the cost-benefit analysis. Combating Nuclear Smuggling: DHS's Cost-Benefit Analysis to Support the Purchase of New Radiation Detection Portal Monitors Was Not Based on Available Performance Data and Did Not Fully Evaluate All the Monitors' Costs and Benefits. [hyperlink, http://www.gao.gov/products/GAO-07-133R]. Washington, D.C.: October 17, 2006. Combating Nuclear Smuggling: DHS's Decision to Procure and Deploy the Next Generation of Radiation Detection Equipment Is Not Supported by Its Cost-Benefit Analysis. [hyperlink, http://www.gao.gov/products/GAO-07-581T]. Washington, D.C.: March 14, 2007. * Key findings. DNDO's cost-benefit analysis issued in response to our March 2006 recommendation did not provide a sound analytical basis for DNDO's decision to purchase and deploy ASPs. We identified a number of problems with the analysis of both the performance of the new portal monitors and the costs. With regard to performance, DNDO did not use the results of its own tests and instead relied on assumptions of the new technology's anticipated performance level. In addition, the analysis focused on identifying highly enriched uranium (HEU) and did not consider how well the new portal monitors can correctly detect or identify other dangerous radiological or nuclear materials. With regard to costs, DNDO did not follow the DHS guidelines for performing cost- benefit analyses and used questionable assumptions about the procurement costs of portal monitor technology. * Recommendations. We recommended that DHS and DNDO conduct a new cost- benefit analysis using sound analytical methods, including actual performance data and a complete accounting of all major costs and benefits as required by DHS guidelines, and that DNDO conduct realistic testing for both ASPs and current-generation portal monitors. Combating Nuclear Smuggling: DNDO Has Not Yet Collected Most of the National Laboratories' Test Results on Radiation Portal Monitors in Support of DNDO's Testing and Development Program. [hyperlink, http://www.gao.gov/products/GAO-07-347R]. Washington, D.C.: March 9, 2007. * Key findings. DNDO had not collected a comprehensive inventory of testing information on current-generation portal monitors. Such information, if collected and used, could improve DNDO's understanding of how well portal monitors detect different radiological and nuclear materials under varying conditions. In turn, this understanding would assist DNDO's future testing, development, deployment, and purchases of portal monitors. * Recommendations. We recommended that the Secretary of Homeland Security, working with the Director of DNDO, collect reports concerning all of the testing of current-generation portal monitors and review the test reports in order to develop an information database on how the portal monitors perform in both laboratory and field tests on a variety of indicators, such as their ability to detect specific radiological and nuclear materials. Combating Nuclear Smuggling: Additional Actions Needed to Ensure Adequate Testing of Next Generation Radiation Detection Equipment. [hyperlink, http://www.gao.gov/products/GAO-07-1247T]. Washington, D.C.: September 18, 2007. * Key findings. We found that tests conducted by DNDO in early 2007 were not an objective and rigorous assessment of the ASPs' capabilities. Specifically, we raised concerns about DNDO using biased test methods that enhanced the apparent performance of ASPs; not testing the limitations of ASPs' detection capabilities--for example, by not using a sufficient amount of the type of materials that would mask or hide dangerous sources and that ASPs would likely encounter at ports of entry; and not using a critical Customs and Border Protection (CBP) standard operating procedure that is fundamental to the performance of handheld radiation detectors in the field. * Recommendations. We recommended that the Secretary of Homeland Security delay Secretarial certification and full-scale production decisions on ASPs until all relevant tests and studies had been completed and limitations to tests and studies had been identified and addressed. We further recommended that DHS determine the need for additional testing in cooperation with CBP and other stakeholders and, if additional testing was needed, that the Secretary of DHS appoint an independent group within DHS to conduct objective, comprehensive, and transparent testing that realistically demonstrates the capabilities and limitations of ASPs. Combating Nuclear Smuggling: DHS's Program to Procure and Deploy Advanced Radiation Detection Portal Monitors Is Likely to Exceed the Department's Previous Cost Estimates. [hyperlink, http://www.gao.gov/products/GAO-08-1108R]. Washington, D.C.: September 22, 2008. * Key findings. Our independent cost estimate suggested that from 2007 through 2017 the total cost of DNDO's 2006 project execution plan (the most recent official documentation of the program to equip U.S. ports of entry with radiation detection equipment) would likely be about $3.1 billion but could range from $2.6 billion to $3.8 billion. In contrast, we found that DNDO's cost estimate of $2.1 billion was unreliable because it omitted major project costs, such as elements of the ASPs' life cycle, and relied on a flawed methodology. DNDO officials told us that the agency was no longer following the 2006 project execution plan and that the scope of the agency's ASP deployment strategy had been reduced to only the standard cargo portal monitor. Our analysis of DNDO's summary information outlining its scaled-back plan indicated the total cost to deploy standard cargo portals over the period 2008 through 2017 would be about $2 billion but could range from $1.7 billion to $2.3 billion. Agency officials acknowledged the program requirements that would have been fulfilled by the discontinued ASPs remained valid, including screening rail cars and airport cargo, but the agency had no plans for how such screening would be accomplished. * Recommendations. We recommended that the Secretary of Homeland Security direct the Director of DNDO to work with CBP to update the projection execution plan to guide the entire radiation detection program at U.S. ports of entry, revise the estimate of the program's cost and ensure that the estimate considers all of the costs associated with its project execution plan, and communicate the revised estimate to Congress so that it is fully apprised of the program's scope and funding requirements. Combating Nuclear Smuggling: DHS Needs to Consider the Full Costs and Complete All Tests Prior to Making a Decision on Whether to Purchase Advanced Portal Monitors. [hyperlink, http://www.gao.gov/products/GAO-08-1178T]. Washington, D.C.: September 25, 2008. * Key findings. In preliminary observations of the 2008 round of ASP testing, we found that DNDO had made progress in addressing a number of problems we identified in previous rounds of ASP testing. However, the DHS criteria for significant increase in operational effectiveness appeared to set a low bar for improvement--for example, by requiring ASPs to perform at least as well as current-generation equipment when nuclear material is present in cargo but not specifying an actual improvement. In addition, the ASP certification schedule did not allow for completion of computer simulations that could provide useful data on ASP capabilities prior to the Secretary's decision on certification. Finally, we questioned the replacement of current-generation equipment with ASPs until DNDO demonstrates that any additional increase in security would be worth the ASPs' much higher cost.[Footnote 7] Combating Nuclear Smuggling: DHS's Phase 3 Test Report on Advanced Portal Monitors Does Not Fully Disclose the Limitations of the Test Results. [hyperlink, http://www.gao.gov/products/GAO-08-979]. Washington, D.C.: September 30, 2008. * Key findings. DNDO's report on the second group of ASP tests in 2007 (the Phase 3 tests) did not appropriately state test limitations. As a result, the report did not accurately depict the results and could potentially be misleading. The purpose of the Phase 3 tests was to conduct a limited number of test runs in order to identify areas in which the ASP software needed improvement. While aspects of the Phase 3 report addressed this purpose, the preponderance of the report went beyond the test's original purpose and made comparisons of the performance of the ASPs with one another or with currently deployed portal monitors. We found that it would not be appropriate to use the Phase 3 test report in determining whether the ASPs represent a significant improvement over currently deployed radiation equipment because the limited number of test runs did not support many of the comparisons of ASP performance made in the report. * Recommendations. We recommended that the Secretary of DHS use the results of the Phase 3 tests solely for the purposes for which they were intended--to identify areas needing improvement--and not as a justification for certifying whether the ASPs warrant full-scale production. If the Secretary intends to consider the results of the Phase 3 tests in making a certification decision regarding ASPs, we further recommended that the Secretary direct the Director of DNDO to revise and clarify the Phase 3 test report to more fully disclose and articulate the limitations present in the Phase 3 tests and clearly state which insights from the Phase 3 report are factored into any decision regarding the certification that ASPs demonstrate a significant increase in operational effectiveness. Finally, we recommended that the Secretary direct the Director of DNDO to take steps to ensure that any limitations associated with the 2008 round of testing are properly disclosed when the results are reported. Combating Nuclear Smuggling: DHS Improved Testing of Advanced Radiation Detection Portal Monitors, but Preliminary Results Show Limits of the New Technology. [hyperlink, http://www.gao.gov/products/GAO-09-655]. Washington, D.C.: May 21, 2009. * Key findings. We reported that the DHS criteria for a significant increase in operational effectiveness require a large reduction in innocent alarms but a marginal improvement in the detection of certain weapons-usable nuclear materials. In addition, the criteria do not take the current-generation portal monitors' full potential into account because DNDO has not completed efforts to improve their performance. With regard to ASP testing, we found that DHS increased the rigor in comparison with previous tests, thereby adding credibility to the test results, but that preliminary results were mixed. The results showed that the new portal monitors performed better than current-generation portal monitors in detection of certain nuclear materials concealed by light shielding approximating the threat guidance for setting detection thresholds, but differences in sensitivity were less notable when shielding was slightly below or above that level. Testing also uncovered multiple problems in ASPs meeting the requirements for successful integration into operations at ports of entry. Finally, we found that DNDO did not plan to complete computer simulations that could provide additional insight into ASP capabilities and limitations prior to certification even though delays to testing allowed more time to conduct the simulations. * Recommendations. We recommended that the Secretary of Homeland Security direct the Director of DNDO to assess whether ASPs meet the criteria for a significant increase in operational effectiveness based on a valid comparison with current-generation portal monitors' full performance potential and revise the schedule for ASP testing and certification to allow sufficient time for review and analysis of results from the final phases of testing and completion of all tests, including computer simulations. If ASPs are certified, we further recommended that the Secretary of Homeland Security direct the Director of DNDO to develop an initial deployment plan that allows CBP to uncover and resolve any additional problems not identified through testing before proceeding to full-scale deployment. [End of section] Footnotes: [1] DNDO was established within DHS in 2005; its mission includes developing, testing, acquiring, and supporting the deployment of radiation detection equipment at U.S. ports of entry. CBP began deploying portal monitors in 2002, prior to DNDO's creation, under the radiation portal monitor project. [2] GAO, Combating Nuclear Smuggling: DHS's Program to Procure and Deploy Advanced Radiation Detection Portal Monitors Is Likely to Exceed the Department's Previous Cost Estimates, [hyperlink, http://www.gao.gov/products/GAO-08-1108R] (Washington, D.C.: Sept. 22, 2008). [3] Consolidated Appropriations Act, 2008, Pub. L. No. 110-161, 121 Stat. 1844, 2069 (2007); Consolidated Security, Disaster Assistance, and Continuing Appropriations Act, 2009, Pub. L. No. 110-329, 121 Stat. 3574, 3679 (2008). [4] GAO, Combating Nuclear Smuggling: DHS Improved Testing of Advanced Radiation Detection Portal Monitors, but Preliminary Results Show Limits of the New Technology, [hyperlink, http://www.gao.gov/products/GAO-09-655] (Washington, D.C.: May 21, 2009). [5] GAO, Department of Homeland Security: Billions Invested in Major Programs Lack Appropriate Oversight, [hyperlink, http://www.gao.gov/products/GAO-09-29] (Washington, D.C.: Nov. 18, 2008). [6] GAO, Best Practices: A More Constructive Test Approach Is Key to Better Weapon System Outcomes, [hyperlink, http://www.gao.gov/products/GAO/NSIAD-00-199] (Washington, D.C.: July 31, 2000). [7] This testimony also summarized our September 2008 report on the life cycle cost estimate to deploy ASPs [hyperlink, http://www.gao.gov/products/GAO-08-1108R]. [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO‘s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO‘s Web site, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: