Combating Nuclear Smuggling
DHS's Program to Procure and Deploy Advanced Radiation Detection Portal Monitors Is Likely to Exceed the Department's Previous Cost Estimates Gao ID: GAO-08-1108R September 22, 2008Since the attacks of September 11, 2001, combating terrorism has been one of the nation's highest priorities. As part of that effort, preventing nuclear and radioactive material from being smuggled into the United States--perhaps to be used by terrorists in a nuclear weapon or in a radiological dispersal device (a "dirty bomb")--has become a key national security objective. On April 15, 2005, the president directed the establishment, within the Department of Homeland Security (DHS), of the Domestic Nuclear Detection Office (DNDO), whose duties include acquiring and supporting the deployment of radiation detection equipment. In October 2006, Congress enacted the SAFE Port Act, which made DNDO responsible for the development, testing, acquisition and deployment of a system to detect radiation at U.S. ports of entry. An important component of this system is the deployment of radiation portal monitors, large stationary detectors through which cargo containers and trucks pass as they enter the United States. Prior to DNDO's creation, another DHS agency--U.S. Customs and Border Protection (CBP)--managed programs for deployment of radiation detection equipment. In 2002, CBP began the radiation portal monitor project, deploying radiation detection equipment at U.S. ports of entry. This program initially deployed portal monitors, known as polyvinyl toluene monitors (PVT), and handheld detection technologies, such as radioactive isotope identification devices (RIID). CBP also established a system of standard operating procedures to guide its officers in the use of this equipment. Current procedures include conducting primary inspections with PVTs to detect the presence of radioactivity, and secondary inspections with PVTs and RIIDs to confirm and identify the source and determine whether it constitutes a threat. After its creation, DNDO assumed responsibility for the development, testing, and deployment of radiation detection equipment, while CBP maintained its role of operating the equipment at U.S. ports of entry. Currently deployed PVTs are capable of detecting radiation, but they have an inherent limitation because they are unable to identify specific radioactive isotopes and therefore cannot distinguish between dangerous and benign materials. CBP officers also use RIIDs to identify different types of radioactive material. However, RIIDs are limited in their ability to identify nuclear material. DNDO believes that these deficiencies may delay legitimate commerce at ports of entry, and that CBP may use an inordinate amount of inspection resources for radiation detection at the expense of other missions, such as drug interdiction.
Our independent cost estimate suggests that from 2007 through 2017 the total cost of DNDO's program to equip U.S. ports of entry with radiation detection equipment will likely be about $3.1 billion, but could range from $2.6 billion to $3.8 billion. We based our estimate on the anticipated costs of DNDO implementing its 2006 project execution plan, the most recent official documentation of the program. According to this plan, DNDO will buy and deploy multiple types of ASPs, including those designed to screen rail cars, and airport and seaport cargo, as well as mobile ASPs--spectroscopic equipment mounted on vehicles--to provide greater flexibility in screening commerce. The project execution plan also targets several types of PVTs for purchase and deployment. DNDO's cost estimate of $2.1 billion to equip U.S. ports of entry with radiation detection equipment is unreliable because it omits major project costs and relies on a flawed methodology. For example, although the normal life expectancy of the standard cargo ASP is about 10 years, DNDO's estimate considers only 8 years--fiscal years 2006 through 2013. According to DNDO officials, OMB's budget submission software allows only a limited number of years of costs to be included. Furthermore, DNDO's cost estimate does not include all of the elements of the ASPs' life cycle, as it omits estimates for maintenance and operational sustainment of ASPs. Finally, contrary to OMB and DHS guidelines, DNDO did not provide detailed documentation of ASP costs, which raises questions about the adequacy and reliability of the agency's estimates. DNDO officials told us on several occasions during the course of our review the agency is no longer following the 2006 project execution plan. These officials told us the scope of the agency's current ASP deployment strategy has been reduced to only the standard cargo portal monitor. Although we repeatedly requested documentation of DNDO's current official deployment strategy, the agency did not provide such official information. In fact, DNDO officials continued to cite the 2006 project execution plan as the most recent official deployment documentation. In July 2008, the agency provided a 1-page spreadsheet of summary information outlining DNDO's current plans to buy and deploy ASPs and PVTs. Our analysis of these summary data indicates the total cost to deploy standard cargo portals over the period 2008 through 2017 will be about $2.0 billion, but could range from $1.7 billion to $2.3 billion. These data also indicate that between fiscal years 2008 and 2014, DNDO plans to deploy 717 ASP and 1,005 PVT standard cargo portals. Furthermore, agency officials acknowledged the program requirements that would have been fulfilled by the discontinued ASPs remain valid, including screening rail cars, airport cargo, and cargo at seaport terminals, but the agency has no current plans for how such screening will be accomplished. These officials told us the technology to accomplish these requirements likely will not be ASP monitors. We believe a comprehensive estimate of the cost to provide radiation detection equipment for U.S. ports of entry should account for meeting these objectives, even if DNDO decides that ASP technology is not suited to them. However, a DNDO official responsible for overseeing the agency's operations told us in August 2008 that DNDO's ASP deployment strategy could change dramatically depending on the outcome of ongoing ASP testing. In our view, it is difficult to assess the total costs of the ASP program because of the frequent changes in DNDO's deployment strategy. Furthermore, the Congress needs a complete understanding of DNDO's deployment strategy before approving additional ASP program funds.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-08-1108R, Combating Nuclear Smuggling: DHS's Program to Procure and Deploy Advanced Radiation Detection Portal Monitors Is Likely to Exceed the Department's Previous Cost Estimates
This is the accessible text file for GAO report number GAO-08-1108R
entitled 'Combating Nuclear Smuggling: DHS's Program to Procure and
Deploy Advanced Radiation Detection Portal Monitors Is Likely to Exceed
the Department's Previous Cost Estimates' which was released on
September 22, 2008.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
GAO-08-1108R:
United States Government Accountability Office:
Washington, DC 20548:
September 22, 2008:
Congressional Requesters:
Subject: Combating Nuclear Smuggling: DHS's Program to Procure and
Deploy Advanced Radiation Detection Portal Monitors Is Likely to Exceed
the Department's Previous Cost Estimates:
Since the attacks of September 11, 2001, combating terrorism has been
one of the nation's highest priorities. As part of that effort,
preventing nuclear and radioactive material from being smuggled into
the United States--perhaps to be used by terrorists in a nuclear weapon
or in a radiological dispersal device (a "dirty bomb")--has become a
key national security objective. On April 15, 2005, the president
directed the establishment, within the Department of Homeland Security
(DHS), of the Domestic Nuclear Detection Office (DNDO), whose duties
include acquiring and supporting the deployment of radiation detection
equipment. In October 2006, Congress enacted the SAFE Port Act,
[Footnote 1] which made DNDO responsible for the development, testing,
acquisition and deployment of a system to detect radiation at U.S.
ports of entry. An important component of this system is the deployment
of radiation portal monitors, large stationary detectors through which
cargo containers and trucks pass as they enter the United States.
Prior to DNDO's creation, another DHS agency--U.S. Customs and Border
Protection (CBP)--managed programs for deployment of radiation
detection equipment. In 2002, CBP began the radiation portal monitor
project, deploying radiation detection equipment at U.S. ports of
entry. This program initially deployed portal monitors, known as
polyvinyl toluene monitors (PVT), and handheld detection technologies,
such as radioactive isotope identification devices (RIID). CBP also
established a system of standard operating procedures to guide its
officers in the use of this equipment. Current procedures include
conducting primary inspections with PVTs to detect the presence of
radioactivity, and secondary inspections with PVTs and RIIDs to confirm
and identify the source and determine whether it constitutes a threat.
After its creation, DNDO assumed responsibility for the development,
testing, and deployment of radiation detection equipment, while CBP
maintained its role of operating the equipment at U.S. ports of entry.
Currently deployed PVTs are capable of detecting radiation, but they
have an inherent limitation because they are unable to identify
specific radioactive isotopes and therefore cannot distinguish between
dangerous and benign materials. CBP officers also use RIIDs to identify
different types of radioactive material. However, RIIDs are limited in
their ability to identify nuclear material. DNDO believes that these
deficiencies may delay legitimate commerce at ports of entry, and that
CBP may use an inordinate amount of inspection resources for radiation
detection at the expense of other missions, such as drug interdiction.
To address the limitations of the PVTs and RIIDs, DNDO sponsored the
development of a next-generation portal monitor--the advanced
spectroscopic portal (ASP). In contrast to PVTs, ASPs offer the ability
to both detect and identify radioactive material, according to DNDO.
The use of ASPs may eliminate, or at least greatly reduce, the need for
RIIDs while minimizing both missed threats and false alarms.
In September 2006, DNDO worked with CBP to produce a project execution
plan that defines the radiation portal monitor project's objectives,
scope, schedule, costs, and funding requirements to deploy radiation
detection systems at U.S. ports of entry.[Footnote 2] DNDO plans to
deploy ASPs at high-volume ports of entry that also experience high
numbers of false positive and nuisance alarms. In these locations, ASPs
would be deployed in primary and secondary inspection areas. In low-
volume ports of entry--that can tolerate a higher false alarm rate--
PVTs would be placed in primary inspection areas and ASPs in secondary
inspection areas. DNDO expects this approach to be cost effective and
to provide a balance of cost and performance. DNDO is now testing a
variation of the ASP, designed to screen cargo on standard sized
trucks. Eventually, however, the project execution plan calls for DNDO
to deploy several variations of the ASP in order to screen cargo on
other types of conveyance, such as rail cars and at seaport terminals.
In the meantime, DNDO is working toward deploying ASPs in secondary
inspection locations, and possibly also in some primary inspection
locations, perhaps as soon as the fall of 2008.
Congressional committees have expressed concern over DNDO's estimates
of ASP program costs. In February 2003, CBP completed the initial
estimate of the total cost of the radiation portal monitor project. At
that time, the agency reported that equipping U.S. ports of entry with
PVT portal monitors would likely range from $399 million to $484
million. By September 2003, the scope of the program had increased and
its estimated cost had grown to $496 million. In December 2004, when
CBP began to anticipate deploying ASPs, the expected cost of the
project rose again to $1.3 billion. This significant increase was due
at least in part to the higher cost of the ASPs. DNDO took control of
the program in April 2005, and in February 2007, raised the cost
estimate to $1.7 billion. By March 2008, it had reached $2.1 billion.
Further, between May 2006 and March 2008, DNDO's unit cost estimate for
the standard cargo version of the ASP (including deployment costs)
increased from about $576,400 to about $800,000. DNDO's current unit
cost estimate of PVT standard cargo portals is about $425,000
(including deployment costs). Congressional committee concern regarding
these costs was evident in the explanatory note accompanying DHS's 2008
appropriation: "While the current generation technologies may not have
the expected capabilities of ASP systems, they are substantially less
expensive, proven to work within known limits, and can be quickly
deployed to POEs [ports of entry]."
In an attempt to understand the cost of DHS's deployment of radiation
detection portal monitors, we were requested to review DNDO's
assessment of the program's costs and benefits. In October 2006, we
reported, among other things, that DNDO's independent cost estimate of
the program was incomplete.[Footnote 3] Specifically, we found the cost-
estimating methodology used for the agency's analysis did not adhere to
DHS guidelines and it omitted several key factors that could impact the
ASPs' cost. For example, DNDO did not capture all of the ASPs'
developmental costs, nor did the agency estimate the ASPs' full life-
cycle costs. We concluded that DNDO's cost-benefit analysis did not
provide sufficient justification for buying and deploying ASPs.
In this context, you asked us to review (1) the projected costs to
implement DNDO's 2006 project execution plan for the radiation portal
monitor program, which includes the costs to develop, procure, deploy,
operate, and maintain ASPs and other RPMs over a 10-year life cycle;
and (2) the reliability of DNDO's projections given what is known about
the costs to procure, install, and operate radiation detection
equipment.
During the course of our review, DNDO officials told us the agency's
2006 radiation portal monitor deployment strategy had been revised.
According to these officials, the only ASP equipment DNDO now plans to
deploy to U.S. ports of entry is the standard cargo portal.[Footnote 4]
DNDO first notified us of this new approach in October 2007. Although
we requested detailed documentation of the agency's revised portal
monitor strategy, the extent of the documentation we received consisted
of a 1-page spreadsheet of summary information regarding the quantity
and costs of its revised deployment plans that DNDO provided in late
July 2008. Based on this incomplete information, we are also providing
a limited analysis of the summary data DNDO provided regarding its
revised portal monitor deployment strategy, although this revised
strategy has not been officially documented.
To address these issues, we contracted with a company whose expertise
is in estimating the life cycle costs of major federal acquisitions.
Together we developed an independent cost estimate of 2006 DHS's
strategy to deploy radiation detection portal monitors, covering fiscal
years 2007 through 2017, to ensure that the Congress has authoritative
information on all the life-cycle costs associated with a full scale
acquisition of radiation portal monitors. We briefed your staffs on the
results of our work in May, June, and July 2008. This report presents
the details of those briefings, which focused mainly on our estimate of
the life-cycle costs associated with the deployment of radiation
detection portal monitors at U.S. ports of entry, and how our estimate
compares with DNDO's estimate of those costs (see encl. I).
To conduct our review and develop our independent cost estimate, we
held discussions with officials from DNDO, CBP, and Pacific Northwest
National Laboratory--which manages the deployment of radiation
detection equipment for DHS. We compiled and analyzed information on
the historical costs of PVTs and ASPs, as well as data on their
development and deployment. In addition, we developed a cost-estimating
model to generate specific estimates of present and future radiation
portal monitor project costs. We analyzed the uncertainty associated
with our estimates to help gauge their accuracy. In March 2008, we
provided DNDO an extensive briefing on our cost-estimating methodology,
including all supporting documentation and analysis, so that DNDO could
better understand our approach and perhaps provide additional
information to enhance our approach. DNDO offered suggestions to
improve our cost-estimating methodology. We incorporated DNDO's
suggestions as we deemed appropriate. A detailed explanation of our
methodology is provided in enclosure II.
We conducted this performance audit from October 2007 to August 2008 in
accordance with generally accepted government auditing standards. Those
standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.
Summary:
Our independent cost estimate suggests that from 2007 through 2017 the
total cost of DNDO's program to equip U.S. ports of entry with
radiation detection equipment will likely be about $3.1 billion, but
could range from $2.6 billion to $3.8 billion. We based our estimate on
the anticipated costs of DNDO implementing its 2006 project execution
plan, the most recent official documentation of the program. According
to this plan, DNDO will buy and deploy multiple types of ASPs,
including those designed to screen rail cars, and airport and seaport
cargo, as well as mobile ASPs--spectroscopic equipment mounted on
vehicles--to provide greater flexibility in screening commerce. The
project execution plan also targets several types of PVTs for purchase
and deployment.
DNDO's cost estimate of $2.1 billion to equip U.S. ports of entry with
radiation detection equipment is unreliable because it omits major
project costs and relies on a flawed methodology. For example, although
the normal life expectancy of the standard cargo ASP is about 10 years,
DNDO's estimate considers only 8 years--fiscal years 2006 through 2013.
According to DNDO officials, OMB's budget submission software allows
only a limited number of years of costs to be included. Furthermore,
DNDO's cost estimate does not include all of the elements of the ASPs'
life cycle, as it omits estimates for maintenance and operational
sustainment of ASPs. Finally, contrary to OMB and DHS guidelines, DNDO
did not provide detailed documentation of ASP costs, which raises
questions about the adequacy and reliability of the agency's estimates.
DNDO officials told us on several occasions during the course of our
review the agency is no longer following the 2006 project execution
plan. These officials told us the scope of the agency's current ASP
deployment strategy has been reduced to only the standard cargo portal
monitor. Although we repeatedly requested documentation of DNDO's
current official deployment strategy, the agency did not provide such
official information. In fact, DNDO officials continued to cite the
2006 project execution plan as the most recent official deployment
documentation. In July 2008, the agency provided a 1-page spreadsheet
of summary information outlining DNDO's current plans to buy and deploy
ASPs and PVTs. Our analysis of these summary data indicates the total
cost to deploy standard cargo portals over the period 2008 through 2017
will be about $2.0 billion, but could range from $1.7 billion to $2.3
billion. These data also indicate that between fiscal years 2008 and
2014, DNDO plans to deploy 717 ASP and 1,005 PVT standard cargo
portals. Furthermore, agency officials acknowledged the program
requirements that would have been fulfilled by the discontinued ASPs
remain valid, including screening rail cars, airport cargo, and cargo
at seaport terminals, but the agency has no current plans for how such
screening will be accomplished. These officials told us the technology
to accomplish these requirements likely will not be ASP monitors. We
believe a comprehensive estimate of the cost to provide radiation
detection equipment for U.S. ports of entry should account for meeting
these objectives, even if DNDO decides that ASP technology is not
suited to them. However, a DNDO official responsible for overseeing the
agency's operations told us in August 2008 that DNDO's ASP deployment
strategy could change dramatically depending on the outcome of ongoing
ASP testing. In our view, it is difficult to assess the total costs of
the ASP program because of the frequent changes in DNDO's deployment
strategy. Furthermore, the Congress needs a complete understanding of
DNDO's deployment strategy before approving additional ASP program
funds.
We provided a draft of this report to DHS for its review and comment.
In its comments, the Department agreed with our recommendations and
part of our conclusions, but strongly disagreed other parts of our
conclusions. In addition, DHS commented that we used inconsistent terms
in referring to "cost" and that as a result it is impossible to verify
whether our assessments and conclusions are valid, or to conduct an
"apples to apples" comparison. We disagree with DHS's assertion that we
used inconsistent terms for "cost" and as a result it was impossible to
verify our conclusions or to conduct a true "apples to apples"
comparison. Our use of the various terms related to cost is accurate
and consistent with the technical vocabulary of cost estimating. Even
so, we made minor changes to the report to make the terminology more
uniform and improve its readability. With regard to DNDO's comment
about assessing the validity of our conclusions, over the course of our
review, we provided extensive briefings and documentation on the
components and results of our life-cycle cost estimate. In fact,
according to a DNDO document (dated May 29, 2008) that summarized
agency efforts to develop its own life-cycle cost estimate, DNDO used
our cost estimate as its initial baseline model in starting to develop
its own cost estimate.
DHS commented that comparing our cost estimate to DNDO's is misleading
because we included operation and maintenance costs--incurred by CBP--
in our analysis.
We disagree that the draft report is misleading when it compares DNDO
and GAO life-cycle costs. In fact, we believe our estimate helps
clarify the program's true cost, while DNDO's estimate obscures it. The
source for DNDO's $2.1 billion cost estimate is the agency's most
recent OMB Exhibit 300, dated March 25, 2008. According to OMB's
Exhibit 300 instructions for completing multi-agency investments,
agencies should include: (a) the costs associated with the entire life
cycle of the investment; and (b) the funding from the lead and all
partner agencies [emphasis added]. By omitting major project costs
(such as maintenance costs), DNDO has understated the life-cycle costs
of the program and violated the intent of OMB policy and guidance. The
effect of these omissions increases the probability of misinformed
decisions and inadequate budget formulation for partner agencies. DHS
offered several other comments which are discussed at the end of this
letter.
DNDO's Program to Deploy Radiation Detection Portal Monitors at U.S.
Ports of Entry Is Likely to Cost About $3 Billion:
Our independent cost estimate suggests the total cost of DNDO's program
to equip U.S. ports of entry with radiation detection equipment will
likely be about $3.1 billion, but could range between $2.6 billion and
$3.8 billion. We based our estimate on the anticipated costs of DNDO
implementing its 2006 project execution plan. According to this plan,
DNDO plans to buy and deploy several types of ASPs, including those
designed to screen rail cars and seaport cargo; as well as mobile ASPs
to provide greater flexibility in screening commerce. The plan also
provides for the deployment of several types of PVTs. Clearly, the
numbers and types of portal monitors deployed will significantly affect
the total cost of the radiation portal monitor project. In all, the
project execution plan calls for the purchase of 2,754 portal monitors,
with a total of 2,582 scheduled for deployment--approximately 1,034
ASPs and 1,548 PVTs--and 172 held in excess at the project's
completion.
In developing our estimate, we categorized radiation portal monitor
project costs according to program phases--design and development,
procurement, deployment, maintenance, and operational sustainment. We
did not incorporate operational costs, in particular the cost of CBP
officers operating the radiation detection equipment. DNDO and CBP
believe that deploying ASPs will reduce the use of CBP staff resources
for radiation detection tasks, but currently there are no usable
estimates of how CBP's staffing would change with the deployment of
ASPs. Finally, our analysis includes a period of 11 years, actual life-
cycle expenses from fiscal year 2007 and estimated life cycle costs
from fiscal year 2008 through 2017. The details of this analysis are
presented in enclosure III.
DNDO's Cost Estimates Are Unreliable and Could Result in Significant
Cost Overruns:
In submitting its budget request to OMB in March 2008, DNDO estimated
the total cost of the radiation portal monitor project at $2.1 billion.
However, DNDO's estimate is unreliable because it omits major project
costs and relies on flawed methodology. As a result, DNDO's cost
estimates and budget requests for the radiation portal monitor project
are too low, which could lead to significant cost overruns later in the
project.
DNDO's estimate contains three major deficiencies. First, it does not
appear to include the costs of all variations of ASPs contained in
DNDO's project execution plan. DNDO's current baseline considers only
the standard ASP cargo portal and ignores the costs of other types of
ASPs, such as those designed to screen rail cars, airport and seaport
cargo, and mobile detectors. According to senior DNDO officials, the
current approved project execution plan no longer reflects the agency's
procurement and deployment plans for ASPs. In acknowledging that the
project execution plan should be revised and updated, these officials
told us that DNDO currently plans to field only the standard cargo ASP
portal--mainly at high-volume ports of entry--but DNDO did not provide
us detailed documentation to support this major programmatic decision.
Second, although the normal life expectancy of the ASP is about 10
years, DNDO's estimate considers only fiscal years 2006 through 2013.
DNDO officials told us this was because OMB's budget submission
software limits the number of years of costs that can be included.
However, DHS cost guidance maintains that a project's life cycle can be
estimated over that period of time during which equipment will remain
available before it is exhausted, that is, decayed or deteriorated. The
manufacturer of the sodium iodide crystals that comprise a key
component of ASP systems expects an operational life of about 10 years
for its crystals.[Footnote 5] DNDO officials agreed that a 10-year life
cycle cost estimate would have been more appropriate, and added that
they would have used a 10-year estimate had they not been constrained
by the OMB software. Third, DNDO's cost estimate does not include all
of the elements of the ASPs' life cycle. For example, the agency's
estimate does not include cost estimates for maintenance or operational
sustainment of ASPs. These costs are approximately $999.2 million and
$364.9 million, respectively, under our cost estimate.
Furthermore, DNDO's cost methodology and documentation contain
significant weaknesses, which may further reduce the reliability of the
agency's cost estimates. For example, DNDO was unable to provide
detailed documentation of the costs used in its estimates. Such
documentation is necessary, according to OMB, DHS, and GAO guidelines,
to establish the basis of the estimates and to provide assurances that
the estimates are credible.[Footnote 6] For fiscal years 2008 through
2012, the years included in both our analysis and the DNDO analysis, we
estimate a $753 million budget shortfall for the radiation portal
monitor project. Additionally, we estimate that DNDO will require
another $833 million from fiscal year 2013 through fiscal year 2017 to
complete the entire life cycle of the program. Our analysis projects
some of the greatest deficits in fiscal years 2008, 2009, and 2010,
when DNDO plans to acquire large numbers of portal monitors.
DNDO's Revised Program to Deploy Radiation Detection Portal Monitors at
U.S. Ports of Entry Is Likely to Cost about $2.0 Billion:
We met with DNDO officials in March and June 2008 to discuss the
preliminary findings of this report. At those times, they noted--as
they had in October 2007--that the project execution plan no longer
guides the radiation portal monitor project. Hence, in their view,
because our cost estimate is based on the project execution plan, it
likely will result in estimates higher than the program's current true
cost since DNDO has eliminated all types of ASPs, except the standard
cargo portal.
In our view, however, estimates of any program's costs must be based on
the agency's documented program. In the absence of more recent
documentation, we believe our independent cost estimate must be based
on the agency's most recent approved plan--the project execution plan.
Furthermore, agency officials acknowledged the program requirements
that would have been fulfilled by the discontinued ASPs remain valid,
including screening rail cars, airport cargo, and cargo at seaport
terminals, but the agency has no current plans for how such screening
will be accomplished. These officials told us the technology to
accomplish these requirements likely will not be ASP monitors. We
believe a comprehensive estimate of the cost to provide radiation
detection equipment for U.S. ports of entry should account for meeting
these objectives, even if DNDO decides that ASP technology is not
suited to them.
At our June 2008 meeting, DNDO agreed to update its project execution
plan, so that we could better estimate the costs of the agency's
current plans. In addition, in light of the fact that DNDO has decided
to procure and deploy only the standard cargo ASP monitor, we agreed to
use DNDO's revised data to re-estimate the standard cargo portal's life-
cycle costs. Furthermore, DNDO also agreed to provide its own updated
estimate of the standard portal's life cycle costs, and to meet with us
in mid-August 2008 to reconcile our two estimates. However, DNDO
delivered neither the promised revised project execution plan nor a
revised cost estimate. Instead, in July 2008, the agency provided a 1-
page spreadsheet of summary information outlining DNDO's plans to buy
and deploy portal monitors--ASPs and PVTs--for the 7-year period 2008
through 2014. DNDO's summary data indicate that during this time period
the agency plans to deploy 717 ASPs and 1,005 PVTs. The summary data do
not provide the breadth and depth of information needed to generate
detailed and fully documented cost estimates. Furthermore, according to
subsequent discussions with a senior DNDO official, if ongoing tests
indicate the ASPs' performance warrants it, the agency may speed its
deployment of ASPs over the next few months. In our view, the frequent
changes in deployment plans, and the lack of available cost
documentation, raises concerns about the overall management of the
radiation portal monitor project, and whether it is guided by a sound
and stable strategy. Despite DNDO's failure to provide revised detailed
program data, we used the agency's summary data to perform a more
limited cost estimate for only the standard cargo portal. The details
of that analysis are presented in enclosure IV. In general, we found
that from 2008 to 2017 the total program cost for buying and deploying
standard cargo portals would likely be about $2 billion, but could
range from about $1.7 billion to $2.3 billion.
Conclusions:
Although combating nuclear smuggling is one of the nation's highest
priorities and the deployment of radiation detection portal monitors is
one of our main weapons in that struggle, DNDO has not yet provided
Congress the full scope or costs of the radiation portal monitor
program. Our estimate of $3.1 billion is based on the cost to implement
the program, as specified in the agency's September 2006 official
program planning documentation. However, in maintaining that these
plans no longer reflect the agency's goals and objectives for deploying
portal monitors at the nation's ports of entry, and that agency plans
currently include only the standard cargo portal, senior DNDO officials
acknowledge a deployment program that is dramatically different in
scope than the one presented to and approved by the Congress. Program
officials now state the program includes only the standard cargo ASP--
a significant reduction in planned ASP equipment. However, DNDO
officials agreed that while the program requirements to screen rail
cars and extra-wide trucks have not been eliminated, the agency has no
current plans for how such screening will be accomplished. These
officials told us that the technology to accomplish these requirements
likely will not be ASP monitors. Regardless, there will be costs
associated with whatever technology DNDO adopts and those costs should
be included in the agency's cost estimate--as ideally DNDO would report
on the total costs to equip U.S. ports of entry with radiation
detection capability, regardless of the technology used.
If DNDO does attempt to implement its authorized 2006 project execution
plan, we anticipate a $1.6 billion shortfall--$753 million for fiscal
years 2008 through 2012, with an additional $833 million needed to
complete the program in fiscal years 2013 through 2017. This represents
additional financial risk for the U.S. government over the life of the
program. DNDO will have to make up this shortfall somehow, either by
returning to the Congress for additional funding, or by cutting
expenses in other parts of the program, which may reduce the efficacy
of the program. DNDO's decision to eliminate rail, extra-wide, and
mobile ASPs from its program plans may be an effort to trim costs. In
either case, the Congress will not be getting the radiation portal
monitor program it initially approved. The Congress should have full
knowledge and complete understanding of DNDO's deployment strategy and
the cost-benefit trade-offs inherent in DNDO's portal monitor decisions
before approving additional program funds.
Recommendations for Executive Action:
We recommend the Secretary of Homeland Security direct the Director of
DNDO to take the following three actions:
* Work with the Commissioner of CBP to update the projection execution
plan to guide the entire radiation detection program at U.S. ports of
entry. The new project execution plan should be based on documented
requirements, and it should provide the agencies a flexible roadmap to
acquiring, deploying, and using the most appropriate and cost-effective
equipment available.
* Revise DNDO's estimate of the program's cost and ensure that the
estimate considers all the costs--design and development, sustainment,
maintenance, deployment, and procurement--associated with its project
execution plan.
* Communicate this revised estimate to the Congress so that it is fully
apprised of the program's scope and funding requirements.
Agency Comments and Our Evaluation:
We provided DHS with a draft of our report for its review and comment.
In its comments, the Department agreed with our recommendations and
part of our conclusions, but strongly disagreed with other parts of our
conclusions. DHS's comments, and our responses, are provided below.
DHS commented that the introductory pages of the draft report did not
always place DNDO in its proper historical context and that the extent
of growth in the portal monitor program was not properly attributed.
Our draft report clearly stated that CBP managed the portal monitor
program prior to the creation of DNDO. Nonetheless, we revised one
paragraph of the final report to better clarify that DNDO was not
managing the radiation portal monitor project prior to 2005.
DHS also commented that we used inconsistent terms in referring to
"cost" and that as a result it is impossible to verify whether our
assessments and conclusions are valid, or to conduct an "apples to
apples" comparison. We disagree with DHS's assertion that we used
inconsistent terms for "cost" and, as a result, it was impossible to
verify our conclusions or to conduct a true "apples to apples"
comparison. Our use of the various terms related to cost is accurate
and consistent with the technical vocabulary of cost estimating. Even
so, we made minor changes to the report to make the terminology more
uniform and improve its readability. With regard to DNDO's comment
concerning assessing the validity of our conclusions, in March 2008, we
provided DNDO a 5-hour briefing, with highly detailed briefing slides,
which described our cost estimating methods, data, and preliminary
results. At the conclusion of that meeting, we gave DNDO our slides and
all the information we collected to that point, including the cost
estimating models, spreadsheets, and data ready for the agency to use.
Furthermore, according to a DNDO document (dated May 29, 2008) that
summarized agency efforts to develop its own life-cycle cost estimate,
DNDO used our cost estimate as its initial baseline model in starting
to develop its own cost estimate. In fact, the structure of DNDO's
estimate was, according to this document, being "Based on GAO ICE
[independent cost estimate]." After providing DNDO with such extensive
information and documentation, and having it use this information in
support of the development of its own cost estimates, we believe that
DNDO had all the data it needed to assess the validity of our analysis
and conclusions.
DHS commented that comparing our cost estimate to DNDO's is misleading
because we included operation and maintenance costs--incurred by CBP--
in our analysis. We disagree that the draft report is misleading when
it compares DNDO and GAO life-cycle costs. In fact, we believe our
estimate helps clarify the program's true cost, while DNDO's estimate
obscures it. The source for DNDO's $2.1 billion cost estimate is the
agency's most recent OMB Exhibit 300, dated March 25, 2008. According
to OMB's Exhibit 300 instructions for completing multi-agency
investments, agencies should include: (a) the costs associated with the
entire life cycle of the investment; and (b) the funding from the lead
and all partner agencies [emphasis added]. According to OMB, the
purpose of the Exhibit 300 is to provide OMB with the information it
needs to make both quantitative decisions about budgetary resources
consistent with the Administration's program priorities, and
qualitative assessments about whether the agency's programming
processes are consistent with OMB policy and guidance. By omitting
major project costs (such as maintenance costs), DNDO has understated
the life cycle costs of the program and violated the intent of OMB
policy and guidance. The effect of these omissions increases the
probability of misinformed decisions and inadequate budget formulation
for partner agencies. Our independent life-cycle cost estimate is more
comprehensive, and closer to OMB's guidelines, than DNDO's as our
estimate includes a significant portion of CBP's expected expenses.
DHS commented that our analysis included costs for ASP variants that
DNDO and CBP no longer plan to deploy. Further, DHS noted that the 2006
Project Execution Plan--upon which we based our estimate--is outdated
and that consequently our cost estimates are speculative. We disagree
that we have mischaracterized the radiation portal monitor project as
it has been presented to OMB and the Congress.[Footnote 7] DNDO's March
2008 OMB Exhibit 300 states that "RPM/ASP units will be deployed to
screen cargo entering the United States across land crossings,
seaports, rail lines, airports, and other ports of entry." Clear
definition of a program's characteristics is a key component in
developing a high-quality cost estimate. DNDO's March 2008 budget
submission should provide an accurate picture of the agency's vision,
while the Project Execution Plan (PEP) should provide many of the
characteristics as it defines the project objectives, work scope, and
schedules. The PEP was accepted in September 2006 and approved jointly
by CBP and DNDO in January 2007. On multiple occasions, we asked DNDO
to provide updates to that plan that have been endorsed by DNDO's
management:
* In October 2007, agency officials indicated that the PEP dated
September 11, 2006, was the most current version.
* In March 2008, agency officials stated that the 2006 PEP was out of
date. We asked them to provide an updated PEP. Agency officials would
not commit to providing such an update but stated that DNDO is
currently developing an ASP life-cycle estimate, and that the specific
quantity of ASPs procured will be based, in part, on ASP test results.
* In June 2008, agency officials agreed to provide an updated PEP with
detailed information and appropriate management signatures. However,
this information was never provided. Instead, DNDO provided a one-page
summary spreadsheet that listed the number of ASPs and PVTs that would
be purchased over the next 7 years. This spreadsheet did not contain
the detailed information that DNDO promised, nor did it carry the
signature endorsement of any DNDO official.
To date, we have not received any official plan from DNDO that outlines
the technical and programmatic baseline of the program other than the
2006 PEP. We therefore relied on this document to provide programmatic
characteristics. Indeed, had the Secretary certified ASPs last year as
initially planned, the program would have included rail, and the other
ASP variants. However, in response to DNDO's statement that the 2006
PEP no longer reflected DNDO's plans for the program, we generated a
second independent cost estimate based on the simplified one-page
spreadsheet that DNDO provided. It is important to note, however, that
a senior DNDO official told us in August 2008 that the number of ASPs
DNDO plans to buy is subject to dramatic change based on the results of
on-going testing.
DHS noted that our cost estimate included the same developmental costs
for additional ASP variants that were required to develop the 4-panel
standard cargo portal. DHS maintains that in so doing, we overstated
the ASPs' developmental cost. On this point, we advised DNDO in March
2008 that we needed more information to be more precise in the area of
developmental costs. Some of this information would need to come
directly from the contractors developing the ASPs. Despite our request,
and because of limitations DNDO place on the contractors in their
contacts with GAO, we did not have meaningful contacts with contractor
officials. Nonetheless, we estimate that developmental costs for the
additional variants account for only about 3 percent of total life-
cycle costs.
DHS commented that our draft report states that there is a need to
screen personally owned vehicles (POV) with ASP technology when, in
DNDO's opinion, PVTs currently do an adequate job in screening POVs.
While the 2004 version of the PEP included ASPs for POVs, the 2006 PEP
did not. The narrative of our draft report inadvertently included POVs
among the types of ASP variants that were included in the 2006 PEP. We
have revised the narrative of our report to reflect this change.
Importantly, this did not affect our cost estimate because the model
for our life cycle cost estimate properly reflected the plans from the
2006 PEP for using PVTs to screen POVs.
DHS commented that DNDO did not instruct its ASP contractors to refuse
our requests for information and interviews, as was stated in the
attachment to our draft report. DNDO instructed its contractors to
respond to our requests for data through DNDO, and then DNDO would
provide us access to these contractors' documents only in a "reading
room" that was established in DNDO offices. The result of the agency's
instructions to its contractors hindered our work in some respects. For
example, in response to our requests for interviews and documents, two
of the three ASP contractors followed DNDO's instructions and insisted
that we make all such requests through DNDO. One of the contractors
directly provided us with all of the information we requested. As is
our normal practice, we sought to obtain documents directly from the
contractors in order to assure complete reporting of all documents
requested. In addition, we terminated one contractor interview because
of repeated interruptions by DNDO officials. Other contractor
interviews did not take place due to DNDO's insistence that agency
officials also participate in those interviews, which we believed could
have hindered or prevented candid responses to our questions. Although
DNDO's actions at times hindered the collection of data from
contractors, we were nevertheless able to collect sufficient
information to support our reports findings, conclusions, and
recommendations.
Finally, DHS commented that DNDO's cost estimate for deploying only
standard cargo portals--$2.1 billion--is consistent with our estimate
of $1.9 billion. We disagree that DNDO's cost estimate is comparable to
ours. For example, our estimate--which ranges from $1.7 billion to $2.3
billion--is based on different technical and programmatic baseline
parameters than the DNDO estimate and includes costs that DNDO has
omitted (such as maintenance). Furthermore, as of August 22, 2008, DNDO
was unable to provide us with a documented life-cycle estimate to
reconcile against our cost estimate. According to OMB guidance, such
documentation would normally be expected to justify a $2.1 billion
acquisition.
We are sending copies of this correspondence to interested
congressional committees and members, the Secretary of Homeland
Security, and other interested parties. We will also make copies
available upon request. In addition, this correspondence will be
available at no charge on GAO's web site at [hyperlink,
http://www.gao.gov]. Should you or your staffs have any questions,
please contact me at (202) 512-3841 or by e-mail at aloisee@gao.gov.
Contact points for our Office of Congressional Relations and Public
Affairs may be found on the last page of this correspondence. Key
contributors to this report include Jennifer Echard, Brian Octeau,
Karen Richey, Benjamin Shouse, Eugene Wisnoski, and Ned Woodward.
Signed by:
Gene Aloise:
Director, Natural Resources and Environment:
Enclosures - 5:
List of Requesters:
The Honorable Joseph I. Lieberman:
Chairman:
Committee on Homeland Security and Governmental Affairs:
United States Senate:
The Honorable Henry A. Waxman:
Chairman:
Committee on Oversight and Government Reform:
House of Representatives:
The Honorable Bennie G. Thompson:
Chairman:
The Honorable Peter T. King:
Ranking Member:
Committee on Homeland Security:
House of Representatives:
The Honorable James R. Langevin:
Chairman:
The Honorable Michael T. McCaul:
Ranking Member:
Subcommittee on Emerging Threats, Cybersecurity, and Science and
Technology:
Committee on Homeland Security:
House of Representatives:
The Honorable Bart Gordon:
Chairman:
Committee on Science and Technology:
House of Representatives:
The Honorable Charles E. Schumer:
United States Senate:
[End of section]
Enclosure I: Slide Presentation:
GAO‘s Independent Cost Estimate of DNDO‘s Radiation Portal Monitor
Program:
Interim Briefing for Congressional Requesters:
Agenda:
* Objectives:
* Summary:
* DNDO Baseline:
* Point Estimates and Uncertainty:
* Evaluation of DNDO Estimates:
* Scope & Methods:
Figure: Photograph of portal monitors in use at the U.S. POE in Blaine,
WA.
Source: GAO.
[End of figure]
[End of section]
Objectives:
In a May 2007 letter to the Comptroller General, you asked us to review
the:
(1) projected costs to implement DNDO‘s project execution plan for the
radiation portal monitor program, which includes the costs to develop,
procure, deploy, operate, and maintain ASPs and other RPMs over a 10-
year life cycle, and the;
(2) validity of DNDO‘s projections given what is known about the costs
to procure, install, and operate radiation detection equipment.
Summary:
* DNDO‘s Program to Deploy Radiation Detection Portal Monitors at U.S.
Ports of Entry Is Likely to Cost About $3 Billion or More:
- The likely cost will be about $3.1 billion, but could range between
$2.6 billion and $3.8 billion.
- Our estimate is based on the anticipated implementation costs of
DNDO‘s most recent plans.
* DNDO‘s Cost Estimates Are Unreliable and Could Result in Significant
Cost Overruns:
- DNDO‘s estimate of $2.0billion fails to take into account several
major cost elements.
- DNDO‘s estimate includes only 8 years of life cycle costs.
- DNDO did not document its cost estimating approach.
Radiation Portal Monitor Program Baseline: Planned Deployments of ASP
Equipment:
* ASP configurations are consistent with those proposed and contracted
for by DNDO in July 2006, and contained in the PEP.
Table:
Type of ASP Planned: Standard Cargo Portal (NaI);
Number Planned: 717.
Type of ASP Planned: Standard Cargo Portal (HPGe);
Number Planned: 9.
Type of ASP Planned: Wide Cargo;
Number Planned: 39.
Type of ASP Planned: Rail;
Number Planned: 51.
Type of ASP Planned: Mobile;
Number Planned: 56.
Type of ASP Planned: Mobile RIID (MRIID);
Number Planned: 162.
Total ASP Systems in 2006 PEP:
Number Planned: 1,034.
[End of table]
Radiation Portal Monitor Program Baseline: Planned Deployments of PVT
Equipment:
* PVT configurations are consistent with those contained in the PEP.
Table:
Type of PVT Planned: 1 Panel;
Number Planned: 3.
Type of PVT Planned: 2 Panel;
Number Planned: 338.
Type of PVT Planned: 4 Panel;
Number Planned: 367.
Type of PVT Planned: 8 Panel;
Number Planned: 23.
Type of PVT Planned: 12 Panel;
Number Planned: 1.
Type of PVT Planned: 4 Panel conversion to 2 Panel;
Number Planned: 184.
Type of PVT Planned: 4 Panel refurbishment;
Number Planned: 126.
Type of PVT Planned: 8 Panel conversion to 4 Panel;
Number Planned: 16.
Type of PVT Planned: 2 Panel Mobile;
Number Planned: 26.
Total PVT Systems in 2006 PEP:
Number Planned: 1,084.
[End of table]
Figure: GAO‘s Estimates of Program Costs:
[Refer to PDF for image]
This figure is a pie-chart depicting the following data:
GAO‘s Estimates of Program Costs, Summary (dollars in millions):
Maintenance: $999.2;
Deployment: $689.1;
Procurement: $603.7;
Design and Develop: $464.5;
Sustainment: $364.9.
Total cost: $3.1 billion, with a minimum estimate of $2.6 billion and a
maximum of $3.8 billion. In constant fiscal year 2008 dollars.
[End of figure]
Figure: ASP Unit Cost Estimates for Standard Cargo Portals (in
thousands, constant fiscal 2008 dollars):
[Refer to PDF for image]
This figure is a multiple vertical bar graph depicting the following
data:
PVT:
DNDO: $994;
GAO: $308.
ASP:
DNDO: $658;
GAO: $822.
[End of figure]
GAO‘s Evaluation of DNDO‘s Program Costs: DNDO‘s Cost Estimates Are
Unreliable:
DNDO‘s current baseline cost estimate omits major project costs. For
example:
* It only includes estimates for the ASP standard cargo portal, even
though:
- The PEP and DHS‘s BY2009 OMB Exhibit 300 call for ASP deployments at
land border crossings, as well as at seaports, rail lines, and other
ports of entry.
- DHS‘s OMB Exhibit 300 requests funding for about 10,400 NaI crystals,
enough to build the standard cargo, rail, extra-wide cargo, and mobile
ASPs (less MRIIDs) called for in DNDO‘s plans (see slide 5, infra).
* Further, DNDO‘s cost estimate contains only 8 years of life cycle
costs, despite:
- The sodium crystal maker estimates that its crystal should last about
10 years, and;
- DHS guidance states that a project‘s life cycle should be estimated
over the period of time during which equipment will remain usable.
* Finally, DNDO‘s baseline cost estimate does not include the ASPs‘
complete life cycle costs.
- It omits cost estimates for maintenance and sustainment.
- These are substantial costs”$999 million and $365 million,
respectively”in our estimate.
DNDO did not document its cost estimating methodology.
* DNDO was unable to provide detailed documentation of the life cycle
costs used in its estimate, even though such documentation is
necessary, according to OMB and DHS guidelines, to establish the basis
of the agency‘s estimates and to provide assurances that the estimates
are credible.
* DHS‘s OMB Exhibit 300 indicates the ASP program‘s cost variance is
greater than 10 percent, but does not provide the basis for the
variance, or corrective actions taken.
GAO‘s Evaluation of DNDO‘s Program Costs: Omissions from DNDO Estimates
Could Lead to Cost Overruns:
GAO‘s independent cost estimate includes:
* 10-year life cycle;
* All variations of ASP portals;
* All cost categories, i.e., development, deployment, procurement,
maintenance, and sustainment.
DNDO‘s ASP baseline cost estimate includes:
* 8-year life cycle;
* Only the standard ASP cargo portals;
* Three of the 5 cost categories, i.e., development, deployment, and
procurement.
GAO‘s Evaluation of DNDO‘s Program Costs: DNDO‘s Estimates Could Result
in Significant Cost Overruns:
The differences between GAO and DNDO cost estimates suggests a
potential cost overrun that could total about $1.1 billion by fiscal
year 2017.
Figure: In thousands, constant fiscal year 2008 dollars:
GAO ICE:
Design and Development: approximately $425,000;
Procurement: approximately $600,000;
Deployment: approximately $700,000;
Maintenance: approximately $1,000,000;
Sustainment: approximately $375,000;
Total: $3.1 billion.
DNDO Estimate:
Design and Development: approximately $425,000;
Procurement: approximately $775,000;
Deployment: approximately $750,000.
Total: $2.0 billion.
Note: DNDO BY09 estimate allocation based on details provided within
the BY08 OMB Exhibit 300.
[End of figure]
Even when GAO ’normalized“ the cost data, i.e., removed the maintenance
and sustainment factors from our estimate, significant differences
remained.
The largest differences occur in fiscal years 2008, 2009, and 2010,
when according to the PEP, DNDO would be buying large numbers of ASPs.
Figure: Comparison of GAO and DNDO ’Normalized“ Costs: Includes only
design, procurement, and deployment (in constant, fiscal year 2008
dollars):
[Refer to PDF for image]
This figure is a multiple line graph depicting the following
approximated data:
Year: FY 2007;
DNDO: $320;
GAO: $150.
Year: FY 2008;
DNDO: $140;
GAO: $375.
Year: FY 2009;
DNDO: $210;
GAO: $360.
Year: FY 2010;
DNDO: $275;
GAO: $350.
Year: FY: 2011;
DNDO: $440;
GAO: $210.
Year: FY 2012;
DNDO: $250;
GAO: $140.
Year: FY 2013;
DNDO: $220;
GAO: $45.
[End of figure]
For the 5 years that overlap the GAO and DNDO cost studies”fiscal years
2008 to 2012”we estimate a $753 million budget shortfall.
Additionally, we estimate that DNDO will require another $833 million
to complete the entire life cycle from fiscal year 2013 through fiscal
year 2017.
ASP contracts have already experienced unfavorable cost and schedule
variances since DNDO awarded the contracts.
* Earned value management analysis indicates that one ASP contractor is
about 13 percent ($1.6 million) over budget and 3 percent behind
schedule. Another ASP contractor is about 25 percent ($1.0 million)
over budget and 23 percent behind schedule.
These unfavorable variances are not likely to improve, but quite likely
will worsen, over the course of the ASP contracts.
Conclusions:
* DNDO has not yet provided the full cost of the radiation portal
monitor project to the Congress.
* The anticipated $1.6 billion shortfall we estimate represents
additional financial risk that the U.S. government will have to
confront over the life of the program.
* DNDO will have to make up this shortfall somehow, either by returning
to the Congress for additional funding, or by cutting expenses in other
parts of the program, which may reduce the efficacy of the ASP systems.
* The Congress should have full knowledge and complete understanding of
the cost-benefit trade-offs inherent in DNDO‘s decisions.
Scope and Methods: GAO‘s Approach to Estimating Costs:
* Establish DNDO‘s technical & programmatic baseline:
- Consult DNDO and PNNL officials;
- Review DHS, CBP, DOE, and Congressional documents;
- Extract historical cost, and technical/program data.
* Develop cost estimate:
- Develop estimating methodology;
- Generate point estimate;
- Estimate probability distributions;
- Perform uncertainty analysis.
* Documentation, Validation, and Verification:
- Trace and verify all data and analyses to GAO standards;
- Align GAO‘s independent cost estimate with DNDO‘s budget and ASP
baseline;
- Reference all data and inputs to source documentation.
* Align GAO‘s independent cost estimate with DNDO‘s budget and baseline
estimates and identify major differences;
* Review and analyze DNDO documentation;
* Present GAO‘s cost model and documentation to DNDO for review and
comment;
* Respond to all DNDO comments.
Scope and Methods: GAO‘s Assumptions:
GAO‘s estimate of program costs:
* Includes fiscal years 2007 through 2017;
* Is consistent with DNDO‘s most recent Project Execution Plan”the
agency‘s detailed plan for implementing the radiation portal monitor
program;
* Includes all program costs associated with development, procurement,
deployment, maintenance, and sustainment, regardless of funds‘ origin;
* Does not include Port Radiation, Inspection, Detection and Evaluation
system or operational costs;
* Are estimated in total year and fiscal year 2008 constant dollars;
* Includes an escalation rate of 3 percent per year.
Scope and Methods: Limitations on GAO‘s Analysis:
* DNDO declined to provide any ’pre-decisional“ information, thus the
estimate could be inconsistent with DNDO‘s latest strategies and data.
* DNDO instructed its ASP contractors to refuse GAO requests for
interviews and data, thus GAO collected most contractor data through
DNDO.
* GAO‘s cost model varied according to work breakdown structure
element, and was dependent upon:
- Data availability, and;
- Granularity of available data.
Figure: Scope and Methods: GAO‘s Cost Model Architecture:
[Refer to PDF for image]
This figure is an illustration of GAO‘s Cost Model Architecture.
[End of figure]
[End of enclosure]
Enclosure II: Scope and Methodology:
Our approach to generating an independent cost estimate of the
radiation portal monitor project (RPMP) was to follow the twelve step
process outlined within the GAO Cost Assessment Guide. That process,
outlined below was followed in generating independent cost estimates
for the total radiation portal monitor project, as well as for the
limited cost estimate for only the standard cargo portal.
Step 1: Define Estimate's Purpose:
The purpose of the estimate was to satisfy the Congressional requests
to GAO pertaining to life cycle costs.
Step 2: Develop Estimating Plan:
Upon receipt of the Congressional request, we identified resources and
developed a design matrix that included identification of information
required and sources, and the proposed scope and methodology for
addressing the Congressional requests. Additionally, in accordance with
Generally Accepted Government Auditing Standards (GAGAS), we planned an
entrance conference with DNDO, issuance of data collection instruments,
and interviews with program officials.
Step 3: Define Program Characteristics:
We collected information directly from DNDO that enabled us to define
the program's technical and programmatic baseline. Specifically, the
program's project execution plan (PEP) was provided to us, outlining
the program's time-phased procurement quantity requirements, and
detailed deployment schedules by port-of-entry. Additionally, we
collected contract documents containing contractor statements of work
(SOW) and system specification documents.
Step 4: Determine Estimating Approach:
We developed an estimating structure in the form of a work breakdown
structure (WBS) reflecting the radiation portal monitor project
outlined in DNDO's project execution plan as well as other program
documents reviewed, such as contractor cost reports. For each WBS
element, the most appropriate estimating approach was identified
according to data availability and credibility. Estimating approaches
applied included analogy, engineering build-up, expert opinion, and
parametric methods.
Step 5: Identify Ground Rules and Assumptions:
Estimating ground rules and assumptions were identified and documented.
Assumptions were minimized, relying instead on real data when possible.
Step 6: Obtain Data:
Cost, technical, and programmatic data was solicited and collected from
a number of sources through the use of data collection instruments
(DCIs). DCI's were provided to DNDO, PNNL, Raytheon, Thermo Eberline,
and:
Canberra. Additional data was collected through interviews and
correspondence with program officials.
Step 7: Develop Point Estimate:
Data collected was thoroughly analyzed, resulting in the development of
cost estimating methodology. The methodology was integrated into a
comprehensive cost model using Microsoft ® Excel, resulting in a
detailed and time-phased point estimate.
Step 8: Conduct Sensitivity Analysis:
A comprehensive sensitivity analysis was conducted on the cost model
using Crystal Ball ® software. The contribution to forecast variance
and rank correlation coefficients were examined for each cost model
input, enabling the development of spider charts and tornado charts
which aid in the identification of key cost drivers.
Step 9: Conduct Risk and Uncertainty Analysis:
Risk and uncertainty analysis was conducted in order to transform the
static point estimate into a probabilistic range based on the risk and
uncertainty inherent in the model input variables. For each model input
variable, underlying data used for the basis of estimate was examined
to establish an uncertainty range and probability distribution type.
These parameters were entered into the Crystal Ball ® software which
was then used to simulate 10,000 estimate trials using Latin Hypercube
sampling.[Footnote 8] The simulation resulted in establishing
probability distributions about the estimated values, enabling the
identification of the confidence level of the point estimate.
Step 10: Document the Estimate:
Detailed documentation was generated that provides the cost estimate
results, as well as ground rules and assumptions, and all underlying
cost estimating methodology. The cost model was thoroughly referenced
to source documents, traced and verified in accordance with GAGAS.
Additional auto-generated documentation consisting of risk and
uncertainty input parameters, forecasts, and statistics was produced.
Step 11: Present Estimate to Management for Approval:
The resulting cost estimates were presented to various levels of GAO
management, as well as to Congressional staffers. Additionally, GAO
presented the cost estimate and underlying methodology in detail to
DNDO in March 2008.
Step 12: Update the Estimate to Reflect Actual Costs and Changes:
Upon presenting the cost estimate and underlying methodology in detail
to GAO, DNDO provided comment, questions, and some additional data
which became the basis for some minor cost estimate revisions. The
revised cost estimate was integrated into a subsequent briefing to DNDO
in June of 2008. DNDO then requested that GAO prepare a cost estimate
excursion that addresses just the 4 panel PVT and ASP variants. That
excursion was then prepared by GAO and included in this report.
Because DNDO instructed its contractors to respond to our requests for
data through DNDO, we collected most contractor data through DNDO. DNDO
also declined to provide any "pre-decisional" information or produce a
revised version of its 2006 PEP. As a result, our estimate may not be
consistent with DNDO's latest procurement strategy. While these actions
limited the data that were available to us, we were nonetheless able to
develop sufficient information to support our findings, conclusions,
and recommendations.
We conducted this performance audit from October 2007 through September
2008 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on out audit objectives.
We believe that the evidence obtained provides a reasonable basis for
our findings and conclusions based on our audit objectives.
[End of enclosure]
Enclosure III: Radiation Portal Monitor Program Independent Cost
Estimate:
Our independent cost estimate for the radiation portal monitor project
was structured according to the level of program definition detail
available within official program documents, contract documents, and
historical RPM program data. In accordance with GAO's Cost Assessment
Guide, a product-oriented work breakdown structure[Footnote 9] (WBS)
tailored by program phase was generated at this level of detail. The
following table provides the first 3 indenture levels of the WBS:
[Footnote 10]
Table:
Element: 1.0;
Description: Total RPM Program.
Element: 1.1;
Description: Design and Development.
Element: 1.1.1;
Description: ASP 4 Panel, Medium Resolution (Raytheon).
Element: 1.1.2;
Description: ASP 4 Panel, Medium Resolution (Thermo Eberline).
Element: 1.1.3;
Description: ASP 4 Panel, High Resolution (Canberra).
Element: 1.1.4;
Description: ASP 8 Panel, Medium Resolution.
Element: 1.1.5;
Description: ASP 12 Panel, Medium Resolution.
Element: 1.1.6;
Description: ASP 2 Panel, Mobile, Medium Resolution.
Element: 1.1.7;
Description: Multi-variant Engineering and Spiral Development.
Element: 1.2;
Description: Procurement.
Element: 1.2.1;
Description: RPM Procurement.
Element: 1.2.2;
Description: MRIID Procurement.
Element: 1.2.3;
Description: Computer Procurement.
Element: 1.2.4;
Description: Spares Procurement.
Element: 1.3;
Description: Deployment.
Element: 1.3.1;
Description: Integration/Installation/Site Design & Construction.
Element: 1.3.2;
Description: Initial Training and Operations Support.
Element: 1.3.3;
Description: Deployment Integration & Coordination.
Element: 1.3.4;
Description: Technical Support.
Element: 1.3.5;
Description: Project Operations.
Element: 1.3.6;
Description: GSA Construction.
Element: 1.4;
Description: Maintenance.
Element: 1.4.1;
Description: Maintenance (PVT Portal Monitors).
Element: 1.4.2;
Description: Maintenance (ASP Portal Monitors).
Element: 1.4.3;
Description: Maintenance (Computers).
Element: 1.5;
Description: Sustainment.
Element: 1.5.1;
Description: Program Office (In-House).
Element: 1.5.2;
Description: Program Office (In-House Contractor Support).
[End of table]
A cost estimating methodology was developed for each lowest level WBS
element level based on historical data, contract data, and other
official program data. Developing cost estimating methodology at the
lowest levels facilitates cost estimate visibility, identification of
cost drivers, and clarification of interrelationships among cost
elements. For example, under element number 1.2.1 RPM Procurement, cost
estimating methodologies were developed at the major component level
for 14 unique RPM variants. Those variants span multiple configurations
(fixed vs. mobile; 1, 2, 4, 8, and 12 panel configurations) and
multiple technologies (polyvinyl toluene (PVT), sodium iodide (NaI),
and high purity germanium (HPGe)). The resulting estimating framework
for element 1.2.1 RPM Procurement therefore allows for the
identification and categorization of cost by configuration or
technology type, and enables sensitivity analysis for identifying and
examining the effect of changing key model assumptions. The primary
cost estimating methods applied were analogy[Footnote 11] and
parametric.[Footnote 12]
Cost estimates generated at the lowest WBS levels were summed up to
each successive level of indenture to generate the $3.1 billion point
estimate. Level 2 of the WBS provides a breakdown by program phase as
illustrated by figure 1.
Figure 1: Radiation Portal Monitor Project Costs by Program Phases:
[Refer to PDF for image]
This figure is a pie-chart depicting the following data:
Radiation Portal Monitor Project Costs by Program Phases:
Maintenance: $999,242,368.91 (32%);
Deployment: $689,107,870.32 (22%);
Procurement: $603,678,392.66 (19%);
Design and Development: $464,477,572.69 (15%);
Sustainment: $364,988,656.21 (12%).
Source: GAO analysis.
[End of figure]
The $3.1 billion point estimate was translated into a cost estimate
range through the application of cost risk and uncertainty analysis.
[Footnote 13] The data underlying the cost model inputs was examined
for variability, thereby establishing uncertainty bounds and
probability distributions for most of the WBS elements. These
distributions were combined into a simulation model, in which the
distributions for each cost element were treated as individual
populations from which random samples were taken. The model was
recalculated 10,000 times by repeatedly drawing values from each WBS
distribution using a Latin Hypercube Sampling (LHS) method.[Footnote
14]
The resulting risk and uncertainty cumulative distribution function
illustrates the probability range of the independent cost estimate
along with the relative placement of the $3.1 billion point estimate on
the distribution. Figure 2 shows this distribution.
Figure 2: Risk and Uncertainty Range of the Radiation Portal Monitor
Project:
[Refer to PDF for image]
This figure is a chart depicting the risk and uncertainty range of the
radiation portal monitor project. Frequency is depicted from 0 to
10,000 and probability is indicated from 0 to 1. A minimum of $2.616
billion and a maximum of $3.804 for the project is indicated. The
following specific data is depicted:
50% probability level: $3.116 billion;
Point estimate (51%): $3.121 billion;
70% probability level: $3.196 billion.
Source: GAO analysis.
[End of figure]
The independent cost estimate ranges from a low of $2.6 billion to a
high of $3.8 billion. The $3.1 billion point estimate approximately
equates to a 50 percent confidence level. An increase to the point
estimate by approximately $0.1 billion results in a 70 percent
confidence level.
Program requirements collected from DNDO such as procurement quantity
profiles, development plans and deployment schedules enabled a cost
estimate profile to be developed. The GAO life cycle cost estimate
profile is provided in figure 3 along with the corresponding DNDO
profile extracted from the program's budget year 2009 Exhibit 300
submission.
Figure 3: Funding Shortfall in the Radiation Portal Monitor Project,
Fiscal Years 2007 to 2017 (dollars in millions):
[Refer to PDF for image]
This figure is a multiple line graph depicting the following data:
Fiscal year: 2007;
DNDO: $302;
GAO ICE: $188.
Fiscal year: 2008;
DNDO: $145;
GAO ICE: $466.
Fiscal year: 2009;
DNDO: $216;
GAO ICE: $488.
Fiscal year: 2010;
DNDO: $289;
GAO ICE: $511.
Fiscal year: 2011;
DNDO: $474;
GAO ICE: $387.
Fiscal year: 2012;
DNDO: $285;
GAO ICE: $330.
Fiscal year: 2013;
DNDO: $246;
GAO ICE: $225.
Fiscal year: 2014;
GAO ICE: $210.
Fiscal year: 2015;
GAO ICE: $212.
Fiscal year: 2016;
GAO ICE: $214.
Fiscal year: 2017;
GAO ICE: $217.
Source: GAO analysis.
[End of figure]
The chart illustrates a $753 million shortfall during the fiscal years
2008 through 2012. The DNDO estimate does not include any costs beyond
fiscal year 2013, contributing to an additional shortfall of $833
million to complete entire life cycle of the program from fiscal year
2013 to 2017.
[End of enclosure]
Enclosure IV: Radiation Portal Monitor Program-GAO's Limited Cost
Estimate for Only the Standard Cargo Portal.
Our limited independent cost estimate for DNDO's radiation portal
monitor project that only included standard cargo portals was
structured according to the level of program definition detail
available within official program documents, contract documents, and
historical radiation portal monitor project data. In accordance with
GAO's Cost Assessment Guide, a product-oriented Work Breakdown
Structure[Footnote 15] tailored by program phase was generated at this
level of detail. The following table provides the first 3 indenture
levels of the WBS:[Footnote 16]
Element: 1.0;
Description: Total RPM Program.
Element: 1.1;
Description: Design and Development.
Element: 1.1.1;
Description: ASP 4 Panel, Medium Resolution (Raytheon).
Element: 1.1.2;
Description: ASP 4 Panel, Medium Resolution (Thermo Eberline).
Element: 1.1.3;
Description: Multi-variant Engineering and Spiral Development.
Element: 1.2;
Description: Procurement.
Element: 1.2.1;
Description: RPM Procurement.
Element: 1.2.2;
Description: Computer Procurement.
Element: 1.2.3;
Description: Spares Procurement.
Element: 1.3;
Description: Deployment.
Element: 1.3.1;
Description: Integration/Installation/Site Design & Construction.
Element: 1.3.2;
Description: Initial Training and Operations Support.
Element: 1.3.3;
Description: Deployment Integration & Coordination.
Element: 1.3.4;
Description: Technical Support.
Element: 1.3.5;
Description: Project Operations.
Element: 1.3.6;
Description: GSA Construction.
Element: 1.4;
Description: Maintenance.
Element: 1.4.1;
Description: Maintenance (PVT Portal Monitors).
Element: 1.4.2;
Description: Maintenance (ASP Portal Monitors).
Element: 1.4.3;
Description: Maintenance (Computers).
Element: 1.5;
Description: Sustainment.
Element: 1.5.1;
Description: Program Office (In-House).
Element: 1.5.2;
Description: Program Office (In-House Contractor Support).
[End of table]
A cost estimating methodology was developed for each lowest level WBS
element level based on historical data, contract data, and other
official program data. Developing cost estimating methodology at the
lowest levels facilitates cost estimate visibility, identification of
cost drivers, and clarification of interrelationships among cost
elements. For example, under element number 1.2.1 RPM Procurement, cost
estimating methodologies were developed at the major component level
for 2 unique RPM variants. Those variants span multiple technologies
(polyvinyl toluene (PVT), sodium iodide (NaI)). The resulting
estimating framework for element 1.2.1 RPM Procurement therefore allows
for the identification and categorization of cost by technology type,
and enables sensitivity analysis for identifying and examining the
effect of changing key model assumptions. The primary cost estimating
methods applied were analogy[Footnote 17] and parametric.[Footnote 18]
Cost estimates generated at the lowest WBS levels were summed up to
each successive level of indenture to generate the $1.9 billion point
estimate. Level 2 of the WBS provides a breakdown by program phase as
illustrated by figure 4.
Figure 4: Radiation Portal Monitor Project - Limited Cost Estimate for
Only the Standard Cargo Portal, Costs by Program Phases:
[Refer to PDF for image]
This figure is a pie-chart depicting the following data:
Radiation Portal Monitor Project - Limited Cost Estimate for Only the
Standard Cargo Portal, Costs by Program Phases:
Deployment: $761,370,624 (39%);
Maintenance: $393,719,522 (20%);
Procurement: $348,335,548 (18%);
Sustainment: $337,224,628 (18%);
Design and Development: $97,970,733 (5%).
Source: GAO analysis.
[End of figure]
The $1.9 billion point estimate was translated into a cost estimate
range through the application of cost risk and uncertainty analysis.
[Footnote 19] The data underlying the cost model inputs was examined
for variability, thereby establishing uncertainty bounds and
probability distributions for most of the WBS elements. These
distributions were combined into a simulation model, in which the
distributions for each cost element were treated as individual
populations from which random samples were taken. The model was
recalculated 10,000 times by repeatedly drawing values from each WBS
distribution using a Latin Hypercube Sampling (LHS) method.[Footnote
20]
The resulting risk and uncertainty cumulative distribution function in
figure 5 illustrates the probability range of the independent cost
estimate along with the relative placement of the $1.9 billion point
estimate on the distribution.
Figure 5: Risk and Uncertainty Range of the Radiation Portal Monitor
Project - Limited Cost Estimate for Only the Standard Cargo Portal:
[Refer to PDF for image]
This figure is a chart depicting the risk and uncertainty range of the
radiation portal monitor project. Frequency is depicted from 0 to
10,000 and probability is indicated from 0 to 1. A minimum of $1.7
billion and a maximum of $2.289 for the project is indicated. The
following specific data is depicted:
50% probability level (Point estimate): $1.939 billion;
70% probability level: $1.976 billion.
Source: GAO analysis.
[End of figure]
The independent cost estimate ranges from a low of $1.7 billion to a
high of $2.3 billion. The $1.9 billion point estimate approximately
equates to a 50 percent confidence level. An increase to the point
estimate by approximately $0.04 billion results in a 70 percent
confidence level.
Program requirements collected from DNDO such as procurement quantity
profiles, development plans and deployment schedules enabled a cost
estimate profile to be developed. The GAO life cycle cost estimate
profile is provided in figure 6 along with the corresponding DNDO
profile extracted from the program's budget year 2009 Exhibit 300
submission.
The DNDO estimate does not include any costs beyond fiscal year 2013,
contributing to a shortfall of $518 million over the entire life cycle
of the program from fiscal year 2008 to 2017.
Figure 6: Funding Shortfall in the Radiation Portal Monitor Project -
Limited Cost Estimate for Only the Standard Cargo Portal, Fiscal Years
2008 to 2017:
[Refer to PDF for image]
This figure is a multiple line graph depicting the following data:
Fiscal year: 2008;
DNDO: $145;
GAO ICE: $221.
Fiscal year: 2009;
DNDO: $216;
GAO ICE: $265.
Fiscal year: 2010;
DNDO: $289;
GAO ICE: $266.
Fiscal year: 2011;
DNDO: $474;
GAO ICE: $255.
Fiscal year: 2012;
DNDO: $285;
GAO ICE: $263.
Fiscal year: 2013;
DNDO: $246;
GAO ICE: $259.
Fiscal year: 2014;
GAO ICE: $208.
Fiscal year: 2015;
GAO ICE: $143.
Fiscal year: 2016;
GAO ICE: $146.
Fiscal year: 2017;
GAO ICE: $148.
Source: GAO analysis.
[End of figure]
[End of enclosure]
Enclosure V:
Comments from the Department of Homeland Security:
U.S. Department of Homeland Security:
Washington, DC 20528:
[hyperlink, http://www.dhs.gov]
September 9, 2008:
Mr. Gene Aloise:
Director, Natural Resources and Environment:
Government Accountability Office:
441 G Street NW:
Washington, DC 20548:
Dear Mr. Aloise:
Re: Draft Report GAO-08-1108R, Combating Nuclear Terrorism: DHS's
Program to Procure and Deploy Advanced Radiation Detection Portal
Monitors Is Likely to Exceed the Department's Previous Cost Estimates.
The Department of Homeland Security (DHS) has reviewed the above-
referenced draft report. Although DHS generally concurs with the
recommendations and with GAO's conclusion that the revised Radiation
Portal Monitor Program (RPMP) is likely to cost about $1.9 billion, or
a range from about $1.7 billion to $2.3 billion, we strongly disagree
with the remaining conclusions in the report.
First, the introductory section from pages 1-5 contains numerous
inaccuracies. In an attempt to show steady cost growth, GAO notes that
DNDO's estimates of cost had "grown" during the period from February
2003 to December 2004. Yet, the Domestic Nuclear Detection Office was
not created until April 2005, and its first OMB-300 was not submitted
until fiscal year 2006. In another attempt to show steady cost growth,
GAO uses inconsistent terms for "cost" such as program cost, total
cost, estimated cost, expected cost, cost, cost estimate, unit cost,
deployment cost, program's costs, independent cost estimate, ASP's
cost, developmental costs, and full life cycle costs. Because of this
imprecise and varying use of the term "cost" throughout this section,
it is impossible to verify whether the GAO's assessment or conclusions
are valid, or to conduct a true "apples to apples" comparison.
Second, the draft report also makes a very misleading comparison when
it compares its estimated life-cycle cost of $3.1 billion for ASP to
the DNDO budget submission of $2.1 billion. These numbers are
essentially comparing "apples to oranges." GAO incorrectly assumes
DNDO's budget submission for ASP should include operation and
maintenance costs when in fact those are costs assumed by Customs and
Border Protection (CBP). These costs are important but would never be
included in a DNDO budget submission. We advised GAO of this during
their March presentation to us and again during the exit conference
held on June 12, 2008; however, GAO continues to make the comparison
anyway.
Third, the GAO cost estimate incorrectly includes costs for ASP
variants that are not presently part of the joint deployment plan
between CBP and DNDO. The 2006 Program Execution Plan (PEP),
acknowledged by all to be outdated, contemplated the potential need for
162 systems called a mobile Radioactive Isotope Identification Device
(RIID). Since that point in time, CBP no longer thinks that such a
system will be required. The 2006 PEP also included 51 systems to
screen international rail cars. It is also doubtful that ASP or any
passive portal technology is a viable technology solution to this
problem. It is not known what type (or cost) of technology might be
used. Therefore, any costs included in the GAO estimate are purely
speculative. We advised GAO of this during their March presentation to
us and again during the exit conference held on June 12, 2008; however,
GAO continues to include these costs.
Fourth, the GAO cost estimate also included the same development costs
for additional variants such as a 2-panel or 8-panel system as was
required to develop the 4-panel cargo system. These systems are just
different configurations of the same portal technology and do not
require extensive development costs. We advised GAO of this during
their March presentation to us and again during the exit conference
held on June 12, 2008; however, GAO continues to include these costs.
Fifth, the GAO report also states that there remains a need to cover
"personally owned vehicle" (POV) sites. This is misleading since POV
sites are currently screened with the existing polyvinyl toluene (PVT)
systems. Since POV sites do not have substantial background suppression
issues or Naturally Occurring Radioactive Material (NORM) traffic, the
current PVT technology has sufficient sensitivity to the threat to meet
mission needs at POV crossings and the next generation ASP technology
is not expected to be necessary. Once again, the GAO was informed of
this discrepancy.
Finally, the GAO report states that "DNDO instructed its ASP
contractors to refuse GAO requests for interviews and data, thus GAO
collected most contractor data through DNDO." As DNDO advised GAO in
March and again in June, this is incorrect. DNDO merely requested the
opportunity to participate in contractor interviews, and that all data
provided by the contractors be provided through DNDO so that DNDO could
ensure that the data was current and accurate. No contractor was
advised to refuse interviews, numerous interviews did take place, and
written responses were provided to GAO questions when interviews could
not be arranged.
DNDO provided to the GAO its current buy plan to cover all commercial
cargo crossings, including land borders and seaports. Using these
numbers and scope, GAO calculated a-range of $1.7 billion to $2.3
billion. Despite the GAO's claims of cost increases, overruns, and
unfunded requirements, DNDO's $2.1 billion budget submission falls
directly within GAO's calculated range. In addition, DNDO is in the
process of developing a life-cycle cost estimate (LCCE) and deployment
plan to support the full-rate production decision Key Decision Point
(KDP-3) and the Secretarial certification.
We will not proceed to full-rate production on Advanced Spectroscopic
Portal (ASP) systems until the Secretary certifies that they have a
significant increase in operational effectiveness over current systems.
The Department has been following a prudent path leading to
certification of ASP systems and a full-rate production decision
Thank you for the opportunity to review and provide comments to the
draft report.
Sincerely,
Signed by:
Jerald E. Levine:
Director, Departmental GAO/OIG Liaison Office:
[End of enclosure]
Footnotes:
[1] P.L. 109-347.
[2] CBP and DNDO, Radiation Portal Monitor Project: Project Execution
Plan (Revision 0), PIET-43741-PM-100 (Richland, Wash.: Sept. 11, 2006).
[3] See GAO, Combating Nuclear Smuggling: DHS's Cost-Benefit Analysis
to Support the Purchase of New Radiation Detection Portal Monitors Was
Not Based on Available Performance Data and Did Not Fully Evaluate All
the Monitors' Costs and Benefits, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-06-133R] (Washington, D.C.: Oct. 17, 2006).
[4] ASP and PVT "standard cargo portals" are 4-panel portal monitors
intended to screen standard commercial trucks.
[5] The ASP panels are comprised mainly of sodium iodide crystals. It
is these crystals that give ASPs the ability to both detect radiation
and identify its isotopes.
[6] DHS, DHS Acquisition Planning Requirements, Appendix A - Chapter
3007 (Washington, D.C.: Oct. 26, 2004). OMB, OMB Circular A-11, Part 7,
Capital Programming Guide (Washington, D.C.: June 2006). GAO, Cost
Assessment Guide: Best Practices for Estimating and Managing Program
Costs--Exposure Draft, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-07-1134SP] (Washington, D.C.: July 2, 2007).
[7] According to OMB Circular A-11, "The Exhibit 300 is designed to
coordinate OMB's collection of agency information for its reports to
the Congress as required by the Federal Acquisition Streamlining Act of
1994 (FASA Title V) and the Clinger-Cohen Act of 1996; to ensure the
business case for investments are made and tied to the mission
statements, long-term goals and objectives, and annual performance
plans developed pursuant to the [Government Performance and Results Act
of 1993]."
[8] Latin Hypercube sampling is a statistical method that ensures the
ensemble of sample points is representative of the real variability.
[9] A Work Breakdown Structure is a hierarchical framework that
reflects the requirements, resources, and tasks that must be
accomplished to develop a program.
[10] The complete WBS extends to five levels of indenture.
[11] Analogy-based methods use actual costs from a similar program.
Those costs may be adjusted to account for technical and/or
programmatic differences to the program being estimated.
[12] Parametric-based methods use statistical relationships developed
between historical costs and program, physical, and performance
characteristics.
[13] Cost risk and uncertainty analysis is a cost-estimating best
practice addressed in many guides and references, including the GAO
Cost Assessment Guide.
[14] Latin Hypercube sampling is a form of stratified sampling that
ensures the distribution function is sampled evenly, thereby reducing
the number of trials necessary for a Monte Carlo simulation to achieve
a more accurate random distribution.
[15] A Work Breakdown Structure is a hierarchical framework that
reflects the requirements, resources, and tasks that must be
accomplished to develop a program.
[16] The complete WBS extends to five levels of indenture.
[17] Analogy-based methods use actual costs from a similar program.
Those costs may be adjusted to account for technical and/or
programmatic differences to the program being estimated.
[18] Parametric-based methods use statistical relationships developed
between historical costs and program, physical, and performance
characteristics.
[19] Cost risk and uncertainty analysis is a cost estimating best
practice addressed in many guides and references, including the GAO
Cost Assessment Guide.
[20] Latin Hypercube sampling is a form of stratified sampling that
ensures the distribution function is sampled evenly, thereby reducing
the number of trials necessary for a Monte Carlo simulation to achieve
a more accurate random distribution.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
