Homeland Security
Summary of Challenges Faced in Targeting Oceangoing Cargo Containers for Inspection Gao ID: GAO-04-557T March 31, 2004After the attacks of September 11, 2001, concerns intensified that terrorists would attempt to smuggle a weapon of mass destruction into the United States. One possible method is to use one of the 7 million cargo containers that arrive at our seaports each year. Addressing the potential threat posed by the movement of oceangoing cargo containers falls to the Department of Homeland Security's U.S. Customs and Border Protection (CBP). Since CBP cannot inspect all arriving cargo containers, it uses a targeting strategy, including an Automated Targeting System. This system targets containers for inspection based on perceived level of risk. In this testimony, GAO summarizes its work on (1) whether the development of CBP's targeting strategy is consistent with recognized key risk management and modeling practices and (2) how well the strategy has been implemented at selected seaports.
CBP has taken steps to address the terrorism risks posed by oceangoing cargo containers, but its strategy neither incorporates all key elements of a risk management framework nor is it entirely consistent with recognized modeling practices. Actions CBP has taken included refining the Automated Targeting System to target cargo containers that are a high risk for terrorism, or other smuggling, for physical inspection. CBP has also implemented national targeting training and sought to improve the quality and timeliness of manifest information, which is one of the inputs for its Automated Targeting System. However, regarding risk management, CPB has not performed a comprehensive set of assessments vital for determining the level of risk for oceangoing cargo containers and the types of responses necessary to mitigate that risk. Regarding recognized modeling practices, CBP has not subjected the Automated Targeting System to adequate external peer review or testing. It has also not fully implemented a process to randomly examine containers in order to test the targeting strategy. Without incorporating all key elements of a risk management framework and recognized modeling practices, CBP cannot be reasonably sure that its targeting strategy provides the best method to protect against weapons of mass destruction entering the United States at its seaports. GAO's visits to selected seaports found that the implementation of CBP's targeting strategy faces a number of challenges. Although port officials said that inspectors were able to inspect all containers designated by the Automated Targeting System as high-risk, GAO's requests for documentation raised concerns about the adequacy of CBP's data to document these inspections. CBP lacks an adequate mechanism to test or certify the competence of students who participate in their national targeting training. Additionally, CBP has not been able to fully address longshoremen's safety concerns related to inspection equipment. Addressing these concerns is important to ensure that cargo inspections are conducted safely and efficiently. Challenges to both the development and the implementation of CBP's targeting strategy, if not addressed, may limit the effectiveness of targeting as a tool to help ensure homeland security.
GAO-04-557T, Homeland Security: Summary of Challenges Faced in Targeting Oceangoing Cargo Containers for Inspection
This is the accessible text file for GAO report number GAO-04-557T
entitled 'Homeland Security: Summary of Challenges Faced in Targeting
Oceangoing Cargo Containers for Inspection' which was released on March
31, 2004.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Testimony before the Subcommittee on Oversight and Investigations,
Committee on Energy and Commerce, House of Representatives:
United States General Accounting Office:
GAO:
For Release on Delivery Expected at 10:00 a.m. EST Wednesday, March 31,
2004:
HOMELAND SECURITY:
Summary of Challenges Faced in Targeting Oceangoing Cargo Containers
for Inspection:
Statement of Richard M. Stana, Director Homeland Security and Justice
Issues:
GAO-04-557T:
GAO Highlights:
Highlights of GAO-04-557T, testimony before the Subcommittee on
Oversight and Investigations, House Committee on Energy and Commerce
Why GAO Did This Study:
After the attacks of September 11, 2001, concerns intensified that
terrorists would attempt to smuggle a weapon of mass destruction into
the United States. One possible method is to use one of the 7 million
cargo containers that arrive at our seaports each year. Addressing the
potential threat posed by the movement of oceangoing cargo containers
falls to the Department of Homeland Security‘s U.S. Customs and Border
Protection (CBP). Since CBP cannot inspect all arriving cargo
containers, it uses a targeting strategy, including an Automated
Targeting System. This system targets containers for inspection based
on perceived level of risk. In this testimony, GAO summarizes its work
on (1) whether the development of CBP‘s targeting strategy is
consistent with recognized key risk management and modeling practices
and (2) how well the strategy has been implemented at selected
seaports.
What GAO Found:
CBP has taken steps to address the terrorism risks posed by oceangoing
cargo containers, but its strategy neither incorporates all key
elements of a risk management framework nor is it entirely consistent
with recognized modeling practices. Actions CBP has taken included
refining the Automated Targeting System to target cargo containers that
are a high risk for terrorism, or other smuggling, for physical
inspection. CBP has also implemented national targeting training and
sought to improve the quality and timeliness of manifest information,
which is one of the inputs for its Automated Targeting System. However,
regarding risk management, CPB has not performed a comprehensive set of
assessments vital for determining the level of risk for oceangoing
cargo containers and the types of responses necessary to mitigate that
risk. Regarding recognized modeling practices, CBP has not subjected
the Automated Targeting System to adequate external peer review or
testing. It has also not fully implemented a process to randomly
examine containers in order to test the targeting strategy. Without
incorporating all key elements of a risk management framework and
recognized modeling practices, CBP cannot be reasonably sure that its
targeting strategy provides the best method to protect against weapons
of mass destruction entering the United States at its seaports.
GAO‘s visits to selected seaports found that the implementation of
CBP‘s targeting strategy faces a number of challenges. Although port
officials said that inspectors were able to inspect all containers
designated by the Automated Targeting System as high-risk, GAO‘s
requests for documentation raised concerns about the adequacy of CBP‘s
data to document these inspections. CBP lacks an adequate mechanism to
test or certify the competence of students who participate in their
national targeting training. Additionally, CBP has not been able to
fully address longshoremen‘s safety concerns related to inspection
equipment. Addressing these concerns is important to ensure that cargo
inspections are conducted safely and efficiently. Challenges to both
the development and the implementation of CBP‘s targeting strategy, if
not addressed, may limit the effectiveness of targeting as a tool to
help ensure homeland security.
What GAO Recommends:
GAO recommends that CBP incorporate all the key elements of a risk
management framework and recognized modeling practices in its targeting
strategy and the Automated Targeting System. GAO also recommends, among
other things, that CBP improve management controls to better implement
the targeting strategy at seaports.
The department cited corrective actions taken or planned to address the
issues GAO identified.
www.gao.gov/cgi-bin/getrpt?GAO-04-557T.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Rich Stana at (202)
512-8777 or StanaR@gao.gov.
[End of section]
Mr. Chairman and Members of the Subcommittee:
I appreciate the opportunity to be here today to provide a summary of
our recent report for you on the Department of Homeland Security's
(DHS) programs to target oceangoing cargo containers for inspection.
This testimony represents a publicly available summary of that report,
which DHS designated as Limited Official Use due to the sensitive and
specific nature of the information it contains. My prepared statement
today also includes appendixes that detail the risk management
framework that we developed and the recognized modeling practices that
we identified to evaluate DHS's program to target oceangoing cargo
containers for inspection.
In the aftermath of the terrorist attacks of September 11, 2001, there
is heightened concern that terrorists may try to smuggle weapons of
mass destruction into a U.S. port using one of the millions of cargo
containers that arrive at our nation's seaports each year. If
terrorists did so and detonated such a weapon (e.g., a nuclear, or
radiological, explosive device) at a seaport, the incident could cause
widespread death and damage to the immediate area, perhaps shut down
seaports nationwide, cost the U.S. economy billions of dollars, and
seriously hamper international trade.
DHS and its U.S. Customs and Border Protection (CBP) are responsible
for addressing the threat posed by terrorist smuggling of weapons in
oceangoing containers. To carry out this responsibility, CBP uses a
targeting strategy, which includes a computerized model called the
Automated Targeting System, to help select (or target) containers for
additional review and/or inspection. Organizations that are involved in
security matters, such as CBP, frequently employ certain risk
management practices, including computer modeling, to help them
prioritize their activities and use of resources. In essence, risk
management is a systematic process to analyze threats, vulnerabilities,
and critical assets (e.g., port facilities) to better support
management decisions.
This statement presents a summary of our latest effort in a series of
GAO reports that evaluate CBP's response to the terrorist
threat.[Footnote 1] Based upon our ongoing assessment of CBP's
targeting strategy for this subcommittee, I will provide a summary of
our findings on (1) whether CBP's development of its targeting strategy
is consistent with recognized risk management and computer modeling
practices and (2) how well the targeting strategy has been implemented
at selected seaports around the country. Our findings are based on
extensive data collection and analysis at CBP, consultations with
experts in terrorism and risk management, visits to six seaports, and
related interviews with federal and local government and private sector
officials responsible for port security and operations. Additional
information on our scope and methodology can be found at the end of
this statement. Our work focused primarily on the targeting system
rather than the sufficiency of inspections at the ports once a
container has been targeted. We conducted our work from January 2003 to
February 2004 in accordance with generally accepted government auditing
standards.
Summary:
While CBP has taken steps to address the terrorism risks posed by
oceangoing cargo containers, its targeting strategy neither
incorporates all key elements of a risk management framework nor is
consistent with certain recognized practices associated with modeling.
To its credit, CBP established the National Targeting Center to serve
as the national focal point for targeting imported cargo and for
distributing periodic intelligence alerts to the ports. CBP has refined
its targeting system, which was originally designed to identify
narcotics contraband, to help identify containers posing potential
terrorist threats for possible physical screening and inspection. It
also instituted a national training program for its personnel that
perform targeting. Further, CBP promulgated regulations aimed at
improving the quality and timeliness of transmitted cargo manifest data
for use in the targeting system. However, while its strategy
incorporates some elements of risk management, CBP has not performed a
comprehensive set of threat, criticality, vulnerability, and risk
assessments that experts said are vital for determining levels of risk
for each container and the types of responses necessary to mitigate
that risk. Regarding recognized modeling practices, CBP has not
subjected the targeting system to external peer review or testing as
recommended by the experts we contacted. In addition, CBP has a program
to augment the targeting strategy by randomly selecting and inspecting
containers in order to compare the results of the random inspections
with those generated by the targeting system. However, our review
disclosed methodological problems with the random inspection program.
By incorporating the missing elements of a risk management framework
and following recognized modeling practices, CBP would have better
information to make management decisions related to preventing
terrorists from smuggling weapons of mass destruction into the United
States.
CBP faces a number of challenges in implementing the targeting strategy
at the six ports we visited, and these challenges could limit the
strategy's effectiveness. First, we found deficiencies in CBP's
national system for reporting and analyzing inspection statistics. CPB
officials told us they have just implemented enhancements to their
targeting system to better collect national data on the results of
inspections, but it is too soon to tell whether it will provide
consistent, complete inspection data for analyzing and improving the
targeting strategy. In addition, we found deficiencies in CBP's
national targeting training program. Further, we found that space
limitations and safety concerns about inspection equipment constrain
some ports in their utilization of screening equipment, a fact that has
affected the efficiency of examinations.
Our Limited Official Use report contains several recommendations to DHS
on how to better incorporate elements of a risk management framework
and recognized modeling practices. Additionally, the report contains
recommendations to improve management controls to better implement the
targeting strategy at seaports.
DHS provided us with written comments on a draft of our Limited
Official Use report. In commenting on that report, DHS stated that in
general the report was constructive and that CBP has taken corrective
actions and will take further corrective actions to address the issues
that we identified. DHS also outlined completion dates to implement
these corrective actions.
Background:
Maritime Cargo Containers Are Important and Vulnerable:
Cargo containers are an important segment of maritime commerce.
Approximately 90 percent of the world's cargo moves by container. In
2002, approximately 7 million containers arrived at U.S seaports,
carrying more than 95 percent of the nation's non-North American trade
by weight and 75 percent by value. Many experts on terrorism--including
those at the Federal Bureau of Investigation and at academic, think
tank and business organizations--have concluded that oceangoing cargo
containers are vulnerable to some form of terrorist action. A terrorist
incident at a seaport, in addition to killing people and causing
physical damage, could have serious economic consequences. In a 2002
simulation of a terrorist attack involving cargo containers, every
seaport in the United States was shut down, resulting in a simulated
loss of $58 billion in revenue to the U.S. economy, including spoilage,
loss of sales, and manufacturing slowdowns and halts in
production.[Footnote 2]
CBP Has Layered Approach to Select and Inspect Cargo Containers:
CBP is responsible for preventing terrorists and weapons of mass
destruction from entering the United States. As part of its
responsibility, it has the mission to address the potential threat
posed by the movement of oceangoing containers. To perform this
mission, CBP has inspectors at the ports of entry into the United
States. Inspectors assigned to seaports help determine which containers
entering the country will undergo inspections, and then perform
physical inspections of such containers. These determinations are not
just based on concerns about terrorism, but also concerns about illegal
narcotics and/or other contraband.
The CBP Commissioner said that the large volume of imports and CBP's
limited resources make it impossible to physically inspect all
oceangoing containers without disrupting the flow of commerce. The
Commissioner also said it is unrealistic to expect that all containers
warrant such inspection because each container poses a different level
of risk based on a number of factors including the exporter, the
transportation providers, and the importer. These concerns led to CBP
implementing a layered approach that attempts to focus resources on
potentially risky cargo containers while allowing other cargo
containers to proceed without disrupting commerce.
As part of its layered approach, CBP employs its Automated Targeting
System (ATS) computer model to review documentation on all arriving
containers and help select or target containers for additional
scrutiny. The ATS was originally designed to help identify illegal
narcotics in cargo containers, but was modified to help identify all
types of illegal contraband used by smugglers or terrorists. In
addition, CBP has a program, called the Supply Chain Stratified
Examination, which supplements ATS by randomly selecting additional
containers to be physically examined. The results of the random
inspection program are to be compared with the results of ATS
inspections to improve targeting. If CBP officials decide to inspect a
particular container, they might first conduct a nonintrusive
inspection with equipment such as the Vehicle and Cargo Inspection
System (VACIS), which takes a gamma-ray image of the container so
inspectors can detect any visual anomalies. With or without VACIS,
inspectors can open a container and physically examine its contents.
Other components of the layered approach include the Container Security
Initiative (CSI) and the Customs-Trade Partnership Against Terrorism
(C-TPAT). CSI is an initiative whereby CBP places staff at designated
foreign seaports to work with foreign counterparts to identify and
inspect high-risk containers for weapons of mass destruction before
they are shipped to the United States. C-TPAT is a cooperative program
between CBP and members of the international trade community in which
private companies agree to improve the security of their supply chains
in return for a reduced likelihood that their containers will be
inspected.[Footnote 3] A supply chain consists of all stages involved
in fulfilling a customer request, including stages conducted by
manufacturers, suppliers, transporters, retailers, and customers.
Risk Management and Modeling Are Important Security Practices:
Risk management is a systematic process to analyze the threats,
vulnerabilities, and criticality (or relative importance) of assets in
a program to better support key decisions linking resources and program
results. Risk management is used by many organizations in both
government and the private sector. In recent years, we have
consistently advocated the use of a risk management approach to help
implement and assess responses to various national security and
terrorism issues.[Footnote 4] We have concluded that without a risk
management approach that provides insights about the present threat and
vulnerabilities as well as the organizational and technical
requirements necessary to achieve a program's goals, there is little
assurance that programs to combat terrorism are prioritized and
properly focused. Risk management helps to more effectively and
efficiently prepare defenses against acts of terrorism and other
threats. Key elements of a risk management approach are listed below.
* Threat assessment: A threat assessment identifies adverse events that
can affect an entity, and may be present at the global, national, or
local level.
* Criticality assessment: A criticality assessment identifies and
evaluates an entity's assets or operations based on a variety of
factors, including importance of an asset or function.
* Vulnerability assessment: A vulnerability assessment identifies
weaknesses in physical structures, personnel protection systems,
processes, or other areas that may be exploited by terrorists.
* Risk assessment: A risk assessment qualitatively and/or
quantitatively determines the likelihood of an adverse event occurring
and the severity, or impact, of its consequences.
* Risk characterization: Risk characterization involves designating
risk on a scale, for example, low, medium, or high. Risk
characterization forms the basis for deciding which actions are best
suited to mitigate risk.
* Mitigation evaluation: Mitigation evaluation is the identification of
mitigating alternatives to assess the effectiveness of the
alternatives. The alternatives should be evaluated for their likely
effect on the risk and their cost.
* Mitigation selection: Mitigation selection involves a management
decision on which mitigation alternatives should be implemented.
Selection among alternatives should be based on preconsidered criteria.
* Systems approach: An integrated systems approach to risk management
encompasses taking action in all organizational areas, including
personnel, processes, technology, infrastructure, and governance.
* Monitoring and evaluation: Monitoring and evaluation is a continuous
repetitive assessment process to keep risk management current and
relevant. It includes external peer review, testing, and validation.
Modeling can be an important part of a risk management approach. To
assess modeling practices related to ATS, we interviewed terrorism
experts and representatives of the international trade community who
were familiar with modeling related to terrorism and/or ATS and
reviewed relevant literature. There are at least four recognized
modeling practices that are applicable to ATS as a decision support
tool.
* Conducting external peer review: External peer review is a process
that includes an assessment of the model by independent and qualified
external peers. While external peer reviews cannot ensure the success
of a model, they can increase the probability of success by improving
the technical quality of projects and the credibility of the decision-
making process.
* Incorporating additional types of information: To identify
documentary inconsistencies, targeting models need to incorporate
various types of information to perform complex "linkage" analyses.
Using only one type of information will not be sufficient to yield
reliable targeting results.
* Testing and validating through simulated terrorist events: A model
needs to be tested by staging simulated events to validate it as a
targeting tool. Simulated events could include "red teams" that devise
and deploy tactics in an attempt to define a system's weaknesses, and
"blue teams" that devise ways to mitigate the resulting vulnerabilities
identified by the red team.
* Using random inspections to supplement targeting: A random selection
process can help identify and mitigate residual risk (i.e., the risk
remaining after the model-generated inspections have been done), but
also help evaluate the performance of the model relative to other
approaches.
Positive Steps Taken, but Targeting Strategy Lacks Some Key Components
of Risk Management and Modeling:
CBP Has Taken Several Steps to Improve Its Targeting Strategy:
CBP has recognized the potential vulnerability of oceangoing cargo
containers and has reviewed and updated some aspects of its layered
targeting strategy. According to CBP officials, several of the steps
that CBP has taken to improve its targeting strategy have resulted in
more focused targeting of cargo containers that may hold weapons of
mass destruction. CBP officials told us that, given the urgency to take
steps to protect against terrorism after the September 11, 2001,
terrorist attacks, they had to take an "implement and amend" approach.
That is, they had to immediately implement targeting activities with
the knowledge they would have to amend them later. Steps taken by CBP
include the following:
* In November 2001, the U.S. Customs Service established the National
Targeting Center to support its targeting initiatives.[Footnote 5]
Among other things, the National Targeting Center interacts with the
intelligence community and manages a national targeting training
program for CBP targeters.
* In August 2002, CBP modified the ATS as an antiterrorism tool by
developing terrorism-related targeting rules and implementing them
nationally. CBP is now in the process of enhancing the ATS terrorism-
related rules.
* In 2002, CBP also developed a 2-week national training course to
train staff in targeting techniques. The course is intended to help
ensure that seaport targeters have the necessary knowledge and ability
to conduct effective targeting. The course is voluntary and is
conducted periodically during the year at the Los Angeles, Long Beach,
and Miami ports, and in the future it will also be conducted at the
National Targeting Center.
* In February 2003, CBP began enforcing new regulations about cargo
manifests--called the "24 hour rule"--which requires the submission of
complete and accurate manifest information 24 hours before a container
is loaded on a ship at a foreign port.[Footnote 6] Penalties for non-
compliance can include a CBP order not to load a container on a ship at
the port of origin or monetary fines. The rule is intended to improve
the quality and the timeliness of manifest information submitted to
CBP, which is important because CBP relies extensively on manifest
information for targeting. According to CBP officials we contacted,
although no formal evaluations have been done, the 24-hour rule is
beginning to improve both the quality and timeliness of manifest
information. CBP officials acknowledged, however, that although
improved, manifest information still is not always accurate or reliable
data for targeting purposes.
Targeting Strategy Does Not Incorporate Some Key Elements of Risk
Management:
While CBP's targeting strategy incorporates some elements of risk
management, our discussions with terrorism experts and our comparison
of CBP's targeting system with recognized risk management practices
showed that the strategy does not fully incorporate all key elements of
a risk management framework. Elements not fully incorporated are
discussed below.
* CBP has not performed a comprehensive set of assessments for cargo
containers. CBP has attempted to assess the threat of cargo containers
through contact with governmental and nongovernmental sources. However,
it has not assessed the vulnerability of cargo containers to tampering
or exploitation throughout the supply chain, nor has it assessed which
port assets are the most critical to carrying out its mission--and
therefore in the most need of protection. These assessments, in
addition to threat assessments, are needed to understand and identify
actions to mitigate risk.
* CBP has not conducted a risk characterization for different forms of
cargo or the different modes of transportation used to import cargo.
Further, CBP has not performed a risk characterization to assess the
overall risk of cargo containers. These characterizations would enable
CBP to better assess and prioritize the risks posed by oceangoing cargo
containers and incorporate mitigation activities in an overall
strategy.
* CBP actions at the ports to mitigate risk are not part of an
integrated systems approach. Risk mitigation encompasses taking action
in all organizational areas, including personnel, processes,
technology, infrastructure, and governance. An integrated approach
would help ensure that taking action in one or more areas would not
create unintended consequences in another. For example, taking action
in the areas of personnel and technology--adding inspectors and
scanning equipment at a port--without at the same time ensuring that
the port's infrastructure is appropriately reconfigured to accept these
additions and their potential impact (e.g., more physical examinations
of containers), could add to already crowded conditions at that port
and ultimately defeat the purpose of the original actions.
We recognize that CBP implemented the ATS terrorist targeting rules in
August 2002 because of the pressing need to utilize a targeting
strategy to protect cargo containers against terrorism, and that CBP
intends to amend the strategy as necessary. In doing so, implementing a
comprehensive risk management framework would help CBP ensure that
information is available to management to make choices about the best
use of limited resources. This type of information would help CBP
obtain optimal results and would identify potential enhancements that
are well conceived, cost-effective, and work in tandem with other
system components. Thus, it is important for CBP to amend its targeting
strategy within a risk management framework that takes into account all
of the system's components and their vital linkages.
Targeting Strategy Not Fully Consistent with Key Recognized Modeling
Practices:
Interviews with terrorism experts and representatives from the
international trade community who are familiar with CBP's targeting
strategy and/or terrorism modeling told us that ATS is not fully
consistent with recognized modeling practices. Challenges exist in each
of the four recognized modeling practice areas that these individuals
identified: external peer review, incorporating different types of
information, testing and validating through simulated events, and using
random inspections to supplement targeting.
* With respect to external review, CBP had limited external
consultations when developing the ATS rules related to terrorism.
* With respect to the sources and types of information, ATS relies on
the manifest as one of its sources of data, and CBP does not mandate
the transmission of entry data before a container's risk level is
assigned. Terrorism experts, members of the international trade
community, and CBP inspectors at the ports we visited characterized the
ship's manifest as one of the least reliable or useful types of
information for targeting purposes. In this regard, one expert
cautioned that even if ATS were an otherwise competent targeting model,
there is no compensating for poor input data. Accordingly, if the input
data are poor, the outputs (i.e., the risk assessed targets) are not
likely to be of high quality. Another problem with manifests is that
shippers can revise them up to 60 days after the arrival of the cargo
container. These problems with manifest data increase the potential
value of additional types of information.
* With respect to testing and validation, the only two known instances
of simulated tests of the targeting system were conducted without CBP's
approval or knowledge by the American Broadcast Company (ABC) News in
2002 and 2003. In an attempt to simulate a terrorist smuggling highly
enriched uranium into the United States, ABC News sealed depleted
uranium into a lead-lined pipe that was placed in a suitcase and later
put into a cargo container. In both instances, CBP targeted the
container that ABC News used to import the uranium, but it did not
detect a visual anomaly from the lead-lined pipe using VACIS and
therefore did not open the container.
* With respect to instituting random inspections, CBP has a program to
randomly select and examine containers regardless of their risk, titled
the Supply Chain Stratified Examination. However, our review disclosed
methodological problems with this program.
Targeting Strategy Faces Implementation Challenges:
CBP Lacks National System to Track Cargo Container Inspections by Risk
Category:
We found a number of deficiencies in CBP's national system for
reporting and analyzing inspection statistics. While officials at all
the ports we visited provided us with inspection data, we observed
problems with the available data. In addition, we had to contact ports
several times to obtain these data, indicating that basic data on
inspections were not readily available.
Separately, CBP officials said that they are trying to capture the
results of cargo inspections through an enhancement to ATS. These
enhancements were not implemented to an extent that we could evaluate
their potential effectiveness.
Staff Testing and Certification Could Help Strengthen Targeting
Process:
CBP does not have an adequate mechanism to test or certify the
competence of targeters in their national targeting training program.
The targeters taking the training must have a thorough understanding of
course contents and their application at the ports. Because the
targeters who complete the training are not tested or certified on
course materials, CPB has little assurance that the targeters could
perform their duties effectively or that they could train others to
perform effectively.
Space Limitations and Safety Concerns Constrain Use of Inspection
Equipment:
One of the key components of the CBP targeting and inspection process
is the use of nonintrusive inspection equipment. CBP uses nonintrusive
inspection equipment, including VACIS gamma-ray imaging technology, to
screen selected cargo containers and to help inspectors decide which
containers to further examine. A number of factors constrain the use of
inspection equipment, including crowded port terminals, mechanical
breakdowns, inclement weather conditions, and the safety concerns of
longshoremen at some ports. Some of these constraints, such as space
limitations and inclement weather conditions, are difficult if not
impossible to avoid.
According to CBP and union officials we contacted, concern about the
safety of VACIS is a constraint to using inspection equipment. Union
officials representing longshoremen at some ports expressed concerns
about the safety of driving cargo containers through VACIS because it
emits gamma rays when taking an image of the inside of the cargo
container. Towing cargo containers through a stationary VACIS unit
reportedly takes less time and physical space than moving the VACIS
equipment over stationary cargo containers that have been staged for
inspection purposes. As a result of these continuing safety concerns,
some longshoremen are unwilling to drive containers through VACIS.
CBP's response to these longshoremen's concerns has been to stage
containers away from the dock, arraying containers in rows at port
terminals so that the VACIS can be driven over a group of containers
for scanning purposes. However, as seaports and port terminals are
often crowded, and there is often limited space to expand operations,
it can be space-intensive and time-consuming to stage containers. Not
all longshoremen's unions have safety concerns regarding VACIS
inspections. For example, at the Port of New York/New Jersey,
longshoremen's concerns over the safety of operating VACIS were
addressed after the union contacted a consultant and received
assurances about the safety of the equipment. Similar efforts by CBP to
convince longshoremen's unions about the safety of VACIS have not been
successful at some of the other ports we visited.
Conclusions and Recommendations:
One legacy of the September 11, 2001 terrorist attacks is uncertainty.
It is unclear if, where, when, and how other attacks might occur and
what steps should be taken to best protect national security. In the
context of possible smuggling of weapons of mass destruction in cargo
containers at our nation's seaports, it is vital that CBP use its
resources to maximize the effectiveness of its targeting strategy to
reduce this uncertainty. Without incorporating all elements of a risk
management framework and utilizing recognized modeling practices, CBP
cannot be sure that its targeting strategy is properly focused and
prioritized. In addition, risk management and the use of recognized
modeling practices will not ensure security if there are lapses in
implementing these practices at the ports. Finally, without instituting
a national inspection reporting system, testing and certifying CBP
officials that receive the targeting training, and resolving the safety
concerns of longshoremen unions, the targeting system's effectiveness
as a risk management tool may be limited.
Our Limited Official Use report contains several recommendations to DHS
on how to better incorporate key elements of a risk management
framework and recognized modeling practices. Additionally, the report
contains recommendations to improve management controls to better
implement the targeting strategy at seaports.
This concludes my statement. I would now be pleased to answer any
questions for the subcommittee.
Contacts and Acknowledgments:
For further information about this testimony, please contact me at
(202) 512-8816. Seto Bagdoyan, Stephen L. Caldwell, Kathleen Ebert, Jim
Russell, and Brian Sklar also made key contributions to this statement.
Additional assistance was provided by David Alexander, Katherine Davis,
Scott Farrrow, Ann Finley, and Keith Rhodes.
[End of section]
Appendix I: Scope And Methodology:
To assess whether CBP's development of its targeting strategy is
consistent with recognized risk management and modeling practices, we
compiled a risk management framework and a list of recognized modeling
practices, drawn from an extensive review of relevant public and
private sector work, prior GAO work on risk management, and our
interviews with terrorism experts. We selected these individuals based
on their involvement with issues related to terrorism, specifically
concerning containerized cargo, ATS, and modeling. Several of the
individuals that we interviewed were referred from within the expert
community, while others were chosen from public texts on the record. We
did not assess ATS's hardware or software, the quality of the threat
assessments that CBP has received from the intelligence community, or
the appropriateness or risk weighting of its targeting rules.
To assess how well the targeting strategy has been implemented at
selected seaports in the country, we visited various CBP facilities and
the Miami, Los Angeles-Long Beach, Philadelphia, New York-New Jersey,
New Orleans, and Seattle seaports. These seaports were selected based
on the number of cargo containers processed and their geographic
dispersion. At these locations, we observed targeting and inspection
operations; met with CBP management and inspectors to discuss issues
related to targeting and the subsequent physical inspection of
containers; and reviewed relevant documents, including training and
operational manuals, and statistical reports of targeted and inspected
containers. We used these statistical reports to determine the type of
data available; we did not assess the reliability of the data or use it
to make any projections. At the seaports, we also met with
representatives of shipping lines, operators of private cargo
terminals, the local port authorities, and Coast Guard personnel
responsible for the ports' physical security. We also met with
terrorism experts and representatives from the international trade
community to obtain a better understanding of the potential threat
posed by cargo containers and possible approaches to countering the
threat, such as risk management.
We conducted our work from January 2003 to February 2004 in accordance
with generally accepted government auditing standards.
[End of section]
Appendix II: Risk Management Framework for Homeland Security and
Terrorism:
Development and Application of Risk Management Framework:
This appendix details the risk management framework that GAO developed
in order to assess CBP's overall targeting strategy. In recent years,
GAO has consistently advocated the use of a risk management approach as
an iterative analytical tool to help implement and assess responses to
various national security and terrorism issues.[Footnote 7] We have
concluded that without a risk management approach, there is little
assurance that programs to combat terrorism are prioritized and
properly focused. Risk management principles acknowledge that while
risk cannot be eliminated, enhancing protection from known or potential
threats can help reduce it. Drawing on this precedent, we compiled a
risk management framework--outlined below--to help assess the U.S.
government's response to homeland security and terrorism risk. One way
in which the Department of Homeland Security's U.S. Customs and Border
Protection has already begun to manage risk is by developing and
implementing the Automated Targeting System to target high-risk
oceangoing containerized cargo for inspection.
Applied to homeland security and terrorism risk, the framework assumes
that the principal classes of risk from terrorism are to (1) the
general public; (2) organizational, governmental, and societal
infrastructure; (3) cyber and physical infrastructure; and (4) economic
sectors/structures. Terrorism risk is framed by and is a function of
(1) a strategic intent of inflicting extreme damage and disruption; (2)
operational, logistical, and technological capabilities including the
ability to obtain and deploy various classes of weapons against targets
of least resistance (targets are chosen and prioritized according to
their attractiveness or utility, based, in turn, on the potential for
economic or human loss, their symbolic value, and name recognition);
and (3) rational responses to moves designed to counteract them. This
last aspect includes the identification and exploitation of loopholes
in the response. A principal example of potential homeland security or
terrorism risk is the global supply chain, a complex system of multiple
interacting components with interdependent risk, and with the potential
for this risk to be transferred from any weak links in the chain. The
risk posed to the supply chain at the operational, or tactical, level
is manifested, for example, in the movement of oceangoing containerized
cargo.
Importance, Benefits, and Limitations of Risk Management:
In terms of the importance of risk management, an entity exists to
provide value for its stakeholders in an environment of uncertainty,
which is a function of the ability to determine the likelihood of
events occurring and quantify the resulting outcomes. As applied to
homeland security, "value" is realized as protection (security)
provided by the U.S. government against terrorism risk at an acceptable
cost (function of time and money) for the recipients of the valued
service (for example, the general public and the business community).
This value might, on occasion, be at risk (worst-case loss scenario)
that needs to be managed, thus risk management can be viewed as an
integral part of managing homeland security.
In terms of its benefits, risk management enables entities to operate
more effectively in environments filled with risks by providing the
discipline and structure to address them; risk management is not an end
in itself but an important means of an entity's management process. As
such, it is interrelated with, among other things, an entity's
governance, performance management, and internal control. Further, risk
management provides the rigor necessary to identify and select among
alternative risk responses whose cumulative effect is intended to
reduce risk, and the methodologies and techniques for making selection
decisions. Also, risk management enables entities to have an enhanced
capability to identify potential events, assess risks, and establish
integrated responses to reduce "surprises," and related costs and
losses.
In terms of its limitations, ultimately, risk management cannot
eliminate risk and the environment of uncertainty that helps sustain
it, but risk management can help reduce risk, with a goal of providing
reasonable assurance that an entity's objectives will be achieved. Risk
management combines elements of science and judgment (human dimension
to conflict), and ultimately relies on a set of estimates about risk
that lies in the future, which is inherently uncertain. Accordingly,
the results of risk management might be called into question because
of, among other things, the potential for human errors in judgment and
the potentially poor quality of information driving the risk management
process.
Risk Management Framework:
The framework is a composite of risk management best practices gleaned
from our interviews with terrorism and risk-modeling experts and our
extensive review of relevant reports on risk management, such as those
by GAO, the Congressional Research Service, Booz Allen Hamilton (on
contract to the U.S. intelligence community), and the Committee of the
Sponsoring Organizations of the Treadway Commission (in conjunction
with PricewaterhouseCoopers).[Footnote 8]
For purposes of the risk management framework, we used the following
definitions:
* Risk--an event that has a potentially negative impact, and the
possibility that such an event will occur and adversely affect an
entity's assets and activities and operations, as well as the
achievement of its mission and strategic objectives. As applied to the
homeland security context, risk is most prominently manifested as
"catastrophic" or "extreme" events related to terrorism, i.e., those
involving more that $1 billion in damage or loss and/or more than 500
casualties.
* Risk management--a continuous process of managing, through a series
of mitigating actions that permeate an entity's activities, the
likelihood of an adverse event happening and having a negative impact.
In general, risk is managed as a portfolio, addressing entity-wide risk
within the entire scope of activities. Risk management addresses
"inherent," or pre-action, risk (i.e., risk that would exist absent any
mitigating action) as well as "residual," or post-action, risk (i.e.,
the risk that remains even after mitigating actions have been taken).
The risk management framework--which is based on the proposition that a
threat to a vulnerable asset results in risk--consists of the following
components:
* Internal (or implementing) environment--the internal environment is
the institutional "driver" of risk management, serving as the
foundation of all elements of the risk management process. The internal
environment includes an entity's organizational and management
structure and processes that provide the framework to plan, execute,
and control and monitor an entity's activities, including risk
management. Within the organizational and management structure, an
operational unit that is independent of all other operational
(business) units is responsible for implementing the entity's risk
management function. This unit is supported by and directly accountable
to an entity's senior management. For its part, senior management (1)
defines the entity's risk tolerance (i.e., how much risk is an entity
willing to assume in order to accomplish its mission and related
objectives) and (2) establishes the entity's risk management philosophy
and culture (i.e., how an entity's values and attitudes view risk and
how its activities and practices are managed to deal with risk). The
operational unit (1) designs and implements the entity's risk
management process and (2) coordinates internal and external evaluation
of the process and helps implement any corrective action.
* Threat (event) assessment--threat is defined as a potential intent to
cause harm or damage to an asset (e.g., natural environment, people,
man-made infrastructures, and activities and operations). Threat
assessments consist of the identification of adverse events that can
affect an entity. Threats might be present at the global, national, or
local level, and their sources include terrorists and criminal
enterprises. Threat information emanates from "open" sources and
intelligence (both strategic and tactical). Intelligence information is
characterized as "reported" (or raw) and "finished" (fully fused and
analyzed).
As applied to homeland security and terrorism risk, and from the
perspective of the source of the threat (for example, a terrorist),
beginning with intent (the basis of the threat), adverse event
scenarios consist of six stages, as shown in table 1.
Table 1: Adverse Event Scenario Stages:
Stage: Intent;
Description: The terrorist develops malice and an intent to harm.
Stage: Target acquisition;
Description: The terrorist chooses specific target(s) among assets.
Stage: Planning;
Description: The terrorist researches the targets and various attack
options.
Stage: Preparation;
Description: Full commitment stage--the terrorist prepares to launch
the attack.
Stage: Execution;
Description: The terrorist carries out the attack.
Stage: "Grace period";
Description: Depending on the nature and success of the attack, there
could be a time lag between the attack and its impact.
Source: GAO Analysis:
[End of table]
* Criticality assessment--criticality is defined as an asset's relative
importance. Criticality assessments identify and evaluate an entity's
assets based on a variety of factors, including the importance of its
mission or function, the extent to which people are at risk, or the
significance of a structure or system in terms of, for example,
national security, economic activity, or public safety. Criticality
assessments are important because they provide, in combination with the
framework's other assessments, the basis for prioritizing which assets
require greater or special protection relative to finite resources.
* Vulnerability assessment--vulnerability is defined as the inherent
state (either physical, technical, or operational) of an asset that can
be exploited by an adversary to cause harm or damage. Vulnerability
assessments identify these inherent states and the extent of their
susceptibility to exploitation, relative to the existence of any
countermeasures. As applied to the global supply chain, a vulnerability
assessment might involve, first, establishing a comprehensive
understanding of the business and commercial aspects of the chain (as a
complex system with multiple interacting participants); and, second,
"mapping" the chain and identifying vulnerability points that could be
exploited.
* Risk assessment--risk assessment is a qualitative and/or quantitative
determination of the likelihood (probability) of occurrence of an
adverse event and the severity, or impact, of its consequences. Risk
assessments include scenarios under which two or more risks interact
creating greater or lesser impacts.
* Risk characterization--risk characterization involves designating
risk as, for example, low, medium, or high (other scales, such as
numeric, are also be used). Risk characterization is a function of the
probability of an adverse event occurring and the severity of its
consequences. Risk characterization is the crucial link between
assessments of risk and the implementation of mitigation actions, given
that not all risks can be addressed because resources are inherently
scarce; accordingly, risk characterization forms the basis for deciding
which actions are best suited to mitigate the assessed risk.
* Mitigation evaluation. Mitigation evaluation is the identification of
mitigation alternatives to assess the effectiveness of the
alternatives. The alternatives should be evaluated for their likely
effect on risk and their cost.
* Mitigation selection. Mitigation selection involves a management
decision on which mitigation alternatives should be implemented among
alternatives, taking into account risk, costs, and the effectiveness of
mitigation alternatives. Selection among mitigation alternatives
should be based upon preconsidered criteria. There are as of yet no
clearly preferred selection criteria, although potential factors might
include risk reduction, net benefits, equality of treatment, or other
stated values. Mitigation selection does not necessarily involve
prioritizing all resources to the highest-risk area, but in attempting
to balance overall risk and available resources.
* Risk mitigation--Risk mitigation is the implementation of mitigation
actions, in priority order and commensurate with assessed risk;
depending on its risk tolerance, an entity may choose not to take any
action to mitigate risk (this is characterized as risk acceptance). If
the entity does choose to take action, such action falls into three
categories: (1) risk avoidance (exiting activities that expose the
entity to risk), (2) risk reduction (implementing actions that reduce
likelihood or impact of risk), and (3) risk sharing (implementing
actions that reduce likelihood or impact by transferring or sharing
risk). In each category, the entity implements actions as part of an
integrated "systems" approach, with built-in redundancy to help address
residual risk (the risk that remains after actions have been
implemented). The systems approach consists of taking actions in
personnel (e.g., training, deployment), processes (e.g., operational
procedures), technology (e.g., software or hardware), infrastructure
(e.g., institutional or operational--such as port configurations), and
governance (e.g., management and internal control and assurance). In
selecting actions, the entity assesses their costs and benefits, where
the amount of risk reduction is weighed against the cost involved and
identifies potential financing options for the actions chosen.
* Monitoring and evaluation of risk mitigation--Monitoring and
evaluation of risk mitigation entails the assessment of the functioning
of actions against strategic objectives and performance measures to
make necessary changes. Monitoring and evaluation includes, where and
when appropriate, peer review and testing and validation; and an
evaluation of the impact of the actions on future options; and
identification of unintended consequences that, in turn, would need to
be mitigated. Monitoring and evaluation helps ensure that the entire
risk management process remains current and relevant, and reflects
changes in (1) the effectiveness of the actions and (2) the risk
environment in which the entity operates--risk is dynamic and threats
are adaptive. The risk management process should be repeated
periodically, restarting the "loop" of assessment, mitigation, and
monitoring and evaluation.
[End of section]
Appendix III: Recognized Modeling Practices Applicable to the Review of
ATS:
This appendix details the recognized modeling practices that GAO used
to assess CBP's computerized targeting model, known as the ATS. CBP
characterized ATS as a knowledge, or rule-based, expert system or model
that serves as a "decision support tool" in implementing its targeting
strategy.[Footnote 9] Accordingly, for purposes of this report, we
identified four practices that are applicable to our review of ATS as
such a tool. We identified these practices through our interviews with
terrorism experts and representatives of the international trade
community--who were familiar with modeling related to terrorism or to
ATS--and GAO's chief scientist; and our review of relevant literature,
such as reports by the U.S. Department of Energy's Office of Science
and Technology and the National Research Council (part of the National
Academies)[Footnote 10] and GAO.[Footnote 11] The four practices are:
* Initiating an external peer review of ATS. Many agencies conduct
various types of internal reviews of projects and programs. However,
these reviews are usually conducted by managers or supervisors and thus
are not independent. Peer review is a process that includes an
independent, documented, critical assessment of the technical,
scientific merit of research or programs by external peers who are
highly qualified scientists with knowledge and expertise equal to that
of those whose work they review. In this regard, peers must be capable
of making independent judgments about the merit and relevance of what
they are reviewing and have no conflicts of interest. If the results
are to be used in programmatic decision making, peer reviews can
improve the technical quality of projects by recognizing technical
weaknesses and suggesting improvements that might be overlooked by
those too close to the project; peer review can also enhance the
credibility of the decision-making process by offering frank
assessments not constrained by organizational concerns and by avoiding
the reality and the perception of conflicts of interest. Peer review
cannot ensure the success of a program, but it can increase the
probability of success.
* Instituting a process of random inspections to supplement targeting.
The experts we spoke with told us that the absence of a process to
randomly select containerized cargo for screening or physical
examination to supplement ATS was a shortcoming of CBP's targeting
strategy. Randomness pertains to a process whose outcome or value
depends on chance or on a process that simulates chance, with the
implication that all possible outcomes or values have a known, non-zero
probability of occurrence--for example, the outcome of flipping a coin
or executing a computer-programmed random number generator. A random
selection process would not only help mitigate residual risk (i.e., the
risk remaining after the original risk mitigation actions have been
implemented), but also help evaluate the performance of targeting
relative to other approaches.
* Enhancing the sources and types of information input into ATS.
Terrorism experts and representatives of the international trade
community told us that ATS needed to incorporate additional types of
information in order to be able to perform complex "linkage" analyses
in an attempt to identify documentary inconsistencies that must be
detected to target suspicious containers. They also told us that the
ship's manifest (or transportation document that lists a summary of the
cargo on board) does not contain enough information in sufficient
detail to be useful, by itself, in targeting suspicious containers.
These individuals further told us that the movement of containers
through the global supply chain generated an additional amount of
commercial documentation that could be used for this purpose.[Footnote
12] Examples of commercial documentation that could be used include
purchase orders, commercial invoices, shippers' letters of instruction,
and certificates of origin.
* Testing and validating ATS by staging simulated terrorist events. The
experts we spoke with emphasized the need to test ATS by staging
simulated terrorist events in order to validate it as a targeting
tool.[Footnote 13] Simulated events could include "red teams"
attempting to smuggle a fake WMD into the United States hidden in an
oceangoing cargo container. Red teaming is an approach to "model" a
system's adversary and define its weaknesses by devising attack
tactics. A blue team may also be used to devise ways to mitigate
vulnerabilities in an attempt to defend against the red team. Simulated
events would determine whether ATS targeted the suspicious container
for screening and/or physical examination, and whether the subsequent
screening or examination actually detected the fake WMD.
[End of section]
Related GAO Products:
Maritime Security: Progress Made in Implementing Maritime
Transportation Security Act, but Concerns Remain. GAO-03-1155T.
Washington, D.C.: September 9, 2003.
Container Security: Expansion of Key Customs Programs Will Require
Greater Attention to Critical Success Factors. GAO-03-770. Washington,
D.C.: July 25, 2003.
Homeland Security: Challenges Facing the Department of Homeland
Security in Balancing its Border Security and Trade Facilitation
Missions. GAO-03-902T. Washington, D.C.: June 16, 2003.
Container Security: Current Efforts to Detect Nuclear Material, New
Initiatives, and Challenges (GAO-03-297T. Washington, D.C.: November
18, 2002.
Customs Service: Acquisition and Deployment of Radiation Detection
Equipment. GAO-03-235T. Washington, D.C.: October 17, 2002.
Port Security: Nation Faces Formidable Challenges in Making New
Initiatives Successful. GAO-02-993T. Washington, D.C.: August 5, 2002.
Homeland Security: A Risk Management Approach Can Guide Preparedness
Efforts. GAO-02-208T. Washington, D.C.: October 31, 2001.
Homeland Security: Key Elements of a Risk Management Approach. GAO-02-
150T. Washington, D.C.: October. 12, 2001.
Federal Research: Peer Review Practices at Federal Science Agencies
Vary. GAO/RCED-99-99. Washington, D.C.: March 17, 1999.
FOOTNOTES
[1] A listing of related GAO reports appears at the end of this
statement.
[2] The consulting firm Booz Allen Hamilton and the Conference Board
sponsored the simulation in 2002. In the simulation, representatives
from government and industry participated in a scenario involving the
discovery and subsequent detonation of radioactive bombs hidden in
cargo containers.
[3] For more information on these programs, see U.S. General Accounting
Office, Container Security: Expansion of Key Customs Programs Will
Require Greater Attention to Critical Success Factors, GAO-03-770
(Washington, D.C.: July 2003).
[4] For example, see U.S. General Accounting Office, Homeland Security:
A Risk Management Approach Can Guide Preparedness Efforts, GAO-02-208T
(Washington, D.C.: July 2003).
[5] The commercial operations and inspection programs at the U.S.
Customs Service (in the Department of the Treasury) were incorporated
into CBP (in the new Department of Homeland Security) effective March
1, 2003.
[6] This rule is also known as the Advance Manifest Regulation, 67 Fed.
Reg. 66318 (2002). The final regulation was issued October 31, 2002,
with implementation beginning February 1, 2003.
[7] U.S. General Accounting Office, Homeland Security: A Risk
Management Approach Can Guide Preparedness Efforts, GAO-02-208T
(Washington, D.C.: October, 31, 2001) and U.S. General Accounting
Office, Key Elements of a Risk Management Approach, GAO-02-150T
(Washington, D.C.: October 12, 2001).
[8] The framework is adapted from primary sources, including reports by
GAO; the Congressional Research Service; Department of Energy's Office
of Science and Technology; National Academies/National Research
Council; Committee of the Sponsoring Organizations of the Treadway
Commission/PricewaterhouseCoopers; Risk Management Solutions, and
RiskMetrics (private risk management consulting firms advising
insurance, reinsurance, and financial services companies on terrorism
and other catastrophic events); Booz Allen Hamilton, on contract to the
U.S. government intelligence community; academic and think-tanks (e.g.,
Brookings Institution, Council on Foreign Relations) papers on
responses to terrorism, including risk management; and interviews with
terrorism and risk modeling experts.
[9] An expert system is a knowledge collection combined with an
inference engine capable of interpreting queries and chaining together
separate items of knowledge to develop new inferences; a model is the
physical, mathematical, or otherwise logical representation of a
system, entity, phenomenon, or process. The knowledge is typically
represented as a system of rules or algorithms. An algorithm is a
prescribed set of well-defined unambiguous rules or processes for the
solution of a problem in a finite number of steps.
[10] The National Academies brings together committees of experts in
all areas of scientific and technological endeavor. Four organizations
constitute the academies: the National Academy of Sciences, the
National Academy of Engineering, the Institute of Medicine, and the
National Research Council. The National Research Council was organized
by the Academy to associate the broad community of science and
technology with the academy's purpose of furthering knowledge and
advising the federal government.
[11] U.S. Department of Energy's Office of Science and Technology, Peer
Review in Environmental Technology Development Programs, (Washington,
D.C., 1998); U.S. General Accounting Office, Federal Research: Peer
Review Practices at Federal Science Agencies Vary, GAO/RCED-99-99
(Washington, D.C.: March 1999).
[12] International trade is a tremendously complex business. A typical
trade will involve multiple parties--for example, importers, exporters,
ocean carriers, financiers, and governments--and may generate 30 to 40
documents.
[13] Validation is the process of determining the degree to which a
model or simulation is an accurate representation of the real world
from the perspective of the intended uses of the model or simulation.
