Anthrax Detection
Agencies Need to Validate Sampling Activities in Order to Increase Confidence in Negative Reults Gao ID: GAO-05-493T April 5, 2005In September and October 2001, letters laced with Bacillus anthracis (anthrax) spores were sent through the mail to two U.S. senators and to members of the media. These letters led to the first U.S. cases of anthrax disease related to bioterrorism. In all, 22 individuals, in four states and Washington, D.C., contracted anthrax disease; 5 died. These cases prompted the Subcommittee to ask GAO to describe and assess federal agencies' activities to detect anthrax in postal facilities, assess the results of agencies' testing, and assess whether agencies' detection activities were validated.
The U.S. Postal Service, Centers for Disease Control and Prevention (CDC), and Environmental Protection Agency (EPA) conducted several interdependent activities, including sample collection and analytic methods, to detect anthrax in postal facilities in 2001. They developed a sampling strategy and collected, transported, extracted, and analyzed samples. They primarily collected samples from specific areas, such as mail processing areas, using their judgment about where anthrax would most likely be found--that is, targeted sampling. The agencies did not use probability sampling in their initial sampling strategy. Probability sampling would have allowed agencies to determine, with some defined level of confidence, when all results are negative, whether a building is contaminated. This is important, considering that low levels of anthrax could cause disease and death in susceptible individuals. The results of the agencies' testing in 286 postal facilities were largely negative--no anthrax was detected. But negative results do not necessarily mean that a facility is free from anthrax. In addition, agencies' detection activities (for example, sample collection and analytical methods) were not validated. Validation is a formal, empirical process in which an authority determines and certifies the performance characteristics of a given method. Consequently, the lack of validation of agencies' activities, coupled with limitations associated with their targeted sampling strategy, means that negative results may not be reliable. In preparing for future incidents, the agencies have (1) made some changes based on what has been learned about some of the limitations of their sampling strategies, (2) made some revisions to their guidelines, and (3) funded some new research. In addition, the Department of Homeland Security (DHS) has taken on the role of coordinating agencies' activities and has undertaken several new initiatives related to anthrax and other bio-threat agents. However, while the actions DHS and other agencies have taken are important, they do not address the issue of validating all activities related to sampling. Finally, the agencies have not made appropriate and prioritized investments to develop and validate all activities related to anthrax and other bio-threat agents.
GAO-05-493T, Anthrax Detection: Agencies Need to Validate Sampling Activities in Order to Increase Confidence in Negative Reults
This is the accessible text file for GAO report number GAO-05-493T
entitled 'Anthrax Detection: Agencies Need to Validate Sampling
Activities in Order to Increase Confidence in Negative Results' which
was released on April 5, 2005.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Testimony before the Chairman, Subcommittee on National Security,
Emerging Threats, and International Relations, House Committee on
Government Reform, House of Representatives:
United States Government Accountability Office:
GAO:
For Release on Delivery Expected at 2:00 p.m.
Tuesday, April 5, 2005:
Anthrax Detection:
Agencies Need to Validate Sampling Activities in Order to Increase
Confidence in Negative Results:
Statement of Keith A. Rhodes, Chief Technologist, Center for Technology
and Engineering, Applied Research and Methods:
GAO-05-493T:
GAO Highlights:
Highlights of GAO-05-493T, a testimony before Chairman, Subcommittee on
National Security, Emerging Threats, and International Relations, House
Committee on Government Reform, House of Representatives:
Why GAO Did This Study:
In September and October 2001, letters laced with Bacillus anthracis
(anthrax) spores were sent through the mail to two U.S. senators and to
members of the media. These letters led to the first U.S. cases of
anthrax disease related to bioterrorism. In all, 22 individuals, in
four states and Washington, D.C., contracted anthrax disease; 5 died.
These cases prompted the Subcommittee to ask GAO to describe and assess
federal agencies' activities to detect anthrax in postal facilities,
assess the results of agencies' testing, and assess whether agencies'
detection activities were validated.
What GAO Found:
GAO recommends that the Department of Homeland Security (DHS) develop a
coordinated approach to working with federal agencies, so that
appropriate validation studies of various activities involved in
detecting anthrax are conducted. The DHS Secretary should also ensure
that an agreed-on definition of validation is developed; appropriate
investments are made to explore improved sampling strategies; and
agencies' policies, procedures, and guidelines reflect the results of
all these efforts. DHS stated that while it has the overall
responsibility for coordination, EPA and HHS have the lead roles in
responding to biological attacks. DHS said that it would coordinate
with EPA to ensure that appropriate investments are made to explore
improved sampling.
What GAO Recommends:
The U.S. Postal Service, Centers for Disease Control and Prevention
(CDC), and Environmental Protection Agency (EPA) conducted several
interdependent activities, including sample collection and analytic
methods, to detect anthrax in postal facilities in 2001. They developed
a sampling strategy and collected, transported, extracted, and analyzed
samples. They primarily collected samples from specific areas, such as
mail processing areas, using their judgment about where anthrax would
most likely be found--that is, targeted sampling. The agencies did not
use probability sampling in their initial sampling strategy.
Probability sampling would have allowed agencies to determine, with
some defined level of confidence, when all results are negative,
whether a building is contaminated. This is important, considering that
low levels of anthrax could cause disease and death in susceptible
individuals.
[See PDF for image]
[End of figure]
The results of the agencies' testing in 286 postal facilities were
largely negative--no anthrax was detected. But negative results do not
necessarily mean that a facility is free from anthrax. In addition,
agencies' detection activities (for example, sample collection and
analytical methods) were not validated. Validation is a formal,
empirical process in which an authority determines and certifies the
performance characteristics of a given method. Consequently, the lack
of validation of agencies' activities, coupled with limitations
associated with their targeted sampling strategy, means that negative
results may not be reliable.
In preparing for future incidents, the agencies have (1) made some
changes based on what has been learned about some of the limitations of
their sampling strategies, (2) made some revisions to their guidelines,
and (3) funded some new research. In addition, the Department of
Homeland Security (DHS) has taken on the role of coordinating agencies'
activities and has undertaken several new initiatives related to
anthrax and other biothreat agents. However, while the actions DHS and
other agencies have taken are important, they do not address the issue
of validating all activities related to sampling. Finally, the agencies
have not made appropriate and prioritized investments to develop and
validate all activities related to anthrax and other biothreat agents.
www.gao.gov/cgi-bin/getrpt?GAO-05-493T.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Keith Rhodes, 202-512-
6412, rhodesk@gao.gov.
[End of section]
April 5, 2005:
Mr. Chairman and Members of the Subcommittee:
We are pleased to participate in this hearing by presenting our
assessment of the federal agencies'--U.S. Postal Service (USPS),
Centers for Disease Control and Prevention, and Environmental
Protection Agency (EPA)--activities conducted to detect anthrax in
postal facilities in 2001. My statement is based on our report,
entitled Anthrax Detection: Agencies Need to Validate Sampling
Activities in Order to Increase Confidence in Negative Results, which
was issued on March 31, 2005.[Footnote 1]
As you know, in September and October 2001, contaminated letters laced
with Bacillus anthracis, or anthrax spores,[Footnote 2] were sent
through the mail to two senators, Thomas Daschle and Patrick Leahy, and
members of the media. The postal facilities in New Jersey and
Washington, D.C., that processed the senators' letters became heavily
contaminated.[Footnote 3] Other mail routed through these facilities,
as well as additional ones in the postal network, also became
contaminated. In addition, numerous federal facilities in the
Washington, D.C., area were later found to be contaminated. The letters
led to the first cases of anthrax disease related to bioterrorism in
the United States. In all, 22 individuals contracted anthrax disease in
four states (Connecticut, Florida, New Jersey, and New York) as well as
in Washington, D.C. Five of these 22 individuals died.
The threat of bioterrorism has been recognized for a considerable time.
Long before the anthrax incidents, several hoax letters indicating the
presence of anthrax had been mailed to federal and state agencies, as
well as to private sector organizations. In calendar year 2000, the
Federal Bureau of Investigation (FBI) responded to about 250 cases
potentially involving weapons of mass destruction. Of these, 200 were
related to anthrax, although all turned out to be hoaxes. Nevertheless,
these events raised the possibility that facilities could become
contaminated and would therefore have to be evaluated for environmental
contamination. However, federal agencies have not been fully prepared
to deal with environmental contamination, that is, anthrax released
through the mail, including the potential for multiple dispersals in
indoor environments.[Footnote 4]
Before I discuss our assessment, let me first present some background.
(See appendix I for a discussion of our scope and methodology.)
Background:
Although anthrax can infect humans, it is most commonly found in plant-
eating animals. Human anthrax infections are rare in the United States,
and when infection does occur, it usually results from occupational
exposure to infected animals or contaminated animal products, such as
wool, hides, or hair. Anthrax infection can occur (1) cutaneously,
usually from a cut or abrasion on the skin; (2) gastrointestinally, by
ingesting undercooked, contaminated meat; and (3) through inhalation,
by breathing aerosolized, or airborne, spores into the lungs.
The response to the incident in the American Media Incorporated
building in Florida in September 2001 led to the identification of mail
as the potential source of contamination; eventually, it led to the
sampling of the postal facilities. The agencies began sampling on
October 12, 2001, in Florida and stopped on April 21, 2002, when the
Wallingford, Connecticut, facility was sampled for the last time. Four
contractors conducted USPS sampling.
The mission of USPS is to provide affordable, universal mail service.
As of May 28, 2004, more than 800,000 workers processed more than 200
billion pieces of mail a year. The USPS headquarters office is in
Washington, D.C. USPS has nine area offices; approximately 350 P&DCs;
and about 38,000 post offices, stations, and branches; the P&DCs vary
widely in size and capacity. The USPS mail system is involved in
collecting, distributing, and delivering letters, flats (that is,
catalogs and magazines), and parcels, as well as other items that vary
in size and capacity.
The federal agencies involved in the response in the postal facilities
had differing responsibilities. The Centers for Disease Control and
Prevention (CDC) and state and local health departments primarily
provided public health advice and assistance to USPS. CDC has had
primary responsibility for national surveillance of specific diseases,
including anthrax; it has also conducted epidemiologic investigations
to determine, among other things, the source of the disease. The FBI
has been responsible for criminal investigations involving interstate
commerce and the mail and crimes committed on federal property. The
Environmental Protection Agency (EPA) has been the nation's lead agency
for responding to a release of hazardous substances into the
environment.
On October 8, 2001, the President created the Office of Homeland
Security to develop and coordinate a comprehensive national strategy
for dealing with domestic terrorist threats or attacks. The office,
which had limited involvement in the 2001 response, was superseded by
the Homeland Security Act of 2002, which transferred many of its
functions to the Department of Homeland Security (DHS); it became
operational in 2003. DHS was created by combining many previously
separate agencies and is assigned a lead role in coordinating the
efforts of federal agencies that respond to acts of terrorism in the
United States.
In addition, the Laboratory Response Network (LRN) was developed in
1999 to coordinate clinical diagnostic testing for bioterrorism. The
primary purpose on the biological side was to detect the presence of
biothreat agents in a number of specimen and sample types. These
laboratories function as first responders that can perform standard
initial tests to rule out, but not definitively confirm, anthrax.
Now I will discuss our assessment of the following federal agencies'
activities: (1) federal agencies' activities to detect anthrax
contamination in the postal facilities; (2) the results of the federal
agencies' testing in the postal facilities; and (3) whether agencies'
activities were validated and, if not, discuss any issues that arose
from the lack of validation and any actions they took to address these
issues.
Federal Agencies' Activities to Detect Anthrax Contamination in the
Postal Facilities:
CDC, EPA, and USPS, the federal agencies involved in sampling the
postal facilities in 2001 to detect anthrax, undertook several
activities: (1) sampling strategy development, followed by (2) sample
collection, (3) transportation, (4) extraction, and (5) analysis of the
samples (see fig. 1).
Figure 1: Agency Sampling Activities:
[See PDF for image]
[End of figure]
Neither these activities nor the overall process has been validated for
anthrax testing. Consequently, the agencies had only limited
information available for reliably choosing one method over another and
no information on the limits of detection to use when evaluating
negative results. In addition, the sampling strategy used by the
agencies could not provide any statistical confidence with regard to
the basic question: Is this building contaminated? Therefore, in the
future, in the absence of a positive result, a different strategy is
needed that will provide statistical confidence, at a defined level, to
answer this question.
Activity 1: Sampling Strategy Development:
The first activity involved agencies' developing a sampling strategy,
which included deciding how many samples to collect, where to collect
them from, and what collection methods to use. The agencies primarily
used a targeted strategy: They collected samples from specific areas
considered more likely to be contaminated, based on judgments. Such
judgments can be effective in some situations, for example, in
determining (1) the source of contamination in a disease outbreak
investigation or (2) whether a facility is contaminated when
information on the source of potential contamination is definitive.
However, in the case of a negative finding, when the source of
potential contamination is not definitive, the basic question--Is this
building contaminated?--will remain unanswered.
Agencies Primarily Used a Targeted Strategy:
The targeted strategy the agencies used was reflected in their site-
specific sampling activities. Sample sizes varied by facility and
circumstances, increased over time, and excluded probability sampling.
In the beginning, in each USPS facility, 23 samples were to be
collected from specific areas relating to mail processing and up to 20
additional "discretionary" samples were to be collected, depending on
the type and size of the facility. Later, USPS increased the number of
samples required to a minimum of 55, with up to 10 additional
discretionary samples for larger facilities. Consequently, the number
of samples collected varied by facility, from a low of 4 to a high of
148. CDC's and EPA's site-specific strategies were primarily
discretionary. The number of samples CDC collected varied by facility,
ranging from a low of 4 to a high of 202. The number of samples EPA
collected ranged from a low of 4 to a high of 71.
According to CDC, a targeted sampling strategy may be effective in
detecting contamination in a facility when sufficient site-specific
information exists to narrow down the locations in which the release
and contamination are most likely to have occurred. CDC's assumptions
for this strategy are that at the outset, (1) a scenario where all
locations have an equal chance of being contaminated is generally the
exception rather than the rule; (2) information collected about the
event, combined with technical judgment about exposure pathways, can be
used to identify locations where contamination is most likely to be
found; (3) contamination levels of the highest public health concern
can usually be detected using a variety of available methods, despite
their limitations; and (4) there is important public health value in
quickly identifying contaminated locations. However, these assumptions
may not always apply. For example, there may be limitations in the
available information that restrict the ability to reliably identify
target locations. The method of contamination spread could conceivably
be via a mechanism where there is an equal chance of any area being
contaminated. Lastly, all results may be negative, which will lead to a
requirement for additional testing, as was the case in Wallingford.
This, in turn, will result in the loss of the critical time needed for
public health intervention.
CDC and USPS officials said that they used a targeted strategy for
several reasons, including limitations on how many samples could be
collected and analyzed. They also said that in 2001 they lacked the
data necessary to develop an initial sampling strategy that
incorporated probability sampling. We disagree with this
interpretation. Probability sampling is statistically based and does
not depend solely on empirical criteria regarding the details of
possible contamination.
Incorporating Probability Sampling Would Allow Greater Confidence in
Negative Results:
We consider probability sampling to be a viable approach that would
address not only the immediate public health needs but also the wider
public health protection, infrastructure cleanup, and general
environmental contamination issues. We recognize that in a major
incident, the number of samples that may need to be collected and
analyzed may challenge available laboratory resources. Accordingly,
there is a need to develop innovative approaches to use sampling
methods that can achieve wide-area coverage with a minimal number of
individual samples to be analyzed. For example, high-efficiency
particulate air (HEPA) vacuum techniques, in combination with other
methods, appear to be one such approach that could achieve this. In
addition, because of limited laboratory capacity, samples may need to
be stored after collection for subsequent analysis, on a prioritized
basis.
The situation in 2001 was unique, and the agencies were not fully
prepared to deal with environmental contamination. In the future, if
the agencies decide to use a targeted rather than a probability
sampling strategy, they must recognize that they could lose a number of
days if their targeted sampling produces negative test results. In this
case, additional samples would need to be collected and analyzed,
resulting in critical time, for public health interventions, being
lost. This was so at the Wallingford postal facility in the fall of
2001, when about 3 weeks elapsed between the time the first sampling
took place and the results of the fourth testing, which revealed
positive results. Furthermore, about 5 months elapsed between the time
of the first sampling event and the time anthrax was found in the
Wallingford facility's high-bay area.
Therefore, in the future, strategies that include probability sampling
need to be developed in order to provide statistical confidence in
negative results. Further, even if information on all the performance
characteristics of methods is not yet available, a probability sampling
strategy could be developed from assumptions about the efficiency of
some of the methods. And even if precise data are not available, a
conservative, approximate number could be used for developing a
sampling strategy. This would enable agencies and the public to have
greater confidence in negative test results than was associated with
the sampling strategy used in 2001.
Activity 2: Collecting Samples:
The agencies used a variety of sample collection methods. USPS
primarily used the dry swab method. CDC and EPA used premoistened and
dry sterile, synthetic (noncotton) swabs, wet synthetic wipes, and HEPA
vacuums for sampling. To determine whether anthrax was airborne, CDC
performed air sampling in the Brentwood facility 12 days after the
contaminated letters were processed. Airborne anthrax spores pose a
health risk because they can cause inhalational anthrax, the most
serious form of the disease. Agency officials stated that laboratory
requirements had influenced the choice of sample collection methods.
For example, in the New York area, CDC used only dry swabs, following a
requirement by New York public health laboratories.
The majority of the samples were collected by the dry swab method,
which experts and others we interviewed considered the least effective.
Single methods were involved in 304 sampling events--that is, CDC and
USPS collecting dry swab samples (185) and CDC and others collecting
premoistened swabs (119).[Footnote 5] However, for some sampling
events, CDC used wet wipes, HEPA vacuum, and air samples at Brentwood
and swabs, wet wipes, and HEPA vacuum samples at Wallingford.
USPS officials said that the choice of dry swabs was based on advice
from CDC and an APHL working group, which had coordinated with the head
of LRN. CDC stated that the reason for the use of swabs was an
accommodation USPS had reached with APHL. According to APHL officials,
the working group consulted with CDC's NCID in November 2001. APHL said
that an NCID official, who was a member of the group, agreed that the
dry synthetic swab method could be used but that premoistened swabs
would pick up more spores.
During our fieldwork, we tried to determine what specific advice CDC
gave the Association of Public Health Laboratories (APHL) on using dry
swabs. In responding to our inquiry, CDC did not specifically deny
APHL's statement that an official from CDC's National Center for
Infectious Diseases (NCID) told APHL that dry swabs could be used.
However, an official from CDC's National Institute for Occupational
Safety and Health (NIOSH), which was not a member of the working group,
said that CDC has always recommended using premoistened swabs.
Nevertheless, according to APHL, "the NIOSH recommendation was not
known by the NCID working group members, nor did they advocate on its
behalf."
The decision to use dry rather than premoistened swabs stemmed partly
from the concern of some public health officials, including APHL
officials we interviewed, that moistened swabs would allow anthrax
spores to germinate, growing into vegetative cells instead of remaining
as spores. Other public health officials we interviewed said it was
highly unlikely that anthrax spores would germinate into vegetative
cells in a premoistened swab. APHL officials said that it was feared
that such vegetative cells would be destroyed during certain analytic
procedures. However, none of the agencies' collection methods were
evaluated for anthrax detection in environmental samples. In the
absence of empirical research, agencies had no information available
for reliably choosing one method over another and no information on the
limits of detection to use when evaluating negative results.[Footnote 6]
Activity 3: Transporting Samples:
Agencies transported samples by land or air to laboratories for
extraction and analysis (activities 4 and 5). The USPS sample
collection plan included shipping instructions that were based on
regulations for shipping infectious substances and designed to prevent
their inadvertent release. EPA's sample collection plan did not refer
to transportation requirements. According to CDC's guidelines, anthrax
samples were to be considered infectious substances and packaged
according to applicable federal regulations enforced by the Department
of Transportation. These regulations were aimed at "ensuring that the
public and the workers in the transportation chain are protected from
exposure to any agent that might be in the package."[Footnote 7] Among
other potential requirements, infectious material must be contained in
a securely sealed, pressure resistant, watertight, primary receptacle
surrounded by an absorbent and cushioning material. This material must,
in turn, be enclosed in a securely sealed, watertight, and durable
secondary packaging, which has to be enclosed in an outer packaging
constructed of fiberboard or equivalent material, as well as shock
absorbent material if more than 50 milliliters are shipped in one
package.
However, these regulations did not address one of the most important
issues--maintaining the biological integrity of samples while being
transported. Failure to do so could result in false negative test
results. For example, analysis by culture requires that spores can
germinate, divide and multiply, so that tests can determine whether a
sample contains anthrax. Temperature and exposure to certain kinds of
light, such as ultraviolet light, can be deleterious to some
microorganisms. Therefore, it is important that every sample collected
retain its original physical form before and during transportation.
We did not attempt to ascertain (1) the specific transit times for
delivering all the samples to laboratories, (2) whether sample
transportation was delayed, and (3) if it was, how long it was delayed.
We also did not attempt to ascertain the environmental conditions the
samples were shipped under or when they were received at the
laboratories. Finally, we did not attempt to ascertain the degree to
which spores could have been exposed to varying environmental
conditions from the time of release to the time of sample collection,
which could have affected sample integrity. Anthrax spores are robust,
compared with other pathogenic microorganisms, but whether
transportation affected their viability cannot be known because the
conditions of their transportation were not validated. Transport
conditions, once validated, would have to be standardized to ensure
reproducibility.
Activity 4: Extracting Samples:
LRN protocols required that sample material be extracted with specific
extraction procedures and fluids (such as sterile saline or water) and
that the extracted fluid be subjected to specific analytic methods. For
the samples USPS collected under the APHL agreement, the extraction
methods included adding a sample processing solution to the conical
tubes containing the dry swabs before "plating." This process was
adapted from LRN protocols for extracting swabs. However, the private
laboratory (not part of LRN) that originally analyzed the samples for
USPS did not use an extraction fluid; it inoculated the noncotton,
rayon-tipped dry swab directly onto a culture plate.
Several factors could have affected extraction efficiency. For example,
according to public health officials and other experts, the degree to
which swabs or wipes can retain spores depends on the material they are
made of. Cotton is more retentive than some artificial fibers like
rayon and may be more difficult for extraction of spores for analysis.
Other factors affecting spore extraction are the physical nature of the
collection device and surface properties. For example, swabs are easier
to manipulate and immerse in extract fluid than more bulky wipes are.
CDC has acknowledged that "the recovery efficiency of the analytical
methods has not been adequately evaluated."
The reproducibility of the results when an extraction fluid is used can
also be an issue. For example, a U. S. Army Medical Research Institute
for Infectious Diseases (USAMRIID) official we interviewed told us of
an unpublished USAMRIID study conducted to determine the efficiency of
extracting anthrax from swabs; the study showed that even if the same
procedure was followed, the results were not always the same.[Footnote
8] Although the importance of reproducibility has been recognized,
definitive scientific information regarding extraction efficiency is
lacking. In its absence, it is not clear whether sampling results were
affected, particularly with respect to samples that may have contained
few spores. Without knowing the extraction efficiency, a false negative
result may potentially be seen as a true negative.
Activity 5: Analyzing Samples:
Analyzing the samples involved a variety of methods and required two
steps--preliminary and confirmatory--to generate a final result. The
laboratory analytic methods that were used for detecting anthrax in
clinical samples already existed, but they had not been used for
environmental samples. As a result, different analytic approaches were
taken at the preliminary step, involving adaptations of such protocols.
Samples deemed positive at the preliminary step were not always
confirmed as positive, as was to be expected. However, this could cause
problems for the agencies. In addition, some agencies considered
preliminary analyses by field-based instruments unreliable, while
others maintained that they were reliable but had been used
inappropriately. However, once sample extracts were subjected to the
required confirmatory tests, a positive result was indeed a positive.
In analyzing the postal samples, laboratories used a variety of methods
for preliminary and confirmatory testing. Preliminary tests included
colony morphology, Gram's stain, hemolysis, and motility
tests.[Footnote 9] Any culture isolates that could not be ruled out in
the preliminary step of testing were considered presumptively positive
and referred for confirmatory testing. Confirmatory tests included
culture analyses (traditional microbiological and biochemical
analyses), gamma phage lysis (a test that identifies the susceptibility
of the organism to anthrax-specific viruses that create a kill zone in
anthrax cultures), and direct fluorescent antibody assay, or antibody
analyses employing a two-component test that detects the cell wall and
capsule, or outer covering, produced by vegetative cells of anthrax.
Other specialized tests, such as molecular subtyping, were also
conducted to determine what strain of anthrax was involved. The test
results were reported as positive--anthrax was found--or negative--
anthrax was not found. Traditional microbiological analyses require 18
to 24 hours before a result can be generated, depending on the
laboratory protocols and procedures. In a few instances, results were
also reported as number of colony forming units (CFU) per gram of
sample material.
According to CDC guidelines, LRN laboratories were to analyze samples
by appropriate LRN protocols. According to CDC, all LRN laboratories
were qualified to perform the preliminary tests, and most could perform
confirmatory and other specialized tests. While a lower level of LRN
laboratory could analyze swab samples for preliminary testing, all
other samples--such as bulk, wipes, air samples, or vacuum samples--
were to be analyzed at a higher level of LRN laboratory. Samples could
also be analyzed at CDC laboratories. Presumptive positives found at a
lower level LRN laboratory had to be referred to an appropriately
qualified laboratory for confirmatory testing.
The problems agencies encountered in preliminary testing included
issues related to training and quality control, as well as problems
with using field-based analytic methods with limitations that were not
well understood. In preliminary testing, a suspect organism must first
be selected; at this point, human error or quality control issues can
affect the results. For example, we identified a problem involving
culture in the preliminary tests--that is, a reliance on the naked
human eye to identify and select the growth of anthrax on the petri
dish. Many different types of organisms could be growing that looked
like, but were not, anthrax. This is significant because when negative
results were obtained during preliminary testing, no further testing
was to be done.
The agencies were also faced with problems when deciding how to respond
to preliminary positive results that might eventually turn out to be
confirmed otherwise. For example, agencies did not have clear criteria
for when to close facilities. In addition, although hand-held assays
(HHA) were considered preliminary tests, concerns were raised that the
negative results might lead to a false sense of security.[Footnote 10]
During the 2001 incidents, USPS kept the Brentwood facility open,
following CDC's advice that closing it was not warranted. According to
USPS officials, the correctness of this advice appeared to be confirmed
by the HHA results obtained on October 18, 2001. When CDC confirmed a
case of inhalation anthrax in a Brentwood employee on October 21, 2001,
the facility was closed that day. According to USPS, it was not until
October 22, 2001, that the laboratory's culture tests of the other
samples, collected on October 18, revealed positive results. In a more
recent instance, on November 6, 2003, USPS shut down 11 postal
facilities in and around Washington, D.C., after a preliminary test--
not a confirmed result--from a routine air sample taken on November 5
indicated that a naval mail processing facility might be contaminated
with anthrax. USPS tracked the flow of mail through its own facilities
and closed 11 postal facilities that delivered mail to the naval
facility. The subsequent confirmatory tests were negative, and the
facilities were reopened about 3 days later.
All the activities discussed above are interdependent, and many
variables for each one can affect the results. Further, problems
associated with any one of these activities could affect the validity
of the results generated by the overall process. Given that there are
so many variables, the use of different sample collection strategies,
reflected in site-specific plans, could yield different results. For
example, three potential sample collection plans could be used in one
facility--plan A, using one collection method (for example, a swab);
plan B, using two methods (for example, a swab and wipe); and plan C,
using three methods (for example, swab, wipe, and HEPA vacuum). How
these collection methods are to be applied--that is, how they are
physically used and how much area each sample covers--is a variable.
Within each plan, sample transportation protocols could differ,
involving variables such as temperature--plans A and B might require
transporting at ambient temperature, while plan C might require
freezing temperature--the sample collection method's moistness during
transport, and the size and construction of the packaging.
In addition, within each plan, laboratory extraction and analysis
protocols could differ, involving variables such as (1) different
manufacturers' different formulations of extraction fluids, (2)
different ways to physically release spores from a particular
collection method (such as a swab) into the liquid extract (such as by
shaking or vortexing), and (3) a combination of analytic methods, such
as culture or polymerase chain reaction (PCR) for deoxyribonucleic acid
(DNA) amplification to identify anthrax. Any problems experienced with
any of these variables across any of these plans could affect the final
result.
The Sampling Results Were Largely Negative:
The results of the CDC, EPA, and USPS testing in 286 postal facilities
were largely negative. Of 286 facilities, 23 tested positive. For 2 of
these 23 facilities, test results were negative at first but positive
on a subsequent testing. However, in 1 of these facilities--the
Wallingford, Connecticut, facility--it was not until the fourth testing
that positive results were obtained.
Testing results differed between the primary facilities and
Wallingford. First, in the three primary facilities, results were
positive each time a facility was tested, with the important exception
of the two quick tests in Brentwood. In Wallingford, considered less
likely to be contaminated, results were positive only on the fourth
sampling. Second, in the primary facilities, sampling with a single
method produced some positive results, regardless of the sample
collection method. In Wallingford, neither dry nor premoistened swabs
produced any positive results. Third, in the primary facilities, both
single and multiple methods produced positive results; in Wallingford,
only multiple methods produced positive results.
When comparing the positive results, obtained with dry swabs, across
the primary facilities, the proportions differed. For example, in one
sampling event in Brentwood, out of 29 samples collected using dry
swabs, 14 were positive (48 percent), whereas in Morgan, out of 56,
only 7 were positive (13 percent). In addition, for the West Palm
Beach, Florida, facility, sampled several times during one sampling
event, out of 38 dry swab samples collected, only 1 was positive (about
3 percent). While we did not define this facility as primary, it was
suspected of processing a contaminated letter, although none was found.
However, the use of both wet and dry swabs produced positive results in
this facility.
USPS and CDC sampled facilities that processed mail from the primary
facilities to determine whether any other facilities had become
contaminated. The majority of test results from these facilities were
negative: Of 286 facilities sampled, 23 tested positive, including the
3 primary facilities, and 263 tested negative.
For some of the positive facilities, excluding the primary ones:
* Generally, only 1 or 2 of the total samples collected for each
facility were positive, such as several post offices that received mail
from Brentwood, including Dulles (11 samples collected, 1 positive),
Friendship Station (32, 1 positive), Pentagon Station (17, 2 positive),
and Raleigh, North Carolina (42, 1 positive). These facilities were
considered cross-contaminated.
* West Palm Beach and Wallingford tested positive only on retesting,
whereas initially they had tested negative. The West Palm Beach
facility tested positive on the second testing. According to CDC, the
sampling strategy used in this facility was found to have limitations
and was not used again. However, Wallingford did not test positive
until the fourth testing. These results underscore the importance of
retesting and cast doubt on the efficiency of the testing.
Of the 263 facilities that tested negative, only 9 were sampled more
than once. A facility in West Trenton tested negative, even though an
employee had contracted cutaneous anthrax. The facility in West Trenton
was tested twice by the FBI and once by CDC, during which a total of 57
samples were collected, with negative results.
Final, or confirmed, results will be negative if contamination is not
present in a facility. However, a result can be negative for several
other reasons, such as (1) the sampling method was not efficient
enough, (2) samples were not collected from places where contamination
was present, (3) not enough samples were collected, (4) not enough
spores were recovered from the sample material, or (5) analysis of the
sample extract was not sensitive enough to detect anthrax spores that
were present (that is, the result was a false negative).
Agencies' Activities Were Not Validated:
None of the agencies' activities to detect anthrax contamination in the
postal facilities were validated. Without validation, the sampling
activities could have been based on false assumptions. Using an
ineffective method or procedure could result in a finding of no
contamination when in fact there is contamination--a false negative.
Because the sampling methods are not validated, it is not known to what
extent they will underestimate contamination. Thus, in the case of a
negative result, agencies would have no sound basis for taking public
health measures for the occupants of the contaminated facility.
Validation, as it is generally understood, is a formal, empirical
process in which the overall performance characteristics of a given
method are determined and certified by a validating authority as (1)
meeting the requirements for the intended application and (2)
conforming with applicable standards. Because the agencies did not use
an empirical process to validate their testing methods, the agencies
had limited information available for reliably choosing one method over
another and no information on the detection limit to use when
evaluating negative results.
Validating the overall process is important because operational and
health-related decisions are made on the basis of testing results
generated by that process. In addition, validation would offer
assurance that the results of using a particular method, which is part
of that process, are robust enough to be reproduced, regardless of
which agency, contractor, or laboratory is involved. Thus, agencies and
the public could be reasonably confident that any test results
generated by a process that includes that method would be reliable and,
in particular, that any negative results would mean that a sample was
free from contamination (within the method's limits of detection).
In preparing for future incidents, the agencies have (1) made some
changes based on what has been learned about some of the limitations of
their sampling strategies, (2) made some revisions to their guidelines
to reflect some of this knowledge and experience or developed new ones,
(3) funded some new research, and (4) planned or conducted conferences
addressing some of the issues we have identified. In addition, DHS has
taken on the role of coordinating agencies' activities and has
undertaken several new initiatives related to dealing with anthrax and
other biothreat agents.
However, while the actions DHS and other agencies have taken are
important, they do not address the issue of validating all activities
related to sampling. Since the fall of 2001, studies have been
performed, or are under way, that may contribute to the validation of
the individual activities. Nonetheless, these studies address only some
aspects of an individual activity rather than the overall process.
Finally, the agencies have not made appropriate and prioritized
investments to develop and validate all activities related to anthrax
and other biothreat agents.
Conclusions:
The lack of validated methods for assessing contamination in postal
facilities impeded the agencies in responding to the incidents. The
need that all methods, from sampling to final analysis, be validated,
so that their performance characteristics can be clearly understood, is
not in doubt. But any combination of methods that makes up the overall
process should also be validated because the effect of different
permutations of methods may not be predictable. It must be recognized,
however, that an inability to validate the entire process reduces, to
some degree, the level of confidence in the results. To assess the
impact of relying on the validation of individual activities,
experiments could be performed with a limited number of processes,
combining different methods.
The issues we have raised in this report apply to any anthrax incident,
including the March 2005 incident involving DOD facilities in the
Washington, D.C. area. In addition, while the 2001 events involved
anthrax, many other biothreat agents exist. Differences in their
characteristics mean different solutions. Accordingly, efforts to
develop sampling strategies and to validate methods should address
requirements specific to those biothreat agents as well. However, since
addressing other agents would consume resources and time, these efforts
should be prioritized in a long-term strategy.
The several agencies that dealt with the anthrax attacks generally
worked well together, but we have identified areas that would have
benefited from one agency's taking the lead in coordinating the
response. Given the mission of DHS and its responsibilities, it appears
that DHS is now well positioned to take a lead role in promoting and
coordinating the activities of the various agencies that have technical
expertise related to environmental testing. In addition, it is
important that all participating agencies recognize and support DHS in
that role and that they have an effective structure for participating
in identifying and addressing the appropriate issues.
Recommendations for Executive Action:
Accordingly, in our report, we recommended that to improve the overall
process for detecting anthrax and to increase confidence in negative
test results generated by that process, the Secretary of Homeland
Security develop a coordinated approach. This approach would include
working with agencies to ensure that appropriate validation studies of
the overall process of sampling activities, including the methods, are
conducted. Specifically, the Secretary should (1) take a lead role in
promoting and coordinating the activities of the various agencies with
technical expertise related to environmental testing; (2) ensure that a
definition of validation is developed and agreed on; (3) guarantee that
the overall process of sampling activities, including methods, is
validated so that performance characteristics, including limitations,
are clearly understood and results can be correctly interpreted; (4)
see that appropriate investments are made in empirical studies to
develop probability-based sampling strategies that take into account
the complexities of indoor environments; (5) ensure that appropriate,
prioritized investments are made for all biothreat agents; and (6)
ensure that agency policies, procedures, and guidelines reflect the
results of such efforts.
We obtained written comments on a draft of this report from CDC, DHS,
and USPS. We also obtained written comments from APHL on excerpts from
the draft that pertained to its role in anthrax testing. Although we
requested comments from DOD and EPA, DOD said it had no comments and
EPA provided only technical comments.
CDC, DHS, and USPS, as well as APHL, agreed with our conclusion--
methods for detecting anthrax contamination in facilities were not
validated--and with the thrust of our recommendations--calling for a
coordinated, systematic effort to validate the methods to be used for
such testing.
In response, DHS stated that while it has the overall responsibility
for coordination for future biological attacks, EPA has "the primary
responsibility of establishing the strategies, guidelines, and plans
for the recovery from a biological attack while HHS has the lead role
for any related public health response and guidelines." DHS further
stated that EPA "is developing specific standards, protocols, and
capabilities to address the risks of contamination following a
biological weapons attack and developing strategies, guidelines, and
plans for decontamination of persons, equipment, and facilities." DHS
pointed out that in the Conference Report on H.R. 4818, the conferees
expressed their expectation that EPA will enter into a comprehensive
MOU [memorandum of understanding] with DHS no later than August 1, 2005
that will define the relationship and responsibilities of these
entities with regard to the protection and security of our Nation. The
Conferees expect the MOU to specifically identify areas of
responsibilities and the potential costs (including which entity pays,
in whole or part) for fully meeting such responsibilities. EPA shall
[is to] submit to the House and Senate Committees on Appropriations a
plan no later than September 15, 2005 that details how the agency will
meet its responsibilities under the MOU, including a staffing plan and
budget.
Finally, DHS stated, "Even though DHS is in charge during a biological
attack, EPA is primarily responsible for the coordination of the
recovery process. So, DHS will coordinate with EPA to ensure
appropriate investments are made to explore improved sampling." With
respect to our recommendation that DHS develop probability-based
sampling strategies, DHS said that it must first define the necessary
requirements for the sampling process and then evaluate targeted and
probability-based sampling strategies against those requirements. DHS
said that targeted sampling may be beneficial for some applications. We
agree with DHS on the need to define the requirements for the sampling
process and to evaluate sampling approaches against those requirements.
On the basis of the work we have done on this review, we believe that
(1) DHS will find that targeted sampling will not always meet all the
requirements to answer the question of whether a facility is
contaminated and (2) probability-based sampling will be necessary when
information on the source and path of potential contamination is not
definitive. In our view, probability sampling will be necessary in
order for DHS to achieve its goal of having a "scientifically
defensible sampling strategy and plan."
Mr. Chariman, this concludes my prepared statement. I will be happy to
answer any questions you or Members of the Subcommittee may have.
Contacts and Acknowledgments:
If you or your staff have any questions about this report or would like
additional information, please contact me at (202) 512-6412, or Sushil
Sharma, PhD., DrPH, at (202) 512-3460. We can also be reached by e-mail
at rhodesk@gao.gov and sharmas@gao.gov.
Other staff that contributed to this report include Hazel Bailey,
Heather Balent, Venkareddy Chennareddy, Crystal Jones, Jack Melling,
Penny Pickett, Laurel Rabin, Mark Ramage, and Bernard Ungar.
[End of section]
Appendix I: Scope and Methodology:
To respond to your request, we interviewed officials from federal
agencies involved in sampling the postal facilities. The federal
agencies included the Centers for Disease Control and Prevention (CDC),
the Environmental Protection Agency (EPA). We also interviewed U.S.
Postal Service (USPS), Association of Public Health Laboratories
(APHL), public health and private sector laboratories, and experts on
microbial detection in indoor environments.
We reviewed documentation provided or developed by CDC, EPA, and USPS,
including sample collection strategies, guidance, environmental
collection and analytical methods and protocols. In addition, we
reviewed and analyzed test results data, that is, sample collection and
analytical data collected by federal agencies, their contractors, and
public health laboratories. We did not independently verify these data.
We conducted site visits to some postal facilities affected by anthrax
and some public health and private sector laboratories that were
involved in analyzing samples. We also reviewed studies on sampling
methods for detecting biological substances, including anthrax, on
surfaces and in the air. We conducted our review from May 2003 through
November 2004 in accordance with generally accepted government auditing
standards.
Although our study focused on anthrax testing relating to 2001 anthrax
incident, we believe that the issue we identified concerning the need
for validated methods and sound sampling strategies would apply to
similar incidents in future. This is particularly evident given the
consequences arising from the March 2005 incident involving facility
closures following preliminary anthrax testing in the Washington, D.C.
area.
FOOTNOTES
[1] GAO, Anthrax Detection: Agencies Need to Validate Sampling
Activities in Order to Increase Confidence in Negative Results, GAO-05-
251 (Washington, D.C.: March 31, 2005). www.gao.gov.
[2] "Anthrax" in this testimony reflects commonly used terminology.
Technically, the term refers only to the disease caused by the
microorganism Bacillus anthracis, not the bacterium itself or its
spores.
[3] Anthrax contamination had been found earlier in several Florida
postal facilities that processed mail for the American Media
Incorporated building there. However, no letter containing anthrax was
ever found.
[4] According to the head of the Postal Inspection Service, more than
7,000 hoaxes, threats, and suspicious letters and packages--an average
of almost 600 a day--were reported to his agency in the weeks following
the first anthrax incident. As a result, nearly 300 postal facilities
had to be evacuated.
[5] We use "sampling event" to refer to initial sample collection by a
specific agency on a specific day and at a specific time in a specific
facility. Multiple agencies collected samples on the same day in some
of the same facilities; therefore, each agency's sample collection is
considered a separate sampling event. As a result, there were more
sampling events than the total number of facilities sampled.
[6] The published literature provided some information on the
efficiency of a few sample collection methods. In all the methods
studied, swabs were always premoistened before samples were collected.
However, according to one study, the most efficient method caused
problems when used with certain analytic methods.
[7] Department of Transportation, 49 C.F.R. subchapter C--Hazardous
Materials Regulation. The USPS regulations mirror the Department of
Transportation regulations. However, to be transported as mail,
material must be classified as mailable. By statute, infectious
materials, such as anthrax spores, that are "disease germs or scabs,
[or] other natural or artificial articles, compositions, or material
which may kill or injure another" cannot be mailed. Such materials are
termed "nonmailable matter." Knowingly mailing such material is a
criminal offense, and doing so with the intent to kill or injure is a
felony. When an etiologic material is not "outwardly or of [its] own
force dangerous or injurious to life, health, or property," USPS may
allow it to be mailed, subject to appropriate rules and regulations
governing its preparation and packing. As a result, USPS allows the
mailing of small quantities of appropriately packaged infectious
material, but only if it is intended for medical or veterinary use,
research, or laboratory certification related to public health.
[8] Using synthetic swabs and a particular type of buffer could lead to
70 to 75 percent extraction. However, repeating the test with the same
type of buffer made by different companies yielded different results.
The official said that this test showed that there were too many
variables. Even when analysts followed the same procedure, the results
were not always reproducible, casting doubt on the reliability of the
test results.
[9] When bacteria stained with Gram's stain retained the color of the
primary stain (crystal violet), they were considered gram-positive, a
characteristic of anthrax. Hemolysis, a procedure involving culturing,
identified whether the colonies gave no evidence of red blood cell
lysis, a characteristic of anthrax. Motility refers to whether the
colonies showed no movement in microscopic observation, another
characteristic of anthrax.
[10] See GAO, U.S. Postal Service: Better Guidance Is Needed to Ensure
an Appropriate Response to Anthrax Contamination, GAO-04-239
(Washington D.C.: Sept. 9, 2004).
