Darfur Crisis
Death Estimates Demonstrates Severity of Crisis, but Their Accuracy and Credibility Could Be Enhanced Gao ID: GAO-07-24 November 9, 2006In 2003, violent conflict in Darfur, Sudan, broke out between rebel groups and government troops and government-supported Arab militias. While few would dispute that many thousands of Darfur civilians have died, less consensus exists about the total number of deaths attributable to the crisis. Estimates by the Department of State (State) and other parties report death tolls up to about 400,000 for varying populations and periods of time between February 2003 and August 2005. Based on the views of experts convened by GAO and the National Academy of Sciences, interviews with estimate authors, and a review of relevant literature, this report (1) evaluates six Darfur death estimates, (2) identifies general challenges to estimating deaths in such crises, and (3) discusses measures to improve death estimates.
The experts GAO consulted did not consistently rate any Darfur death estimate as having a high level of accuracy; moreover, they noted that all the studies had methodological strengths and shortcomings. Most of the experts had the highest overall confidence in estimates by the Centre for Research on the Epidemiology of Disasters (CRED) and had a slightly lower level of confidence in State's estimate. Many experts believed State's lower-end estimate was too low. Additionally, the published documents describing State's estimate lacked sufficient information about its data and methods to allow it to be replicated and verified by external parties. Estimating deaths in a humanitarian crisis such as that in Darfur involves numerous challenges. For example, in Darfur, difficulties in collecting mortality data, such as lack of access to particular geographical regions, impacted the data's quality and led to data gaps. Because of such data gaps, some Darfur death estimates relied on potentially risky assumptions and limited contextual information. Further, limitations in estimates of Darfur's population before and during the crisis may have led to over- or underestimates of the death toll. Finally, varying use of baseline mortality rates--the rate of deaths that would have occurred without the crisis--may have led to overly high or low death estimates. The experts proposed and rated a wide range of measures that U.S. agencies could take to improve the quality and reliability of death estimates for Darfur and future humanitarian crises. Among these measures, the most highly rated was ensuring that public documentation of the data and methods used contain sufficient information to enable external replication and verification of the estimates. Other very highly rated measures include collecting and maintaining data for specific periods of time and geographic areas and housing the responsibility for making estimates in a reputable independent body.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-07-24, Darfur Crisis: Death Estimates Demonstrates Severity of Crisis, but Their Accuracy and Credibility Could Be Enhanced
This is the accessible text file for GAO report number GAO-07-24
entitled 'Darfur Crisis: Death Estimates Demonstrate Severity of
Crisis, but Their Accuracy and Credibility Could Be Enhanced' which was
released on December 11, 2006.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
United States Government Accountability Office:
GAO:
November 2006:
Darfur Crisis:
Death Estimates Demonstrate Severity of Crisis, but Their Accuracy and
Credibility Could Be Enhanced:
GAO-07-24:
GAO Highlights:
Highlights of GAO-07-24, a report to congressional requesters
Why GAO Did This Study:
In 2003, violent conflict in Darfur, Sudan, broke out between rebel
groups and government troops and government-supported Arab militias.
While few would dispute that many thousands of Darfur civilians have
died, less consensus exists about the total number of deaths
attributable to the crisis. Estimates by the Department of State
(State) and other parties report death tolls up to about 400,000 for
varying populations and periods of time between February 2003 and
August 2005. Based on the views of experts convened by GAO and the
National Academy of Sciences, interviews with estimate authors, and a
review of relevant literature, this report (1) evaluates six Darfur
death estimates, (2) identifies general challenges to estimating deaths
in such crises, and (3) discusses measures to improve death estimates.
What GAO Found:
The experts GAO consulted did not consistently rate any Darfur death
estimate as having a high level of accuracy; moreover, they noted that
all the studies had methodological strengths and shortcomings. Most of
the experts had the highest overall confidence in estimates by the
Centre for Research on the Epidemiology of Disasters (CRED) and had a
slightly lower level of confidence in State‘s estimate. Many experts
believed State‘s lower-end estimate was too low. Additionally, the
published documents describing State‘s estimate lacked sufficient
information about its data and methods to allow it to be replicated and
verified by external parties.
Estimating deaths in a humanitarian crisis such as that in Darfur
involves numerous challenges. For example, in Darfur, difficulties in
collecting mortality data, such as lack of access to particular
geographical regions, impacted the data‘s quality and led to data gaps.
Because of such data gaps, some Darfur death estimates relied on
potentially risky assumptions and limited contextual information.
Further, limitations in estimates of Darfur‘s population before and
during the crisis may have led to over- or underestimates of the death
toll. Finally, varying use of baseline mortality rates”the rate of
deaths that would have occurred without the crisis”may have led to
overly high or low death estimates.
The experts proposed and rated a wide range of measures that U.S.
agencies could take to improve the quality and reliability of death
estimates for Darfur and future humanitarian crises. Among these
measures, the most highly rated was ensuring that public documentation
of the data and methods used contain sufficient information to enable
external replication and verification of the estimates. Other very
highly rated measures include collecting and maintaining data for
specific periods of time and geographic areas and housing the
responsibility for making estimates in a reputable independent body.
Figure: Darfur Village Burning:
[See PDF for Image]
Source: United States Holocaust Museum, Photograph by Brian Steidle.
[End of Figure]
What GAO Recommends:
To safeguard the U.S. government‘s credibility as a source of reliable
death estimates, GAO recommends ensuring greater transparency regarding
the data and methods used for such estimates. GAO also recommends that
the U.S. government consider other measures suggested by the experts to
help address gaps in data and improve the quality of any future death
estimates. State and the U.S. Agency for International Development
agreed with GAO‘s recommendations.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-24].
To view the full product, including the scope and methodology, click on
the link above. For more information, contact David Gootnick at (202)
512-3190 or gootnickd@gao.gov.
[End of Section]
Contents:
Letter:
Results in Brief:
Background:
Some Death Estimates Judged More Accurate and Methodologically
Stronger, Despite Shortcomings:
Estimating Deaths in Humanitarian Crises Involves Many Challenges:
Wide Range of Measures Could Improve Death Estimates for Darfur and
Future Humanitarian Crises:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: List of Experts and Meeting Agenda:
Appendix III: List of Death Estimates and Mortality Surveys Provided to
Experts and Additional Bibliographical References:
Appendix IV: Summary Description of Death Estimates Reviewed:
Appendix V: Additional Follow-Up Survey Results:
Appendix VI: Comments from the Department of State:
GAO Comments:
Appendix VII: Comments from the U.S. Agency for International
Development:
Appendix VIII: Summary of Authors' Comments:
Appendix IX: GAO Contacts and Staff Acknowledgments:
Tables:
Table 1: Accuracy of Darfur Death Estimates Rated by Experts:
Table 2: Measures Rated by Experts as Likely to Greatly Improve Death
Estimates for Future Crises, in Order of Ranking and Number of
Endorsements:
Figures:
Figure 1: Map of Sudan:
Figure 2: Map of Destroyed and Damaged Darfur Villages, as of February
2005:
Figure 3: Example of Generation of a Death Estimate:
Figure 4: Death Estimates Based on Reported Figures and Time Included:
Abbreviations:
CDC: Centers for Disease Control and Prevention:
CE-DAT: Complex Emergency Database:
CIJ: Coalition for International Justice:
CMR: crude mortality rate:
CRED: Centre for Research on the Epidemiology of Disasters:
IDP: internally displaced persons:
JEM: Justice and Equality Movement:
NGO: nongovernmental organization:
SLM/A: Sudan Liberation Movement/Army:
SMART: Standardized Monitoring and Assessment of Relief and
Transitions:
USAID: U.S. Agency for International Development:
UN: United Nations:
WHO: World Health Organization:
United States Government Accountability Office:
Washington, DC 20548:
November 9, 2006:
The Honorable Tom Lantos:
Ranking Minority Member:
Committee on International Relations:
House of Representatives:
The Honorable Mike DeWine:
The Honorable Richard J. Durbin:
United States Senate:
In 2003, violent conflict broke out in the Darfur region of western
Sudan when rebel groups, believing that the region was marginalized by
the Sudanese government, led attacks against the government. In
response, the government armed and supported local Arab and tribal
militias, commonly known as the Janjaweed, to defeat the rebels.
Attacks on the civilian population by these militias, sometimes in
conjunction with the Sudanese army, have resulted in widespread death
and disruption. The crisis has affected an estimated 3.76 million
people in Darfur, including approximately 1.85 million--"internally
displaced persons" (IDPs)--who now live in camps. To aid these
populations, the United States has provided more than $1 billion in
assistance for Darfur since fiscal year 2004, largely through the U.S.
Agency for International Development (USAID).[Footnote 1]
While few would dispute that many civilians have died in Darfur owing
to violence, disease, and malnutrition, less consensus exists about the
total number of deaths during, or directly resulting from, the crisis.
The U.S. Department of State (State) reported that a total of 98,000 to
181,000 people died between March 2003 and January 2005. Five other
studies--conducted for varying purposes by international institutions,
academics, and individual researchers--have produced estimates ranging
up to about 400,000 deaths for various periods of time between February
2003 and August 2005.[Footnote 2]
Because policymakers require an accurate estimate of the death toll in
Darfur to understand the dimensions of the crisis and determine the
U.S. response, we were asked to examine the estimates, the methods used
to produce them, and their relative accuracy. This report (1) evaluates
the relative accuracy and methodological strengths and shortcomings of
the six death estimates for Darfur;[Footnote 3] (2) identifies general
challenges to estimating the total death toll in Darfur and similar
humanitarian crises; and (3) discusses measures that the U.S.
government could take to improve its death estimates for Darfur and any
similar, future crisis.
To evaluate the estimates, we reviewed and analyzed public information
on the estimates and interviewed the estimate authors regarding their
studies' data, methods, and objectives. We provided this information
and summaries of the interviews to a group of 12 experts in
epidemiology, demography, statistics, and the Darfur crisis convened in
April 2006 in collaboration with the National Academy of
Sciences.[Footnote 4] These experts discussed their review of this
information and evaluation of the estimates during an all-day session
and also assessed the estimates in a follow-up survey. State's Bureau
of Intelligence and Research, which conducted the department's death
estimate for Darfur, declined to speak with us or provide additional
information, limiting the experts' ability to fully understand State's
methods of analysis. However, despite this limitation, the experts were
able to discuss State's estimate in detail and assess its accuracy and
methodologies. To identify challenges of estimating total deaths in
Darfur, we asked the group of experts to highlight key problems, and we
reviewed literature related to death estimates and mortality data in
crises such as the one in Darfur. To identify ways in which the U.S.
government could improve death estimates for that and any future
humanitarian crisis, we asked the experts to provide suggestions during
their discussion, and we solicited further opinions on these
suggestions in the follow-up survey. Additionally, to learn about
current practices related to the experts' suggestions for improving
death estimates, we spoke with officials from USAID, State, and the
Centers for Disease Control and Prevention (CDC) in the U.S. Department
of Health and Human Services. Appendix I provides additional details of
our objectives, scope and methodology. We conducted our work from
September 2005 to November 2006 in accordance with generally accepted
government auditing standards.
Results in Brief:
The experts we consulted did not consistently rate any of the death
estimates as having a high level of accuracy and noted that all of the
studies had methodological strengths and shortcomings. Most of the
experts had the highest overall confidence in the estimates by the
Centre for Research on the Epidemiology of Disasters (CRED), which
relied primarily on a statistical analysis of about 30 mortality
surveys, and they rated the CRED estimates' accuracy and methodological
strengths highest among the six.[Footnote 5] The experts had a slightly
lower level of confidence in the State estimate and gave it slightly
lower ratings for accuracy and methodological strengths. Further, many
experts believed that the lower end of State's estimate was too low and
found that published documents describing State's estimate lacked
sufficient information about its data and methods to allow it to be
replicated and verified by external researchers. Most experts rated an
estimate by the World Health Organization (WHO), which primarily
extrapolated findings from its 2004 mortality survey, somewhat lower
than the estimates by CRED and State. Finally, most of the experts
expressed the least confidence in three estimates that reported the
highest number of deaths. They cited several shortcomings in these
estimates, such as a reliance on unrealistic assumptions regarding
fixed levels of mortality for all populations and time periods included
in the estimate.[Footnote 6]
Estimating deaths in a humanitarian crisis such as that in Darfur
involves numerous challenges. In Darfur, difficulties related to the
collection of survey data, including a lack of access to particular
geographical regions, challenging survey conditions, and limited
resources and training for research staff, impacted the data's quality
and resulted in data gaps. Because of the lack of complete mortality
data, some of the estimates rely on potentially risky assumptions and
limited contextual information. For example, some estimates assume that
mortality rates in accessible areas can be applied to inaccessible
areas, without accounting for differences between the two due to
factors such as patterns of violence and humanitarian relief efforts.
In addition, limitations in estimates of Darfur's population before and
during the crisis may have led to over-or underestimations of the death
toll. Finally, varying use of baseline mortality rates--the rate of
deaths that would have occurred regardless of the crisis--may also have
led to overly high or low death estimates.
The group of experts proposed and rated a wide range of measures that
U.S. agencies could take, directly or by supporting other
organizations, to improve the quality and reliability of death
estimates and relevant data for Darfur and any future such humanitarian
crises. The most highly rated measure was ensuring that publicly
available documentation of U.S. government estimates contains
sufficient information on data and methods, so that external
researchers can replicate the estimates and verify their credibility
and objectivity. Other very highly rated measures that the government
could consider included collecting and maintaining temporal and spatial
data (i.e., data covering specific periods of time and geographic
areas); housing the responsibility for making estimates in a reputable
independent body; improving the training of nongovernmental
organization (NGO) staff who collect mortality survey data; and
promoting an interdisciplinary approach to estimating mortality.
Through various initiatives already under way, U.S. agencies are
carrying out efforts that may address some of the experts' suggestions,
for example, providing guidance on the design and implementation of
survey instruments and supporting the collection and maintenance of
temporal and spatial data.
To safeguard State's credibility as a source of accurate and reliable
death estimates, we recommend that the Secretary of State provide, for
this and any future estimates of mortality that State conducts,
sufficiently detailed descriptions of its data and methodology to
enable other parties to assess and replicate its findings.[Footnote 7]
Additionally, to enhance the U.S. government's capacity to assess and
respond to any future humanitarian crisis, we recommend that the
Secretary of State and the Director of U.S. Foreign Assistance and
USAID Administrator consider the experts' other suggestions to help
address gaps in data and improve any future death estimates.
We provided a draft of this report to the Department of State and
USAID. State and USAID responded with formal comments, agreeing with
our recommendations, and State provided additional perspectives on
reporting and documentation regarding its death estimate. Reproductions
of these letters, as well as our responses to the letters, can be found
in appendixes VI and VII. We also provided a draft to the CDC for
technical review, and we received technical comments from both the CDC
and State, which we incorporated in the report as appropriate. Finally,
we provided the authors of the other five estimates the portions of the
report pertaining to their individual estimates. They provided
comments, which we incorporated in the report as appropriate. These
authors' comments and our responses are summarized in appendix VIII.
Background:
Sudan is the largest country in Africa, with a population estimated at
about 40 million (see fig. 1). Darfur is in the western region of Sudan
and comprises three states, with an estimated preconflict population of
around 6 million.[Footnote 8]
Figure 1: Map of Sudan:
[See PDF for image]
Source: GAO based on Map Resources (map).
[End of figure]
Conflict in Darfur:
In early 2003, Darfur rebels attacked Sudanese police stations and the
airport in El Fasher, the capital of North Darfur. In El Fasher, the
rebels destroyed numerous military aircraft, killed many Sudanese
soldiers, and kidnapped a Sudanese general. In response, the government
armed and supported the local tribal and Arab militias (the Janjaweed).
Fighting between the principal rebel groups--the Sudan Liberation
Movement/Army (SLM/A) and the Justice and Equality Movement (JEM)--and
the Sudan military and Janjaweed intensified during late 2003. In
addition to displacing approximately 1.85 million Darfurians in the
region, attacks on civilians by the Sudan military and Janjaweed have
forced an estimated 220,000 Darfur refugees to take shelter in
Chad[Footnote 9] and so affected approximately 1.91 million additional
Darfur residents that they require humanitarian assistance. (Fig. 2
shows the locations of Darfur villages destroyed or damaged in the
conflict.)
Figure 2: Map of Destroyed and Damaged Darfur Villages, as of February
2005:
[See PDF for image]
Source: GAO based on a map from State's Humanitarian Information Unit
and map Resources (map).
[End of figure]
Peace negotiations under the mediation of the Chad government in 2003
led to a September 2003 cease-fire agreement between the SLM/A and the
Sudan government; however, the agreement collapsed in December of that
year. A second cease-fire agreement was signed by the Sudanese
government, the SLM/A, and the JEM in April 2004.[Footnote 10] At this
point, the African Union was brought in to monitor compliance with the
cease-fire agreement between the three parties, and peace negotiations
continued under African Union auspices with Chadian
participation.[Footnote 11] After a relatively calm 2005, cease-fire
violations and violent incidents reportedly began to increase in the
final months of the year and into 2006.
On May 5, 2006, the government of Sudan and one faction of the SLM/A
signed the Darfur Peace Agreement, which establishes agreements in key
areas such as power sharing, wealth sharing, and security
arrangements.[Footnote 12] The U.S. government and other parties
support a transition of peacekeeping responsibilities from the African
Union to the UN. In August 2006, the UN Security Council adopted a
resolution expanding the mandate of the UN Mission in the Sudan and
calling for the mission's deployment to Darfur.[Footnote 13] As of
October 2006, the Sudanese president had rejected a UN transition but
expressed support for a September offer by the UN Secretary-General to
assist the African Union Mission in Sudan by providing equipment and
dedicated personnel. Meanwhile, the environment in Darfur remained
insecure, with attacks and displacement continuing and, during some
periods, worsening over time.
U.S. Response to Darfur Crisis:
The U.S. government has been active in addressing the conflict. On July
22, 2004, the U.S. House and the Senate each passed separate
resolutions citing events in Darfur as acts of genocide.[Footnote 14]
Further, on September 9, 2004, in testimony before the Senate Foreign
Relations Committee, the U.S. Secretary of State announced that
genocide had been committed in Darfur and that the Sudanese government
had supported the Janjaweed, directly and indirectly, as they carried
out a "scorched earth" policy toward the rebels and the African
civilian population in Darfur.[Footnote 15]
According to State officials, the administration's declaration of
genocide was influenced by findings from interviews with 1,136 Darfur
refugees in eastern Chad in July and August 2004, which demonstrated a
pattern of abuse against members of Darfur's non-Arab communities. The
interviews were conducted by an "Atrocities Documentation Team"
assembled by State and consisting of officials from State and USAID, as
well as members of the Coalition for International Justice[Footnote 16]
and the American Bar Association. State reported that 61 percent of
those interviewed said that they had directly witnessed the killing of
a family member in addition to other crimes.[Footnote 17]
For fiscal years 2004 through 2006, the United States obligated $996
million in humanitarian assistance for Darfur.[Footnote 18] Although
more than 68 percent of this assistance has consisted of food aid, U.S.
assistance to Darfur has also supported other vital needs, such as
water and sanitation, shelter, and primary health care services. Partly
as a result of U.S. assistance, NGOs and UN agencies have made
significant progress in increasing the number of internally displaced
persons and affected residents receiving aid. The number of
international and national humanitarian aid workers in Darfur expanded
from 202 in April 2004 to 13,500 staff members of 84 NGOs and 13 UN
agencies in July 2006. In 2005 and early 2006, malnutrition and
mortality rates in Darfur had dropped since 2004, a trend that U.S. and
other officials attribute in part to international humanitarian
assistance efforts. Further, according to State sources, the U.S.
government, via private contractors, provided about $280 million,
between June 2004 through September 2006, primarily to build and
maintain facilities throughout Darfur to house African Union troops
assigned to monitor compliance with the April 2004 cease-fire
agreement.[Footnote 19]
Data Sources and Methods for Darfur Death Estimates:
A key data source for the Darfur death estimates is health, nutrition,
and mortality surveys conducted in the field by NGOs delivering
humanitarian services (e.g., Médecins Sans Frontières, known in English
as Doctors without Borders, and Save the Children)[Footnote 20] as well
as by UN or governmental agencies (e.g., the WHO, the World Food
Program, and the CDC). These surveys are discrete data collection
exercises carried out at a specific time with a particular sample of
the affected population, such as people in a certain IDP camp.
Surveys that ask about mortality are often combined with those
collecting health and nutrition data. "Retrospective" mortality data
are collected by asking a sample of respondents to recall the number of
deaths that occurred in their household during an earlier defined
period of time.[Footnote 21] Interviewers may also ask respondents
questions that allow them to categorize the cause of death (e.g.,
deaths due to violence, disease, or malnutrition). Households surveyed
may be located in a single area (e.g., displacement camp) or multiple
areas (e.g., multiple camps or sites within a region.) Because an
absolute number of deaths is difficult to interpret, organizations
conducting surveys calculate mortality rates for the population sampled
and for the time included in the "recall period." These mortality rates
allow for comparison among different population groups and with
different time periods, such as when no humanitarian crisis is
occurring. In acute emergencies, when mortality may change
significantly during a short time interval, mortality rates are often
expressed as the number of deaths per 10,000 people per day. Typically,
a crude mortality rate--that is, the rate of death for the entire
population, including both sexes and all ages[Footnote 22]--is
reported, as well as mortality rates for specific groups (such as those
younger than 5 years or of a specific sex). Data from CRED's Complex
Emergency Database (CE-DAT) show that at least 68 surveys conducted
with IDPs and affected residents reporting crude mortality rates were
conducted in Darfur between 2004 and 2006.[Footnote 23] Two
retrospective mortality studies used in most Darfur death estimates
were conducted, respectively, by Médecins Sans Frontières and the WHO.
* Médecins Sans Frontières Mortality Surveys in West Darfur. The
organization conducted retrospective mortality surveys with 3,175
households in four sites in West Darfur.[Footnote 24] The surveys,
conducted between April and June 2004, asked respondents about the
number and cause of deaths in their household up to 6 months
earlier,[Footnote 25] before they fled their villages, as well as after
they arrived at their IDP camps or settlements. Médecins Sans
Frontières reported crude mortality rates ranging from 5.9 to 9.5
deaths per 10,000 per day during the period when respondents were in
their villages and in flight and rates ranging from 1.2 to 5.6 deaths
per 10,000 per day during the camp period. Médecins Sans Frontières
also reported that violence caused 68 to 93 percent of deaths during
the village and flight periods.
* 2004 World Health Organization Retrospective Mortality Survey. The
WHO surveyed 3,140 households in accessible IDP settlements in the
three regions of Darfur to determine mortality rates between June and
August of 2004.[Footnote 26] The organization reported crude mortality
rates of 1.5 per 10,000 per day for North Darfur; 2.9 per 10,000 per
day in West Darfur; and 3.8 per 10,000 per day for Kalma camp in South
Darfur for the defined recall period. Diarrhea accounted for most of
the reported deaths in all three regions, while violence or injury
accounted for a smaller percentage of deaths, ranging from 10 to 21
percent.
Researchers estimating death tolls in humanitarian crises such as that
in Darfur generally extrapolate mortality surveys' various mortality
rates to longer time periods and larger populations. In addition, to
estimate an "excess" number of deaths directly attributable to the
conflict, some researchers subtract a baseline mortality rate--that is,
an expected number of deaths that would have occurred absent the
conflict--from the total number of deaths estimated for the time period
and population. (Fig. 3 illustrates the generation of a death
estimate.)
Figure 3: Example of Generation of a Death Estimate:
[See PDF for image]
Source: GAO.
[End of figure]
The death estimates that we and the group of experts reviewed were
produced for varying purposes, according to their authors.
* The authors of the CRED estimates[Footnote 27] said that their aim in
conducting their estimate was to develop a method that, rather than
extrapolating mortality from a single survey to the entire region and
conflict period, took into account variations over time and space.
* Dr. Coebergh[Footnote 28] noted that his estimate was intended as a
political statement to increase public awareness of the crisis. He also
stated that the estimate was produced as a response to earlier total
death tolls quoted in the media, which he believed were too low and did
not adequately capture violent deaths.
* According to Dr. Hagan,[Footnote 29] his estimate in the spring of
2005 responded to earlier total death figures, quoted in the media,
that he believed were too low and did not adequately capture violent
deaths. He stated that one of his goals was to provide greater
transparency about the available estimates.
* Dr. Reeves[Footnote 30] said that he produced his first estimate of
the Darfur death toll, in June 2004, because he believed that the
figures being quoted by the UN significantly understated the Darfur
death toll and were not supported by the data.
* According to State, its purpose was to provide information for
internal policymakers.
* A WHO[Footnote 31] official told us that the organization sought to
assess the order of magnitude of deaths in Darfur to assist it in
planning humanitarian relief for IDPs in camps.
The estimates that we and the experts reviewed vary quantitatively and
in the time that they cover--from 35,000 excess deaths or 45,000 total
deaths for IDPs in camps over 7 months of the conflict, to almost
400,000 total deaths for Darfur over 26 months of the conflict. (See
fig. 4 for a summary description of the estimates and their findings,
and see app. IV for more details.)
Figure 4: Death Estimates Based on Reported Figures and Time Included:
[See PDF for image]
Source: GAO analysis of estimates from above organizations.
[A] CRED conducted two death estimates: one for the period of September
2003-January 2005 and another for the period of February-June 2005. The
bar on the left for "estimated deaths" is the combined total of these
two estimates.
[B] This estimate was conducted by John Hagan and colleagues and
released by the CIJ.
[End of figure]
We and the group of experts also reviewed a preliminary version of a
death estimate for West Darfur by John Hagan and Alberto Palloni;
however, this estimate is not discussed in our report, because the
version that the experts reviewed was preliminary and not publicly
available when they convened in April 2006. In the estimate's final
version, which became publicly available in September 2006, Hagan and
Palloni estimated that a range of 57,506 to 85,346 deaths had occurred
in West Darfur over 31 months. Using the same ratio of deaths to
displacement, they estimated a range of 170,000 to 255,000 deaths in
all three states of Darfur over the same time period.[Footnote 32]
In addition, the estimates differ methodologically, incorporating
varying source data, mortality rates and causes of death, assumptions
regarding variable or fixed mortality rates, estimates of the affected
population, consideration of baseline mortality rates, and reporting
methods. (See app. IV for more information.)
* Source data. The estimates' respective sources of data range from
about 30 mortality surveys to a single mortality survey. The estimates
also use nonsurvey or contextual information, such as the retrospective
reporting of attacks or displacement patterns, to varying degrees. In
addition, the amount of source data used varies according to the
availability of such data when the estimates were conducted. For
example, fewer data were available for estimates conducted in
2004[Footnote 33] than for those conducted in 2005.
* Mortality rates and cause of death. The estimates differ in whether
they applied mortality rates that include all causes of death or
calculated and applied mortality rates due to violence and nonviolence
separately. However, most of the estimates, as well as the mortality
surveys we reviewed, express mortality rates in deaths per 10,000
people per day.
* Assumptions of variable or fixed mortality rates. The estimates
include different assumptions regarding whether mortality rates varied
over the period of the conflict or according to the affected region or
remained constant for all populations in all areas over the entire
period of conflict.
* Affected population estimates. The estimates rely primarily on
information from the UN Humanitarian Profiles in determining the total
affected population, but they differ in their assumptions about whether
the affected population changed over time or remained constant.
Additionally, one estimate also includes only IDPs in camps and
excludes affected residents in Darfur and refugees in Chad.
* Baseline mortality rates. The estimates varied in their use of a
baseline mortality rate--that is, the number of expected deaths absent
the conflict--from using no baseline to using a baseline of 0.5 deaths
per 10,000 per day.
* Reporting methods. Some estimates report a range of possible total
deaths, others report point estimates --that is, single numbers.
Additionally, some estimates are precise figures, such as 396,563,
while others are rounded to fewer significant digits, such as 134,000.
Some Death Estimates Judged More Accurate and Methodologically
Stronger, Despite Shortcomings:
Although none of the death estimates was consistently considered
accurate or methodologically strong, the experts we consulted rated
some of the estimates more highly than others. Overall, the experts
expressed the highest level of confidence in CRED's estimates and
slightly lower levels of confidence in State's and the WHO's estimates.
They expressed the lowest level of confidence in the three estimates
that report the highest number of deaths, citing multiple shortcomings,
such as a reliance on unrealistic assumptions about populations' level
of risk over periods of time.
No Estimate Consistently Considered Accurate:
Although the experts clearly had greater confidence in some estimates
than in others, they did not consistently express a high level of
confidence in any of the estimates or consistently rate any of them as
accurate.[Footnote 34]
The experts expressed a slightly higher level of confidence in the CRED
estimates than the State estimate. However, only 2 of the 12 experts
expressed a high level of confidence, and most had a moderate level of
confidence, in the CRED estimates. Experts had a moderate to low level
of confidence in the WHO estimate. Almost all experts expressed a low
level of confidence in the estimates by Coebergh, Hagan, and
Reeves.[Footnote 35] (See app. V for additional survey results.)
Similarly, the experts did not consistently rate any of the estimates
as accurate; the majority of experts rated almost all of the reported
figures as either too high or too low. Only CRED's second estimate--
36,000 total deaths for February-June 2005--was viewed by half of the
experts as "about right." (See table 1.)
Table 1: Accuracy of Darfur Death Estimates Rated by Experts:
Estimate (number of months in estimate): Coebergh: 218,449 excess
deaths (21);
Too high: 7;
About right: 1;
Too low: 2.
Estimate (number of months in estimate): Coebergh: 253,573 excess
deaths (21);
Too high: 8;
About right: 0;
Too low: 2.
Estimate (number of months in estimate): Coebergh: 306,130 excess
deaths (21);
Too high: 8;
About right: 1;
Too low: 1.
Estimate (number of months in estimate): CRED's two estimates: Sept.
2003-Jan. 2005: 134,000 total deaths (17);
Too high: 1;
About right: 3;
Too low: 6.
Estimate (number of months in estimate): CRED's two estimates: Feb.
2005-July 2005: 36,237 total deaths (5);
Too high: 2;
About right: 5;
Too low: 2.
Estimate (number of months in estimate): Hagan et al: 396,563 total
deaths (26);
Too high: 10;
About right: 0;
Too low: 0.
Estimate (number of months in estimate): Reeves: Over 370,000 excess
deaths (31);
Too high: 10;
About right: 0;
Too low: 0.
Estimate (number of months in estimate): State's lower-and higher-end
estimates: Lower-end estimate: 98,000 total deaths (23);
Too high: 0;
About right: 1;
Too low: 9.
Estimate (number of months in estimate): State's lower-and higher-end
estimates: Higher-end estimate: 181,000 total deaths (23);
Too high: 4;
About right: 4;
Too low: 2.
Estimate (number of months in estimate): WHO's lower-and higher-end
estimates: Lower-end estimate: 45,000 total deaths (7);
Too high: 1;
About right: 2;
Too low: 7.
Estimate (number of months in estimate): WHO's lower-and higher-end
estimates: Higher-end estimate: 80,000 total deaths (7);
Too high: 3;
About right: 1;
Too low: 6.
Source: GAO.
Note: We asked experts to rate the accuracy of total deaths when these
were reported in the estimate. Coebergh and Reeves reported excess
deaths only. Additionally, in responding to this question, one expert
said that there was no basis to judge or that he or she was not sure
about the accuracy of the estimates; one expert chose not to respond to
this question; another expert chose two responses for the second CRED
estimate, and his or her responses are excluded.
[End of table]
In some cases, experts tended to agree as to whether the reported
figures were too high or too low; in other cases, there was less
agreement. For example, 9 of 10 experts rated the lower-end of State's
estimate as too low, and the majority of the experts viewed the
estimates by Coebergh, Hagan, and Reeves as too high. In contrast,
experts did not agree as to whether State's higher estimate was too
high, too low, or about right. For example, a few experts believed the
higher end of State's estimate was likely to be closer to a reasonable
midrange estimate. These experts also thought that the mortality
surveys State used for the estimate may have been conducted at places
with higher levels of aid and subsequently lower levels of mortality,
or that disease outbreaks may have been missed. However, one expert
believed that State may have overestimated mortality by applying
elevated mortality rates for too many months of the crisis.
Experts Found Methodological Shortcomings in Each Estimate:
The experts' overall assessment of the estimates' methodological
strengths in terms of their data, methods, objectivity, and reporting
did not produce any consistently high ratings, and experts noted
shortcomings in each estimate.[Footnote 36]
CRED Estimates:
Overall, the experts rated CRED's estimates most highly in terms of
data, methods, objectivity, and reporting of limitations. However,
several experts found shortcomings in the CRED estimates' data and
methods and thought that CRED could have provided more information and
clarity in its reporting.
* Source data. Most experts said that the data CRED used for its two
estimates (drawing from a total of about 30 mortality surveys in Darfur
and Chad[Footnote 37]) were generally sound.[Footnote 38] CRED reported
checking the reliability and validity of the surveys included in its
estimates, which experts found to be a strength. However, several
experts cited some shortcomings in CRED's data sources. For example,
some said that CRED could have better articulated the criteria used to
select the survey data or weighted the surveys it used by sample size.
A few experts also thought that CRED should have considered other
sources besides mortality surveys, such as surveillance data on
morbidity or nutrition.
* Methods used, including extrapolations and assumptions. Some experts
found CRED's method of using disparate data sources to estimate total
deaths to be innovative and logical.[Footnote 39] Additionally, more
than half of the experts rated CRED's assumptions and extrapolations as
somewhat appropriate or reasonable.[Footnote 40] For example, several
experts found the Sudan baseline mortality rate that CRED used more
accurate than the baseline mortality rates derived from a larger region
of sub-Saharan Africa used in some of the other estimates. However,
some assumptions and extrapolations were questioned. For example,
several experts thought that CRED's assumption regarding a generally
stable rate for nonviolent mortality during much of 2004 may have been
inaccurate, owing to possible changes in the affected population, camp
formations, the level of aid, and the outbreak of disease.
Additionally, some experts considered the method that CRED used to
estimate deaths among refugees in Chad to be somewhat unsystematic.
* Level of objectivity. Overall, experts viewed CRED's death estimates
as having the highest level of objectivity.[Footnote 41] Two authors of
other estimates also concluded that the CRED estimates were likely to
be more reliable and more scientific than other Darfur death estimates,
including their own.
* Sufficiency of reporting. Experts noted that, among the estimates
they evaluated, CRED most sufficiently reported the limitations and
potential sources of over-or underestimation in its estimates. However,
several experts believed that better descriptions of the methods used,
including information on specific formulas and calculations, could have
been provided. An author of another estimate also noted that the
mortality surveys used in CRED's estimates lacked complete citation
information. In our review of CRED's first estimate, we were able to
replicate it to some degree only after the authors provided a
substantial amount of information, such as specific mortality rates and
formulas used and citations for source studies, in addition to the
information in the published document.
State Estimate:
In assessing State's estimate, the experts identified methodological
strengths related to each of the four elements but also noted some
shortcomings. Strengths included its use of multiple types of
information, including contextual data from other sources besides
surveys, such as reporting of attacks. Many experts also believed the
estimate had a high level of objectivity. However, the experts, as well
as authors of other estimates, cited several shortcomings in State's
estimate. For example, many believed that the lower end of the estimate
was too low, owing to several factors including the use of some data
that underestimated mortality rates. Additionally, experts and other
authors thought that the published documents containing the estimate
lacked sufficient information, such as a clear description of the
mortality rates used for all populations and time periods included in
the estimate.
* Source data. Many experts cited as one strength the estimate's use of
different types of data, including mortality survey data and contextual
information, to triangulate findings and estimate mortality, and one
expert deemed this approach a "pioneering attempt" in the field of
death estimates in humanitarian crises.[Footnote 42] However, just over
half of the experts thought that the data used were methodologically
sound.[Footnote 43] Some experts said that several of the mortality
surveys used in State's estimate may have had methodological
limitations in areas such as survey design, implementation, or
accessibility to insecure regions, resulting in unrealistically low
mortality rates. These experts believed that such limitations in source
data, in addition to other problems--for example, the estimate's lack
of clarity regarding how missing populations are accounted for and use
of a relatively higher baseline mortality rate--may have pulled down
State's estimate, in particular, its lower end.
* Methods used, including extrapolations and assumptions. About half of
the experts felt that State's estimate applied somewhat appropriate
extrapolations, and a similar proportion thought that it made
reasonable assumptions. A few experts thought that its depiction of
varying levels of mortality over time and affected regions was
appropriate to estimate total deaths throughout the Darfur crisis.
However, several experts thought that the assumptions used, in some
cases, were based on insufficient rationale and evidence and that
additional sensitivity analysis could have been conducted. For example,
the State estimate assumes that mortality rates in Darfur are 20
percent lower for affected residents than for IDPs, an assumption that
some experts believed lacked sufficient rationale.
* Level of objectivity. Nine of the 12 experts rated the State
estimate's level of objectivity as high.[Footnote 44] Several experts
generally believed that the estimate represented a "good faith effort"
to use available evidence in an unbiased way.
* Sufficiency of reporting. Many of the experts found that the
published documents containing State's estimate lacked sufficient
information to allow them to replicate the estimate and verify the
accuracy and reliability of the data and methods. For example, some
experts noted that the data used were not sufficiently described and
that more specific citations, as well as a description of the criteria
applied to select the data, would have been useful. Additionally, some
experts felt that they did not have a good sense of the reliability and
validity of the contextual data used in the estimate and the
application of these data to determine mortality rates or total deaths.
Moreover, several experts said that State should have included the
specific formulas or mortality rates used for all populations, time
periods, and regions. Similar comments were made by several authors of
other estimates. Our review of the State estimate also showed that it
could not be replicated with the information contained in the report.
Further, one expert noted that the published document available on
State's Web site particularly lacked sufficient description of the
estimate's methods, data, and potential limitations.
WHO Estimate:
Several experts found strengths in the data and level of objectivity of
the WHO's estimate, which it presented in a short briefing in October
2004. However, several experts observed shortcomings in the WHO's
reporting of its estimate.
* Source data. The WHO estimate of IDP deaths in Darfur for 7 months in
2004 relied primarily on findings from the organization's 2004
mortality survey. Several experts noted that this survey followed
standard methods and was generally reliable. However, a few experts and
estimate authors said that the 2004 WHO survey may have underestimated
mortality, owing to local government restrictions that prevented
researchers from asking respondents detailed questions about mortality.
* Methods used, including extrapolations and assumptions. Experts
provided mixed ratings on the appropriateness of the extrapolations and
the reasonableness of the assumptions used in the WHO estimate. For
example, one expert believed that because the methodology of the
organization's 2004 survey was appropriate to gauge levels of mortality
among a limited IDP population for 2 months, but not aggregate
mortality, the survey findings should not have been extrapolated to
generate a death estimate for the total IDP population for 7 months.
* Level of objectivity. Half of the experts rated the level of
objectivity of the WHO estimate as high, three rated it as moderate,
and two rated it as low.[Footnote 45] About half of the experts thought
that the WHO estimate's level of objectivity was equal to CRED's and
State's.
* Sufficiency of reporting. Several experts thought that the WHO's
reporting of its estimate had several shortcomings. For example, they
thought that the briefing document describing the death estimate did
not make it sufficiently clear that the estimate included deaths of
IDPs from all causes--both violent and nonviolent--but excluded deaths
of affected residents in Darfur and refugees in Chad. The experts found
that the lack of a clear description of such issues allowed for
misinterpretation by readers, including the media and academia.
Three Other Estimates:
The experts cited several methodological shortcomings in the Coebergh,
Hagan, and Reeves estimates, including the use of problematic data and
application of unrealistic assumptions about the levels of mortality
over time and affected populations.
* Source data. Many experts found shortcomings in each of the three
estimates' use of certain survey data.[Footnote 46] A number of experts
noted problems in the design, sampling, and data collection in the
Atrocities Documentation Team's survey of Chad refugees on which all
three estimates based, at least in part, their numbers of violent
deaths. Experts also pointed out that, because the survey's intended
purpose was to document levels and types of victimization, the
estimates by Dr. Coebergh, Dr. Hagan, and Dr. Reeves should not have
extrapolated the survey findings to a broader population or time period
in order to estimate total deaths. In addition, many experts observed
that the estimates by Coebergh and Hagan inappropriately used findings
from the 2004 WHO survey to calculate only nonviolent deaths without
taking into account the fact that some deaths reported by the WHO were
due to violence or injury.[Footnote 47]
* Methods used, including extrapolations and assumptions. Most experts
found that the Coebergh, Hagan, and Reeves estimates used unrealistic
extrapolations and assumptions to fill information gaps and estimate
total deaths. For example, many experts thought that each of the three
estimates relied on too few data points extrapolated to an excessive
degree. As a result of this type of extrapolation, the experts
observed, a sensitivity analysis changing one or two assumptions could
swing the total number of deaths from 100,000 to half a million, making
the estimates unreliable. Moreover, several experts believed that some
of the authors had inappropriately assumed constant rates of mortality
for different population groups in Darfur at different periods in the
conflict.[Footnote 48] Two of the estimates (Coebergh's and Hagan's)
also used "fixed" estimates of the affected population over time, a
method that some experts thought was inappropriate because the affected
population grew over the course of the conflict. Additionally, the
three estimates involve assumptions that some experts viewed as
questionable, such as using unsupported numbers to estimate the number
of deaths among populations inaccessible to aid or assuming all missing
persons were likely to be dead.
* Level of objectivity. Most experts rated the level of objectivity of
the three estimates as low, particularly those by Drs. Coebergh and
Reeves. The experts thought that the estimates were more characteristic
of advocacy or journalistic material than of objective analysis.
* Sufficiency of reporting. One expert noted that the estimate by Dr.
Hagan was very straightforward and one could follow the logic of the
data and methods applied. However, most of the experts found that the
three estimates did not sufficiently describe limitations that may have
resulted in under-or overestimation of total deaths. In reviewing the
estimates, we found we were able to replicate Dr. Hagan's entire
estimate based on its description in public documents. We were also
generally able to replicate Reeves' estimate after receiving additional
information about, among other things, his calculation of some numbers
and the sources of his data. We had more difficulty in replicating Dr.
Coebergh's estimate based on the information in the published article
and were able to do so only after the author provided, at our request,
details about the specific data, methods, and formulas that were used.
Estimating Deaths in Humanitarian Crises Involves Many Challenges:
Estimating the total deaths in a humanitarian crisis such as that in
Darfur involves a number of challenges, most notably related to
collecting data in the field and extrapolating from limited data.
Difficulties in collecting sound, consistent survey data--including
lack of access to particular geographical regions, the conditions under
which the surveys are conducted, and limited resources and training for
field staff conducting surveys--affect the quality of the data
collected and result in data gaps. Because of such limitations, as well
as the unavailability of data from other sources, the death estimates
that we reviewed rely on potentially risky assumptions and limited
contextual information. Limitations in estimates of Darfur's population
before and during the crisis also impacted the quality of the death
estimates. Additionally, varying approaches to the use of baseline
mortality rates may have somewhat affected their accuracy.
Challenges in Collecting Source Data Affected Data Quality and Led to
Data Gaps:
Difficulties in conducting health, nutrition, and mortality surveys in
a crisis such as Darfur's can affect survey data's quality and
completeness and, thus, the soundness of death estimates based on the
data. In Darfur, these difficulties include lack of access to certain
geographical regions, difficult survey conditions, researchers' limited
resources and training, and lack of consensus over sampling methods. In
addition, the findings from the surveys in Darfur are not always
publicly available, and few other reliable sources of mortality data
exist in Darfur.
* Lack of access to some geographical regions. Researchers' lack of
access to some geographic areas in Darfur raises concerns about the
completeness of the data collected. Because of security problems,
humanitarian aid organizations that typically conduct the surveys, such
as Médecins Sans Frontières, and other researchers found it difficult
to access all areas of Darfur, particularly South Darfur, according to
several experts. In one instance, a survey conducted by the WHO in 2005
in South Darfur was suspended because of security concerns. Other
surveys were also curtailed because of security concerns or attacks on
NGO staff. In general, surveys were conducted primarily in camps where
humanitarian relief was being provided, which could lead to
underestimates of actual mortality. For example, the WHO and World Food
Program note in their respective reports that their 2004 surveys may
underestimate mortality, owing to their samples' exclusion of people in
inaccessible areas. Additionally, one expert reported that the Sudanese
government at times placed restrictions on the relief organizations
collecting data and limited their access to particular camps and
regions.
* Challenging survey conditions. Challenges in conducting household-
based mortality surveys during and after humanitarian crises such as
that in Darfur can affect the data's accuracy, consistency, and
completeness and lead to over-or underreporting of mortality. Such
challenges include linguistic and other cultural factors, difficulties
resulting from the surveys' conflict or postconflict environment, and
issues of recall and precision.
- Accurate translation of surveys into foreign languages can be
difficult.
- Surveys might not be conducted in all local languages.[Footnote 49]
- Definitions of a household vary.[Footnote 50]
- Some people are reluctant to talk about death.
- Some cultures will not report the deaths of infants.[Footnote 51]
- If all household members die, none remain to be surveyed.[Footnote
52]
- The presence of government employees or parties to the conflict can
lead to over-or underreporting.
- A reluctance to forgo food rations may lead to underreporting of
deaths in the household.
- Dating deaths that occurred months prior to survey can be
difficult.[Footnote 53]
- The length of the survey's recall period may lead to under-or
overreporting of deaths and affect the precision of estimated mortality
rates.[Footnote 54]
- Identifying some causes of death can be difficult.[Footnote 55]
* Insufficient training and resources among research staff.
Difficulties in training staff and assembling resources may also have
affected the quality of the surveys conducted. Mortality surveys
require staff who are proficient in matters such as designing
questionnaires, selecting samples of the local population,[Footnote 56]
and conducting interviews in local languages. Although larger
organizations that routinely conduct surveys generally have staff that
are experienced in designing and implementing surveys, other groups may
not have as much knowledge and experience in collecting data, according
to experts and the research literature. Although the NGOs provide some
training, high turnover rates make it difficult to retain the levels of
knowledge and skill that are required.
* Challenges related to sampling methods. The research literature notes
that samples drawn in IDP or refugee camps may not provide an accurate
count of deaths that occurred in attacks on villages and when IDPs were
fleeing to the camps.[Footnote 57] However, although it recognizes the
difficulty of selecting and implementing an appropriate sampling method
to measure violent deaths outside camps, the literature does not offer
any definitive solutions.
In addition, the NGOs that conduct mortality surveys in Darfur do so
primarily to monitor conditions in the camps they serve, and they
generally do not disseminate the survey results. To address this
problem, CE-DAT was established in 2003, under the Standardized
Monitoring and Assessment of Relief and Transitions (SMART)
initiative,[Footnote 58] to provide quick access to accurate and
reliable data needed by humanitarian decision makers. However, the
usefulness of the database is limited, because NGOs are not reporting
the survey results systematically and because the quality of many of
the reported surveys is problematic.
Moreover, mortality data from other sources, such as prospective or
ongoing surveillance systems[Footnote 59] that systematically record
deaths, are generally unavailable in Darfur. The research literature
notes that existing systems for registering or reporting deaths
generally collapse when crises occur and that prospective surveillance
systems established in crisis situations have limitations.[Footnote
60]However, the literature also notes that data from graveyard counts
and regular reporting and surveillance systems have been used in some
crises, despite such limitations, in conjunction with survey-based data
to estimate mortality in crisis situations.[Footnote 61] In Darfur,
data from surveillance systems were generally not available, although
one expert reported that systems had been set up in some of the larger
camps by the middle of 2004. However, the expert also reported that
these systems were capturing data in ways that would allow them to be
used only qualitatively, rather than quantitatively, for death toll
estimates.
Gaps in Data Leads to Reliance on Assumptions and Limited Information:
Gaps in data on mortality can lead to reliance on extrapolations based
on potentially risky assumptions and limited contextual information.
Because the available data for Darfur cover certain geographic areas
and time periods, the estimates that we examined assume that mortality
rates for surveyed locations and specific periods can apply to
unsurveyed locations and longer periods. Most of the experts we
consulted voiced concerns about such extrapolations. Some experts noted
that factors such as patterns of attack and displacement, humanitarian
relief efforts, and the incidence of disease might cause mortality
rates in the surveyed areas to differ from rates in the unsurveyed
areas. One expert stated that he could only speculate on conditions and
mortality rates in the areas that had not been surveyed. In addition,
several experts expressed concern about extrapolations from limited
time periods to longer periods, noting that mortality rates can change
rapidly.
Some of the estimates' assumptions are informed by contextual
information in the absence of data; however, this information also had
limitations. For example, some estimates relied on anecdotal reports of
conditions in Darfur, satellite imagery on attacks of villages, and
information about weather conditions to determine appropriate mortality
rates. While the experts generally approved of the use of contextual
information, they did not believe that the information used could
compensate for the gaps in the data.
Limitations in Population Data Affect Reliability of Death Estimates:
Limitations in the population data before and during a crisis such as
Darfur's can also impact researchers' ability to produce reliable death
estimates. No definitive estimate of Darfur's pre-crisis population
exists, and estimates of the current population vary considerably, from
around 4 to 7 million. The difficulty of estimating the region's
population is compounded by the fact that migration was widespread in
Darfur before the crisis, making it difficult to get accurate counts.
Because the death estimates essentially extrapolate the mortality rates
from the surveys to the entire population affected by the crisis, an
estimate of the affected population that is too high or too low will
lead to an over-or underestimate of the death toll. The estimates we
reviewed generally used data for the affected population that were
reported in the UN Humanitarian Profiles; however, the collection and
reliability of the profile data had limitations.[Footnote 62] For
example, the profiles estimated a population of more than 160,000 IDPs
in Kalma camp, the largest in Darfur, prior to October 2005. Yet, a
subsequent count by a relief agency in October 2005 reported only
87,000 IDPs in the camp. An expert also noted that figures based on
registration can be unreliable and depend greatly on the agency
responsible for registration. He stated that sources of bias include
poor coverage by the agency (underestimation), as well as problems with
populations' being "double-counted" to increase the amount of aid
delivered (overestimation).
Varying Use of Baseline Mortality Rates Can Affect Estimates' Accuracy:
Varying approaches to the selection of a baseline for normal mortality
can raise or lower death estimates, possibly making the totals overly
high or low. All but one of the Darfur death estimates that we examined
subtract baselines for normal or expected mortality from the total
deaths, based on the assumption that some deaths from disease, old age,
or malnutrition would have occurred without the crisis. For example,
CRED's use of a baseline mortality rate of 0.3 subtracted about 16,000
deaths from the total estimate of 134,000, resulting in 118,000
"excess" deaths. If CRED had used a baseline of 0.5, it would likely
have subtracted about 26,000 deaths, obtaining a somewhat lower
estimate of deaths from the crisis.[Footnote 63]
The experts we convened proposed various baseline mortality rates,
ranging from 0.3 to 0.75 deaths per 10,000 affected persons per
day.[Footnote 64] One reason for the difference in the suggested rates
is the range of methods that the experts used to select them--for
example, a rate that prevailed in the country before the crisis, a
standard based on prior humanitarian crises, or a rate from a
comparable country in the region. However, each of these methods has
acknowledged limitations that could result in inaccurate estimates of
expected rates and, therefore, in over-or underestimations of mortality
due to the crisis.
In addition, the experts debated whether a baseline of any sort was
justified for a humanitarian crisis such as Darfur, arguing ethical and
philosophical, rather than technical, considerations. About half of the
experts said that deaths that would have occurred regardless of the
crisis should be subtracted from the death toll attributed to the
crisis. However, two experts took a contrary position, arguing that the
concept of expected or normal levels of mortality was not appropriate
in the presence of genocide or ethnic cleansing because the
perpetrators of those crimes against humanity should be considered
culpable for all deaths that resulted from the crises they
instigated.[Footnote 65] Using a baseline to estimate mortality would
lead to a somewhat smaller excess death toll than not using a baseline.
For example, State's estimate of total deaths ranged from 98,000 to
181,000, minus 35,000 expected deaths; thus, State estimated 63,000 to
146,000 excess deaths directly resulting from the crisis.
Wide Range of Measures Could Improve Death Estimates for Darfur and
Future Humanitarian Crises:
The group of experts proposed and rated a wide range of measures that
U.S. agencies could take directly, or support other entities in taking,
to improve the quality and reliability of death estimates and relevant
data for Darfur and future humanitarian crises. The measure rated most
likely to produce the most improvements was ensuring sufficient public
documentation of estimates' data and methods to allow replication of
the methods, verification of the findings, and confirmation of the
estimates' credibility and objectivity. Other highly rated measures
included collecting and maintaining temporal and spatial data, housing
responsibility for making the estimates in a reputable independent
body, improving the training of nongovernmental organizations' staff
who collect survey mortality data, and promoting an interdisciplinary
approach to estimating mortality. U.S. agencies are engaged in several
initiatives that address some of these measures.
Proposed Measures Cover Wide Range of Activities:
The experts suggested a series of measures covering a broad range of
activities that U.S. agencies or other, U.S.-funded or -supported
entities could take to improve death estimates for future humanitarian
crises.[Footnote 66] (See table 2.) Although the experts acknowledged
the importance and necessity of estimating mortality--for example, to
help hold perpetrators accountable and to have a complete historical
record--they generally believed that death estimates should be
conducted with caution. To assist in this endeavor, the experts
proposed and rated 19 measures as likely to produce some improvements
in data collection and mortality estimation. Some experts differed as
to whether government agencies or other entities would be best placed
to implement particular suggestions.
Table 2: Measures Rated by Experts as Likely to Greatly Improve Death
Estimates for Future Crises, in Order of Ranking and Number of
Endorsements:
Measure: Ensure that publicly available documents on U.S. government
estimates provide sufficient information on methods, data, assumptions,
and limitations;
Number of endorsements: 11.
Measure: Support the collection and maintenance of temporal and spatial
data;
Number of endorsements: 10.
Measure: House responsibility for mortality estimates in a reputable,
independent body or group;
Number of endorsements: 9.
Measure: Improve training of NGO staff who collect mortality survey
data;
Number of endorsements: 9.
Measure: Promote an interdisciplinary approach to estimating mortality
(include epidemiologists and demographers);
Number of endorsements: 9.
Measure: Create technical teams, under the auspices of an international
body, that can conduct mortality estimates as needed;
Number of endorsements: 8.
Measure: Report mortality and morbidity information more routinely and
systematically to provide an ongoing sense of the situation;
Number of endorsements: 8.
Measure: Promote data collection by NGOs on the ground at routine
service points in addition to periodic assessments;
Number of endorsements: 7.
Measure: Promote the use of other measures of a conflict's severity
(e.g., displaced persons, number of attacks) in addition to mortality
estimates;
Number of endorsements: 7.
Measure: Improve existing surveying techniques (e.g., cluster sampling)
by incorporating spatial or temporal information;
Number of endorsements: 7.
Measure: Provide guidance (minimum standards) on how to design and
implement survey instruments in the affected region (e.g., pretesting
or translation techniques);
Number of endorsements: 6.
Measure: Provide guidance on amalgamating existing mortality/morbidity
surveys;
Number of endorsements: 5.
Measure: Define criteria for selecting and using data;
Number of endorsements: 4.
Measure: Make satellite tapes and imagery available to researchers;
Number of endorsements: 3.
Measure: Provide guidance on triangulating different types of data;
Number of endorsements: 3.
Measure: Provide assistance to local statistical agencies to improve
data collection;
Number of endorsements: 3.
Measure: Develop algorithms to track population change over time;
Number of endorsements: 3.
Measure: Create a statistical unit under appropriate agency (possibly
under auspices of the Committee on National Statistics) to be
responsible for these types of estimates;
Number of endorsements: 3.
Measure: Tap other resources, such as pro bono groups of statisticians,
that could provide assistance;
Number of endorsements: 1.
Source: GAO.
Note: The table is based on analysis of 12 experts in epidemiology,
demography, statistics, and the Darfur crisis. Numbers shown under
"Number of endorsements" represent the number of experts who rated the
measure as likely to "very greatly" or "greatly" to improve the
estimates. In several instances, one or two experts noted problems with
suggestions that had been highly rated by others. The experts rated an
additional measure for the Darfur crisis, namely, whether an
independent agency should conduct a retroactive assessment.
[End of table]
Majorities of the experts rated 10 of the 19 measures, and 9 of the 12
experts rated 5 of the measures, as likely to "very greatly" or
"greatly" improve the estimates for future crises.[Footnote 67]
Following are the 10 most highly rated measures:
* Ensure that publicly available documents on the U.S. government
estimates provide sufficient information on methodology. Sufficient
documentation and transparency of data and methods are needed to allow
independent researchers to verify the reliability and validity of
estimates. Although this suggestion would not improve the estimates per
se, experts felt that it was important for establishing the credibility
of the estimates.
* Support the collection and maintenance of temporal and spatial data.
Temporal and spatial databases would allow researchers to track
mortality over time and across regions and could improve researchers'
ability to estimate mortality. Experts suggested that temporal and
spatial data could be collected in a number of ways, including through
the use of satellite imagery.
* House responsibility for mortality estimates in a reputable,
independent entity or group. Experts indicated that an independent
entity could be perceived as free of bias, political goals, or both.
However, such an entity would need adequate funding, and experts had
differing views as to where it should be housed. One expert felt it
could be placed under a U.S. organization such as the CDC, while
another felt it should be under an international organization such as
the WHO.
* Improve training of NGO staff who collect mortality survey data.
Because NGO staff conduct most of the surveys, improving their training
would help improve the quality of data collected. Several experts
reported the existence of initiatives to improve and standardize NGOs'
data collection in emergencies, such as the SMART initiative, and of
guidance on the topic from the World Food Program and the WHO. Some
experts suggested expanding these efforts with additional U.S.
government support.
* Promote an interdisciplinary approach to estimating mortality
(include epidemiologists and demographers). Because epidemiologists and
demographers bring different expertise and perspectives to mortality
estimation, collaboration between the disciplines could lead to greater
understanding of each other's approaches and, ultimately, to more
integrated methodologies for death estimates.
* Create technical teams, under the auspices of an international body,
that can conduct mortality estimates as needed. Technical teams,
assembled as needed and operating under independent organizations,
could--rather than a single, permanent independent entity--be
responsible for conducting death estimates. Such teams might be
perceived as free of bias and political goals, which would increase the
credibility of their findings. However, some experts reported that such
technical teams already exist, citing as examples the WHO, the
International Rescue Committee, and Médecins Sans Frontières, among
others.[Footnote 68]
* Report mortality and morbidity information more routinely and
systematically to provide an ongoing sense of the situation. This
suggestion is intended to allow researchers and governments to monitor
crises as they develop. Although this suggestion was relatively highly
rated, experts had some questions about who would do the reporting, how
the data would be gathered, and how the estimates would be made. One
expert noted that groups like CRED are currently reporting such
mortality and morbidity information somewhat routinely and
systematically.
* Promote data collection by NGOs on the ground at routine service
points, in addition to periodic assessments. Proponents stated that
humanitarian relief organizations could collect data on famine-related
deaths and that human rights organizations could collect narrative
testimonies about political violence and conflict-related displacement.
Some experts believed that the NGOs could collect these data with
little additional time or expense and that the data collection could
help create an early warning system for famine and diseases. However,
several experts were concerned that data collected at routine service
points would provide biased estimates because certain segments of the
affected population would not be likely to pass through these points.
* Promote the use of other measures of a conflict's severity (e.g.,
displaced persons, number of attacks) in addition to mortality
estimates. Such alternative estimates could provide insights into
crises and help check the accuracy of the mortality data. These
estimates could include conflict-related displacement, property
destruction, property looting, exposure to violence as a marker for
psychological trauma, incidence of sexual and gender-based violence,
forced displacement, and data from security incidence reports. One
expert indicated that some of these data are probably being collected
by various human rights agencies and organizations and suggested
creating a body to collate and report their data.
* Improve existing surveying techniques (e.g., cluster sampling) by
incorporating spatial or temporal information. A few experts felt that
improvements in surveying techniques were needed because of limitations
in certain aspects of cluster sampling, the most commonly used
technique; however, another expert stated that cluster
sampling[Footnote 69] was extremely useful. One expert argued that
existing surveying techniques should take into account the spatial and
temporal distribution of the affected population.
Existing U.S. Initiatives May Improve Some Aspects of Data and Death
Estimates:
Several current U.S. initiatives may enhance the availability and
quality of mortality data and produce more accurate death estimates for
Darfur and other similar crises in the future. These initiatives embody
several of the measures rated by the group of experts as likely to
improve death estimates; however, several of the experts, as well as
U.S. officials knowledgeable about the initiatives, whom we spoke with
indicated that more can be done.
* CE-DAT may improve death estimates through its promotion of more
routine reporting of mortality and morbidity information and defining
some criteria based on methodological requirements for assessing the
quality of data. In addition, CE-DAT includes a cartographical database
to link indicators to maps, which aligns with the experts'
recommendation for collecting and maintaining spatial and temporal
data. CE-DAT staff discussed some limitations, with respect to the data
in the database, including the fact that information is not provided to
CE-DAT routinely and CE-DAT is not widely known. CE-DAT staff also said
that there is currently no systematic means of verifying data
reliability and validity of data and making data publicly available in
a timely manner.
* SMART, an interagency initiative coordinated by USAID, may improve
death estimates through its promotion of a standard methodology to
design and implement survey instruments that measure crude mortality,
as well as other indicators of need.[Footnote 70] According to a USAID
official, SMART is also intended to help build the technical capacity
of NGO field workers and host government partners to collect and report
more reliable data. To this end, the SMART initiative aligns with some
of the recommendations made by our group of experts, including,
improving the training of NGO staff that collect mortality data and
reporting mortality and morbidity data more routinely and
systematically. Experts and officials whom we spoke with generally
agreed that the idea behind SMART represents positive steps toward
improving mortality data collection. However, they also discussed
various limitations of the initiative. For example, an official
involved with the effort noted that that funding has been at times
sporadic and that limited resources have stymied outreach efforts.
Others stated that some of SMART's methodological principles need
further research and testing in the field and that no consensus
currently exists regarding some of these principles. Officials from
USAID indicated that SMART's major components, including mortality,
nutrition, and food security, have been piloted in several countries
and that the SMART methodology is meant to be iterative with continuous
upgrading based on further research and best practices.
* The Humanitarian Information Unit, housed in State's Bureau of
Intelligence and Research, may enhance the quality and availability of
spatial and temporal information used in future death estimates through
its current efforts to develop of maps and other visual images of
humanitarian crises. For example, for the Darfur crisis, it has
developed a series of maps showing the number of destroyed villages or
reported attacks in the region at different points in time based on
data from satellite imagery or reports from on the ground organizations
such as the African Union, the UN, and USAID. The unit's work also
addresses two additional suggested measures--promoting the use of other
measures of conflict and improving existing surveying techniques by
incorporating spatial and temporal data. However, the Humanitarian
Information Unit, in some cases, has had difficulty obtaining
standardized, reliable, or complete data. Additionally, the unit does
not consistently and systematically collect data, such as preconflict
population estimates, in part because the unit performs its work upon
request.
* The International Emergency and Refugee Health Branch of the CDC may
improve death estimates and the collection of mortality data through
its provision of technical assistance to UN and U.S. agencies
collecting data in complex emergencies and its support of the
collection of spatial and temporal data.[Footnote 71] However,
officials from the International Emergency and Refugee Health Branch
stated that because it often works as a consultative body and its
resources are constrained, the scope of its work is limited.
Several international initiatives also address problems with data
collection and analysis. The WHO plans to implement a Health and
Nutrition Tracking Service that would routinely monitor mortality and
malnutrition during major crises and provide a central help desk,
remotely located, for field staff conducting surveys. Additionally, the
World Food Program has created a manual on survey techniques to measure
health and nutrition indicators, including crude mortality.
The group of experts we convened were generally aware of, or involved,
with these U.S. government and international initiatives, as well as
some NGO initiatives, that addressed certain aspects of their
suggestions. For example, three experts assisted in the development of
the SMART guidelines, and two of the three were also technical advisors
for the CE-DAT initiative. Some experts felt that certain suggestions
could be met by investing more in these existing initiatives rather
than by creating new ones. U.S. officials responsible or aware of these
initiatives also acknowledged limitations with several of these or
thought that more could be done with respect to the measures suggested
by the experts.
Conclusions:
Despite variance among the death estimates we reviewed, each of the
estimates shows, tragically and unquestionably, that many thousands of
civilians died in Darfur between February 2003 and August 2005. Given
the continuing conflict, insecurity, and displacement in the region,
many more have undoubtedly died since the estimates were conducted, as
a direct result of violence or because of increased vulnerability to
disease and malnutrition. However, despite the importance of the death
estimates in showing the severity of the crisis, none of the estimates
consistently received high ratings in terms of accuracy or
methodological strengths from the experts we convened. Some of the
shortcomings in the estimates' source data, methods, and objectivity
may be attributed to challenges in mortality data collection and
extrapolation, characteristic of a humanitarian crisis such as Darfur.
However, in certain cases, a lack of transparent reporting of an
estimate's data, methods, assumptions, or limitations hindered the
experts from replicating it and thus verifying its accuracy and
credibility. In particular, many experts noted the unavailability of
these published details--and the resulting lack of transparency--for
State's estimate.
Several ongoing U.S. initiatives may produce enhancements that align
with the experts' suggested measures for addressing gaps in data and
improving death estimates conducted for Darfur and any similar
humanitarian crises in the future. For example, the SMART initiative
and CE-DAT, respectively, provide guidance on how to design and
implement surveys and promote more routine reporting of mortality and
morbidity data. However, experts and U.S. government officials observed
that gaps in these areas still exist. Among the measures that have not
been addressed, the one that the experts rated most highly--ensuring
the public availability of information on estimate methods, data,
assumptions, and limitations--is essential to protect the credibility
of U.S. government death estimates for Darfur and any future
humanitarian crises.
Recommendations for Executive Action:
To safeguard the Department of State's credibility as a source of
accurate and reliable death estimates, we recommend that the Secretary
of State promote greater transparency in any of its future death
estimates for Darfur or other humanitarian crises by ensuring that
publicly available documents contain sufficient detail on the
estimates' data, methods, assumptions, and limitations to allow
external researchers to replicate and verify the estimates.
Additionally, to enhance the U.S. government's capacity to assess the
dimensions of, and respond appropriately to, any future humanitarian
crises, we recommend that the Secretary of State and the Director of
U.S. Foreign Assistance and USAID Administrator consider the experts'
other suggested measures to help address gaps in data and improve the
quality and reliability of any future death estimates.
Agency Comments and Our Evaluation:
We provided a draft of this report to the Department of State and
USAID. State and USAID responded with formal comments, agreeing with
our recommendations, and State provided additional perspectives on
reporting and documentation regarding its death estimate. Reproductions
of these letters, as well as our responses to the letters, can be found
in appendixes VI and VII. We also provided a draft to the CDC for
technical review, and we received technical comments from both the CDC
and State, which we incorporated in the report as appropriate.
We provided the authors of the other five estimates the portions of the
report pertaining to their individual estimates. The authors of the
CRED estimates agreed with the experts' evaluation. Jan Coebergh did
not provide any major comments regarding the relevant portion of the
report and indicated that he was aware of the limitations of his
estimate. John Hagan expressed concerns regarding the experts'
relatively positive evaluation of the estimates by CRED and State and
believed we should have included the estimate he did with Alberto
Palloni in our evaluation. David Nabarro from the WHO disagreed with
our inclusion of his estimate with the others in the report and said
that we had not clearly conveyed the estimate's purpose. Eric Reeves
disagreed with the experts' criticism of the Atrocities Documentation
Team's survey of Chad refugees, stating that the survey was well
conducted and a critical source of data to estimate violent deaths,
particularly early on in the conflict when little data was available.
More details on the authors' comments and our responses can be found in
appendix VIII.
As agreed with your offices, unless you publicly announce the contents
of this report earlier, we plan no further distribution until 30 days
from the report date. At that time, we will send copies of this report
to the Secretary of State, the Director of U.S. Foreign Assistance and
USAID Administrator, the Director of the CDC, relevant congressional
committees, and other interested parties. We will also make copies
available to others on request. In addition, the report will be
available on GAO's Web site at [Hyperlink, http://www.gao.gov].
If you or your staff have any questions about this report, please
contact either David Gootnick at (202) 512-3149 or gootnickd@gao.gov,
or Nancy Kingsbury at (202) 512-2700 or kingsburyn@gao.gov. Contact
points for our Offices of Congressional Relations and Public Affairs
may be found on the last page of this report. Key contributors are
listed in appendix IX.
Signed by:
David Gootnick, Director:
International Affairs and Trade:
Signed by:
Nancy Kingsbury, Managing Director:
Applied Research and Methods:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
This report (1) evaluates the relative accuracy and methodological
strengths and shortcomings of six death estimates for Darfur, including
one by the Department of State (State); (2) identifies general
challenges to estimating the total death toll in Darfur and similar
humanitarian crises; and (3) discusses measures that the U.S.
government could take to improve its death estimates for Darfur and any
similar, future crises.
To evaluate the relative accuracy and methodological strengths and
shortcomings of the Darfur death estimates, we selected estimates that
had been made public prior to March 2006. We imposed this cutoff point
so that we could interview the authors, prepare materials for the
experts, and give the experts time to review the materials. Because our
cutoff point was March 2006, the convening experts' discussion was of
death estimates that spanned various time periods between February 2003
and August 2005. Our findings do not, therefore, discuss any estimates
that were made after March 2006.
Additionally, to address these objectives, we convened, in
collaboration with the National Academy of Sciences, 12 experts for an
all-day meeting on April 7, 2006, in Washington, D.C., to discuss the
six death estimates. (Two additional experts also participated by phone
for parts of the day.) The academy proposed lists of experts, and we
approved their final selection. The selected experts had extensive
knowledge of estimating mortality rates and death totals in conflict
and postconflict situations or other types of humanitarian crises and
were familiar with the sorts of data and estimates that have been
produced for the Darfur crisis or other such crises in Africa.
Additionally, the selected experts represented a range of professional
experiences and backgrounds, including epidemiologists, demographers,
statisticians, and directors of aid programs currently in Darfur. They
were affiliated with various organizations, including universities,
U.S. government and United Nations (UN) agencies, and humanitarian aid
and nonprofit organizations based in the United States, as well as
overseas. All of the experts signed a form from the National Academy of
Sciences certifying that they had no conflicts of interest that could
compromise their ability to assess the death estimates objectively.
Some experts reported they had had professional contact with several of
the authors.
During the meeting, the experts discussed their evaluation of each of
the Darfur death estimates, as well as challenges to estimating total
deaths for Darfur and similar crises and measures to improve such
estimates. The meeting was recorded and transcribed to ensure that we
accurately captured the experts' statements, and we reviewed the
transcripts as a source of evidence. (See app. II for a list of the
experts' names and affiliations and a summary of the meeting agenda and
discussion questions.)
In addition, prior to the meeting, we reviewed and analyzed public
information on the estimates, including documents describing the
estimates and source data for the estimates and provided this
information to the experts.[Footnote 72] The source data documents that
we reviewed included 15 reports of health, nutrition, and mortality
surveys; State publication describing the July-August 2004 survey with
refugees in Chad; and volumes of the UN Nutrition Information in Crisis
Situations containing descriptions of additional health, nutrition, and
mortality surveys of which we were unable to obtain full reports. (See
app. III for a complete list of published documents that were provided
to the experts.) We also provided the experts supplemental information
on the design, sampling and implementation of the Atrocities
Documentation Team's survey of Chad refugees and a copy of the survey
instrument, which we obtained from representatives of the Coalition of
International Justice. Additionally, we interviewed the estimates'
authors regarding the data, methods, and objectives of their work and
replicated their estimates when we had sufficient information. We
provided summaries of these interviews to the experts, as well as
replications of the estimates, reviewed and approved by the authors, in
advance of the meeting. We spoke with all of the estimate authors
except State's Bureau of Intelligence and Research, which declined to
speak with us or provide additional information. Although this limited
the ability of our experts to fully understand State's methods of
analysis as noted during their discussion, the experts were able to
discuss State's estimate in detail and provide assessments of its data,
methods, objectivity, limitations, and accuracy.
Following the meeting, we also asked the experts to answer additional
questions on each estimate, as well as questions on suggested measures
to improve estimates through a follow-up data collection instrument. We
developed the instrument with the help of survey specialists and based
the questions in the instrument on the meeting agenda and points that
arose during the meeting deliberations. We pretested the instrument
with two experts and made changes based on their input. We administered
the instrument via e-mail and received responses from all 12 of the
experts who attended the meeting in Washington, D.C. In some instances,
we contacted the respondents by e-mail or phone to obtain greater
clarity or details regarding their answers.
Further, to identify challenges involved in estimating total deaths in
humanitarian crises, such as that in Darfur, we asked the group of
experts to highlight key challenges during their discussion, and we
reviewed literature related to death estimates and mortality data for
humanitarian crises. We summarized the parts of the experts' discussion
that most directly addressed challenges, and we identified themes that
were raised by the experts. The literature we reviewed included
articles that we identified through databases, such as ProQuest, Lexis
Nexis, and Medline, using various search terms, such as mortality
estimates, death estimates, humanitarian crise(s), conflict(s), or
(complex) emergency/ emergencies. We also reviewed literature cited in
these articles and on Web sites related to humanitarian assistance or
data analysis in humanitarian crises, as well as literature recommended
by the group of experts[Footnote 73] and the authors of the estimates.
In total, we reviewed about 20 articles. We organized the individual
challenges identified by the experts and in the literature according to
the two overarching themes we identified from the experts' discussion
and from the literature: (1) challenges affecting the source data used
for the death estimates and (2) challenges affecting the generation of
the death estimates. We also provided portions of the draft pertaining
to the challenges to conducting death estimates as well as the
background regarding mortality surveys to several of the experts we
convened and incorporated their technical comments into the final
version of the report.
Finally, to identify measures that the U.S. government could take to
improve death estimates for Darfur and in future humanitarian crises,
we asked the experts to provide suggestions during the meeting and
solicited further opinions on these suggestions in the follow-up
instrument. The instrument listed 19 measures that the experts had
suggested as likely to improve death estimates for Darfur and such
crises that may occur in the future and asked the experts to rate them
with a five-point scale, ranging from "Very greatly improve" to "Not
improve."[Footnote 74] We ranked the measures according to the numbers
of respondents that rated them "Very greatly improve" and "Greatly
improve." The instrument also asked for experts' comments on each of
the suggested measures, and we followed-up with some experts to get
additional information regarding comments on suggested measures,
particularly with respect to current efforts. Additionally, we spoke
with officials from the U.S. Agency for International Development,
State, and the Department of Health and Human Services' Centers for
Disease Control and Prevention to learn of any current U.S. government
initiatives related to the suggested measures. We also reviewed public
information and documents provided by the officials and available on
the Internet that describe efforts by the U.S. government, as well as
other international initiatives.
We conducted our work in San Francisco and Washington, D.C., from
September 2005 to November 2006 in accordance with generally accepted
government auditing standards.
[End of section]
Appendix II: List of Experts and Meeting Agenda:
This appendix provides the names and affiliation of the experts
participating in the meeting held on April 7, 2006, in Washington,
D.C., and a summary of the day's discussion questions.
List of Experts:
The following experts attended the meeting:
* Jana Asher, American Association for the Advancement of Science:
* Richard Brennan, Health Unit, International Rescue Committee:
* Francesco Checchi, London School of Hygiene and Tropical Medicine:
* Allan Hill, Harvard School of Public Health, Harvard University:
* Arif Husain, Vulnerability Analysis and Mapping Unit, United Nations
World Food Program:
* Mark Myatt, University College of London:
* W. Courtland Robinson, Bloomberg School of Public Health, Johns
Hopkins University:
* William Seltzer, Department of Sociology and Anthropology, Fordham
University:
* Romesh Silva, Human Rights Data Analysis Group, The Benetech
Initiative:
* Michael VanRooyen, Program on Humanitarian Crises and Human Rights,
Harvard University:
* Ronald Waldman, Mailman School of Public Health, Columbia University:
* Bradley Woodruff, Maternal and Child Nutrition Branch, U.S. Centers
for Disease Control and Prevention:
The following experts participated in the meeting by phone for parts of
the day:
* Bushra Gamar Hussein, Darfur Region, Sudan Social Development
Organization:
* Jennifer Leaning, Harvard School of Public Health, Harvard
University:
Summary of Meeting Discussion Questions:
Darfur Death Estimates:
* Are the source data used in each of the estimates sufficiently
representative and reliable?
* To what extent do the methods used in each of the estimates follow
principles that are generally accepted in the field?
* Do the key assumptions made in the estimates seem reasonable?
* How accurate or inaccurate do you think each of these estimates are
in terms of representing the actual number of deaths that occurred in
Darfur during the time period and regions under consideration?
* For each of the estimates, is sufficient information presented in the
reports to adequately assess the strengths and weaknesses and/or the
reasonableness of the estimates? Do you understand the processes by
which the estimates were derived?
* What are the major strengths and limitations of each of these
estimates for Darfur?
Challenges to Estimating Total Deaths for Darfur and Similar Crisis:
* In your opinion, what sources of data for what time periods and
locations would be necessary to produce reasonably precise estimates of
death totals for Darfur?
* What general observations do you have concerning the challenges of
estimating total deaths in Darfur?
Implications for the U.S. Government: Ways to Improve Death Estimates
for Darfur and Future Crises:
* In your view, are there any lessons learned about conducting death
estimates in Darfur for the U.S. government? If so, what are these?
* Are there measures the U.S. government could take or promote to
improve estimates for Darfur or similar crises in the future?
[End of section]
Appendix III: List of Death Estimates and Mortality Surveys Provided to
Experts and Additional Bibliographical References:
Death Estimates[Footnote 75]
Estimate by the U.S. Department of State:
* State Department's Fact Sheet, "Sudan: Death Toll in Darfur," March
25, 2005. [Hyperlink,
http://www.state.gov/s/inr/rls/fs/2005/45105.htm](Viewed, August 24,
2006):
* M. Phelan in D. Guha-Sapir and O. Degomme, "Darfur: Counting the
Deaths" (Method 2). Centre for Research on the Epidemiology of
Disasters (CRED). Brussels: May 26, 2005. [Hyperlink,
http://www.cred.be/docs/cedat/DarfurCountingTheDeaths-
withClarifications.pdf] (Viewed October 23, 2006):
Estimate by the Centre for Research on the Epidemiology of Disasters:
Debarati Guha-Sapir and Olivier DeGomme:
* Guha-Sapir, D., O. Degomme, and M. Phelan, "Darfur: Counting the
Deaths" (Method 1). Centre for Research on the Epidemiology of
Disasters (CRED). Brussels: May 26, 2005. [Hyperlink,
http://www.cred.be/docs/cedat/DarfurCountingTheDeaths-
withClarifications.pdf] (Viewed October 23, 2006):
* Guha-Sapir, D. and O. Degomme, "Darfur: Counting the Deaths (2): What
are the trends?" Centre for Research on the Epidemiology of Disasters
(CRED). Brussels: December 15, 2005. [Hyperlink,
http://www.cred.be/docs/cedat/DarfurCountingTheDeaths2.pdf] (Viewed
October 23, 2006):
Estimate by Jan Coebergh:
* Coebergh, J. "Sudan: genocide has killed more than the tsunami."
Parliamentary Brief, vol. 9, no. 7, pp. 5-6 (February 2005).
[Hyperlink, http://www.thepolitician.org/] (Viewed, August 24, 2006):
Estimate by John Hagan, Wenona Rymond-Richmond, and Patricia Parker and
announced by the Coalition for International Justice:
* Coalition for International Justice Press Release, "New Analysis
Claims Darfur Deaths Near 400,000." April 21, 2005.
* Hagan, J., W. Rymond-Richmond, and P. Parker, "The Criminology of
Genocide: The Death and Rape of Darfur," Criminology, vol. 43, no. 3,
pp. 525-561 (July 2005).
Estimate by Eric Reeves:
* Portions of mortality updates from www.sudanreeves.org. (Reviewed by
author.) Dr. Reeves began reporting his mortality updates in June 11,
2004. The most recent update reviewed by the experts was from August
31, 2005.
Estimate by the World Health Organization:
* Nabarro, D. "Mortality Projections for Darfur," World Health
Organization Briefing presented October 15, 2004. [Hyperlink,
http://www.who.int/mediacentre/news/briefings/2004/en] (Viewed, August
24, 2006):
Surveys Used in Estimates as Data Sources:
State or regionwide estimates:
* U.S. Centers for Disease Control and Prevention and UN World Food
Program, "Emergency Nutrition Assessment of Crisis Affected Populations
Darfur Region, Sudan, August-September 2004." [Hyperlink,
http://www.reliefweb.int/library/documents/2004/wfp-sdn-26ocr.pdf].
(Viewed, August 24, 2006):
* UN World Food Program, "Emergency Food Security and Nutrition
Assessment in Darfur, Sudan." (Rome, Italy: World Food Program, October
2004). [Hyperlink,
http://www.wfp.org/country_brief/africa/sudan/assessments/041025_food_as
sessment.pdf] (Viewed, August 24, 2006):
* World Health Organization, "Retrospective Mortality Survey Among the
Internally Displaced Population Greater Darfur, Sudan 2004." September
15, 2004. [Hyperlink,
http://www.reliefweb.int/library/documents/2004/who-sdn-15sep.pdf]
(Viewed, August 24, 2006):
* World Health Organization, "Mortality Survey among Internally
Displaced Persons and Other Affected Populations in Greater Darfur,
Sudan." September 2005. [Hyperlink,
http://www.emro.who.int/sudan/pdf/CMS%20Darfur%202005%20final%20report_1
1%2010%2005.pdf] (Viewed, August 24, 2006):
North Darfur:
* Dubray, C., "Health assessment in emergencies: Serif Umra, North
Darfur, Sudan." Epicentre and Medecins Sans Frontiers. November 2004.
* Sibson, V., "Findings of a nutrition survey of Kutum town, Kasab camp
and Fata Borno, North Darfur." GOAL. May 2005.[Footnote 76]
West Darfur:
* Abadallah, S., "Report of rapid baseline survey Azirni, Sanidadi and
Um Tagouk." Center for International Emergency, Refugee and Disaster
Studies, Johns Hopkins University and World Relief. October 12, 2004.
* Deconinick, H. and O. Karouri, "Nutrition assessment; Fur Baranga,
Administrative Unit, Habila Locality, West Darfur, Sudan." Save the
Children, USA. January 14-17, 2005. [Hyperlink,
http://www.humanitarianinfo.org/darfur/uploads/assessments/fb%20nutritio
n%20survey%20report%20jan%2005.doc] (Viewed, August 24, 2006):
* Depoortere, E., F. Checchi, F. Broillet, S. Gerstl, A. Minettia, O.
Gayraud et al., "Violence and mortality in West Darfur, Sudan (2003-
2004): epidemiological evidence from four surveys" Lancet, vol. 364,
no. 9442, pp. 1,315-1,320 (2004). [Hyperlink,
http://www.msf.fr/documents/base/2004-10-01-Depoortere.pdf] (Viewed,
August 24, 2006):
* Depoortere, E., "Health assessment in emergencies: Murnei and
Zalingei, West Darfur, Sudan." Epicentre and Médecins Sans Frontières.
(Paris, France: Médecins Sans Frontières, June 2004). [Hyperlink,
http://www.doctorswithoutborders.org/publications/reports/2004/epicentre
_report_darfur_05-2004.pdf] (Viewed, August 24, 2006):
* Hearns, A. and A. Agar, "Nutritional Survey, Mornei Camp, January-
February 2005." Concern Worldwide in collaboration with State Ministry
of Health and UNICEF.
* Rio, D., "Nutrition and Food Security Assessment, Wade Saleh and
Mukjar Provinces, West Darfur, North-Sudan." Médecins Sans Frontières
Holland. April 2004.
* West Darfur State, Ministry of Health in collaboration with UNICEF,
Concern and Save the Children-US. "Nutrition and Mortality Survey in
Ardamat, Dorti, Riyad and Abu-zar IDP camps." June 14-18, 2005.
South Darfur:
* Grandesso, F., F. Sanderson, J. Kruijit, T. Koene, and V. Brown,
"Mortality and Malnutrition among Populations Living in South Darfur,
Sudan." JAMA, vol. 293, no. 12, pp. 1,490-1,494 (March 2005).
[Hyperlink, http://jama.ama-assn.org/cgi/content/abstract/293/12/1490]
(Viewed, August 24, 2006):
Chad:
* United Nations High Commissioner for Refugees et al., "Emergency
Nutrition and Mortality Surveys Conducted among Sudanese Refugees and
Chadian Villagers, Northeast Chad, June 2004." [Hyperlink,
http://www.cdc.gov/nceh/ierh/ResearchandSurvey/Chad_report04.pdf]
(Viewed, August 24, 2006):
Other Information Provided:
* UN Nutrition Information in Crisis Situations, volumes 3, 4, 6, 7
from 2004 and 2005. [Hyperlink,
http://www.unsystem.org/scn/publications/RNIS/countries/sudan.htm]
(Viewed, August 24, 2006):
* Bureau of Democracy, Human Rights and Labor and Bureau of
Intelligence and Research, Department of State, "Documenting Atrocities
in Darfur," Publication 11182 (Washington, D.C.: Bureau of Democracy,
Human Rights, and Labor and the Bureau of Intelligence and Research,
September 2004). [Hyperlink, http://www.state.gov/g/drl/rls/36028.htm]
(Viewed, August 24, 2006):
Additional Bibliographical References:
* Boss, L., M. Toole, and R. Yip, "Assessment of Mortality, Morbidity
and Nutritional Status in Somalia During the 1991-1992 Famine:
Recommendations for Standardization of Methods," JAMA, vol. 272, no. 5
(1994).
* Checchi, F. and L. Roberts. "Interpreting and using mortality data in
humanitarian emergencies: A primer for non-epidemiologists,"
Humanitarian Practice Network, Network Paper no. 52. (London, UK,
2005).
* Coghlan, B., R. Brennan, P. Ngay, D. Dofara, B. Otto, M. Clements,
and T. Stewart, "Mortality in the Democratic Republic of Congo: a
nationwide survey," The Lancet, vol. 367, pp. 44-51 (2006).
* Guha-Sapir, D. and W. van Panhuis, "Conflict Mortality: Comparing the
data." CE-DAT Working Paper, series no. CD/2004/001. (2004).
* Hofman, C., L. Roberts, J. Shoham, P. Harvey, "Measuring the impact
of humanitarian aid: A review of current practice," Humanitarian Policy
Group. (London, UK, 2004).
* Leaning, J. and M. VanRooyen. "An assessment of mortality studies in
Darfur, 2004-2005," Humanitarian Exchange, vol. 30, pp. 23-26 (2005).
* Myatt M., A. Taylor, and W. Courtland Robinson, "A method for
estimating mortality rates using previous birth history," Field
Exchange, p. 13 (2002).
* Reed, H. (Rapporteur),"Demographic Assessment Techniques in Complex
Humanitarian Emergencies, Summary of a Workshop." National Academy
Press, Washington, D.C. (2002).
* Reed, H. and C. B. Keeley (eds.), Forced Migration and Mortality.
National Academy Press, Washington, D.C. (2001).
* Salama, P., P. Spiegel, L. Talley, and R. Waldman, "Lessons Learned
from complex emergencies over past decade," The Lancet, vol. 364,
pp.1,801-1,813 (2004).p:
* Silva, R. and P. Ball, "The Demography of Conflict-Related Mortality
in Timor-Leste (1974-1999): Empirical Quantitative Measurement of
Civilian Killings, Disappearances & Famine-Related Deaths." In Human
Rights and Statistical Objectivity. J. Asher, D. Banks, F. Schueren,
eds. New York: Springer (forthcoming).
* Spiegel, P., P. Salama, S. Maloney, and A. van der Veen, "Quality of
Malnutrition Assessment Surveys Conducted During Famine in Ethiopia,"
JAMA, vol. 292, no. 5 (2004).
* Thieren, M., "Health information systems in humanitarian
emergencies," Bulletin of the World Heath Organization, vol. 83, no. 8
(2005).
* Toole, M. and R. Waldman, "The Public Health Aspects of Complex
Emergencies and Refugee Situations," Annual Review of Public Health,
vol. 18, pp. 283-312 (1997).
* U.S. Centers for Disease Control and Prevention and The World Food
Program, A Manual: Measuring and Interpreting Malnutrition and
Mortality. Washington and Rome. (2005).
* Woodruff, B., "Violence and mortality in West Darfur." The Lancet,
vol. 364, pg. 1,290 (2004).
* Woodruff, B., "Interpreting mortality data in humanitarian
emergencies," The Lancet, vol. 367, pg. 9 (2006).
* World Food Program, "Emergency Food Security and nutrition assessment
in Darfur, Sudan 2005." Rome. (December 2005). [Hyperlink,
http://documents.wfp.org/stellent/groups/public/documents/ena/wfp089682.
pdf] (Viewed October 23, 2006):
* World Health Organization, "Module 4. Studying Health Status and
Health Needs." [Hyperlink,
http://www.who.int/hac/techguidance/tools/disrupted_sectors/module_04/en
/index2.html] (Viewed October 23, 2006):
[End of section]
Appendix IV: Summary Description of Death Estimates Reviewed:
Table 3: Estimate by Jan Coebergh:
Reported figures;
Three possible point estimates for excess deaths: 218,449; 253,573;or
306,130 represent a range of death estimates all above 200,000.
Time period covered;
April 2003 through December 2004.
Total number of months;
21.
Data sources;
* Uses varied sources including Atrocities Documentation Team's survey
of Chad refugees based on number of people who reported seeing family
member killed, 2004 mortality surveys by World Health Organization
(WHO) and Médicins Sans Frontiers and estimates by WHO officials;
* Does not use any contextual or nonsurvey information.
Mortality rates applied and cause of death;
* Calculates violent and nonviolent deaths separately;
* 306,130: Includes nonviolent and violent deaths. About 170,000 from
violence, 108,000 from health causes, and 25,000 in inaccessible areas;
* 218,449: Includes nonviolent and violent deaths. About 73,700 from
violence and 126,000 from health causes;
* 253,573: Includes nonviolent and violent deaths. About 111,000 from
health causes and about 143,000 from violence;
* Applies some daily mortality rates; other mortality rates are monthly
or for longer periods of time.
Mortality changes over time and/or region;
Assumes the same mortality levels over time and per region.
Baseline crude mortality rate assumed;
0.5.
Affected population included in estimate;
* Uses one "fixed" population estimate of 400,000 which is the average
of an assumed starting point of 0 in February 2003 and 800,000 in April
2004. (Affected population does not change over time.);
* Estimate includes affected population from all regions in Darfur and
refugees in Chad.
[End of table]
Table 4: Centre for Research on the Epidemiology of Disasters (CRED):
Two Estimates:
Reported figures;
* 134,000 total deaths;
* 118,142 excess deaths;
* 36,237 total deaths;
* 23,658 excess deaths.
Time period covered;
* September 2003 through January 2005;
* February 2005 through June 2005.
Number of months;
* 17;
* 5.
Data sources;
* Examines data from about 20 mortality surveys. About half of these
were used to estimate nonviolent mortality rates, and half were used to
analyze the proportion due to violence. (Surveys used depended on what
surveys reported the proportion of deaths due to violence.);
* Some steps were taken to examine the reliability of methods used in
the surveys and exclude outliers;
* Examines about 10 surveys conducted between January and August 2005;
however, primarily uses findings from the 2005 statewide WHO survey
because these results did not differ with other smaller surveys that
covered the same period.
* Relies mainly on mortality survey findings and uses other sources of
contextual or nonsurvey information to refine both estimates.
Mortality rates applied and cause of death;
* Estimates nonviolent and violent deaths separately. Total crude
mortality is nonviolence crude mortality divided by (1 minus the
proportion due to violence).
* Nonviolent mortality is generally assumed stable except for the
hunger (or rainy season) during June through August 2004;
* Proportion due to violence is based on a regression analysis of
survey data;
* 2005 WHO survey findings include crude mortality rates and proportion
of deaths due to violence or injury.
Mortality changes over time and/or region;
* Nonviolent mortality rate is assumed stable at 1.1 for all regions
except June through August 2004 for which rates are higher due to
impact of rainy season. (Rates used are based on results from the 2004
WHO survey for each region.);
* Based on regression analysis, proportion of violence generally
decreases over time beginning December 2003 to January 2005. Prior to
December 2003, assumes steady rate of violence;
* Assumes some difference in violence between regions;
* Assumes different mortality levels per region but no difference over
time from February to June 2005;
* Shows increase in violence for North Darfur from last estimate.
Baseline crude mortality rate assumed;
* 0.3 based on national average from the United Nations Children's Fund
(UNICEF).
Affected population included in estimate;
* Assumes different monthly estimates of the affected population based
on information from the United Nations Humanitarian Profiles for each
Darfur state from September 2003 till January 2005;
* Excludes deaths from isolated areas with high violence rates in South
Darfur after September 2004;
* Adds additional 200,000 for Chad refugees; their deaths estimated as
10% of the total excess deaths estimated;
* Assumes different monthly estimates separated by internally displaced
persons (IDP) and affected residents based on information from the
United Nations Humanitarian Profiles;
* Includes all regions in Darfur;
* Applies different mortality rates to groups of affected residents,
IDPs in camps, IDPs outside camps (did not do this in first estimate).
[End of table]
Table 5: Estimate by John Hagan and Others and Released by the
Coalition for International Justice:
Reported figures;
396,563 total deaths.
Time period covered;
February 2003 through March 2005.
Total number of months;
26.
Data sources;
* Uses Atrocities Documentation Team's survey of Chad refugees to
estimate violent deaths based on number of people who reported seeing a
family member killed;
* Uses 2004 WHO mortality survey findings for North and West Darfur to
estimate nonviolent deaths;
* Does not use any contextual or nonsurvey information.
Mortality rates applied and cause of death;
* Nonviolent mortality rate is on a combination of findings from North
and West Darfur WHO survey;
* Violent mortality rate is calculated based on survey of Chad refugees
and assumptions include that all missing persons have died and that the
average family size is five.
Mortality changes over time and/or region;
* Assumes the same mortality levels over time and per region.
Baseline crude mortality rate assumed;
* 0 (Did not apply a baseline mortality rate because believed it was
not appropriate.)
Affected population included in estimate;
* Uses a single estimate of 1.5 million, as of April 2005, to include
IDP population from all regions in Darfur and refugees in Chad. 1.5
million is the midpoint between 1.2 and 1.8 million. (1.2 comes from
State's Documenting Atrocities Report, and 1.8 was stated as widely
cited as the affected population in early 2005.)
[End of table]
Table 6: Estimate by Eric Reeves:
Reported figures;
Over 370,000 excess deaths.
Time period covered;
February 2003 through August 2005.
Number of months;
31.
Data sources;
* Uses varied sources such as the Atrocities Documentation Team's
survey of refugees in Chad, the 2005 WHO mortality survey,
communication from David Nabarro, and projections from the Department
of U.S. Agency for International Development;
* Uses contextual or nonsurvey data in various calculations, such as
mortality rates for inaccessible populations.
Mortality rates applied and cause of death;
* Generally estimates nonviolent and violent deaths separately;
* Survey of Chad refugees to estimate violent deaths based on number of
people who reported seeing family member killed;
* Estimates some mortality rates per day; estimates other rates per
month or longer periods of time.
Mortality changes over time and/or region;
* Assumes some changes in mortality over time. Generally does not
assume different mortality rates per region except for populations in
inaccessible areas.
Baseline crude mortality rates;
* 0.3 based on UNICEF.
Affected population included in estimate;
* Assumes affected population changes somewhat over time, although not
on a monthly basis. Uses information from UN profiles, as well as other
sources. For example, in some cases, assumes some groups are excluded
from UN profiles and increases numbers accordingly;
* Includes affected population from all regions in Darfur and Chad
refugees.
[End of table]
Table 7: Department of State Estimate:
Reported figures;
* 98,000 to 181,000 total deaths;
* 63,000 to 146,000 excess deaths.
Time period covered;
March 2003 through January 2005.
Total number of months;
23.
Data sources;
* Uses data deemed sound from available mortality surveys in the
Complex Emergency Database at that time for Darfur and refugees in Chad
(refers to list of about 20 surveys included in report);
* Provides some information regarding which surveys and contextual data
sources were used and why;
* Applies contextual information to fill in data gaps and determine if
existing mortality rates should be applied to a broader population
within that region or if a higher or lower rate was warranted. Refers
to information such as displacement patterns, fire mapping remote
sensing data, reporting of attacks, historical trends on humanitarian
intervention, and disease outbreaks.
Mortality rates applied and cause of death;
* Estimates high and low mortality rates that are assumed to include
mortality due to all causes (nonviolent and violent);
* Reports that some mortality rates were adjusted based on contextual
information (e.g., report of attacks). For example, reports using West
Darfur survey data as a basis for estimating mortality in North Darfur
are based on nonsurvey data showing similar patterns of conflict in
2003 and early 2004, but estimates of higher total deaths in North
Darfur are due to larger displaced population and more incidents of
fighting.
Mortality changes over time and/or region;
* Assumes differences in mortality over time and per region;
* Divides conflict into four time periods: (1) the initial outbreak of
violence (March through September 2003), (2) the breakdown of cease-
fire/escalation of conflict (October 2003 through March 2004), (3) the
second cease-fire (April through June 2004) and (4) increased
international humanitarian response (July 2004 through January 2005);
* Reports that highest mortality and level of violence occurred during
December 2003 to February 2004. Mortality began falling in March 2004.
Baseline crude mortality rate assumed;
Uses 0.5 deaths per 10,000 per based on estimates from the WHO of sub-
Saharan African populations.
Affected population included in estimate;
* Assumes different monthly estimates (separated by displaced and
affected populations) based on UN profiles for each Darfur state from
September 2003 until January 2005;
* Assumes an incremental increase prior to September 2003 because no
source of reliable information was available. Uses data from the United
Nations High Commissioner for Refugees (UNHCR) on spontaneous
settlements and camps to estimate Chad population;
* Assumes affected residents experienced mortality rates that were 20%
lower than displaced populations.
[End of table]
Table 8: WHO Estimate:
Reported figures;
* 45,000 to 80,000 total deaths;
* 35,000 to 70,000 excess deaths.
Time period covered;
March 2004 through September 2004.
Total number of months;
7.
Data sources;
* Primarily uses findings from 2004 WHO mortality survey, although
looked at other surveys conducted;
* Uses contextual information, such as the occurrence of disease
outbreaks and the level of humanitarian assistance, to help refine
estimates and define best and worst case scenarios.
Mortality rates applied and cause of death;
Estimates mortality rates include all causes of death (violent and
nonviolent).
Mortality changes over time and/or region;
Estimates an average mortality rate for the entire region of Darfur.
Does not assume different mortality rates over time or per region.
Baseline crude mortality rate assumed;
0.5.
Affected population included in estimate;
* Assumes different monthly estimates of IDPs based on UN profile
reports;
* Estimate includes only IDPs in camps and excludes affected residents
from all Darfur regions and refugees in Chad.
Source: GAO analysis based on published documents of estimates and
interviews with estimate authors.
Note: The summaries of these estimates are based on the published
information, as well as interviews with and additional information
provided by estimate authors with the exception of State who declined
to speak with us. Additionally, mortality rates discussed in table are
crude mortality rates per 10,000 per day.
[End of table]
[End of section]
Appendix V: Additional Follow-Up Survey Results:
This is a summary of responses to additional questions from our follow-
up survey from the 12 experts we convened with the National Academy of
Sciences in April 2006.
(1) In your view, are the mortality estimates by the listed authors
based on methodologically sound source data? Mark only one response in
each row.
Authors/estimates: Coebergh;
Definitely yes: 0;
Generally yes: 1;
Based as much on sound as unsound data: 4;
Generally no: 5;
Definitely no: 2;
No basis to judge/not sure: 0.
Authors/estimates: CRED(Guha-Sapir and Degomme);
Definitely yes: 0;
Generally yes: 9;
Based as much on sound as unsound data: 2;
Generally no: 1;
Definitely no: 0;
No basis to judge/not sure: 0.
Authors/estimates: Hagan, Rymond-Richard, and Parker;
Definitely yes: 0;
Generally yes: 0;
Based as much on sound as unsound data: 2;
Generally no: 7;
Definitely no: 2;
No basis to judge/not sure: 0.
Authors/estimates: Reeves;
Definitely yes: 0;
Generally yes: 0;
Based as much on sound as unsound data: 2;
Generally no: 7;
Definitely no: 3;
No basis to judge/not sure: 0.
Authors/estimates: U.S. Department of State (Phelan);
Definitely yes: 0;
Generally yes: 7;
Based as much on sound as unsound data: 2;
Generally no: 1;
Definitely no: 1;
No basis to judge/not sure: 1.
Authors/estimates: WHO (Nabarro);
Definitely yes: 1;
Generally yes: 6;
Based as much on sound as unsound data: 3;
Generally no: 0;
Definitely no: 2;
No basis to judge/not sure: 0.
[End of table]
(2) In your view, do the authors make appropriate or inappropriate
extrapolations from the source data to the affected population ? Mark
only one response in each row.
Authors/estimates: Coebergh;
Very appropriate: 0;
Somewhat appropriate: 0;
About as appropriate as inappropriate: 1;
Somewhat inappropriate: 5;
Very inappropriate: 5;
No basis to judge/not sure: 1.
Authors/estimates: CRED (Guha-Sapir and Degomme);
Very appropriate: 1;
Somewhat appropriate: 7;
About as appropriate as inappropriate: 2;
Somewhat inappropriate: 1;
Very inappropriate: 0;
No basis to judge/not sure: 1.
Authors/estimates: Hagan, Rymond-Richmond, and Parker;
Very appropriate: 0;
Somewhat appropriate: 0;
About as appropriate as inappropriate: 1;
Somewhat inappropriate: 5;
Very inappropriate: 6;
No basis to judge/not sure: 0.
Authors/estimates: Reeves;
Very appropriate: 0;
Somewhat appropriate: 0;
About as appropriate as inappropriate: 0;
Somewhat inappropriate: 5;
Very inappropriate: 7;
No basis to judge/not sure: 0.
Authors/estimates: U.S. Department of State (Phelan);
Very appropriate: 0;
Somewhat appropriate: 5;
About as appropriate as inappropriate: 3;
Somewhat inappropriate: 2;
Very inappropriate: 0;
No basis to judge/not sure: 2.
Authors/estimates: WHO (Nabarro);
Very appropriate: 0;
Somewhat appropriate: 4;
About as appropriate as inappropriate: 3;
Somewhat inappropriate: 2;
Very inappropriate: 2;
No basis to judge/not sure: 1.
[End of table]
(3) In your view, are the assumptions made by the authors reasonable or
unreasonable? (Mark only one response in each row.
Authors/estimates: Coebergh;
Very reasonable: 0;
Somewhat reasonable: 0;
As many reasonable as unreasonable assumptions: 2;
Somewhat unreasonable: 4;
Very unreasonable: 5;
No basis to judge/Not sure: 1.
Authors/estimates: CRED (Guha-Sapir and Degomme);
Very reasonable: 1;
Somewhat reasonable: 9;
As many reasonable as unreasonable assumptions: 0;
Somewhat unreasonable: 1;
Very unreasonable: 1;
No basis to judge/Not sure: 0.
Authors/estimates: Hagan, Rymond-Richmond, and Parker;
Very reasonable: 0;
Somewhat reasonable: 2;
As many reasonable as unreasonable assumptions: 0;
Somewhat unreasonable: 4;
Very unreasonable: 6;
No basis to judge/Not sure: 0.
Authors/estimates: Reeves;
Very reasonable: 0;
Somewhat reasonable:0;
As many reasonable as unreasonable assumptions: 1;
Somewhat unreasonable: 4;
Very unreasonable: 7;
No basis to judge/Not sure: 0.
Authors/estimates: U.S. Department of State (Phelan);
Very reasonable: 1;
Somewhat reasonable: 6;
As many reasonable as unreasonable assumptions: 1;
Somewhat unreasonable: 2;
Very unreasonable: 0;
No basis to judge/Not sure: 2.
Authors/estimates: WHO (Nabarro);
Very reasonable: 0;
Somewhat reasonable: 5;
As many reasonable as unreasonable assumptions: 3;
Somewhat unreasonable: 2;
Very unreasonable: 1;
No basis to judge/Not sure: 1.
[End of table]
(4) In your view, do the authors sufficiently or insufficiently
describe appropriate limitations, including sources of possible over or
under estimation? (Mark only one response in each row.)
Authors/estimates: Coebergh;
Very sufficiently: 0;
Somewhat sufficiently: 3;
As sufficiently as insufficiently: 0;
Somewhat insufficiently: 3;
Very insufficiently: 6;
No basis to judge/Not sure: 0.
Authors/estimates: CRED (Guha-Sapir and Degomme);
Very sufficiently: 2;
Somewhat sufficiently: 6;
As sufficiently as insufficiently: 2;
Somewhat insufficiently: 2;
Very insufficiently: 0;
No basis to judge/Not sure: 0.
Authors/estimates: Hagan, Rymond-Richmond, and Parker;
Very sufficiently: 0;
Somewhat sufficiently: 0;
As sufficiently as insufficiently: 3;
Somewhat insufficiently: 1;
Very insufficiently: 8;
No basis to judge/Not sure: 0.
Authors/estimates: Reeves;
Very sufficiently: 0;
Somewhat sufficiently: 1;
As sufficiently as insufficiently: 0;
Somewhat insufficiently: 4;
Very insufficiently: 7;
No basis to judge/Not sure: 0.
Authors/estimates: U.S. Department of State (Phelan);
Very sufficiently: 1;
Somewhat sufficiently: 4;
As sufficiently as insufficiently: 2;
Somewhat insufficiently: 4;
Very insufficiently: 0;
No basis to judge/Not sure: 1.
Authors/estimates: WHO (Nabarro);
Very sufficiently: 0;
Somewhat sufficiently: 7;
As sufficiently as insufficiently: 1;
Somewhat insufficiently: 2;
Very insufficiently: 2;
No basis to judge/Not sure: 0.
[End of table]
(5) In your view, how high or low would you rate the level of
objectivity in the authors' mortality estimates? (Mark only one
response in each row.
Authors/estimates: Coebergh;
Very high: 0;
High: 0;
Moderate: 2;
Low: 8;
Very Low: 2;
No basis to judge/Not sure: 0.
Authors/estimates: CRED (Guha-Sapir and Degomme);
Very high: 2;
High: 8;
Moderate: 1;
Low: 0;
Very Low: 0;
No basis to judge/Not sure: 1.
Authors/estimates: Hagan, Rymond-Richmond, and Parker;
Very high: 1;
High: 0;
Moderate: 5;
Low: 3;
Very Low: 2;
No basis to judge/Not sure: 1.
Authors/estimates: Reeves;
Very high: 0;
High: 0;
Moderate: 0;
Low: 8;
Very Low: 4;
No basis to judge/Not sure: 0.
Authors/estimates: U.S. Department of State (Phelan);
Very high: 0;
High: 9;
Moderate: 2;
Low: 0;
Very Low: 0;
No basis to judge/Not sure: 1.
Authors/estimates: WHO (Nabarro);
Very high: 0;
High: 6;
Moderate: 3;
Low: 1;
Very Low: 1;
No basis to judge/Not sure: 1.
[End of table]
(6) Overall, what is your level of confidence in the estimates made by
each of the authors? (Mark only one response in each row.
Authors/estimates: Coebergh;
Very high: 0;
High: 0;
Moderate: 0;
Low: 7;
Very Low: 5;
No basis to judge/Not sure: 0.
Authors/estimates: CRED (Guha-Sapir and Degomme);
Very high: 0;
High: 2;
Moderate: 9;
Low: 1;
Very Low: 0;
No basis to judge/Not sure: 0.
Authors/estimates: Hagan, Rymond-Richmond, and Parker;
Very high: 0;
High: 0;
Moderate: 1;
Low: 3;
Very Low: 8;
No basis to judge/Not sure: 0.
Authors/estimates: Reeves;
Very high: 0;
High: 0;
Moderate: 0;
Low: 4;
Very Low: 8;
No basis to judge/Not sure: 0.
Authors/estimates: U.S. Department of State (Phelan);
Very high: 0;
High: 2;
Moderate: 7;
Low: 2;
Very Low: 1;
No basis to judge/Not sure: 0.
Authors/estimates: WHO (Nabarro);
Very high: 0;
High: 0;
Moderate: 7;
Low: 3;
Very Low: 1;
No basis to judge/Not sure: 1.
[End of Table]
[End of section]
Appendix VI: Comments from the Department of State:
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
United States Department of State:
Assistant Secretary for Resource Management and Chief Financial
Officer:
Washington, D. C. 20520:
Ms. Jacquelyn Williams-Bridgers:
Managing Director:
International Affairs and Trade:
Government Accountability Office 441 G Street, N. W.
Washington, D.C. 20548-0001:
Oct 4 2006:
Dear Ms. Williams-Bridgers:
We appreciate the opportunity to review your draft report, "Darfur
Crisis: Death Estimates Show Crisis Severity, but Their Accuracy and
Credibility Could Be Enhanced," GAO Job Code 320420.
The enclosed Department of State comments are provided for
incorporation with this letter as an appendix to the final report.
If you have any questions concerning this response, please contact Jim
Gray, Congressional Liaison, Bureau of Intelligence and Research, at
(202) 647-2921.
Sincerely,
Signed by:
Bradford R. Higgins:
cc: GAO - Leslie Holen:
INR - Randall M. Fort:
State/OIG - Mark Duda:
Department of State Comments on GAO Draft Report:
Darfur Crisis: Death Estimates Show Crisis Severity, but Their Accuracy
and Credibility Could be Enhanced (GAO-7-24, GAO Code 320420):
Thank you for allowing the Department of State the opportunity to
comment on the draft report Darfur Crisis: Death Estimates Show Crisis
Severity, but Their Accuracy and Credibility Could be Enhanced.
GAO Recommendations:
To safeguard the Department of State's credibility as a source of
accurate and reliable death estimates, we recommend that the Secretary
of State promote greater transparency in any of its future death
estimates for Darfur or other humanitarian crises by ensuring that
publicly available documents contain sufficient detail on the
estimates' data, methods, assumptions, and limitations to allow
external researchers to replicate and verify the estimates.
Additionally, to enhance the U.S. government's capacity to assess the
dimensions of, and respond appropriately to, any future humanitarian
crises, we recommend that the Secretary of State and the Director of
U.S. Foreign Assistance and USAID Administrator consider the experts'
other suggested measures to help address gaps in data and improve the
quality and reliability of any future death estimates.
Department of State Comment:
The Department of State endorses these recommendations and supports
efforts to increase transparency, address gaps in data, and improve the
quality of future death estimates. As the GAO points out, humanitarian
crises present difficult challenges in collecting and presenting
mortality data.
In the case of the Darfur Mortality Estimate, the State Department
declassified an internal assessment of the Bureau of Intelligence and
Research (INR) originally intended for use by Department policymakers.
INR's source documentation and description of methodology were provided
to the Centre for Research on the Epidemiology of Disasters (CRED) for
use in compiling the CRED study titled Darfur: Counting the Deaths, a
source for this GAO report.
Following are GAO's comments on the Department of State's letter dated
October 4, 2006.
GAO Comments:
1. State asserts that its death estimate was intended for internal
purposes. We maintain that because State's estimate was publicly
available and discussed by State officials, sufficient detail on this
and any future such estimates is necessary to safeguard State's
credibility as a source of accurate and reliable death estimates,
particularly where such a serious topic is concerned.
2. When asked, the CRED authors indicated that State provided them with
the text on the methodology as included in the published report, but
did not provide source information on the data used in State's
estimate.
[End of section]
Appendix VII: Comments from the U.S. Agency for International
Development:
USAID:
From The American People:
Oct 17 2006:
David Gootnick:
Director:
International Affairs and Trade:
U.S. Government Accountability Office:
441 G Street, N.W.
Washington, D.C. 20548:
Dear Mr. Gootnick:
I am pleased to provide the U.S. Agency for International Development's
(USAID) formal response on the draft GAO report entitled Darfur Crises:
Death Estimates Show Crisis Severity, but Their Accuracy and
Credibility Could Be Enhanced (GAO-07-24).
Thank you for the opportunity to respond to the GAO draft report and
for the courtesies extended by your staff in the conduct of this
review. We appreciate GAO's comprehensive efforts in addressing this
important subject and we support your recommendation that the U.S.
Government should work to ensure greater transparency regarding the
data and methods used for death estimates. USAID also concurs with your
second recommendation that the Secretary of State and the Director of
U.S. Foreign Assistance and USAID Administrator consider the experts'
other suggested measures to help address gaps in data and improve the
quality and reliability of future death estimates.
USAID looks forward to working to help address gaps in data and improve
the quality of future death estimates. Thank you again for the
opportunity to comment.
Sincerely,
Signed by:
Mosina H. Jordan:
Counselor to the Agency:
[End of section]
Appendix VIII: Summary of Authors' Comments:
To the five other estimate authors, we provided portions of the report
pertaining to their individual estimates. We summarize their comments
and our responses below.
* The authors of the Centre for Research on the Epidemiology of
Disasters (CRED) estimates agreed with the experts' evaluation.
* Jan Coebergh did not provide any major comments regarding the
relevant portion of the report and indicated that he was aware of the
limitations of his estimate.
* John Hagan expressed concern regarding the experts' relatively
positive evaluations of the estimates by CRED and the Department of
State (State), stating that the respective estimates' published
documentation does not provide sufficient information on the data and
methodologies used. As discussed in the report, the experts noted
limitations in the sufficiency of reporting for both these estimates,
but they were able to discuss these estimates in detail and assess
their respective levels of accuracy and methodological strengths and
shortcomings. In addition to providing the experts the published
reports containing the estimates, we provided them information
regarding the source mortality surveys used in the estimates, as listed
in appendix III. For the CRED estimate, we also provided the experts
with additional details about the methodology based on our interviews
with the authors. Dr. Hagan also believed that we should have included
in our evaluation the estimate he did with Alberto Palloni published in
the September 2006 issue of Science. We added information on the key
findings of this estimate to the report, but as we note in the report,
it does not include an analysis of the estimate because a final
publicly available version was not available when the experts convened
in April 2006.
* Eric Reeves disagreed with the experts' criticism of the Atrocities
Documentation Team's survey of Chad refugees. He stated this survey was
well conducted and a critical source of data to estimate violent deaths
especially early in the conflict when little information was available.
In the report, we note that the literature acknowledges difficulties in
capturing deaths due to violence. Nevertheless, some of the estimates
reviewed by the experts used other techniques to account for violent
deaths, such as statistical analysis or contextual information.
Moreover, many experts felt that Atrocities Documentation Team's survey
was not a reliable or appropriate source of data to estimate violent
deaths for a cumulative death estimate on Darfur, based on public
documentation on the survey, as well as supplemental information on the
survey's design, implementation, and sampling we obtained from
representatives at the Coalition for International Justice. In
addition, Dr. Reeves also expressed a concern regarding the 2004 survey
by WHO and believed it did not sufficiently capture violent mortality
in areas outside camps. To address this concern, we added information
regarding the issue of accessibility as a potential source for
underestimation of mortality and specifically discuss this survey. Dr.
Reeves also stated that in a draft version of the report, we had
incorrectly characterized his estimate's use of the 2004 WHO survey to
account for only nonviolent deaths and, accordingly, we removed this
characterization from the report.
* Regarding the WHO estimate, David Nabarro emphasized in his comments
that the estimate was undertaken to provide a rough order of magnitude
of deaths in order to facilitate humanitarian relief efforts. He also
pointed out that it applied to a shorter time period than the other
estimates and was restricted to IDPs located in camps, without
attempting to account for deaths in the larger population. We provided
additional clarification in the report to emphasize the more restricted
scope and coverage of the WHO estimate. Further, we stated that the
purpose of the WHO estimate was to provide an order of magnitude
estimate to assist in humanitarian relief planning.
[End of section]
Appendix IX: GAO Contacts and Staff Acknowledgments:
GAO Contacts:
David Gootnick, (202) 512-3149, gootnickd@gao.gov Nancy Kingsbury,
(202) 512-2700, kingsburyn@gao.gov:
Staff Acknowledgments:
In addition to the individuals named above, Emil Friberg, Assistant
Director; Jim Ashley; Martin de Alteriis; Etana Finkler; Leslie Holen;
Theresa Lo; Reid Lowe; Grace Lui; John F. Miller; and Chhandasi Pandya
made key contributions to this report.
FOOTNOTES
[1] Simultaneously with the release of this report, we are issuing a
separate report on U.S. humanitarian assistance to relieve the crisis
in Darfur; challenges that have affected the delivery of the
assistance; the African Union's efforts to fulfill a mandate to support
peace in Darfur; and factors that have affected its implementation of
this mandate. See GAO, Darfur Crisis: Progress in Aid and Peace
Monitoring Threatened by Ongoing Violence and Operational Challenges,
GAO-07-9 (Washington, D.C.: Nov. 9, 2006).
[2] The five estimates that were publicly available in March 2006 were
conducted by, respectively, (1) Jan Coebergh, a medical doctor who has
worked in Darfur; (2) the Centre for Research on the Epidemiology of
Disasters (CRED) in Brussels, Belgium; (3) John Hagan, Wynnona Rymond-
Richmond, and Patricia Parker (released by the Coalition of
International Justice); (4) Eric Reeves, an researcher and advocate for
Sudan-related issues; and (5) the World Health Organization (WHO).
[3] This report's assessment of the estimates' methodological strengths
and shortcomings is based on experts' opinions of the estimates' data,
methods, objectivity, and sufficiency of reporting.
[4] Two additional experts participated by phone for parts of the day.
[5] CRED conducted two death estimates: one for the period of September
2003-January 2005 and another for the period of February-June 2005.
[6] The three highest estimates reviewed by the experts ranged from
253,573 to 396,563 deaths over varying periods of time.
[7] We recognize that when such estimates draw from classified data,
external researchers reviewing these data would require appropriate
levels of security clearance.
[8] Estimates for the preconflict population of Darfur range from about
4 million to close to 7 million. Experts we interviewed have noted
uncertainty regarding the population estimate for Darfur due to the
lack of a current census and the fact that migration in this region
occurs even during nonconflict times.
[9] U.S. law considers any person who is outside the country of such
person's nationality and who is unable or unwilling to return to, and
is unable or unwilling to avail himself or herself of the protection
of, that country because of persecution or a well-founded fear of
persecution on account of race, religion, nationality, membership in a
particular social group, or political opinion to be a refugee. 8 U.S.C.
§ 1101 (a) (42)A.
[10] "Agreement on Humanitarian Ceasefire on the Conflict in Darfur,"
signed on April 8, 2004, in N'Djamena, Chad.
[11] Additional interim agreements were also reached, including the
July 5, 2005, "Declaration of Principles for the Resolution of the
Sudanese Conflict in Darfur" signed by the Sudanese government, the
SLM/A, and the JEM. This declaration contains 17 principles to guide
future deliberations, such as respect for the diversity of the people
of the Sudan, democracy, a federal system of government, effective
representation in all national government institutions, and equitable
distribution of national wealth.
[12] Darfur Peace Agreement" signed on May 5, 2006, at Abuja, Nigeria.
[13] UN Security Council Resolution 1706, adopted August 31, 2006. The
expanded UN Mission in the Sudan mandate includes, among other things,
supporting the implementation of the 2006 Darfur Peace Agreement and
the 2004 Humanitarian Ceasefire Agreement.
[14] H. Con. Res. 467, 108th Cong. (2004); S. Con. Res. 133, 108th
Cong. (2004).
[15] The Convention on the Prevention and Punishment of the Crime of
Genocide of 1948 defines genocide as any of the following acts
committed with intent to destroy, in whole or in part, a national,
ethnical, racial or religious group, as such: (a) killing members of
the group, (b) causing serious bodily or mental harm to members of the
group, (c) deliberately inflicting on the group conditions of life
calculated to bring about its physical destruction in whole or in
part,(d) imposing measures intended to prevent births within the group,
or (e) forcibly transferring children of the group to another group.
The U.S. government ratified the convention in 1988. According to a
State official, the key factor in the U.S. government's genocide
determination was the intent of the Sudanese government regarding its
actions in Darfur (i.e., its intent to destroy, in whole or in part, a
specific group of people); the number of deaths attributable to the
crisis was not a critical factor.
[16] The Coalition for International Justice (CIJ) was an
international, nonprofit organization that supported international war
crimes tribunals and justice initiatives. It closed its operations in
March 2006.
[17] Department of State, Documenting Atrocities in Darfur, Publication
11182 (Washington, D.C.: Bureau of Democracy, Human Rights, and Labor
and the Bureau of Intelligence and Research, September 2004). Available
at [Hyperlink, http://www.state.gov/g/drl/rls/36028.htm].
[18] In addition to the U.S. assistance provided for Darfur, an
additional $197 million has supported Darfur refugees located in
Eastern Chad for fiscal years 2004 through 2006.
[19] See GAO-07-9.
[20] The estimates by Coebergh, Hagan, and Reeves used the Atrocities
Documentation Team's survey of Chad refugees as an additional key
source of data.
[21] Respondents are also asked to report the number of births and the
numbers of people in their household during the recall period.
[22] According to the Sphere Project, the crude mortality rate can be
expressed with different standard population denominators and for
different time periods; however, the daily crude mortality rate is the
most specific and useful health indicator to monitor in a disaster
situation. [See The Sphere Project, Humanitarian Charter and Minimum
Standards in Disaster Response (Geneva: Switzerland, 2004); also known
as the Sphere Handbook.] Sphere, launched in 1997 by a group of
humanitarian NGOs and the Red Cross, developed minimum standards to be
attained in disaster assistance in five sectors: water supply and
sanitation, nutrition, food aid, shelter, and health services. The
Sphere Handbook reports that a doubling of the baseline crude mortality
rate indicates a significant public health emergency, requiring an
immediate response. If the baseline rate is unknown, health agencies
should aim to maintain the crude mortality rate at below 1.0 per 10,000
per day. According to data from the U.S. National Center for Health
Statistics, the normal crude mortality rate for 2003 in the United
States is about 0.23 deaths per 10,000 per day.
[23] CE-DAT is a searchable database of complex emergencies that
includes information on health and mortality indicators. The database
which receives funds from the State's Bureau of Population, Refugees,
and Migration, is housed in CRED, a WHO Collaborating Center located
within the School of Public Health of the Université Catholique de
Louvain in Brussels. (See [Hyperlink,
http://www.cred.be/cedat/index.htm.]) According to CRED, the database
contains 1,155 surveys covering 36 countries; half of these surveys are
drawn from original reports, and the other half are drawn from
secondary sources. Research staff from CE-DAT we interviewed have told
us that, because they rely on voluntary reporting from organizations
conducting surveys to populate their database, not all surveys that
have been conducted in Darfur may be included in their database.
[24] Across the four sites, Médecins Sans Frontières surveyed 3,175
households representing a total of about 17,500 people and drawn from
an estimated IDP population of 215,400. See Depoortere, Evelyn et al.
"Violence and Mortality in West Darfur, Sudan (2003-2004):
Epidemiological Evidence from Four Surveys." Lancet, 364 (2004): 1,315-
1,320. Médecins Sans Frontières produced this study in collaboration
with Epicentre, a nonprofit organization created in 1987 by Médecins
Sans Frontières, which groups health professionals specialized in
public health and epidemiology. (See Hyperlink,
http://www.epicentre.msf.org/.)
[25] The recall period varied in the surveys conducted across the four
sites. The longest was 183 days for the survey conducted at Zalingei,
and the shortest was 39 days for the survey at El Geneina. Médecins
Sans Frontières reported using a calendar of locally important events
to facilitate recall.
[26] The WHO reported surveying a total of 3,140 households
representing about 21,000 people. The WHO surveyed IDPs from 43
locations in North Darfur and 43 locations in West Darfur, but due to
security problems surveyed only IDPs residing in Kalma camp in South
Darfur. See World Health Organization. Retrospective Mortality Survey
among the Internally Displaced Population Greater Darfur, Sudan 2004,
2004. [Hyperlink, http://www.who.int/disasters/repo/14656.pdf]
[27] CRED is a nonprofit research institution and a World Health
Organization Collaborating Centre based in the School of Public Health
of the Université Catholique de Louvain in Brussels.
[28] Jan Coebergh is a medical doctor in the Netherlands who has worked
in Darfur.
[29] John Hagan, a professor of sociology at Northwestern University,
conducted his estimate with his colleagues, Wynnona Rymond-Richmond and
Patricia Parker; the estimate was released by the Coalition of
International Justice. Dr. Hagan stated that the 2005 estimate used an
assumption of constant levels of mortality over time, because he
believed other estimates had relied on that assumption. He also noted
that he subsequently modified his assumptions and estimating methods in
his estimate with Alberto Palloni. [See John Hagan and Alberto Palloni,
"Social Science: Death in Darfur," Science, vol. 313 (2006): 1,578-
1,579.]
[30] Eric Reeves is a professor of English language and literature at
Smith College and has spent the past 7 years working full-time on
research and advocacy issues related to Sudan. When the experts
convened in April 2006, they reviewed and discussed Reeves's latest
available estimate of 375,000 through August 2005. At the end of April
2006, he reported that total excess mortality in Darfur was greater
than 450,000 deaths.
[31] The WHO estimate was presented by David Nabarro, a senior WHO
official.
[32] See John Hagan and Alberto Palloni, "Social Science: Death in
Darfur," Science, vol. 313 (2006): 1578-1579. In addition, the experts
did not review a study of the Darfur conflict by Bloodhound, a Danish
advocacy group, because it was published after the experts met in April
2006. The study estimates that deaths in Darfur ranged from 57,000 to
128,000 deaths between April 2003 and September 2005 due to attacks on
villages throughout the region. [See Andreas Höfer Petersen and Lise-
Lotte Tullin, The Scorched Earth of Darfur: Patterns in Death and
Destruction Reported by the People of Darfur. January 2001-September
2005 (Copenhagen: Bloodhound, 2006). Available at Hyperlink,
http://www.bloodhound.se/rap_uk.html.]
[33] The WHO conducted an initial estimate in July 2004 and a
subsequent estimate in October 2004. Eric Reeves began his estimates in
June 2004 and has reported ongoing "mortality updates" since that time.
As of October 2006, his last update was conducted at the end of April
2006.
[34] Although they questioned the accuracy of the estimates and had
differing views about the actual number of deaths that have occurred in
Darfur, the experts did not question the severity of the crisis. In
discussing the importance of the estimates' accuracy, several experts
noted that estimates can be used in war crime proceedings, and one
stated that it was important to account for all those who died so that
this could become part of the historical record.
[35] One expert rated having a moderate level of confidence in the
estimate by Hagan.
[36] In evaluating each estimate's methodological strength, the experts
rated the source data and whether such data were methodologically
sound; the methods, including whether extrapolations were appropriate,
assumptions were reasonable, and shortcomings that could result in over-
or underestimation were sufficiently described; the level of
objectivity, based on whether a particular bias appears to be part of
the estimating procedure; and the sufficiency of reporting and
information contained in published documents describing the estimate.
[37] CRED selected or excluded surveys for reasons such as whether
findings from a regional survey differed statistically from findings
for statewide surveys or whether a survey's findings were viewed as
outliers. CRED used the selected surveys to analyze violent or
nonviolent mortality rates depending on, for example, whether the
surveys reported the proportion of deaths due to violence. Although
CRED primarily relied on mortality survey data, it also used contextual
information to refine its estimates and make some assumptions.
[38] Nine experts rated the data CRED used as generally sound, two
rated the data as equally sound and unsound, and two rated the data as
generally unsound.
[39] CRED analyzed findings from the mortality surveys mentioned above
to separately estimate nonviolent and violent mortality rates and
combined these to get overall mortality rates for different regions and
periods of the conflict.
[40] Eight experts rated CRED's extrapolations as very or somewhat
appropriate, two experts rated them about as appropriate as
inappropriate, one expert rated them as somewhat inappropriate, and one
expert found no basis to judge or was not sure. Additionally, 10
experts rated CRED's assumptions as very or somewhat reasonable, and
two experts rated them as somewhat or very unreasonable.
[41] Ten experts rated CRED's level of objectivity as very high or
high, one expert rated it as moderate, and one expert indicated that he
or she had no basis to judge or was not sure.
[42] State reports using various sources of information from sources
other than surveys, such as displacement patterns, village destruction,
and retrospective reporting of attacks to, among other things, fill in
data gaps and determine whether mortality rates derived from existing
surveys could be applied as an overall rate to a broader population or
if higher or lower rates were warranted.
[43] Seven of the 12 experts rated the State's data as generally
methodologically sound, 2 rated the data as equally sound and unsound,
2 rated the data as generally not sound, and 1 indicated that he or she
was not sure or had no basis to judge.
[44] Two experts rated the objectivity as moderate, and one said that
he or she had no basis to judge or was not sure.
[45] One indicated no basis to judge or not sure.
[46] Each of the three estimates based, at least in part, its numbers
of violent deaths on the "Atrocities Documentation Team's" survey of
Chad refugees and its numbers of nonviolent deaths on the 2004 WHO
survey. The Coebergh and Reeves estimates also used additional data
sources.
[47] The 2004 WHO survey reported that violence or injury accounted for
10 to 21 percent of deaths in North and West Darfur and Kalma camp in
South Darfur for a 2-month recall period in the summer of 2004. Jan
Coebergh and John Hagan told us they knew that the WHO survey reported
a percentage of deaths due to violence but believed the survey did not
sufficiently capture violent deaths, particularly during the period
prior to camp arrival.
[48] Reeves's estimate assumes some change in mortality levels over
time.
[49] In Darfur, for example, some surveys were conducted in Arabic and
not in other local languages.
[50] One expert stated that in Darfur, a household was defined as
everyone who ate at the same table the previous night. This definition
is important because household size is a factor in calculating crude
mortality rates.
[51] One expert noted that certain cultures do not report the deaths of
unnamed infants and, in some of these cultures, naming does not occur
until an infant has lived for 40 days. Because the first 40 days of
life present the highest risk of mortality, unreported infant deaths
could compromise mortality estimates.
[52] Known as "survivor bias" in the research literature. For example,
see World Health Organization, "Module 4: Studying Health Status and
Health Needs." (Available at Hyperlink,
http://www.who.int/hac/techguidance/tools/disrupted_sectors/module_04/en
/index2.html p.3)
[53] Known as "recall bias" in the research literature. For example,
see World Health Organization, "Module 4: Studying Health Status and
Health Needs." (Available at Hyperlink,
http://www.who.int/hac/techguidance/tools/disrupted_sectors/module_04/en
/index2.html p.3.)
[54] For example, a survey conducted by the World Food Program in the
fall of 2004 had a recall period of 7 months. However, because the
average length of displacement was 7.5 months, it is likely that the
survey did not capture all mortality that occurred prior to
displacement, some of which could have been due to violence.
[55] For example, identifying death from particular illnesses and
diseases can be difficult for respondents.
[56] The research literature also questions whether appropriate
guidance and training in sampling are routinely provided to field staff
and whether guidance is always followed when provided. See Paul B.
Spiegel, Peter Salama, Susan Maloney, and Albertine van der Veen,
"Quality of Malnutrition Assessment Surveys Conducted During Famine in
Ethiopia," JAMA, vol. 292, no. 5 (2004).
[57] See Jennifer Leaning and Michael VanRooyen, "An Assessment of
Mortality Studies in Darfur, 2004-2005," Humanitarian, vol. 30 (June
2005). Also see Michel Thieren, "Health Information Systems in
Humanitarian Emergencies," Bulletin of the World Heath Organization,
vol. 83, no. 8 (2005). For a discussion of possible approaches to
sampling in these situations, see Holly Reed (Rapporteur), "Demographic
Assessment Techniques in Complex Humanitarian Emergencies," Summary of
a Workshop (Washington, D.C.: National Academy Press, 2002).
[58] SMART coordinated by USAID, involves experts from U.S. government
agencies, multilateral organizations, universities, and nongovernmental
organizations. SMART has issued guidance on how to measure mortality,
nutrition, and food security in crisis situations. CE-DAT serves as
SMART's primary data source for trend analysis and monitoring and
reporting.
[59] Mortality surveillance systems require teams of trained home
visitors to record deaths in the population as they occur. The home
visitors are assigned a specific sector of the camp or village and
instructed to visit their assigned areas regularly. During their
visits, they record the population size of the area and all deaths that
have occurred.
[60] See World Health Organization, pp. 2-3, and Bradley Woodruff,
"Violence and Mortality in West Darfur," The Lancet, vol. 364 (2004),
p. 1,290.
[61] See Bradley Woodruff. "Interpreting mortality data in humanitarian
emergencies," The Lancet, vol. 367 (2006), p. 9. Also see Romesh Silva
and Patrick Ball, "The Demography of Conflict-Related Mortality in
Timor-Leste (1974-1999): Empirical Quantitative Measurement of Civilian
Killings, Disappearances and Famine-related Deaths," in Human Rights
and Statistical Objectivity, J. Asher, D. Banks, F. Schueren, eds. New
York: Springer (forthcoming). Also see Francesco Checchi and Les
Roberts, Interpreting and Using Mortality Data in Humanitarian
Emergencies: A Primer for Non-Epidemiologists, Network Paper no. 52
(London: Humanitarian Practice Network, 2005).
[62] Estimates of the affected population reported in the UN
Humanitarian Profiles are based on data provided by international
humanitarian agencies and their implementing partners; the majority of
information comes from the UN World Food Program food registration.
These estimates do not include residents in the three state capitals of
Darfur, Nyala, El Fasher, and Geneina because their number is
relatively large compared with the IDP population they are hosting.
Officials from the UN Office for the Coordination of Humanitarian
Affairs told us that they had had difficulty verifying the reliability
of data reported and insuring the uniformity of such data.
[63] The Darfur death estimates used different baseline mortality
rates. For example, CRED used a baseline of 0.3 because UNICEF reported
that as a national average prior to the conflict. State used 0.5
because that was the WHO baseline mortality rate for sub-Saharan
Africa.
[64] In addition, some experts felt that it was appropriate to use two
rates, one for the "normal" period and one for the "emergency" period
of the crisis. For example, one expert proposed 0.3 for the normal
period and 0.6 for the emergency period.
[65] The remaining experts either presented views for or against or did
not comment on this issue. An expert who made this argument also noted
that the countries with higher rates of normal mortality would
attribute relatively more deaths to "normal" factors than in countries
with lower mortality rates.
[66] The experts suggested that the U.S. agencies best able to
implement these suggestions are the CDC, USAID, and State. Other,
nongovernment entities include academic institutions, such as the Johns
Hopkins University; multilateral organizations, such as the WHO; and
NGOs such as Médecins Sans Frontières.
[67] Experts rated the measures using the following 5-point scale:
"Very greatly improve," "Greatly improve," Moderately Improve,"
"Somewhat Improve," and "Not improve."
[68] One expert felt that it would be preferable for the U.S.
government to support the existing technical teams.
[69] Cluster sampling is a simple method that can be used to draw a
representative sample even where there is no listing, or known total
number, of households (that is, no individual household sampling frame)
and where households are arranged in a chaotic pattern, as if often the
case in IDP camps. See Francesco Checchi and Les Roberts, Interpreting
and Using Mortality Data in Humanitarian Emergencies: A Primer for Non-
Epidemiologists, Network Paper no. 52 (London: Humanitarian Practice
Network, 2005).
[70] SMART initiative is aimed at improving the monitoring, reporting,
and evaluation of humanitarian assistance interventions. (See
www.smartindicators.org.)
[71] For example, members from the International Emergency and Refugee
Health Branch provided technical training on data collection to the
World Food Program for two mortality surveys conducted in Darfur.
[72] As noted in the report, we and the group of experts also reviewed
a preliminary death estimate for West Darfur by John Hagan and Alberto
Palloni, but we do not discuss the estimate in this report because the
estimate had not been published. The authors told us that the version
of the estimate they gave us was preliminary and that they were working
on a more refined version of the estimate using different methods.
[73] Over half of our experts were also authors or coauthors on some of
the articles we reviewed.
[74] The experts also rated an additional measure for the Darfur crisis
regarding whether an independent agency should conduct a retrospective
assessment.
[75] As noted in the report, the experts reviewed a death estimate of
West Darfur by John Hagan and Alberto Palloni, but this estimate is not
discussed in this report because it was not publicly available.
[76] Survey included in CRED December 2005 update. Note: the report
included here was only publicly available in HTML format. Therefore,
some of the appendixes and formatting did not come through.
[77] This figure is from an August 2005 estimate that was the latest
available at the time the group of experts convened, and this was
reviewed. At the end of April 2006, Eric Reeves provided a new estimate
of over 450,000 people have died from violence, disease, and
malnutrition.
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site (www.gao.gov). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site.
To have GAO e-mail you a list of newly posted products every afternoon,
go to www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125
Washington, D.C. 20548:
Public Affairs:
Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548:
