Drug Control
DOD Needs to Improve Its Performance Measurement System to Better Manage and Oversee Its Counternarcotics Activities Gao ID: GAO-10-835 July 21, 2010The Department of Defense (DOD) leads detection and monitoring of aerial and maritime transit of illegal drugs into the United States in support of law enforcement agencies. DOD reported resources of more than $1.5 billion for fiscal year 2010 to support its counternarcotics activities. Congress mandated GAO report on DOD's counternarcotics performance measurement system. Specifically, this report addresses the extent to which (1) DOD's counternarcotics performance measurement system enables DOD to track progress and (2) DOD uses performance information from its counternarcotics performance measurement system to manage its activities. GAO analyzed relevant DOD performance and budget documents, and discussed these efforts with officials from DOD and the Office of National Drug Control Policy (ONDCP).
DOD does not have an effective performance measurement system to track the progress of its counternarcotics activities; however, it continues efforts to improve the system. GAO has previously reported that measuring performance provides managers a basis for making fact-based decisions. DOD has established performance measures for its counternarcotics activities and a database to collect performance information, including measures, targets, and results. However, these measures lack a number of attributes, such as being clearly stated and objective, which GAO considers key to successful performance measures. In May 2010, DOD issued new guidance for its counternarcotics performance measurement system. However, DOD officials noted the department will face challenges implementing the guidance. These challenges include creating performance measures that assess program outcomes and ensuring adequate resources, such as expertise in performance management, are available to develop measures. DOD rarely uses the information in its performance measurement system to manage its counternarcotics activities and has applied few practices to facilitate its use. GAO has found that the full benefit of collecting performance information is realized only when managers use it to inform key decisions. However, DOD officials responsible for counternarcotics activities throughout the department told us they rarely use data submitted to the system to manage activities. Rather, they tend to manage programs using data not submitted to the system, such as information obtained in weekly program meetings regarding the cost and timeliness of projects. Moreover, officials responsible for oversight of DOD's activities stated they use the system to develop reports for ONDCP, but not to allocate resources. While DOD has applied some practices to facilitate the use of the performance information in its system, it does not utilize certain key practices identified by GAO, such as frequently and effectively communicating performance information. Absent an effective performance management system, DOD lacks critical information to use to improve the management and oversight of its counternarcotics activities. GAO recommends that the Secretary of Defense take steps to improve DOD's counternarcotics performance measurement system by (1) revising its performance measures and (2) applying practices to better facilitate the use of performance data to manage its counternarcotics activities. DOD concurred with GAO's recommendations.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Jess T. Ford Team: Government Accountability Office: International Affairs and Trade Phone: (202) 512-4268GAO-10-835, Drug Control: DOD Needs to Improve Its Performance Measurement System to Better Manage and Oversee Its Counternarcotics Activities
This is the accessible text file for GAO report number GAO-10-835
entitled 'Drug Control: DOD Needs to Improve Its Performance
Measurement System to Better Manage and Oversee Its Counternarcotics
Activities' which was released on July 21, 2010.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Committees:
United States Government Accountability Office:
GAO:
July 2010:
Drug Control:
DOD Needs to Improve Its Performance Measurement System to Better
Manage and Oversee Its Counternarcotics Activities:
GAO-10-835:
GAO Highlights:
Highlights of GAO-10-835, a report to congressional committees.
Why GAO Did This Study:
The Department of Defense (DOD) leads detection and monitoring of
aerial and maritime transit of illegal drugs into the United States in
support of law enforcement agencies. DOD reported resources of more
than $1.5 billion for fiscal year 2010 to support its counternarcotics
activities.
Congress mandated GAO report on DOD‘s counternarcotics performance
measurement system. Specifically, this report addresses the extent to
which (1) DOD‘s counternarcotics performance measurement system
enables DOD to track progress and (2) DOD uses performance information
from its counternarcotics performance measurement system to manage its
activities. GAO analyzed relevant DOD performance and budget
documents, and discussed these efforts with officials from DOD and the
Office of National Drug Control Policy (ONDCP).
What GAO Found:
DOD does not have an effective performance measurement system to track
the progress of its counternarcotics activities; however, it continues
efforts to improve the system. GAO has previously reported that
measuring performance provides managers a basis for making fact-based
decisions. DOD has established performance measures for its
counternarcotics activities and a database to collect performance
information, including measures, targets, and results. However, these
measures lack a number of attributes, such as being clearly stated and
objective, which GAO considers key to successful performance measures.
In May 2010, DOD issued new guidance for its counternarcotics
performance measurement system. However, DOD officials noted the
department will face challenges implementing the guidance. These
challenges include creating performance measures that assess program
outcomes and ensuring adequate resources, such as expertise in
performance management, are available to develop measures.
DOD rarely uses the information in its performance measurement system
to manage its counternarcotics activities and has applied few
practices to facilitate its use. GAO has found that the full benefit
of collecting performance information is realized only when managers
use it to inform key decisions. However, DOD officials responsible for
counternarcotics activities throughout the department told us they
rarely use data submitted to the system to manage activities. Rather,
they tend to manage programs using data not submitted to the system,
such as information obtained in weekly program meetings regarding the
cost and timeliness of projects. Moreover, officials responsible for
oversight of DOD‘s activities stated they use the system to develop
reports for ONDCP, but not to allocate resources. While DOD has
applied some practices to facilitate the use of the performance
information in its system, it does not utilize certain key practices
identified by GAO, such as frequently and effectively communicating
performance information. Absent an effective performance management
system, DOD lacks critical information to use to improve the
management and oversight of its counternarcotics activities.
Figure: DOD‘s Performance Measurement System as Compared to GAO-
Identified Steps:
[Refer to PDF for image: illustration]
GAO-identified steps in an effective performance measurement system:
Measure performance:
* establish performance measures;
* collect data.
Use information obtained from performance measures to:
* inform key decisions;
* improve programs and results.
DOD‘s counternarcotics performance measurement system:
DOD has developed performance measures and a database to collect data:
* measures lack key attributes of successful performance measures.
DOD infrequently uses information from its performance measurement
system to:
* identify problems or manage activities;
* develop strategy or allocate resources;
* identify and share effective approaches.
Source: GAO; Corel Draw (logos).
[End of figure]
What GAO Recommends:
GAO recommends that the Secretary of Defense take steps to improve
DOD‘s counternarcotics performance measurement system by (1) revising
its performance measures and (2) applying practices to better
facilitate the use of performance data to manage its counternarcotics
activities. DOD concurred with GAO‘s recommendations.
View [hyperlink, http://www.gao.gov/products/GAO-10-835] or key
components. For more information, contact Jess Ford at (202) 512-4268
or fordj@gao.gov.
[End of section]
Contents:
Letter:
Background:
DOD Has Not Developed a System to Effectively Track the Progress of
Its Counternarcotics Activities, but Continues to Work to Improve Its
Efforts:
DOD Rarely Uses the Performance Information Contained in Its
Performance Measurement System to Manage Its Counternarcotics
Activities and Has Applied Few Practices to Facilitate Its Use:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Comments from the Department of Defense:
Appendix III: Contact and Staff Acknowledgments:
Tables:
Table 1: DOD Resources in Support of Its Counternarcotics Activities,
Fiscal Years 2005-2010:
Table 2: DOD Goals, Objectives, and Example Performance Measures
Related to Its Counternarcotics Mission to Support U.S. Agencies and
Foreign Partners:
Table 3: GAO's Key Attributes of Successful Performance Measures:
Table 4: Examples of Data Sources Other than DOD's Counternarcotics
Performance Measurement System Used by DOD Components to Manage
Counternarcotics Activities:
Table 5: Status of DOD Efforts to Apply Practices to Facilitate Use of
Performance Information in Its Counternarcotics Performance
Measurement System and Reported Challenges, as of June 2010:
Figure:
Figure 1: Percentages of DOD's Fiscal Year 2009 Counternarcotics
Performance Measures Exhibiting Six Key Attributes:
Abbreviations:
AFRICOM: U.S. Africa Command:
CENTCOM: U.S. Central Command:
DASD-CN>: Office of the Deputy Assistant Secretary of Defense-
Counternarcotics and Global Threats:
DOD: Department of Defense:
DOD-IG: Department of Defense Inspector General:
EUCOM: U.S. European Command:
JIATF-S: Joint Interagency Task Force-South:
JIATF-W: Joint Interagency Task Force-West:
NORTHCOM: U.S. Northern Command:
ONDCP: Office of National Drug Control Policy:
SOUTHCOM: U.S. Southern Command:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
July 21, 2010:
Congressional Committees:
The global drug trade threatens U.S. national security by weakening
the rule of law in affected countries, financing the activities of
global and regional terrorists, and contributing to dangers such as
weapons trafficking. The Department of Defense (DOD) leads detection
and monitoring of aerial and maritime transit of illegal drugs into
the United States in support of law enforcement agencies.
Additionally, DOD's counternarcotics activities include sharing
information with U.S. and foreign agencies, as well as helping foreign
countries build their counternarcotics capacity. DOD reported
resources of more than $1.5 billion for fiscal year 2010 in support of
these activities.
The National Defense Authorization Act for Fiscal Year 2010 mandated
that GAO report on the performance measurement system used by DOD to
assess its counternarcotics activities.[Footnote 1] We have previously
reported that performance measurement systems used by results-oriented
agencies include steps to measure performance to gauge progress and
use the information obtained to make key management decisions.
[Footnote 2] In April 2010 we briefed congressional staff from the
defense committees on our preliminary observations regarding DOD's
counternarcotics performance measurement system.[Footnote 3] This
report contains the final results of our evaluation. Specifically, we
address the extent to which (1) DOD's counternarcotics performance
measurement system enables DOD to track progress and (2) DOD uses
performance information from its counternarcotics performance
measurement system to manage its activities.
To address these objectives, we analyzed DOD strategy, budget, and
performance documents, as well as DOD and Office of National Drug
Control Policy (ONDCP) guidance on performance measures. Further, we
discussed DOD's counternarcotics performance measurement system and
its use of performance information with officials from ONDCP and DOD
components including the Office of the Deputy Assistant Secretary of
Defense for Counternarcotics and Global Threats (DASD-CN>), U.S.
Africa Command (AFRICOM), U.S. Central Command (CENTCOM), U.S.
European Command (EUCOM), U.S. Northern Command (NORTHCOM), U.S.
Southern Command (SOUTHCOM), the Joint Interagency Task Force-South
(JIATF-S), the Joint Interagency Task Force-West (JIATF-W), and the
DOD Inspector General (DOD-IG). We evaluated a generalizable random
sample of DOD's fiscal year 2009 counternarcotics performance measures
(115 of 239 measures) to assess the extent to which these measures
adhered to GAO criteria on the key attributes of successful
performance measures. We also analyzed the extent to which DOD applies
key management practices identified by GAO to facilitate the use of
performance information from its counternarcotics performance
measurement system. Moreover, we visited CENTCOM, SOUTHCOM, and JIATF-
S to examine DOD's use of performance data to support its
counternarcotics mission. (See appendix I for a complete description
of our scope and methodology.)
We conducted this performance audit from December 2009 to July 2010 in
accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives.
Background:
DOD Counternarcotics Strategy and Activities:
According to DOD's Counternarcotics Strategy developed in fiscal year
2009, the department seeks to disrupt the market for illegal drugs by
helping local, state, federal, and foreign government agencies address
the drug trade and narcotics-related terrorism.[Footnote 4] DOD
achieves this mission through three goals--detecting and monitoring
drug trafficking, sharing information on illegal drugs with U.S. and
foreign government agencies, and building the counternarcotics
capacity of U.S. and foreign partners.
DASD-CN>, with oversight from the Under Secretary of Defense for
Policy, exercises management and oversight of DOD's counternarcotics
activities and performance measurement system. DASD-CN>'s
responsibilities include ensuring DOD develops and implements a
counternarcotics program with clear priorities and measured results.
Programs, Resources, and Assessments, a division within DASD-CN>, is
the lead office for the development of counternarcotics resources and
plans. Among other activities, this office directs and manages the
planning, programming, and budgeting system of the DOD
counternarcotics program and is responsible for updating and
disseminating guidance on DOD's counternarcotics performance
measurement system.
DOD's counternarcotics activities are implemented through DOD's
combatant commands, military departments, and defense agencies.
[Footnote 5] According to DOD, these organizations provide assets,
such as aircraft and patrol ships, military personnel, and other
assistance, to support U.S. law enforcement agencies and foreign
security forces in countering narcotics trafficking.
In support of DOD's counternarcotics activities, DOD reported
resources totaling approximately $7.7 billion from fiscal year 2005 to
fiscal year 2010, including more than $6.1 billion appropriated to its
Counternarcotics Central Transfer Account and more than $1.5 billion
in supplemental appropriations (see table 1).
Table 1: DOD Resources in Support of Its Counternarcotics Activities,
Fiscal Years 2005-2010 (Dollars in millions):
Counternarcotics Central Transfer Account[A]:
FY 2005: $905.8;
FY 2006: $936.1;
FY 2007: $1,075.2;
FY 2008: $984.8;
FY 2009: $1,096.7;
FY 2010: $1,158.2;
Total: $6,156.9.
Supplemental appropriations[A,B]:
FY 2005: $242.0;
FY 2006: $86.9;
FY 2007: $202.7;
FY 2008: $328.0;
FY 2009: $300.4;
FY 2010: $369.9;
Total: $1,529.9.
Total:
FY 2005: $1,147.8;
FY 2006: $1,022.9;
FY 2007: $1,277.8;
FY 2008: $1,312.8;
FY 2009: $1,397.2;
FY 2010: $1,528.2;
Total: $7,686.7.
Source: GAO analysis of DOD data.
Note: Totals may not add due to rounding.
[A] DOD funding resources in support of its counternarcotics
activities are annually reported as part of the National Drug Control
Strategy Budget Summary Documents. For fiscal years 2005-2010, these
documents list DOD resources for its Counternaroctics Central Transfer
Account and for supplemental appropriations.
[B] According to DOD, it rolls over unobligated supplemental funding
into the next fiscal year; therefore, the supplemental totals listed
here do not match the total supplemental funding appropriated for that
year.
[End of table]
Of these resources, DOD estimated that approximately $4.2 billion were
in support of its international counternarcotics activities from
fiscal years 2005-2010.
Previous GAO Reporting and Legislation Related to DOD's
Counternarcotics Performance Measures:
DOD efforts to develop performance measures for its counternarcotics
activities are long-standing. We reported in December 1999[Footnote 6]
that DOD had not developed a set of performance measures to assess the
impact of its counternarcotics operations, but had undertaken initial
steps to develop such measures. In January 2002[Footnote 7] and
November 2005,[Footnote 8] we found that DOD was in the process of
developing performance measures focused on its role of detecting and
monitoring the trafficking of illegal drugs into the United States. In
November 2005 we recommended that DOD, in conjunction with other
agencies performing counternarcotics activities, develop and
coordinate counternarcotics performance measures.
In December 2006 Congress directed ONDCP--the organization that
establishes U.S. counternarcotics goals and coordinates the federal
budget to combat drugs--to produce an annual report describing the
national drug control performance measurement system that identifies
the activities of national drug control program agencies, including
DOD. In May 2007 ONDCP issued guidance requiring DOD and other
national drug control program agencies to annually submit to the
Director of ONDCP a performance summary report including performance
measures, targets, and results.[Footnote 9] In addition, ONDCP
officials stated that they have recommended improvements to DOD's
performance measures, both in correspondence and in meetings with DOD
staff.
DOD Has Not Developed a System to Effectively Track the Progress of
Its Counternarcotics Activities, but Continues to Work to Improve Its
Efforts:
DOD does not have an effective system for tracking the progress of its
counternarcotics activities; however, it continues efforts to improve
the system. We have found that measuring performance provides managers
a basis for making fact-based decisions. DOD has established
performance measures for its counternarcotics activities and a
database to collect performance information. However, these measures
lack a number of attributes which we consider key to successful
performance measures and, therefore, do not provide a clear indication
of DOD's progress toward its counternarcotics goals. Recognizing the
need to update and improve its measures, in May 2010, DOD issued new
guidance for its counternarcotics performance measurement system.
However, DOD officials noted the department will faces challenges
implementing the guidance.
DOD Has Developed Performance Measures and a Database for Its
Counternarcotics Activities:
We have previously reported that effective performance measurement
systems include steps to measure performance, such as establishing
performance measures and collecting data.[Footnote 10] In response to
ONDCP's 2007 guidance, DOD developed performance measures for its
fiscal year 2007 counternarcotics activities and established a
centralized database within its performance measurement system to
collect data on those performance measures.[Footnote 11] The
counternarcotics performance measurement system database, maintained
by DASD-CN>, requires DOD components to submit performance
information at specified intervals during the fiscal year, such as
results for performance measures, the mechanisms used to collect
results data, and future performance targets. For fiscal year 2009,
DOD guidance required that all projects funded by its Counternarcotics
Central Transfer Account have a performance measure. As a result, DOD
reported it had 285 performance measures for its fiscal year 2009
counternarcotics activities. Of those, 239 were performance measures
related to DOD's mission of supporting U.S. agencies and foreign
partners in countering narcotics trafficking. (See table 2 for
examples of DOD's counternarcotics performance measures.)
Table 2: DOD Goals, Objectives, and Example Performance Measures
Related to Its Counternarcotics Mission to Support U.S. Agencies and
Foreign Partners:
Goal: Detect and monitor illegal drug traffic;
Related objective:
* Detect and monitor illegal drug trafficking using DOD and contractor
provided air and maritime assets;
Example performance measure[A]:
* On-station ship days (includes U.S. and allied);
Related objective:
* Detect and monitor illegal drug trafficking using DOD radar systems;
Example performance measure[A]:
* System mission capability rate (expressed as a percentage);
Related objective:
* Support detection and monitoring activities by providing operational
facilities;
Example performance measure[A]:
* OPBAT[B] functionality (providing the system required to fuel
resident helicopters) of fueling system capability;
Related objective:
* Provide integrated command and control, voice and data
communications, and connectivity in support of interdiction operations;
Example performance measure[A]:
* Number of sensors integrated and providing reliable and dependable
radar to JIATF-S and/or host nations.
Goal: Share information on illegal drugs and technology support with
U.S. and foreign government agencies;
Related objective:
* Manage intelligence collection against counternarcotics targets;
Example performance measure[A]:
* Percentage of tasked CN[C] missions flown;
Related objective:
* Provide analysis and reporting in response to strategic and tactical
requirements;
Example performance measure[A]:
* Number of formal intelligence products provided to or on behalf of
law enforcement agencies or other U.S. agencies;
Related objective:
* Share information with U.S. and partner nation security and law
enforcement;
Example performance measure[A]:
Related objective:
* Provide collection and analysis training to DOD, U.S., and partner
nation law enforcement personnel;
Example performance measure[A]:
* Number of partner nation law enforcement agencies engaged;
Related objective:
* Develop and deploy technology that disrupts the flow of illegal
drugs;
Example performance measure[A]:
* Number of attendees to Basic Counterdrug Intelligence Course.
Goal: Build the counternarcotics capacity of U.S. and foreign partners;
Related objective:
* Provide training and support to U.S. law enforcement personnel
conducting counternarcotics related activities;
Example performance measure[A]:
* Number of trained military working dog teams trained;
Related objective:
* Provide training and equipment to partner nation forces;
Example performance measure[A]:
* Percent of inland waterways controlled by Colombian Marine Corps
forces.
Related objective:
* Provide infrastructure projects in support of partner nation forces;
Example performance measure[A]:
* Number of infrastructure projects in support of training
requirements.
Related objective:
* Provide support to partner nation forces;
Example performance measure[A]:
* Percentage of positive to negative media references from non-U.S.
media sources for a calendar year.
Source: DOD's counternarcotics performance measurement system's
database.
[A] The example performance measures are reprinted as they appear in
DOD's counternarcotics performance measurement system's database.
[B] OPBAT is an acronym used by DOD meaning "Operation Bahamas Turks
and Caicos."
[C] CN is an acronym used by DOD meaning "counternarcotics."
[End of table]
DOD's Fiscal Year 2009 Counternarcotics Performance Measures Exhibit
Some, but Not All, Key Attributes of Successful Performance Measures:
DOD's current set of counternarcotics performance measures varies in
the degree to which it exhibits key attributes of successful
performance measures. Prior GAO work has identified nine attributes of
successful performance measures.[Footnote 12] Table 3 shows the nine
attributes, their definitions, and the potentially adverse
consequences of not having the attributes.
Table 3: GAO's Key Attributes of Successful Performance Measures:
Key attributes evaluated by reviewing performance measures as a set:
Attribute: Core program activities;
Definition: Key attributes evaluated by reviewing performance measures
as a set: Measures cover the activities that an entity is expected to
perform to support the intent of the program;
Potentially adverse consequences of not meeting attribute: Key
attributes evaluated by reviewing performance measures as a set: Not
enough information available in core program areas to managers and
stakeholders.
Attribute: Balance;
Definition: Key attributes evaluated by reviewing performance measures
as a set: Balance exists when a suite of measures ensures that an
organization's various priorities are covered;
Potentially adverse consequences of not meeting attribute: Key
attributes evaluated by reviewing performance measures as a set: Lack
of balance could create skewed incentives when measures overemphasize
some goals.
Attribute: Limited overlap;
Definition: Key attributes evaluated by reviewing performance measures
as a set: Measure should provide new information beyond that provided
by other measures;
Potentially adverse consequences of not meeting attribute: Key
attributes evaluated by reviewing performance measures as a set:
Managers may have to sort through redundant, costly information that
does not add value.
Key attributes evaluated by reviewing performance measures
individually:
Attribute: Linkage;
Definition: Key attributes evaluated by reviewing performance measures
as a set: Measure is aligned with division and agencywide goals and
mission and clearly communicated throughout the organization;
Potentially adverse consequences of not meeting attribute: Key
attributes evaluated by reviewing performance measures as a set:
Behaviors and incentives created by measures do not support achieving
division or agencywide goals or mission.
Attribute: Governmentwide priorities;
Definition: Key attributes evaluated by reviewing performance measures
as a set: Each measure should cover a priority, such as quality,
timeliness, and cost of service;
Potentially adverse consequences of not meeting attribute: Key
attributes evaluated by reviewing performance measures as a set: A
program's overall success is at risk if all priorities are not
addressed.
Attribute: Reliability;
Definition: Key attributes evaluated by reviewing performance measures
as a set: Measure produces the same result under similar conditions;
Potentially adverse consequences of not meeting attribute: Key
attributes evaluated by reviewing performance measures as a set:
Reported performance data is inconsistent and adds uncertainty.
Attribute: Objectivity;
Definition: Key attributes evaluated by reviewing performance measures
as a set: Measure is reasonably free from significant bias or
manipulation;
Potentially adverse consequences of not meeting attribute: Key
attributes evaluated by reviewing performance measures as a set:
Performance assessments may be systematically over-or understated.
Attribute: Clarity;
Definition: Key attributes evaluated by reviewing performance measures
as a set: Measure is clearly stated, and the name and definition are
consistent with the methodology used to calculate it;
Potentially adverse consequences of not meeting attribute: Key
attributes evaluated by reviewing performance measures as a set: Data
could be confusing and misleading to users.
Attribute: Measurable target;
Definition: Key attributes evaluated by reviewing performance measures
as a set: Measure has a numerical goal;
Potentially adverse consequences of not meeting attribute: Key
attributes evaluated by reviewing performance measures as a set:
Cannot tell whether performance is meeting expectations.
Source: GAO.
[End of table]
Our analysis found that DOD's counternarcotics performance measures
lack several of the key attributes of successful performance measures.
Based on our analysis of a generalizable sample of DOD's fiscal year
2009 performance measures,[Footnote 13] we found the attributes of
core program activities and linkage were generally present, but other
attributes such as balance and limited overlap were missing, and
attributes including governmentwide priorities, reliability,
objectivity, clarity, and measurable targets were present in varying
degrees.
We found that the attribute of core program activities was identified
in the set of measures, while balance and limited overlap did not
appear to be present.
* Core program activities. We estimate that all of DOD's
counternarcotics performance measures cover the department's core
program activities. We have previously reported that core program
activities are the activities that an entity is expected to perform to
support the intent of the program, and that performance measures
should be scoped to evaluate those activities. For the measures we
reviewed, DOD divides its core counternarcotics activities across its
3 goals and 13 objectives (see table 2). In our analysis, we found at
least one performance measure covering each of DOD's counternarcotics
objectives. Therefore, we determined that DOD's core program
activities were covered.
* Balance. DOD's set of performance measures lack balance. We have
previously reported that balance exists when a set of measures ensures
that an organization's various priorities are covered. According to
DOD, performance measures best cover its priorities when five
measurable aspects of performance, as defined by DOD--input, process,
output, outcome, and impact--are present in its performance measures.
As an example, "number of attendees to basic counterdrug intelligence
course" is, in our determination, a measure of output, as it measures
the services provided by DOD. We estimate 93 percent of DOD's fiscal
year 2009 performance measures are input, process, or output measures,
while 6 percent are outcome measures and 0 percent are impact
measures.[Footnote 14] Therefore, given that DOD's set of measures is
highly skewed towards input, process, and output measures and contains
no impact measures, we determined that the set is not balanced by
DOD's criteria. Performance measurement efforts that lack balance
overemphasize certain aspects of performance at the expense of others,
and may keep DOD from understanding the effectiveness of its overall
mission and goals.
* Limited overlap. We determined there to be overlap among DOD's
performance measures. We found instances where the measures and their
results appeared to overlap with other measures and results. When we
spoke with DASD-CN> officials concerning this, they stated that the
set of measures could be conveyed using fewer, more accurate measures.
We have reported that each performance measure in a set should provide
additional information beyond that provided by other measures. When an
agency has overlapping measures, it can create unnecessary or
duplicate information, which does not benefit program management.
Of the remaining six attributes of successful performance measures,
only one attribute--linkage--was present in almost all of the
measures, while the other five attributes--governmentwide priorities,
reliability, objectivity, clarity, and measurable targets--appeared in
varying degrees (see figure 1).
Figure 1: Percentages of DOD's Fiscal Year 2009 Counternarcotics
Performance Measures Exhibiting Six Key Attributes:
[Refer to PDF for image: horizontal bar graph]
Key attributes of performance measure: Governmentwide priorities;
Estimated percentage of DOD performance measures exhibiting key
attribute: 41%.
Key attributes of performance measure: Reliability;
Estimated percentage of DOD performance measures exhibiting key
attribute: 46%.
Key attributes of performance measure: Objectivity;
Estimated percentage of DOD performance measures exhibiting key
attribute: 59%.
Key attributes of performance measure: Clarity;
Estimated percentage of DOD performance measures exhibiting key
attribute: 65%.
Key attributes of performance measure: Measureable target;
Estimated percentage of DOD performance measures exhibiting key
attribute: 66%.
Key attributes of performance measure: Linkage;
Estimated percentage of DOD performance measures exhibiting key
attribute: 99%.
Source: GAO analysis of DOD performance measures.
[End of figure]
DOD's counternarcotics performance measures demonstrate linkage. We
estimate that 99 percent of DOD's measures are linked to agencywide
goals and mission. DOD's counternarcotics performance measurement
system database requires that for each performance measure entered
into the database, a goal and related objective of DOD's
counternarcotics mission be identified. Our analysis found that in all
but one instance, linkage between DOD's goals and performance measures
is easily identified.
However, DOD's counternarcotics performance measures did not fully
satisfy five attributes.
* Governmentwide priorities. We estimate that 41 percent of the
measures we analyzed cover a broader governmentwide priority, such as
quality, timeliness, efficiency, cost of service, or outcome. We
determined, for example, that the governmentwide priority of "quality"
was reflected in the measure "number of sensors integrated and
providing reliable and dependable radar data to JIATF-S and/or host
nations," because it measures the reliability and dependability of
detection services. In the majority of the instances, however,
measures did not address a governmentwide priority. For example, the
measure "number of trained military working dog teams trained" was
determined not to cover a governmentwide priority because it does not
measure the quality or efficiency of training provided. When measures
fail to cover governmentwide priorities managers may not be able to
balance priorities to ensure the overall success of the program.
* Reliability. We estimate that 46 percent of DOD's performance
measures have data collection methods indicated in the database that
generally appear reliable. Reliability refers to whether a measure is
designed to collect data or calculate results such that the measure
would be likely to produce the same results if applied repeatedly to
the same situation. For each entry in the database, users are directed
to enter, among other information, one performance measure and its
associated methodology, target, and result. However, in numerous
instances the system contained multiple performance measures entered
into fields that should contain only one measure. Such entries could
result in errors of collecting, maintaining, processing, or reporting
the data. Additionally, some measures did not provide enough
information on data collection methods or performance targets to
assure reliability. For example, a measure in the database states
"continuous U.S. Navy ship presence in the SOUTHCOM area of
responsibility." The performance target listed for this measure is
"3.5," but to what 3.5 refers--such as days, number of ships, or
percentage points--is not explained. Moreover, the methodology in the
database for this measure is entered as "not applicable." Therefore,
the measure's methodology does not provide insight into how DOD could
measure whether or not it reached its target of 3.5. As a result, we
determined that this measure did not have data collection methods to
gather reliable results. We have previously reported that if errors
occur in the collection of data or the calculation of their results,
it may affect conclusions about the extent to which performance goals
have been achieved.
* Objectivity. We estimate that 59 percent of DOD's performance
measures for its counternarcotics activities are objective. We have
previously reported that to be objective, measures should indicate
specifically what is to be observed, in which population or
conditions, and in what time frame, and be free of opinion and
judgment. We estimate that 41 percent of DOD's measures are not
objective and could therefore face issues of bias or manipulation. For
example, a measure in the database is, "percent of inland waterways
controlled by Colombian Marine Corps forces." For this measure, no
criteria for "controlled" is provided and it is not clear how the
Colombian government reports the percentage of waterways under its
control and over what time frame this control will occur.
* Clarity. We estimate that 65 percent of DOD's performance measures
exhibit the attribute of clarity. A measure achieves clarity when it
is clearly stated and the name and definition are consistent with the
methodology used for calculating the measure. However, we estimate
that 35 percent of DOD's measures are not clearly stated. For example,
one of DOD's measures linked to the objective of sharing information
with U.S. and partner nations is "identify and establish methodology
for implementation." For this measure, no associated methodology is
identified, and it is unclear what is being implemented. We have
previously reported that a measure that is not clearly stated can
confuse users and cause managers or other stakeholders to think that
performance was better or worse than it actually was.
* Measurable target. We estimate that 66 percent of DOD's measures
have measurable targets. Where appropriate, performance goals and
measures should have quantifiable, numerical targets or other
measurable values. Some of DOD's measures, however, lacked such
targets. For example, one performance measure identified its target as
"targets developed by the local commander." As it is not quantifiable,
this target does not allow officials to easily assess whether goals
were achieved because comparisons cannot be made between projected
performance and actual results.
DOD Is Working To Improve Its Counternarcotics Performance Measures,
but Implementation Challenges Exist:
DOD officials have acknowledged that weaknesses exist in the
department's current set of counternarcotics performance measures. In
May 2010 DOD issued revised guidance for its counternarcotics
performance measurement system to guide users in establishing
performance measures that more accurately capture the quantitative and
qualitative achievements of DOD's activities. To do this, the guidance
states that performance measures should be, among other attributes,
useful for management and clearly stated. The guidance describes
different types of performance measures that can be used to monitor
DOD's contribution to its strategic counternarcotics goals, such as
those that measure DOD's efficiency, capability, and effectiveness at
performing its activities. Additionally, according to the guidance,
DOD components should provide evidence of the quality and reliability
of the data used to measure performance.
However, DOD officials noted four specific challenges that the
department faces in developing performance measures consistent with
its revised guidance.
* Creating performance measures that assess program outcomes. Some DOD
officials noted that, because DOD acts as a support agency to partner
nations and other law enforcement entities--and the actual
interdiction of drugs is conducted by other entities--measuring the
outcome of DOD's performance is difficult. While developing outcome
measures can be challenging, we have found that an agency's
performance measures should reflect a range of priorities, including
outcomes. Moreover, we have found that methods to measure program
outcomes do exist. For example, agencies have applied a range of
strategies to develop outcome measures for their program, such as
developing measures of satisfaction based upon surveys of
customers.[Footnote 15] In addition, officials from EUCOM, AFRICOM,
and JIATF-S stated that while developing outcome performance measures
can be difficult, developing such measures for support activities is
possible and is done at other federal agencies. For example, EUCOM
indicated it could track the outcome of the support it provides to
partner nations by tracking the annual percentage increase in
interdictions and arrests related to illicit trafficking.
Additionally, JIATF-W[Footnote 16] indicated that it conducts
quarterly command assessments of current programs, which focus on
aligning resources provided by JIATF-W to the outcomes of its law
enforcement partners.
* Implementing revisions in a timely manner. DOD officials noted that
implementing revisions to the department's performance measures in a
timely fashion will be difficult given that such revisions are
resource and time intensive. Further, while including dates for
submission, DOD's revised guidance does not clearly specify a time
frame by which DOD components should revise the counternarcotics
performance measures that are to be submitted to the database. We have
previously reported that establishing timetables for the development
of performance measures can create a sense of urgency that assists in
the effort being taken more seriously. DASD-CN> officials noted that
time frames by which DOD's measures would be revised are being
discussed. However, these officials do not expect new performance
measures to be established in fiscal year 2010, and said that fiscal
year 2011 would be the earliest year of full implementation of the
guidance.
* Ensuring adequate resources are available. DOD officials noted that
ensuring adequate resources--such as expertise and training in
performance management--are available to develop performance measures
at both DASD-CN> and the combatant commands will be a challenge.
These officials noted that DOD employees tasked with developing
performance measures and tracking the progress towards achieving goals
are not sufficiently trained to design and monitor outcome performance
measures. We have previously reported that access to trained staff
assists agencies in their development of performance measures.
[Footnote 17]
* Ensuring reliable data. DOD officials noted that ensuring data used
to measure DOD performance are reliable is challenging. To measure the
performance of its counternarcotics activities DOD officials told us
they rely heavily on external sources of data, such as U.S. law
enforcement agencies and foreign government officials. This challenge
can pose issues for DOD regarding data verification and ensuring
proper information is recorded for performance measures.
DOD Rarely Uses the Performance Information Contained in Its
Performance Measurement System to Manage Its Counternarcotics
Activities and Has Applied Few Practices to Facilitate Its Use:
DOD makes limited use of its performance measurement system to manage
its counternarcotics activities and has applied few practices to
facilitate its use. We have found that the full benefit of collecting
performance information is realized only when managers use the
information to inform key decisions.[Footnote 18] While DOD has
applied some practices to facilitate the use of the performance
information in its system, it does not utilize certain key practices,
such as frequently and effectively communicating performance
information. Absent an effective performance management system, DOD
lacks critical information to use to improve the management and
oversight of its counternarcotics activities.
Agencies Can Use Performance Information to Manage for Results:
We have previously reported that, in addition to measuring
performance, effective performance measurement systems include steps
to use information obtained from performance measures to make
decisions that improve programs and results.[Footnote 19] We
identified several ways in which agencies can use performance
information to manage for results, including using data to (1)
identify problems and take corrective actions, (2) develop strategy
and allocate resources, and (3) identify and share effective
approaches.
DOD Submits Performance Reports to ONDCP, But Makes Limited Use of the
Information in Its Performance Measurement System to Manage and
Oversee Its Counternarcotics Activities:
DOD officials representing DASD-CN>, AFRICOM, CENTCOM, EUCOM,
NORTHCOM, SOUTCOM, JIATF-S, and JIATF-W told us they rarely use
information from DOD's counternarcotics performance measurement system
to manage counternarcotics activities. Specifically, they rarely use
the system to:
* Identify problems and take corrective actions. Agencies can use
performance information to identify problems or weaknesses in
programs, to try to identify factors causing the problems, and to
modify a service or process to try to address problems. DOD officials
representing DASD-CN> and SOUTHCOM told us that they currently make
limited use of the performance information in DOD's performance
measurement system to manage counternarcotics activities. Officials
from DASD-CN> stated that they use data from the performance
measurement system to produce reports for ONDCP, which may include
information identifying problems in the implementation of DOD's
counternarcotics activities. However, in reviewing these documents, we
found that the reports did not include a clear assessment of DOD's
overall progress toward its counternarcotics goals. For instance, the
report submitted to ONDCP for fiscal year 2009 contained detailed
information on 6 of DOD's 285 counternarcotics performance measures,
but did not clearly explain why the results of these 6 measures would
be critical to the success of DOD's counternarcotics program.[Footnote
20] Moreover, according to ONDCP, DOD's reports for fiscal years 2007,
2008, and 2009 did not fulfill the requirements of ONDCP's guidance
because the reports were not authenticated by the DOD-IG.[Footnote 21]
Further, officials from AFRICOM, CENTCOM, EUCOM, NORTHCOM, JIATF-S,
and JIATF-W told us they do not use the DOD's performance measurement
system to manage counternarcotics activities. While these officials
indicated that they submitted performance information to the system's
database as required by DOD guidance, they stated they tend to manage
programs using information not submitted to the system (see table 4).
For example, CENTCOM officials told us information obtained in weekly
program meetings regarding the timeliness and cost of counternarcotics
projects, not data sent to the system's database, is most often used
to help them identify problems and make program adjustments.
Table 4: Examples of Data Sources Other than DOD's Counternarcotics
Performance Measurement System Used by DOD Components to Manage
Counternarcotics Activities:
DOD component: AFRICOM;
Examples of other data sources used: Information obtained from site
visits and U.S and foreign partners. For instance, an AFRICOM official
told us the command obtained information on the inoperability of
detection equipment installed in Ghana through site visits.
DOD component: CENTCOM;
Examples of other data sources used: Information obtained from
contractors, site visits, and U.S. law enforcement and foreign
partners. For example, CENTCOM officials told us they obtain
information during weekly program meetings with contractors and
program managers.
DOD component: EUCOM;
Examples of other data sources used: Information obtained from site
visits and U.S and foreign partners. For instance, EUCOM officials
told us they engage with U.S. law enforcement liaisons to obtain
information on counternarcotics activities, such as seizures, arrests,
and closed investigations.
DOD component: NORTHCOM;
Examples of other data sources used: Information obtained from site
visits, U.S. law enforcement and foreign partners. For example,
NORTHCOM officials told us they obtain information on detection and
monitoring of drug traffic from the Mexican Navy.
DOD component: SOUTHCOM;
Examples of other data sources used: Information obtained from
contractors, site visits, and U.S. law enforcement and foreign
partners. For example, SOUTHCOM officials told us they obtain
information from their foreign partners, such as Colombia and Peru.
DOD component: JIATF-S;
Examples of other data sources used: Information from detection and
monitoring activities. For example, JIATF-S manages activities using
information stored in databases tracking the effectiveness of
detection and monitoring activities.
DOD component: JIATF-W;
Examples of other data sources used: Information obtained from U.S.
law enforcement and foreign partners. For example, JIATF-W officials
told us they obtain information during quarterly command reviews in
which law enforcement outcomes of JIATF-W activities to build partner
capacity and share information are discussed.
Source: GAO analysis of DOD information.
[End of table]
Recognizing the need improve the information in the system's database,
officials from DASD-CN> told us that for fiscal year 2011 they are
working with DOD components to integrate performance information into
the system's database that can be more useful for decision making.
Officials from several combatant commands stated they could integrate
performance information obtained from outside sources into the
counternarcotics performance measurement system. Officials from JIATF-
S, for example, told us they collect and analyze a variety of data on
counternarcotics activities that they do not input into DOD's
counternarcotics performance measurement system. On a daily basis,
JIATF-S collects information on "cases"--that is, boats or planes
suspected of illegal trafficking. In addition to tracking the number
of cases, JIATF-S compiles information as to whether or not a
particular case was targeted, detected, or monitored, and whether or
not those actions resulted in interdictions or seizures of illegal
drugs. By compiling this information, officials at JIATF-S told us
they can better identify program outcomes, areas in which their
efforts are successful, and ways to take corrective actions.
* Develop strategy and allocate resources. Agencies can use
performance information to make decisions that affect future
strategies, planning, and budgeting, and allocating resources. DASD-
CN>'s role includes both defining the strategic goals and managing
the budgeting system of the DOD counternarcotics program. DOD's
counternarcotics guidance states that information from the
counternarcotics performance measurement system will inform strategic
counternarcotics plans, but it does not clearly state how the system
will be used to inform decisions to allocate resources. Moreover,
officials from DASD-CN> told us that the office does not currently
link performance information from the counternarcotics performance
measurement system's database directly to budget allocation decisions.
In addition, our analysis of DOD's fiscal year 2011 Drug Interdiction
and Counterdrug Activities Budget Estimates--which provides details on
DOD's fiscal year 2011 budget request for its counternarcotics
activities--identified no clear link between budget allocation
decisions and performance information in the system's database. DOD
officials told us they plan to incorporate performance information
from the counternarcotics performance measurement system into future
budget requests provided to Congress.
* Identify and share effective approaches. We have reported that high-
performing organizations can use performance information to identify
and increase the use of program approaches that are working well.
According to DOD's counternarcotics performance measurement system
guidance, DASD-CN> will use performance information submitted to the
system's database to compile reports for ONDCP, which DASD-CN> has
done. However, DASD-CN> officials told us they do not currently use
the system to produce reports for DOD components, which could assist
in identifying and sharing effective approaches between DOD's
components. While indicating performance reports could be a useful
tool, officials from several DOD components told us they had not
received such reports from DASD-CN>. DOD's May 2010 guidance does
not state whether the system will be used to produce such reports in
the future.
DOD Has Applied Few Practices to Facilitate the Use of Its
Counternarcotics Performance Measurement System:
We have found that agencies can adopt practices that can facilitate
the use of performance data[Footnote 22]. These include (1)
demonstrating management commitment to results-oriented management;
(2) aligning agencywide goals, objectives, and measures; (3) improving
the usefulness of performance data to better meet management's needs;
(4) developing agency capacity to effectively use performance
information; and (5) communicating performance information within the
agency frequently and effectively.
As part of its role overseeing DOD's counternarcotics activities, DASD-
CN> manages the DOD counternarcotics performance measurement system.
DASD-CN> applies some practices to facilitate the use of its
counternarcotics performance measurement system. For example, DASD-
CN> has recently taken steps to demonstrate management commitment by
issuing revised guidance emphasizing the development of improved
performance measures and, according to DASD-CN> officials,
conducting working groups with some DOD components[Footnote 23] to
assist them in revising performance measures. Moreover, DASD-CN>
officials told us they are taking steps to increase staffing to better
oversee the performance measurement system. We have found that the
commitment of agency managers to result-oriented management is
critical to increased use of performance information for policy and
program decisions. Further, DASD-CN> has created a results framework
that aligns agencywide goals, objectives, and performance measures for
its counternarcotics activities. As we have previously reported, such
an alignment increases the usefulness of the performance information
collected by decision makers at each level, and reinforces the
connection between strategic goals and the day-to-day activities of
managers and staff.
However, DASD-CN> has not applied certain key practices to
facilitate the use of data, such as improving the usefulness of
performance information in its performance measurement system,
developing agency capacity to use performance information, and
communicating performance information frequently and effectively.
Furthermore, DOD officials told us they face challenges using DOD's
performance measurement system to manage their activities due to (1)
the limited utility of the performance measures and data currently in
DOD's counternarcotics database, (2) insufficient capacity to collect
and use performance information, and (3) infrequent communication from
DASD-CN> regarding performance information submitted to the
database. For instance, DOD's guidance emphasizes the development of
performance measures that are, among other attributes, useful for
management and supported by credible data. However, DOD officials from
several combatant commands told us that the performance measures and
targets currently in the system are of limited utility[Footnote 24]
and will need to be revised. Moreover, officials from several DOD
components emphasized the need to build additional capacity to use
performance data, such as receiving training on how to revise
performance standards and measures. We have found that the practice of
building analytical capacity to use performance information--both in
terms of staff trained to do analysis and availability of research and
evaluation resources--is critical to an agency using performance
information in a meaningful way. Finally, DOD components told us that
they received little feedback or direction from DASD-CN> regarding
performance information they submitted to the system. We have
previously reported that improving the communication of performance
information among staff and stakeholders can facilitate the use of
performance information in key management activities. For more
information see table 5.
Table 5: Status of DOD Efforts to Apply Practices to Facilitate Use of
Performance Information in Its Counternarcotics Performance
Measurement System and Reported Challenges, as of June 2010:
Key practice: Demonstrating management commitment;
Examples of practice: Agency managers can demonstrate commitment to
results-oriented management through leading and involving staff from
different levels in regular performance review meetings to discuss
progress made toward achieving results;
DOD efforts to apply practice:
* DASD-CN> has held working groups with some DOD components to
discuss the development of performance measures;
* DOD revised guidance for its counternarcotics performance
measurement system as of May 2010 with information on the development
of performance measures;
* DASD-CN> told us they are increasing staff to oversee the
counternarcotics performance measurement system;
Reported challenges: Limited feedback and direction from DASD-CN>
regarding performance information submitted to the database.
Key practice: Aligning goals and measures;
Examples of practice: Agencies can encourage greater use of
performance information by aligning program performance measures with
goals and day-to-day activities;
DOD efforts to apply practice:
* Most DOD performance measures clearly link to agency goals and
objectives;
Reported challenges: None reported.
Key practice: Improving the usefulness of performance information;
Examples of practice: To ensure performance information meets users'
needs, agencies can implement practices such as using an assessment
tool to document the intended use of a measure, assess the information
and system in which data are kept, and identify any limitations in
data;
DOD efforts to apply practice:
* DOD's revised guidance for its counternarcotics performance
measurement system as of May 2010 emphasizes standards of data quality;
Reported challenges: Limited utility of the performance measures and
data currently in DOD's counternarcotics database.
Key practice: Developing agency capacity;
Examples of practice: Agencies can build analytical capacity to use
performance information by providing training to staff on setting
performance standards and measures, analyzing data, and using
information to revise standards and measures; as well as by providing
staff access to technical resources and evaluation support staff;
DOD efforts to apply practice:
* DASD-CN> has held working groups and training sessions with some
DOD components on the development of performance measures;
* Some DOD components have hired staff to assist in the development of
performance measures;
Reported challenges: Some DOD components suggest additional training
on topics, such as analyzing performance data, and using information
to revise measures is needed due to limited capacity to collect and
use performance information.
Key practice: Communicating performance information;
Examples of practice: To enhance communication among staff and
stakeholders, agencies can provide performance updates through regular
e-mail; distribute performance review meeting minutes; or use visual
tools such as poster displays, performance score cards, or agency
intranet sites to share performance information;
DOD efforts to apply practice:
* DASD-CN> has held working groups with some DOD components to
discuss the development of performance measures;
Reported challenges: Limited feedback and direction from DASD-CN>
regarding performance information submitted to the database.
Source: GAO analysis of DOD information.
[End of table]
Conclusions:
DOD reported more than $1.5 billion in fiscal year 2010 for its
counternarcotics activities, but has not yet developed an effective
performance measurement system to readily inform progress toward the
achievement of its counternarcotics goals. We have previously reported
that performance measurement systems include steps to measure
performance to gauge progress and use the information obtained to make
key management decisions. DOD acknowledges weaknesses in its
performance measurement system and has taken steps to improve the
system, such as revising its guidance for the development of
performance measures and holding working groups with DOD components.
However, its current set of measures lack key attributes of successful
performance measures, such as balance, objectivity, and reliability.
Moreover, DOD infrequently uses the information presently in its
counternarcotics performance measurement system and has yet to fully
apply key practices to facilitate its use. Absent an effective
performance measurement system, DOD lacks critical performance
information to use to improve its management decisions, eliminate
wasteful or unproductive efforts, and conduct oversight of its
activities.
Recommendations for Executive Action:
To improve DOD's performance measurement system to manage and oversee
its counternarcotics activities, we recommend that the Secretary of
Defense take the following two actions:
1. To address weaknesses identified in DOD's counternarcotics
performance measurement system, we recommend that the Secretary of
Defense direct the Deputy Assistant Secretary for Counternarcotics and
Global Threats to review the department's performance measures for
counternarcotics activities and revise the measures, as appropriate,
to include the key attributes of successful performance measures
previously identified by GAO.
2. To address factors associated with the limited use of DOD's
counternarcotics performance measurement system, we recommend that the
Secretary of Defense direct the Deputy Assistant Secretary for
Counternarcotics and Global Threats to apply practices that GAO has
identified to facilitate the use of performance data.
Agency Comments and Our Evaluation:
We provided a draft of this report to DOD and ONDCP for their review
and comment. We received written comments from DOD, which are
reprinted in appendix II. DOD concurred with our recommendations, and
stated it has developed and begun to implement a plan to improve the
quality and usefulness of its counternarcotics performance measurement
system. ONDCP did not provide written comments.
We received technical comments from DOD and ONDCP, which we have
incorporated where appropriate.
We are sending copies of this report to interested congressional
committees, the Secretary of Defense, and the Director of the Office
of National Drug Control Policy. In addition, the report will be
available at no charge on the GAO Web site at [hyperlink,
http://www.gao.gov].
If you or your staff have any questions about this report, please
contact me at (202) 512-4268 or fordj@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. GAO staff who made major contributions
to this report are listed in appendix III.
Signed by:
Jess T. Ford:
Director, International Affairs and Trade:
List of Congressional Committees:
The Honorable Carl Levin:
Chairman:
The Honorable John McCain:
Ranking Member:
Committee on Armed Services:
United States Senate:
The Honorable Daniel K. Inouye:
Chairman:
The Honorable Thad Cochran:
Ranking Member:
Subcommittee on Defense:
Committee on Appropriations:
United States Senate:
The Honorable Ike Skelton:
Chairman:
The Honorable Howard P. McKeon:
Ranking Member:
Committee on Armed Services:
House of Representatives:
The Honorable Norman D. Dicks:
Chairman:
The Honorable C.W. Bill Young:
Ranking Member:
Subcommittee on Defense:
Committee on Appropriations:
House of Representatives:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
Section 1016 of the National Defense Authorization Act for Fiscal Year
2010 directed GAO to report on the Department of Defense's (DOD)
performance measurement system used to assess its counternarcotics
activities. In response to this mandate, we examined the extent to
which (1) DOD's counternarcotics performance measurement system
enables DOD to track progress and (2) DOD uses performance information
from its counternarcotics performance measurement system to manage its
activities.
Our work focused on the efforts of DOD to develop an effective
counternarcotics performance measurement system. Within DOD, we spoke
with officials from several relevant components involved in the
management, oversight, and implementation of DOD's counternarcotics
activities, including the Office of the Deputy Assistant Secretary of
Defense for Counternarcotics and Global Threats (DASD-CN>), U.S.
Africa Command (AFRICOM), U.S. Central Command (CENTCOM), U.S.
European Command (EUCOM), U.S. Northern Command (NORTHCOM), U.S.
Southern Command (SOUTHCOM), the Joint Interagency Task Force-South
(JIATF-S), the Joint Interagency Task Force-West (JIATF-W), and the
DOD Inspector General (DOD-IG). We also discussed DOD efforts with
officials from the Office of National Drug Control Policy (ONDCP), the
organization that establishes U.S. counternarcotics goals and
coordinates the federal budget to combat drugs.
To examine the extent to which DOD's counternarcotics performance
measurement system enables the department to track its progress we
analyzed DOD strategy, budget, and performance documents, such as
DOD's Counternarcotics Strategy, Drug Interdiction and Counterdrug
Activities Budget Estimates, and Performance Summary Reports. We
reviewed relevant DOD and ONDCP guidance on performance measures, such
as DOD's Standard Operating Procedures for the Counternarcotics
Performance Metrics System and ONDCP's Drug Control Accounting
circular. Further, we evaluated a generalizable random sample of DOD's
fiscal year 2009 counternarcotics performance measures (115 of 239
measures) to assess the extent to which these measures adhered to GAO
criteria on the key attributes of successful performance measures.
Because we followed a probability procedure based on random
selections, our sample is only one of a large number of samples that
we might have drawn. Since each sample could have provided different
estimates, we express our confidence in the precision of our
particular sample's results at a 95 percent confidence interval (e.g.,
plus or minus 6 percentage points). This is the interval that would
contain the actual population value for 95 percent of the samples we
could have drawn. To evaluate the sample, two analysts independently
assessed each of the performance measures against nine attributes of
successful performance measures identified by GAO.[Footnote 25] Those
analysts then met to discuss and resolve any differences in the
results of their analyses. A supervisor then reviewed and approved the
final results of the analysis. In conducting this analysis, we
analyzed information contained in DOD's counternarcotics performance
measurement system database and spoke with DOD officials responsible
for managing counternarcotics activities and entering information into
the database. We did not, however, review supporting documentation
referenced but not included in the system's database, nor did we
assess other databases that might exist at the DOD component level. We
also discussed DOD's performance measures with cognizant officials
from ONDCP and several DOD components, including DASD-CN>, AFRICOM,
CENTCOM, EUCOM, NORTHCOM, SOUTHCOM, JIATF-S, JIATF-W, and the DOD-IG.
To evaluate the extent to which DOD uses performance information from
its counternarcotics performance measurement system to support its
mission, we held discussions with officials from DOD components--
including DASD-CN>, AFRICOM, CENTCOM, EUCOM, NORTHCOM, SOUTHCOM,
JIATF-S, and JIATF-W--to determine the ways in which these components
use information from DOD's system, as well as other sources of
performance information. We also examined DOD's Performance Summary
Reports and fiscal year 2011 Drug Interdiction and Counterdrug
Activities Budget Estimates to assess the extent to which these
materials reported that DOD used performance information from its
counternarcotics performance measurement system database. Further, we
analyzed the extent to which DOD applies key management practices
previously identified by GAO[Footnote 26] to facilitate the use of
performance information from its counternarcotics performance
measurement system. We also traveled to Tampa, Miami, and Key West,
Florida where we visited CENTCOM, SOUTHCOM, and JIATF-S. In these
visits, we met with DOD officials responsible for management and
implementation of counternarcotics activities to discuss DOD's use of
performance data to support its counternarcotics mission.
To determine the completeness and consistency of DOD funding data, we
compiled and compared data from DOD with information from cognizant
U.S. agency officials in Washington, D.C. We also compared the funding
data with budget summary reports from the ONDCP to corroborate their
accuracy. Although we did not audit the funding data and are not
expressing an opinion on them, based on our examination of the
documents received and our discussions with cognizant agency
officials, we concluded that the funding data we obtained were
sufficiently reliable for the purposes of this report.
We conducted this performance audit from December 2009 to July 2010 in
accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives.
[End of section]
Appendix II: Comments from the Department of Defense:
Assistant Secretary Of Defense:
Special Operations, Low-Intensity Conflict & Interdependent
Capabilities:
2500 Defense Pentagon:
Washington, DC 20301-2500:
July 13, 2010:
Mr. Jess T. Ford:
International Affairs and Trade:
U.S. Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Attached is the Department of Defense response to the Government
Accountability Office (GAO) draft report, GAO-10-835, 'DRUG CONTROL: DOD
Needs to Improve Its Performance Measurement System to Better Manage
and Oversee Its Counternarcotics Activities,' dated July 21, 2010 (GAO
Code 320743).
My office has worked diligently to establish a performance metric
system for the Department of Defense counternarcotics (CN) efforts as
required under the Office of National Drug Control Policy's (ONDCP)
Circular "Drug Control Accounting." While it continues to be a work in
progress, I am confident that the progress being made will result in
the DoD CN performance metric program providing information that is
useful and informative to both strategic decision making and
operational tactics. The attached provides the DoD response to the two
issues reported by GAO.
My point of contact for this action is Mrs. Silvia Serban, at 703-614-
8847.
Signed by:
Michael Vickers:
[End of letter]
GAO Draft Report ” Dated July 22,2010:
GAO-10-835 (GAO Code 320743):
"Drug Control: DoD Needs to Improve Its Performance Measurement System
to Better Manage and Oversee Its Counternarcotics Activities"
Department Of Defense Comments To The GAO Recommendations (Draft 8
July 2010):
Recommendation 1: To address weaknesses identified in DoD's
counternarcotics performance measurement system, the GAO recommends
that the Secretary of Defense direct the Deputy Assistant Secretary
for Counternarcotics and Global Threats to review the department's
performance measures for counternarcotics activities and revise the
measures, as appropriate, to include the key attributes of successful
performance measures previously identified by GAO.
DoD Response: Concur. Beginning in June 2009, the Deputy Assistant
Secretary for Counternarcotics and Global Threats (DASD-CN>)
identified the need to improve its performance measurement system for
the Department's counternarcotics program. The DASD-CN> initiated an
effort to review the current process and guidance for performance
measurement and developed a two-year transition plan to improve the
quality and usefulness of the program's performance measurement
system. The first steps of the review included an examination of the
current counternarcotics strategy, prior audits by the DoD Inspector
General, feedback received from the Office of National Drug Control
Policy (ONDCP), and input from the counternarcotics program managers
within the combatant commands. The 285 existing performance
measurements submitted during FY 2008 were individually reviewed along
with the corresponding project code budget justifications. Each
performance measurement was objectively analyzed and critiqued based
on the following criteria: the direct applicability of the stated
measure; the measure's objectivity; the usefulness of the measure for
management; the practicality of the measure; the attributable link
between the measure and its related goal; the timeliness of the data
collection; and the adequateness of the stated measure to capture the
activity.
After the initial review was completed, an assessment report of the
measures and the overall performance measurement system was compiled.
From this initial assessment, DoD identified the following areas for
revision to improve the performance measurement system:
1. Establish a CN strategic results framework that depicts the casual
logic cascading from the ONDCP National Drug Control Strategy through
the DASDCN> strategic goats and objectives, to the individual
theater CN strategies in place at each combatant command. This
strategic framework captures the enabling roles among illicit drug
trafficking disruption, interdiction and apprehension participants and
presents a comprehensive cause and effect framework that clarifies
relationships among CN activities and expected results.
2. Distill the current number of performance indicators to a more
manageable size built around a uniform and consistent set of
performance dimensions to increase program understanding and
accessibility, and lay the foundation for comparing performance across
similar project code investments.
3. Establish meaningful performance targets that coalesce with the
Department's annual planning, program, budgeting, and execution
timeline.
4. Expand the counternarcotics performance metric guidance to
institutionalize the performance metric system and provide
counternarcotics program managers with informational tools to improve
the collection and quality of data.
The DASD-CN> has begun to revise the current performance measures,
as appropriate, to include the key attributes of successful
performance measures previously identified by GAO. On May 18, 2010,
the DASD-CN> issued standard operating procedures (SOPs) for the
counternarcotics performance metric system. The SOPs provide
guidelines and instructions to be used in the documentation of
performance for any counternarcotics activity funded by the
Department's Central Transfer Account (CTA). As part of the two-yeas
transition strategy, the SOPs and resulting FY 2010 revisions to the
counternarcotics performance measurements are focused on creating
performance measures that display key attributes of successful
performance measures. In FY 2011, performance indicators will
incorporate theater level CN strategies and campaign plans to expand
the performance outcomes achieved within each of the combatant
commands. Following the release of the SOPs, the DASD-CN> has
facilitated workshops to assist the combatant commands with the
composition of a theatre results framework and the revision of current
performance measurements. Work to improve the performance metrics and
the overall system for performance measurement is ongoing at the time
of this report.
Recommendation 2: To address factors associated with the limited use
of DoD's counternarcotics performance measurement system, the GAO
recommends that the Secretary of Defense direct the Deputy Assistant
Secretary for Counternarcotics and Global Threats to apply practices
that GAO has identified to facilitate the use of performance data.
DOD Response: Concur. The efforts of the DASD-CN> to improve the DoD
counternarcotics performance measurement system are focused on
capturing information that is useful and informative to both CN
strategy and operational tactics. The DASDCN> has launched an effort
to revise the current DoD Counternarcotics Strategy to establish more
precise goals and robust objectives for the Department's
counternarcotics program. Furthermore, a results framework has been
created that aligns goals, objectives, activities, and performance
measures for each combatant command's counternarcotics activities.
This alignment increases the usefulness of the performance information
collected and reinforces the connection between strategic goals and
tactical activities.
In FY 2011, performance measurement will incorporate CN theater
strategies by combatant commands to produce both theater specific
outcome and output data, thereby assisting counternarcotics program
managers to identify issues or trends and make immediate adjustments
as appropriate. The DASD-CN> has expanded the use of performance
measurements in the Program Objectives Memorandum (POM) process to
make sure that budget justifications and resource allocations are
informed by objective performance data. Access to performance data
will also be improved though the planned use of new technologies that
will allow counternarcotics program managers to access financial data
and its corresponding performance measurement information. The Office
of Counternarcotics and Global Threats also plans to conduct more
frequent and periodic performance measurement reviews with
counternarcotics program managers to ensure more information
management and oversight of the counternarcotics activities.
[End of section]
Appendix III: Contact and Staff Acknowledgments:
Contact:
Jess Ford, Director, International Affairs and Trade, (202) 512-4268
or fordj@gao.gov:
Staff Acknowledgments:
In addition to the individual named above, Juan Gobel, Assistant
Director; Elizabeth Curda; Martin de Alteriis; Karen Deans; Mark
Dowling; Justin Fisher; Richard Geiger; Eileen Larence; Marie Mak;
Christopher Mulkins; John Pendleton; Elizabeth Repko; and Mark Speight
made key contributions to this report.
[End of section]
Footnotes:
[1] The National Defense Authorization Act for Fiscal Year 2010, Pub.
L. No. 111-84 § 1016.
[2] GAO, Government Reform: Goal Setting and Results, [hyperlink,
http://www.gao.gov/products/GAO/AIMD/GGD-95-130R] (Washington, D.C.:
Mar. 27, 1995) and Executive Guide: Effectively Implementing the
Government Performance and Results Act, GAO/GGD-96-118 (Washington,
D.C.: June 1996).
[3] GAO, Preliminary Observations on the Department of Defense's
Counternarcotics Performance Measurement System, [hyperlink,
http://www.gao.gov/products/GAO-10-594R] (Washington, D.C.: Apr. 30,
2010).
[4] In addition to helping U.S. and foreign agencies address the drug
trade, the DOD Counternarcotics Strategy also seeks to maintain DOD
readiness through drug demand reduction programs. As the mandate to
GAO contained in the National Defense Authorization Act for 2010
focused on DOD's international counternarcotics activities, this
report does not contain information on DOD's demand reduction programs.
[5] DOD defines a combatant command as a military command with
geographic or functional responsibilities, such as SOUTHCOM or U.S.
Strategic Command. Military departments include the U.S. Army, Navy,
and Air Force. Defense agencies, such as the Defense Intelligence
Agency, perform selected support and service functions on a department-
wide basis.
[6] GAO, Drug Control: Assets DOD Contributes to Reducing the Illegal
Drug Supply Have Declined, [hyperlink,
http://www.gao.gov/products/GAO/NSIAD-00-9] (Washington, D.C.: Dec.
21, 1999).
[7] GAO, Drug Control: Difficulties in Measuring Costs and Results of
Transit Zone Interdiction Efforts, [hyperlink,
http://www.gao.gov/products/GAO-02-13] (Washington, D.C.: Jan. 25,
2002).
[8] GAO, Drug Control: Agencies Need to Plan for Likely Declines in
Drug Interdiction Assets, and Develop Better Performance Measures for
Transit Zone Operations, [hyperlink,
http://www.gao.gov/products/GAO-06-200] (Washington, D.C.: Nov. 15,
2005).
[9] ONDCP, ONDCP Circular: Drug Control Accounting, (May 1, 2007).
Section 7 of the circular contains the requirements of a performance
summary report.
[10] [hyperlink, http://www.gao.gov/products/GAO/AIMD-GGD-95-130R] and
[hyperlink, http://www.gao.gov/products/GAO/GGD-96-118].
[11] While some components previously developed performance measures
to monitor counternarcotics activities, fiscal year 2007 was the first
year for which DOD centrally collected counternarcotics performance
measures in its counternarcotics performance measurement system
database.
[12] GAO, Tax Administration: IRS Needs to Further Refine Its Tax
Filing Season Performance Measures, [hyperlink,
http://www.gao.gov/products/GAO-03-143] (Washington, D.C.: Nov. 22,
2002).
[13] We randomly sampled 115 of DOD's 239 counternarcotics performance
measures for fiscal year 2009 that were associated with DOD's goals of
detection and monitoring, sharing information, and building capacity
of partner nations. The resulting estimates are subject to a maximum
margin of error of plus or minus 6 percentage points.
[14] We could not determine which of the 5 measurable aspects of
performance were present for 1 of the 115 measures in the sample
because the measure did not contain enough information for a thorough
analysis. As a result, the sum of the percentages does not equal 100.
[15] GAO, Managing for Results: Analytic Challenges in Measuring
Performance, [hyperlink,
http://www.gao.gov/products/GAO/HEHS/GGD-97-138] (Washington, D.C.:
May 30, 1997).
[16] JIATF-W is a taskforce of U.S. Pacific Command with a mission to
combat drug-related transnational organized crime.
[17] [hyperlink, http://www.gao.gov/products/GAO/HEHS/GGD-97-138].
[18] GAO, Managing for Results: Enhancing Agency Use of Performance
Information for Management Decision Making, [hyperlink,
http://www.gao.gov/products/GAO-05-927] (Washington, D.C.: Sept. 9,
2005).
[19] [hyperlink, http://www.gao.gov/products/GAO/GGD-96-118].
[20] We have previously reported that, according to the Government
Performance and Results Act (GPRA), performance reports should contain
elements such as describing whether or not agency performance goals
have been met and discussing performance measures that are most
significant to the success of a program. See GAO, Results-Oriented
Government: GPRA Has Established a Solid Foundation for Achieving
Greater Results, [hyperlink, http://www.gao.gov/products/GAO-04-38]
(Washington, D.C.: Mar. 10, 2004).
[21] In May 2008, the DOD-IG released a review of the department's
fiscal year 2007 performance reporting, see DOD-IG, Independent
Auditor's Report on the FY 2007 Performance Summary Report for DOD
National Drug Control Program Activities, D-2008-085 (May 2, 2008). In
the report, DOD-IG stated that due to delays in receiving material, it
was not able to express an opinion as to whether the report conformed
to ONDCP guidance. DOD-IG has not produced reports authenticating
DOD's fiscal year 2008 or 2009 counternarcotics performance measures.
[22] [hyperlink, http://www.gao.gov/products/GAO-05-927].
[23] As of June 2010, DASD-CN> told us they had conducted working
groups with CENTCOM and EUCOM, and had scheduled working groups with
AFRICOM, NORTHCOM, and SOUTHCOM.
[24] We have previously reported that to be useful, performance
information must meet users' needs for completeness, accuracy,
consistency, timeliness, validity, and ease of use. Other attributes
that affect the usefulness of information include, but are not limited
to, relevance, credibility, and accessibility.
[25] [hyperlink, http://www.gao.gov/products/GAO-03-143].
[26] [hyperlink, http://www.gao.gov/products/GAO-05-927].
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
