Preliminary Observations on the Department of Defense's Counternarcotics Performance Measurement System
Gao ID: GAO-10-594R April 30, 2010The Department of Defense (DOD) leads detection and monitoring of aerial and maritime transit of illegal drugs into the United States in support of law enforcement agencies. Additionally, DOD's counternarcotics activities include sharing information with U.S. and foreign agencies, as well as helping foreign countries build their counternarcotics capacity. In support of these activities, Congress provided just over $6.1 billion to DOD's Counternarcotics Central Transfer Account from fiscal year 2005 through 2010. In November 2005, GAO recommended that DOD, in conjunction with other agencies performing counternarcotics activities, develop and coordinate counternarcotics performance measures. GAO has found that measuring performance allows organizations to track the progress they are making toward their goals and provides managers a basis for making key decisions to improve programs and results. In May 2007, the Office of National Drug Control Policy (ONDCP) released a circular that required agencies, including DOD, to submit a report on their performance-related information for counternarcotics activities, including how the agency's performance measures are used in the management of the program. The National Defense Authorization Act for Fiscal Year 2010 (Pub. L. No. 111-84, 1016) mandated that GAO report on the performance evaluation system used by DOD to assess its counternarcotics activities. To satisfy this mandate, in April 2010, we briefed congressional staff from the defense committees. This letter summarizes the preliminary observations conveyed during those briefings.
In summary, the briefing provided to congressional staff noted that (1) DOD has developed performance measures for its counternarcotics activities. DOD has also developed a database to collect performance information, including measures, targets, and results. However, these performance measures lack a number of the attributes, such as being clearly stated and having measurable targets, which GAO considers key to successful performance measures. (2) DOD is currently reassessing its performance measures for its counternarcotics activities. DOD hired a contractor to assist the department in revising its guidance for the development of counternarcotics performance measures. DOD officials, however, noted that the department still faces challenges measuring the performance of its counternarcotics activities. These challenges include creating performance measures that assess program outcomes; ensuring adequate resources, such as expertise in performance management, are available to develop performance measures; and ensuring that the data used to measure DOD performance is reliable. (3) It is unclear the extent to which DOD uses the performance information it collects through its database to manage its counternarcotics activities. Previous work by GAO has found that the full benefit of collecting performance information is realized only when managers use this information to inform key decisions to improve programs and results, such as identifying corrective actions, allocating resources, and sharing best practices for program implementation. However, officials at U.S. Central Command, U.S. Southern Command, and the Joint Interagency Task Force-South told us they rarely use the database to identify performance problems or manage counternarcotics activities. Rather, they tend to manage programs using measures and data not submitted to the database.
GAO-10-594R, Preliminary Observations on the Department of Defense's Counternarcotics Performance Measurement System
This is the accessible text file for GAO report number GAO-10-594R
entitled 'Preliminary Observations on the Department of Defense's
Counternarcotics Performance Measurement System' which was released on
April 30, 2010.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
GAO-10-594R:
United States Government Accountability Office:
Washington, DC 20548:
April 30, 2010:
Congressional Committees:
Subject: Preliminary Observations on the Department of Defense's
Counternarcotics Performance Measurement System:
The Department of Defense (DOD) leads detection and monitoring of
aerial and maritime transit of illegal drugs into the United States in
support of law enforcement agencies. Additionally, DOD's
counternarcotics activities include sharing information with U.S. and
foreign agencies, as well as helping foreign countries build their
counternarcotics capacity. In support of these activities, Congress
provided just over $6.1 billion to DOD's Counternarcotics Central
Transfer Account from fiscal year 2005 through 2010.
In November 2005, GAO recommended that DOD, in conjunction with other
agencies performing counternarcotics activities, develop and
coordinate counternarcotics performance measures. GAO has found that
measuring performance allows organizations to track the progress they
are making toward their goals and provides managers a basis for making
key decisions to improve programs and results. In May 2007, the Office
of National Drug Control Policy (ONDCP) released a circular that
required agencies, including DOD, to submit a report on their
performance-related information for counternarcotics activities,
including how the agency's performance measures are used in the
management of the program.
The National Defense Authorization Act for Fiscal Year 2010 (Pub. L.
No. 111-84, § 1016) mandated that GAO report on the performance
evaluation system used by DOD to assess its counternarcotics
activities. To satisfy this mandate, in April 2010, we briefed
congressional staff from the defense committees. This letter
summarizes the preliminary observations conveyed during those
briefings. The briefing slides used to provide the briefing are
attached as enclosure I.
We plan to issue the final results of our evaluation of DOD's
counternarcotics performance evaluation system in a report to be
released in the summer of 2010 that will focus on the extent to which:
(1) DOD's counternarcotics performance measures enable DOD to track
progress toward achieving its goals and (2) DOD uses performance
information to support its counternarcotics mission.
In summary, the briefing provided to congressional staff noted that:
* DOD has developed performance measures for its counternarcotics
activities. DOD has also developed a database to collect performance
information, including measures, targets, and results. However, these
performance measures lack a number of the attributes, such as being
clearly stated and having measurable targets, which GAO considers key
to successful performance measures.
* DOD is currently reassessing its performance measures for its
counternarcotics activities. DOD hired a contractor to assist the
department in revising its guidance for the development of
counternarcotics performance measures. DOD officials, however, noted
that the department still faces challenges measuring the performance
of its counternarcotics activities. These challenges include creating
performance measures that assess program outcomes; ensuring adequate
resources, such as expertise in performance management, are available
to develop performance measures; and ensuring that the data used to
measure DOD performance is reliable.
* It is unclear the extent to which DOD uses the performance
information it collects through its database to manage its
counternarcotics activities. Previous work by GAO has found that the
full benefit of collecting performance information is realized only
when managers use this information to inform key decisions to improve
programs and results, such as identifying corrective actions,
allocating resources, and sharing best practices for program
implementation. However, officials at U.S. Central Command, U.S.
Southern Command, and the Joint Interagency Task Force-South told us
they rarely use the database to identify performance problems or
manage counternarcotics activities. Rather, they tend to manage
programs using measures and data not submitted to the database. We
plan to examine this issue in more detail in our report to be released
in the summer of 2010.
We conducted this performance audit from December 2009 to April 2010,
in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives.
We provided drafts of this report to DOD and ONDCP for comment. DOD
provided no comments. ONDCP provided technical comments, which we have
incorporated where appropriate.
We are sending copies of this report to interested congressional
committees, the Secretary of Defense, and the Director of the Office
of National Drug Control Policy. The report also is available at no
charge on GAO's Web site at [hyperlink, http://www.gao.gov].
If you or your staff members have any questions about this
information, please contact me at (202) 512-4268 or fordj@gao.gov.
Contact points for our Offices of Congressional Relations and Public
Affairs may be found on the last page of this report. GAO staff who
made major contributions to this report are listed in enclosure II.
Signed by:
Jess T. Ford:
Director, International Affairs and Trade:
Enclosures:
List of Congressional Committees:
The Honorable Carl Levin:
Chairman:
The Honorable John McCain:
Ranking Member:
Committee on Armed Services:
United States Senate:
The Honorable Daniel K. Inouye:
Chairman:
The Honorable Thad Cochran:
Ranking Member:
Subcommittee on Defense:
Committee on Appropriations:
United States Senate:
The Honorable Ike Skelton:
Chairman:
The Honorable Howard P. McKeon:
Ranking Member:
Committee on Armed Services:
House of Representatives:
The Honorable Norman D. Dicks:
Chairman:
The Honorable:
Ranking Member C.W. Bill Young:
Subcommittee on Defense:
Committee on Appropriations:
House of Representatives:
[End of section]
Enclosure I: Briefing for Congressional Committees:
Preliminary Observations on Department of Defense's Counternarcotics
Performance Measurement System:
A Briefing for Congressional Committees:
April 2010:
Table of Contents:
* Objectives, Scope, and Methodology:
* Background:
* Observations on Department of Defense's (DOD) Counternarcotics
Performance Measures:
* Preliminary Observations on DOD's Use of Performance Information to
Support Its Counternarcotics Mission:
* Agency Comments and Our Evaluation:
Objectives, Scope, and Methodology:
The National Defense Authorization Act of 2010 (Pub. L. No. 111-84, §
1016) mandated that GAO report on the performance evaluation system
used by the DOD to assess its counternarcotics activities.
This briefing contains information on DOD's counternarcotics
performance evaluation system, including the following:
* the extent to which DOD's counternarcotics performance measures
enable DOD to track progress toward achieving its goals, and;
* preliminary observations on DOD's use of performance
information to support its counternarcotics mission.
To address these objectives we:
* Interviewed officials from the Office of National Drug Control
Policy (ONDCP) and DOD components including the following:
- Office of the Deputy Assistant Secretary of Defense for
Counternarcotics and Global Threats (DASD-CN),
- U.S. Central Command (CENTCOM),
- U.S. Southern Command (SOUTHCOM),
- Joint Interagency Task Force-South (JIATF-S), and,
- DOD Inspector General (DOD-IG).
To address these objectives we:
* reviewed and analyzed DOD strategy and budget documents, as well as
DOD guidance on performance measures, and,
* evaluated a generalizable random sample of DOD's unclassified fiscal
year 2009 counternarcotics performance measures (71 of 171 measures)
to assess the extent to which these measures adhered to GAO criteria
on the key attributes of performance measures (see table 1).
[Footnote 1]
We conducted this performance audit from December 2009 to April 2010,
in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives.
After conducting additional audit work, we plan to issue the final
results of our evaluation of DOD's counternarcotics performance
measurement system in a report to be released in the summer of 2010.
[End of section]
Background:
The DOD leads detection and monitoring of aerial and maritime transit
of illegal drugs into the United States in support of law enforcement
agencies.
To accomplish this mission, DOD has identified three goals:
* detecting and monitoring drug trafficking,
* sharing information with U.S. and foreign agencies, and,
* helping foreign countries build their counternarcotics capacity.
In support of these activities, Congress provided just over $6.1
billion to DOD's Counternarcotics Central Transfer Account from fiscal
year 2005 to fiscal year 2010.
In May 2007, ONDCP issued a drug control accounting circular. Section
7 of the circular requires that National Drug Control Program
agencies, including DOD, submit to the Director of ONDCP a performance
summary report including performance measures, targets, and results.
ONDCP cited 21 U.S. Code § 1703(d)(7), § 1705(c)(4), § 1705 (a)(2)(xi)
as the legal authorities that require ONDCP to monitor the
implementation of the National Drug Control Strategy, develop a
performance measurement system, and assess the effectiveness of
achieving federal National Drug Control Strategy goals.
DASD-CN reports that it is the focal point for DOD counternarcotics
efforts and that it will ensure the department develops and implements
a focused counternarcotics program with clear priorities and measured
results.
Programs, Resources, and Assessments, a division within DASDCN, is the
lead office for the development of counternarcotics resources and
plans. Among other activities, this office:
* directs and manages the planning, programming, and budgeting system
of the DOD counternarcotics program, and,
* collects quarterly operational statistics to evaluate the
effectiveness of DOD's contribution to meeting the goals of the
National Drug Control Strategy.
[End of section]
DOD Has Developed Performance Measures for Its Counternarcotics
Activities:
In response to ONDCP guidance, DOD developed and submitted to ONDCP a
set of performance measures for its fiscal year 2007 counternarcotics
activities. Subsequently, DOD revised its performance measures for
fiscal years 2008 and 2009 and submitted a report on these measures to
ONDCP.[Footnote 2]
DOD established a database to collect performance information from its
components, including performance measures, targets, and results. In
June 2008, DOD issued guidance with instructions for entering
counternarcotics performance information into the database.
In fiscal year 2009, DOD reported it had 285 performance measures for
its counternarcotics activities. Of those, there were 171 unclassified
performance measures related to DOD's mission of supporting law
enforcement agencies.
GAO Has Identified Nine Key Attributes of Successful Performance
Measures:
Table 1: GAO's Key Attributes of Successful Performance Measures:
Attribute: Linkage;
Definition: Measure is aligned with division and agencywide goals and
mission and clearly communicated throughout the organization.
Potentially adverse effects of not meeting attribute: Behaviors and
incentives created by measures do not support achieving division or
agencywide goals or mission.
Attribute: Clarity;
Definition: Measure is clearly stated, and the name and definition are
consistent with the methodology used to calculate it.
Potentially adverse effects of not meeting attribute: Data could be
confusing and misleading to users.
Attribute: Measurable target;
Definition: Measure has a numerical goal.
Potentially adverse effects of not meeting attribute: Cannot tell
whether performance is meeting expectations.
Attribute: Objectivity;
Definition: Measure is reasonably free from significant bias or
manipulation.
Potentially adverse effects of not meeting attribute: Performance
assessments may be systematically over- or understated.
Attribute: Reliability;
Definition: Measure produces the same result under similar conditions.
Potentially adverse effects of not meeting attribute: Reported
performance data is inconsistent and adds uncertainty.
Attribute: Core program activities;
Definition: Measures cover the activities that an entity is expected
to perform to support the intent of the program.
Potentially adverse effects of not meeting attribute: Not enough
information available in core program areas to managers and
stakeholders.
Attribute: Limited overlap;
Definition: Measure should provide new information beyond that
provided by other measures.
Potentially adverse effects of not meeting attribute: Manager may have
to sort through redundant, costly information that does not add value.
Attribute: Balance;
Definition: Balance exists when a suite of measures ensures that an
organization's various priorities are covered.
Potentially adverse effects of not meeting attribute: Lack of balance
could create skewed incentives when measures overemphasize some goals.
Attribute: Government-wide priorities;
Definition: Each measure should cover a priority such as quality,
timeliness, and cost of service.
Potentially adverse effects of not meeting attribute: A program's
overall success is at risk if all priorities are not addressed.
Source: GAO-03-143.
[End of table]
DOD's Performance Measures Lack Key Attributes of Successful Measures:
DOD and ONDCP officials acknowledged that there are weaknesses in the
performance measures in DOD's counternarcotics database.
GAO's analysis of a sample of DOD's unclassified fiscal year 2009
performance measures shows that key attributes were not always present
in the measures.
* Two of the attributes, balance and limited overlap, did not appear
to be present in the set of measures.
* The other seven attributes of successful performance measures were
present to varying degrees (see figure 1).
Our preliminary observations suggest that DOD's classified fiscal year
2009 performance measures may have similar weaknesses.
Figure 1: DOD's Fiscal Year 2009 Unclassified Counternarcotics
Performance Measures Relative to Seven Attributes of Successful
Performance Measures:
[Refer to PDF for image: horizontal bar graph]
Key attributes of performance measures:
Reliability:
Percentage of DOD counternarcotics performance measures that exhibited
key attribute: approximately 40%.
Government-wide priorities:
Percentage of DOD counternarcotics performance measures that exhibited
key attribute: approximately 42%.
Objectivity:
Percentage of DOD counternarcotics performance measures that exhibited
key attribute: approximately 60%.
Clarity:
Percentage of DOD counternarcotics performance measures that exhibited
key attribute: approximately 63%.
Measurable target:
Percentage of DOD counternarcotics performance measures that exhibited
key attribute: approximately 65%.
Linkage:
Percentage of DOD counternarcotics performance measures that exhibited
key attribute: approximately 98%.
Core program activities:
Percentage of DOD counternarcotics performance measures that exhibited
key attribute: approximately 100%.
Source: GAO analysis of DOD performance measures.
Note: GAO randomly sampled 71 of DOD's 171 counternarcotics
performance measures for fiscal year 2009 that were not classified and
were associated with DOD's goals of detection and monitoring, sharing
information, and building capacity of partner nations. The resulting
estimates, reported above, are subject to a maximum margin of error of
+/- 10 percentage points.
[End of figure]
DOD Is Working To Improve Its Performance Measures, but Some
Challenges Remain:
DOD officials have acknowledged that the department's counternarcotics
performance measures need to be revised. DOD hired a contractor to
assist it in revising its guidance for the development of
counternarcotics performance measures. Revised guidance, dated January
25, 2010, exists but is not yet finalized.
CENTCOM and SOUTHCOM officials told us that the revised
guidance provides better direction than previous guidance.
We plan to comment on DOD's revised guidance in our summer 2010
product.
DOD officials, however, noted that the department still faces
challenges measuring the performance of its counternarcotics
activities. These challenges include the following:
* creating outcome-based performance measures given DOD's supporting
role in counternarcotics activities,
* ensuring adequate resources, such as expertise in performance
management, are available to develop performance measures, and,
* ensuring data used to measure DOD performance is reliable.
GAO Has Identified Use of Performance Information As Key To Improving
Management:
Figure 2: Uses of Performance Information:
[Refer to PDF for image: illustration]
Uses of Performance Data:
* Identify and share effective approaches;
* Identify problems and take corrective action;
* Develop strategy and allocate resources;
Lead to:
Improved management decision making.
Source: GAO.
Note: GAO has also previously reported that recognizing and rewarding
individual performance is a use of performance data. However, this use
was not applicable to DOD's counternaroctics performance measures, as
they relate only to program activities.
[End of figure]
[End of section]
Preliminary Observations:
DOD's Use of Performance Information to Support Its Counternarcotics
Mission:
Identify problems and take corrective action: CENTCOM, SOUTHCOM, and
JIATF-S officials told us they rarely use the DASD-CN database to
identify performance problems or manage counternarcotics activities.
Rather, they tend to manage programs using information not submitted
to the DASD-CN database. For example, CENTCOM officials told us
information obtained in weekly program meetings, not data sent to the
DASD-CN database, is most often used to help them identify problems
and make program adjustments. Further, it is unclear the extent to
which DASD-CN uses information in the database to oversee
counternarcotics activities.
Develop strategy and allocate resources: DASD-CN officials told us
that they do not currently link performance information directly to
budget allocation decisions.
Identify and share effective approaches: While DOD creates reports for
ONDCP, DASD-CN officials told us they do not produce additional
reports for the department.
[End of section]
Agency Comments and Our Evaluation:
We provided drafts of this report to DOD and ONDCP for comment. DOD
provided no comments. ONDCP provided technical comments, which we have
incorporated where appropriate.
[End of Enclosure I]
Enclosure II: GAO Contact and Staff Acknowledgments:
GAO Contact:
Jess Ford, Director, International Affairs and Trade, (202) 512-4268
or fordj@gao.gov:
Staff Acknowledgments:
In additional to the individuals named above, Juan Gobel, Assistant
Director; Joe Carney; Elizabeth Curda; Martin de Alteriis; Mark
Dowling; Justin Fisher; Richard Geiger; Eileen Larence; Jeff Miller;
Christopher Mulkins; John Pendleton; Elizabeth Repko; and Mark Speight
made key contributions to this report.
[End of Enclosure I]
Footnotes:
[1] Prior GAO work identified nine attributes of successful
performance measures. See GAO, Tax Administration: IRS Needs to
Further Refine Its Tax Filing Season Performance Measures, [hyperlink,
http://www.gao.gov/products/GA0-03-143] (Washington, D.C.: Nov. 22,
2002).
[2] According to ONDCP, DOD's reports for fiscal years 2007, 2008, and
2009 did not fulfill the requirements of ONDCP's drug control
accounting circular because the reports were not authenticated by the
DOD-IG.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
