Defense Management
Tools for Measuring and Managing Defense Agency Performance Could Be Strengthened Gao ID: GAO-04-919 September 13, 2004GAO was mandated to assess the effectiveness of defense agency performance contracts as management tools. As agreed, GAO also reviewed other tools (performance plans and balanced scorecards) and focused on three defense agencies--the Defense Logistics Agency (DLA), the Defense Information Systems Agency (DISA), and the Department of Defense Education Activity (DODEA). GAO addressed (1) the extent that the defense agencies initially used performance contracts, including whether this tool addressed attributes associated with results-oriented management; (2) defense agencies' efforts to implement performance plans using lessons learned from the initial contracts; and (3) the extent DOD established mechanisms to share lessons learned. GAO reviewed the content of these tools, but not the actual or reported performance. DISA has not yet finalized its scorecard, thus this report discusses only DISA's plans for its scorecard.
Since fiscal year 1998, the Department of Defense (DOD) has implemented various tools to help manage and oversee the performance of defense agencies. Between fiscal year 1999 and 2003, DLA, DISA, and DODEA initially used "performance contracts"--internal management agreements--to bring specific problems to the attention of senior DOD and agency leadership. While the contracts produced some useful information for decision makers, this tool would have been more effective for assessing performance, making resource allocation decisions, and taking corrective actions if DOD had required the agencies to include certain attributes associated with results-oriented management. Such attributes include aligning agency performance goals and measures with agency strategic plans and departmentwide goals; identifying individuals accountable for achieving results; providing a comprehensive view of organizational performance; linking resource needs to performance; discussing data quality; and providing contextual information, including external factors that affect reported performance. Beginning in fiscal year 2003, DOD renamed the performance contracts as "performance plans" and gave the defense agencies the option to use a "balanced scorecard" approach, a tool used in the public and private sectors to assess organizational performance. Based on experiences using the initial contracts, DOD took steps to strengthen performance plans and scorecards by revising the oversight and review process, requiring performance measures to align with agency and departmentwide goals, and requiring measures to provide a more comprehensive view of agency performance. DLA's scorecard, DODEA's performance plan, and DISA's plans for the agency's scorecard incorporated these changes and other attributes to varying degrees. While these tools have the potential to provide information useful to decision makers, they would be strengthened if DOD had required the agencies to include additional attributes such as designating specific individuals responsible for achieving results; identifying the relationship between resource needs and performance; reporting on data quality; and providing contextual information to allow top leaders to understand the extent of progress made, take corrective actions to achieve goals, and establish realistic performance goals for future years. With these attributes, decision makers would potentially gain additional insights into agency performance and areas needing greater management attention. DOD has developed mechanisms, such as a performance management Web site and roundtables, to help agencies share lessons learned from implementing performance plans and scorecards. In response to GAO's suggestions during this review, DOD recognized the need to continue to hold roundtables more frequently. DLA and DISA have also proactively shared their experiences with each other.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-04-919, Defense Management: Tools for Measuring and Managing Defense Agency Performance Could Be Strengthened
This is the accessible text file for GAO report number GAO-04-919
entitled 'Defense Management: Tools for Measuring and Managing Defense
Agency Performance Could Be Strengthened' which was released on
September 13, 2004.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Committee on Armed Services, U.S. Senate:
United States Government Accountability Office:
GAO:
September 2004:
DEFENSE MANAGEMENT:
Tools for Measuring and Managing Defense Agency Performance Could Be
Strengthened:
GAO-04-919:
GAO Highlights:
Highlights of GAO-04-919, a report to the Committee on Armed Services,
U.S. Senate:
Why GAO Did This Study:
GAO was mandated to assess the effectiveness of defense agency
performance contracts as management tools. As agreed, GAO also reviewed
other tools (performance plans and balanced scorecards) and focused on
three defense agencies”the Defense Logistics Agency (DLA), the Defense
Information Systems Agency (DISA), and the Department of Defense
Education Activity (DODEA). GAO addressed (1) the extent that the
defense agencies initially used performance contracts, including
whether this tool addressed attributes associated with results-
oriented management; (2) defense agencies‘ efforts to implement
performance plans using lessons learned from the initial contracts;
and (3) the extent DOD established mechanisms to share lessons
learned. GAO reviewed the content of these tools, but not the actual
or reported performance. DISA has not yet finalized its scorecard,
thus this report discusses only DISA‘s plans for its scorecard.
What GAO Found:
Since fiscal year 1998, the Department of Defense (DOD) has implemented
various tools to help manage and oversee the performance of defense
agencies. Between fiscal year 1999 and 2003, DLA, DISA, and DODEA
initially used ’performance contracts“”internal management agreements”
to bring specific problems to the attention of senior DOD and agency
leadership. While the contracts produced some useful information for
decision makers, this tool would have been more effective for
assessing performance, making resource allocation decisions, and
taking corrective actions if DOD had required the agencies to include
certain attributes associated with results-oriented management. Such
attributes include aligning agency performance goals and measures with
agency strategic plans and departmentwide goals; identifying
individuals accountable for achieving results; providing a
comprehensive view of organizational performance; linking resource
needs to performance; discussing data quality; and providing contextual
information, including external factors that affect reported
performance.
Beginning in fiscal year 2003, DOD renamed the performance contracts
as ’performance plans“ and gave the defense agencies the option to use
a ’balanced scorecard“ approach, a tool used in the public and private
sectors to assess organizational performance. Based on experiences
using the initial contracts, DOD took steps to strengthen performance
plans and scorecards by revising the oversight and review process,
requiring performance measures to align with agency and departmentwide
goals, and requiring measures to provide a more comprehensive view of
agency performance. DLA‘s scorecard, DODEA‘s performance plan, and
DISA‘s plans for the agency‘s scorecard incorporated these changes and
other attributes to varying degrees. While these tools have the
potential to provide information useful to decision makers, they would
be strengthened if DOD had required the agencies to include additional
attributes such as designating specific individuals responsible for
achieving results; identifying the relationship between resource needs
and performance; reporting on data quality; and providing contextual
information to allow top leaders to understand the extent of progress
made, take corrective actions to achieve goals, and establish
realistic performance goals for future years. With these attributes,
decision makers would potentially gain additional insights into agency
performance and areas needing greater management attention.
DOD has developed mechanisms, such as a performance management Web
site and roundtables, to help agencies share lessons learned from
implementing performance plans and scorecards. In response to GAO‘s
suggestions during this review, DOD recognized the need to continue to
hold roundtables more frequently. DLA and DISA have also proactively
shared their experiences with each other.
What GAO Recommends:
GAO is making recommendations to DOD aimed at improving guidance to
make performance plans and scorecards more informative and useful and
further strengthen the potential of these tools for measuring and
managing agency performance.
In its comments, DOD generally concurred with GAO‘s recommendations.
www.gao.gov/cgi-bin/getrpt?GAO-04-919.
To view the full product, click on the link above. For more
information, contact Sharon Pickup at (202) 512-4300 or
pickups@gao.gov or Patricia Dalton at (202) 512-6806 or
daltonp@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
While Used to Varying Degrees, Performance Contracts That We Reviewed
Could Have Been More Effective Had They Included Certain Attributes
Associated with Results-Oriented Management:
The Plan and Scorecard We Reviewed Show Improvement, but Could Be
Strengthened by Including Certain Key Attributes:
DOD Captures and Shares Lessons Learned:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Department of Defense's Risk Management Framework:
Appendix III: Fiscal Year When Defense Agencies Implemented Their First
Performance Contract, Plan, or Scorecard:
Appendix IV: Certain Key Attributes Associated with Results-Oriented
Management:
Appendix V: Comments from the Department of Defense:
Appendix VI: Key Contact and Staff Acknowledgments:
Related GAO Products:
Tables:
Table 1: Time Frames for DLA's, DISA's, and DODEA's Use of Performance
Contracts, Performance Plans, or Balanced Scorecards:
Table 2: Extent To Which DLA's, DISA's, and DODEA's Performance
Contracts Incorporated Certain Key Attributes Associated with Results-
Oriented Management:
Table 3: Extent To Which DLA's Scorecard and DODEA's Plan Incorporated
Certain Key Attributes Associated with Results-Oriented Management:
Figures:
Figure 1: Depiction of How DLA's May 2004 Scorecard Supports
Departmentwide Goals and Cascades to Lower Organizational Levels:
Figure 2: Example of the Type of Information in DODEA's Performance
Plan That OSD Officials Monitor, as of May 2004:
Figure 3: Detailed Data on DODEA's Pupil to Teacher Ratio as of May
2004:
Figure 4: Format of a Sample Measure from DISA's Draft September 2003
Balanced Scorecard:
Figure 5: DOD's Risk Management Framework:
Figure 6: Defense Agencies:
Figure 7: Defense Field Activities:
Abbreviations:
DISA: Defense Information Systems Agency:
DLA: Defense Logistics Agency:
DOD: Department of Defense:
DODEA: Department of Defense Education Activity:
OSD: Office of the Secretary of Defense:
PA&E: Program Analysis and Evaluation:
USD/P&R: Under Secretary of Defense for Personnel and Readiness:
United States Government Accountability Office:
Washington, DC 20548:
September 13, 2004:
The Honorable John W. Warner:
Chairman:
The Honorable Carl Levin:
Ranking Minority Member:
Committee on Armed Services:
United States Senate:
Department of Defense (DOD) agencies and field activities, such as the
Defense Logistics Agency, are a key part of DOD's business operations
and provide numerous support services, ranging from information
technology and education of servicemembers' dependents to logistics
support for the department.[Footnote 1] The agencies' customers include
the military services, other agencies, and military personnel and their
families. During the past 2 decades, service officials have expressed
concern about inadequate oversight and performance of the defense
agencies; specifically, the lack of customer responsiveness and
inefficient business processes in light of growing defense agency
budgets.
As part of a departmentwide effort, known as the Defense Reform
Initiative, aimed at improving DOD's business operations, in 1998, the
Deputy Secretary of Defense directed the defense agencies to develop
agreements between the defense agencies and the Under and Assistant
Secretaries to which they report. Known as "performance contracts" at
the time, these agreements were intended to help improve the Office of
the Secretary of Defense's (OSD) oversight of these agencies as well as
their performance. According to a DOD official, these performance
contracts were not intended to be contracts in the legal sense, but
rather were one of a number of management tools--such as strategic
plans--used by the defense agencies to improve performance by setting
performance goals, focusing the attention of OSD and agency leaders on
customer concerns, and bringing management visibility over key areas of
the agencies' performance.[Footnote 2] In fiscal year 2003, DOD refined
its approach to performance management and renamed the contracts as
"performance plans" in order to more accurately reflect the intent of
the agreements. According to DOD, these plans are intended to provide a
more comprehensive view of agency goals and performance and align with
agency strategic plans and departmentwide goals. DOD gave the defense
agencies the option to use a "balanced scorecard"--a form of
performance plan.[Footnote 3] Balanced scorecards organize performance
measures by organizational drivers--including financial, customer, and
internal business processes as well as workforce learning and growth--
to help measure performance, make improvements, and assess how well
organizations are positioned to perform in the future. Although the
plans and scorecards themselves are not intended to resolve or prevent
problems, OSD and defense agency officials intend to use these tools to
monitor agency performance, identify developing problems and corrective
actions to improve business practices, meet customer needs, and set
funding priorities.
We have conducted an extensive body of work over the last decade
identifying leading practices in results-oriented management,
including performance measurement and reporting. In our prior work on
government performance, we identified attributes associated with
results-oriented management that help inform management decision
making.[Footnote 4] For this report, we identified seven key attributes
that we feel are the most important for accurately assessing the
strengths and weaknesses of programs and making improvements. These
attributes include aligning performance measures with strategic goals,
assigning accountability for achieving results, developing measures
that demonstrate results, developing measures that provide a
comprehensive view of agency performance, linking resource needs to
performance, discussing data quality, and including contextual
information.
The Senate Committee on Armed Services report accompanying the National
Defense Authorization Act for Fiscal Year 2004 directed us to assess
the effectiveness of defense agency performance contracts as management
tools.[Footnote 5] Because of the evolution from performance contracts
to plans or balanced scorecards, this report evaluates all of these
tools. Of the 14 defense agencies that presently use these tools, we
focused our work on 3--the Defense Logistics Agency (DLA), the Defense
Information Systems Agency (DISA), and the Department of Defense
Education Activity (DODEA)--based on their levels of experience using
performance contracts, different types of missions, and types of
services and customers.
Table 1 shows the time frames during which these agencies used
performance contracts, performance plans, or balanced scorecards.
Table 1: Time Frames for DLA's, DISA's, and DODEA's Use of Performance
Contracts, Performance Plans, or Balanced Scorecards:
Fiscal year:
Agency: DLA;
Performance contract: 1999-2001;
Performance plan: Not applicable;
Balanced scorecard: 2003-2004.
Agency: DISA;
Performance contract: 2000-2002;
Performance plan: 2003;
Balanced scorecard: 2004.
Agency: DODEA;
Performance contract: 2000-2003;
Performance plan: 2004;
Balanced scorecard: Not applicable.
Sources: DLA, DISA, and DODEA.
Note: Because DLA was transitioning to a balanced scorecard, with DOD's
approval, the agency did not develop a performance contract or plan
during fiscal year 2002. DODEA officials decided to use a performance
plan rather than exercise their option to develop a balanced scorecard
for fiscal year 2004. DISA submitted a sample draft balanced scorecard
to the Office of the Secretary of Defense, Program Analysis and
Evaluation in September 2003 in order to meet reporting requirements.
[End of table]
Although we obtained DISA's draft balanced scorecard dated September
2003, we did not evaluate it against certain key attributes associated
with results-oriented management that we discuss in this report because
the scorecard had not been finalized by the time we completed our
review. However, we do discuss DISA's plans for using a scorecard and
addressing the attributes, as applicable. Our objectives were to
evaluate:
(1) the extent to which the defense agencies initially used performance
contracts to manage and measure agency performance, including whether
the contracts addressed certain key attributes associated with results-
oriented management, such as performance measurement and reporting;
(2) the status of efforts by the defense agencies to implement
performance plans, including the extent to which these plans reflect
any lessons learned from DOD's experience with the initial performance
contracts and address certain key attributes associated with results-
oriented management; and:
(3) the extent to which DOD has established mechanisms to capture and
share lessons learned from the implementation of performance plans.
To address these objectives, we collected data from and interviewed
knowledgeable OSD, defense agency, and service officials about the use
of performance contracts, plans, and balanced scorecards, as
applicable, at DLA, DISA, and DODEA. We obtained these agencies'
performance contracts, plans, and scorecards, as applicable, and
analyzed them against certain key attributes associated with results-
oriented management that we identified in our prior reports. We did not
review actual or reported agency performance, only the content of the
tools and how they were used. We also did not validate procedures that
the defense agencies have in place to ascertain the reliability of data
used in their performance plans and scorecards. We conducted our review
between July 2003 and May 2004 in accordance with generally accepted
government auditing standards. Details about our scope and methodology
appear in appendix I.
Results in Brief:
Between fiscal year 1999 and 2003, DLA, DISA, and DODEA--the three
agencies we reviewed--initially used performance contracts to varying
degrees to identify performance issues, elevate management challenges
and customer concerns to the attention of senior OSD and defense agency
leaders, and take corrective action. The contracts were an important
step toward more results-oriented management practices for the defense
agencies. However, the contracts would have been more effective tools
for informing decision making and managing and measuring agency
performance had they included additional attributes associated with
results-oriented management. The contracts showed progress made and
encouraged discussion among top agency and OSD leaders about actions
needed to achieve targeted results, particularly in terms of quality,
quantity, cost, and timeliness of services. For example, DODEA
officials told us that the agency's fiscal year 2003 performance
contract included a measure to achieve optimum pupil to teacher ratios
that influenced DOD to fund the hiring of 200 full-time teachers.
However, DOD guidance did not require performance contracts to include
certain key attributes that we have identified in prior reports that
are associated with results-oriented management. While other management
documents may reflect some of these attributes, it is important that
they also be reflected in the contracts to provide a comprehensive
picture of goals and performance. Consequently, the performance
contracts could have been strengthened by including the following
attributes:
* Aligns goals and measures with agency and departmentwide goals.
Performance goals and measures for some of the agencies we reviewed
only partially aligned with agency and departmentwide goals and none of
the contracts cascaded goals and measures to lower organizational
levels.
* Assigns accountability for achieving results. Individuals at lower
organizational levels accountable for accomplishing specific goals and
taking corrective actions were not identified.
* Demonstrates results. Contracts were inconsistent in using a
combination of clearly defined output and outcome goals accompanied by
trend data.
* Provides a comprehensive view of agency performance. With the
exception of DODEA, the contracts we reviewed lacked some measures that
could have provided officials with a comprehensive view of factors that
drive organizational performance, such as financial, customer, and
internal business processes. The contracts, to some extent, did reflect
the different concerns of customers and stakeholders.
* Links resource needs to performance. Contracts did not discuss the
relationship between resource needs and performance outcomes.
* Discusses data quality. Contracts did not discuss whether data were
reliable, valid, or verifiable, or identify data sources.
* Provides contextual information. Contracts lacked some contextual
information to help officials assess the significance of factors that
affected reported performance, such as how performance measures help
achieve mission and departmentwide goals or the impact of contingency
operations on achieving performance goals.
Based on experiences using the initial performance contracts, DOD took
steps to strengthen performance plans and scorecards by revising the
oversight and review process as well as guidance to require that,
beginning with fiscal year 2004, performance measures align with agency
and departmentwide goals and provide a more comprehensive view of
agency performance. However, these tools could have been strengthened
had DOD required the defense agencies to include certain key attributes
associated with results-oriented management. For example, while DLA's
scorecard included a combination of output and outcome measures
accompanied by trend data, it did not include clearly defined measures.
DODEA's performance plan measures also were not consistently clearly
defined and not always accompanied by trend data. The scorecard and
plan we reviewed also lacked other attributes--not required by DOD--
that could be useful to officials for discussing and understanding
competing priorities and resource allocation constraints, assessing and
understanding the extent of progress made against existing performance
targets, making improvements, and establishing realistic performance
goals for future years. Such attributes include:
* identification of specific individuals who are responsible for
achieving results;
* linkage of resource needs to performance;
* discussion of data quality, including data sources, as well as
whether data were reliable, valid, and verifiable; and:
* inclusion of contextual information such as external factors that
could affect performance and relevant information about the agencies'
missions, business lines, and customers.
DOD developed some mechanisms--such as a performance management Web
site and roundtable discussions--to help the defense agencies capture
and share experiences and lessons learned from developing performance
plans and balanced scorecards, and some agencies are proactively
sharing such information with each other. Based on our suggestions
during this review, DOD recognized the need to continue to hold
roundtables to discuss balanced scorecards. A DOD official also told us
that DOD plans to revise and update its performance management Web site
to include defense agencies' performance plans and balanced scorecards
as well as guidance. Some defense agencies, such as DLA and DISA, are
sharing their experiences with each other to leverage lessons learned.
This report contains recommendations to DOD aimed at making performance
plans and scorecards more informative for decision making and further
strengthening the potential of these tools for measuring and managing
agency performance. In commenting on a draft of this report, DOD
generally concurred with our recommendations.
Background:
Evolution of Performance Contracts:
In 1997, the Secretary of Defense issued the Defense Reform Initiative
report outlining a plan for reforming the department's business
operations. DOD established a management oversight structure to help
sustain the direction and emphasis of these efforts. As part of its
efforts to improve its business operations, DOD began requiring
selected defense agencies to develop annual performance contracts for
fiscal year 1999. These contracts were internal management agreements
between the principal staff assistants--also known as Under and
Assistant Secretaries--in the Office of the Secretary of Defense, the
Deputy Secretary, and the defense agencies. DOD intended the
performance contracts to improve oversight of the defense agencies by
(1) identifying programming and budgeting issues, (2) providing agency
directors with clear objectives and performance targets, (3)
establishing open and direct communications with customers to, among
other things, demonstrate how their actions affect agency performance,
and (4) changing the way DOD does business. The contracts included
improvement goals for the agencies in terms of cost, productivity,
quality, and responsiveness to customers. DOD also required the defense
agencies to assess their progress toward achieving performance contract
goals in annual performance reports. A Defense Management Council--
chaired by the Deputy Secretary and consisting of key civilian and
military leaders--was created to oversee Defense Reform Initiative
efforts, and a Defense Agency Task Force--chaired by the Deputy
Director of Resource Analysis in the Office of the Secretary of
Defense, Program Analysis and Evaluation (OSD/PA&E) and consisting of
senior-level representatives from the Under Secretaries of Defense, the
service secretaries, and the Joint Staff--was formed to oversee the
development and review of performance contracts.
DOD initially selected 4 defense agencies to implement performance
contracts during fiscal year 1999, and gradually expanded the
requirement to a total of 10 agencies by fiscal year 2002. At that
time, DOD issued guidelines that established a standard format for
performance contracts and recommended that certain attributes
associated with results-oriented management be included. DOD initially
excluded defense agencies with intelligence-or research-related
missions from the requirement because officials believed it would be
more appropriate to focus primarily on certain agencies with
businesslike missions and develop lessons learned based on their
experiences. The 4 agencies DOD initially selected were:
* the Defense Logistics Agency, a revolving fund activity;
* the Defense Finance and Accounting Service, a businesslike agency;
* the Defense Contract Audit Agency, a small and appropriated fund
agency; and:
* the Defense Health Program.
As part of the current administration's focus on business
transformation, DOD issued guidance in February and August 2003 that
refined its approach to performance management and renamed the
contracts as performance plans. These plans, first implemented in
September 2003, build on the performance goals and measures established
in the performance contracts and are intended to include a more
comprehensive set of performance measures that align with agency
strategic plans and departmentwide goals set forth in DOD's risk
management framework. DOD uses the risk management framework,
established in the 2001 Quadrennial Defense Review Report, to consider
trade-offs among defense objectives and resource constraints. The
framework consists of four dimensions of risk:
* Force management--the ability to recruit, retain, train, and equip
sufficient numbers of quality personnel and sustain the readiness of
the force while accomplishing its many operational tasks;
* Operational--the ability to achieve military objectives in a near-
term conflict or other contingency;
* Future challenges--the ability to invest in new capabilities and
develop new operational concepts to dissuade or defeat mid-to long-term
military challenges; and:
* Institutional--the ability to develop management practices and
controls that use resources efficiently and promote the effective
operation of the defense establishment.
These risk areas form the basis for DOD's annual performance goals and
for tracking associated performance results. Appendix II depicts DOD's
risk management framework.
DOD gave the defense agencies the option to use a balanced scorecard to
develop performance goals and report on progress. Currently, according
to an official in OSD/PA&E, DOD requires 10 of 27 defense agencies with
businesslike missions to have either performance plans or balanced
scorecards. Five defense agencies use performance plans, and the
remainder have exercised the option to develop balanced scorecards. In
addition, this official told us that 3 intelligence agencies have
developed performance plans in order to demonstrate how they are
coordinating with DOD on strategic and budgetary planning in the post-
September 2001 environment. Appendix III depicts the first fiscal year
when each defense agency began using performance contracts, plans, or
scorecards.
DOD Organizations Responsible for Development of Performance Contracts
and Plans:
The Director, PA&E, has responsibility for overseeing the development
of defense agency performance plans, including reviewing the progress
of each defense agency. Specifically, this official oversees the
development of defense agency performance plans and balanced
scorecards, provides guidance for interpreting and applying DOD's risk
management framework, and, in conjunction with the relevant Under or
Assistant Secretary of Defense, reviews the progress of each agency in
building its performance plan or scorecard and reporting on its overall
performance. The Under and Assistant Secretaries of Defense are
responsible for overseeing the defense agencies and implementing
performance plans and balanced scorecards.
Our Prior Work Identifies Attributes Associated with Results-Oriented
Management:
Measuring performance allows organizations to track the progress they
are making toward their goals and gives managers critical information
on which to base decisions for improving their programs. Our prior work
on results-oriented management in the federal government indicates that
agencies that are successful in measuring performance and achieving
business transformation goals strive to establish goals and measures at
all levels of the agency that:
* align goals and measures with the agency's strategic plan and
departmentwide goals;
* assign accountability for achieving results;
* demonstrate results;
* provide a comprehensive view of agency performance;
* link resource needs to performance;
* discuss data quality, including reliability and data sources; and:
* provide contextual information to help officials evaluate the
significance of underlying factors that may affect reported
performance.
While these may not cover all attributes associated with results-
oriented management, we feel they are some of the most important for
accurately assessing the strengths and weaknesses of programs and
making improvements. Without these attributes, decision makers may not
have as complete information for measuring and managing an agency's
performance as needed. Each of these attributes is discussed in
appendix IV.
While Used to Varying Degrees, Performance Contracts That We Reviewed
Could Have Been More Effective Had They Included Certain Attributes
Associated with Results-Oriented Management:
Between fiscal year 1999 and 2003, DLA, DISA, and DODEA--the three
agencies we reviewed--used performance contracts to varying degrees to
identify performance issues, demonstrate progress made, and encourage
discussions to improve agency performance. The contracts were an
important step toward more results-oriented management practices for
the defense agencies. However, the contracts could have been more
effective had DOD guidance required the defense agencies to include
certain attributes associated with results-oriented management. For
example, the contracts addressed key business lines, but did not
consistently include information that could have helped demonstrate
results or provide some contextual information that could help
officials assess progress and understand the significance of underlying
factors that may affect reported performance.
Defense Agencies Used Performance Contracts to Varying Degrees:
Officials at the three defense agencies we reviewed used performance
contracts to varying degrees as part of their strategic planning
systems to demonstrate progress made and increase visibility over the
quality, quantity, cost, and timeliness of products and services for
their key business lines. In some instances, the contracts helped OSD
and defense agency officials to identify developing problems and assess
the effectiveness of corrective actions, such as justifying additional
funding to achieve performance goals and implementing or revising
internal processes to meet customer expectations. We did not verify the
actual or reported performance data included in the performance
contracts or in these examples. The agencies we reviewed provided the
following examples showing how they used performance contracts.
DLA officials told us that they used their performance contracts to
focus management attention and monitor efforts to improve the agency's
response time for providing logistics commodities, such as clothing and
textile items, medical and pharmaceutical supplies, and weapon system
spare parts. In its fiscal year 2001 performance contract, DLA
established a performance measure for its supply centers to process a
greater percentage of requisitions within shorter time frames. DLA
reported that by increasing emphasis on logistics response time goals
to officials in its Ohio-based Defense Supply Center and dedicating
more trucks to its Defense Distribution Center, the agency had
processed 96 percent of requisitions of commodities in 5.8 days during
fiscal year 2001 rather than the targeted 9.3 days.
A DISA official told us that the performance contracts helped DISA
reduce the cost of providing services in response to customer concerns.
For example, DISA developed cost-related performance measures for
delivering computing services, such as "Provide mainframe information
processing services while incurring no more than the unit costs
listed." As a result of increased management attention, DISA reported
in its fiscal year 2002 annual performance report, which assessed
progress against contract goals, that it had reduced the cost per unit
for providing OS/390 mainframe processing services by 24 percent, from
$38.26 in fiscal year 2001 to $29.04 in fiscal year 2002. DISA
officials told us that the high-level attention made possible by the
performance contracts helped ensure that cost reduction goals were
established and pursued.
DODEA officials told us that in order to meet OSD requirements, they
developed and submitted performance contracts that included measures
taken directly from their community strategic plan. However, they
primarily used their community strategic plan--containing strategic
goals, objectives, and performance measures--rather than the
performance contract, which mirrored the strategic plan, to manage and
measure systemwide school performance. We determined that DODEA's
contract did not show the relationship between resource needs and
performance, but performance data helped bring attention to initiatives
and programs needing additional funding and provided support for
funding requests. For instance, DODEA officials noted that a measure in
DODEA's fiscal year 2003 performance contract influenced DOD's decision
to provide $114 million to hire 200 full-time teachers so DODEA could
achieve lower pupil to teacher ratios and, thereby, further improve
student performance.
Contracts Did Not Fully Incorporate Certain Key Attributes:
Although useful in some respects, the performance contracts for the
agencies we reviewed would have been stronger tools if they had fully
incorporated certain key attributes that are associated with results-
oriented management and that contribute to effective government
performance. In previous reports, we identified seven such attributes.
However, the contracts we reviewed incorporated aspects of these
attributes to varying degrees. We found that the contracts did not
identify individuals accountable for achieving results or clearly
demonstrate relationships between resource needs and performance. We
also determined that the contracts, with the exception of DODEA, did
not provide officials with a comprehensive view of agency performance,
including factors that drive organizational performance, such as
financial, customer, and internal business processes. Furthermore,
defense agency contracts were inconsistent in demonstrating results by
providing trend data to show agency progress in meeting established
goals over time, and in discussing data quality issues and providing
contextual information to help officials understand the measures used.
We based our analysis on the most recent performance contracts
developed by DLA, DISA, and DODEA because DOD officials told us that
these would reflect lessons learned from prior iterations. Our analysis
did not include other internal management documents, such as strategic
plans, that may have addressed these attributes. We also did not
evaluate the usefulness or appropriateness of the measures themselves.
Table 2 summarizes the extent to which the performance contracts we
reviewed included the key attributes we have identified.
Table 2: Extent To Which DLA's, DISA's, and DODEA's Performance
Contracts Incorporated Certain Key Attributes Associated with Results-
Oriented Management:
Aligns goals and measures with agency and departmentwide goals;
Attribute: Each of the agency's strategic plan goals is supported by
performance measures;
DLA FY 2001: Partially incorporated;
DISA FY 2002: Partially incorporated;
DODEA FY 2003: Partially incorporated.
Aligns goals and measures with agency and departmentwide goals;
Attribute: Measures align with departmentwide goals;
DLA FY 2001: Partially incorporated;
DISA FY 2002: Partially incorporated;
DODEA FY 2003: Fully incorporated.
Aligns goals and measures with agency and departmentwide goals;
Attribute: Goals and measures cascade from the corporate level to the
lowest level of the agency [A];
DLA FY 2001: Not incorporated;
DISA FY 2002: Not incorporated;
DODEA FY 2003: Not incorporated.
Assigns accountability for achieving results;
Attribute: Establishes a foundation to hold top leadership accountable
for achieving results;
DLA FY 2001: Fully incorporated;
DISA FY 2002: Fully incorporated;
DODEA FY 2003: Fully incorporated.
Assigns accountability for achieving results;
Attribute: Performance appraisals for top agency leadership reflect
performance goals and measures;
DLA FY 2001: Partially incorporated[B];
DISA FY 2002: Not incorporated;
DODEA FY 2003: Fully incorporated.
Assigns accountability for achieving results;
Attribute: Identifies individuals accountable for achieving results;
DLA FY 2001: Not incorporated;
DISA FY 2002: Not incorporated;
DODEA FY 2003: Not incorporated.
Demonstrates results;
Attribute: Includes a combination of output-and outcome-oriented
measures;
DLA FY 2001: Partially incorporated;
DISA FY 2002: Partially incorporated;
DODEA FY 2003: Fully incorporated.
Demonstrates results;
Attribute: Measures are clearly defined;
DLA FY 2001: Partially incorporated;
DISA FY 2002: Partially incorporated;
DODEA FY 2003: Partially incorporated.
Demonstrates results;
Attribute: Provides trend data to demonstrate progress against
targeted performance;
DLA FY 2001: Fully incorporated;
DISA FY 2002: Partially incorporated;
DODEA FY 2003: Partially incorporated.
Provides comprehensive view of agency performance;
Attribute: Measures address factors that drive organizational
performance (financial, customer, and internal business processes and
workforce learning and growth);
DLA FY 2001: Partially incorporated;
DISA FY 2002: Partially incorporated;
DODEA FY 2003: Fully incorporated.
Provides comprehensive view of agency performance;
Attribute: Measures address cost, quality, quantity, and timeliness;
DLA FY 2001: Fully incorporated;
DISA FY 2002: Fully incorporated;
DODEA FY 2003: Fully incorporated.
Provides comprehensive view of agency performance;
Attribute: Measures address key business lines;
DLA FY 2001: Fully incorporated;
DISA FY 2002: Fully incorporated;
DODEA FY 2003: Fully incorporated.
Provides comprehensive view of agency performance;
Attribute: Incorporates customer and stakeholder priorities;
DLA FY 2001: Partially incorporated;
DISA FY 2002: Partially incorporated;
DODEA FY 2003: Partially incorporated.
Links resource needs to performance;
Attribute: Measures identify resources (e.g., human capital and
information technology) needed to achieve performance goals;
DLA FY 2001: Not incorporated;
DISA FY 2002: Not incorporated;
DODEA FY 2003: Not incorporated.
Discusses data quality;
Attribute: Discusses data verification and validation procedures,
including reliability;
DLA FY 2001: Not incorporated;
DISA FY 2002: Not incorporated;
DODEA FY 2003: Not incorporated.
Discusses data quality;
Attribute: Identifies sources of data for each measure;
DLA FY 2001: Not incorporated;
DISA FY 2002: Not incorporated;
DODEA FY 2003: Not incorporated.
Provides contextual information;
Attribute: Provides context or explanation for understanding measures
and reported performance;
DLA FY 2001: Partially incorporated;
DISA FY 2002: Partially incorporated;
DODEA FY 2003: Partially incorporated.
Source: GAO analysis of DLA's, DISA's, and DODEA's performance
contracts.
Notes: "Fully incorporated" indicates that all measures exhibited that
particular attribute. "Partially incorporated" indicates that some of
the measures, but not all, exhibited that attribute, whereas "not
incorporated" means that none of the measures did.
[A] Performance contract measures did not cascade to lower
organizational levels. However, agency officials told us that they
shared strategic goals and objectives from their strategic plans with
all organizational levels and made these available to customers and
stakeholders.
[B] According to a DLA official, performance appraisals for top agency
leadership reflected performance contract measures. However, DLA
officials were not able to provide a standardized performance appraisal
template to demonstrate this.
[End of table]
Additional information about the extent to which the performance
contracts for the agencies we reviewed included these attributes is
discussed below.
Performance Contracts Did Not Always Align with Agency and
Departmentwide Goals:
DOD did not require performance contract measures to align with
agencies' strategic plans or departmentwide goals until fiscal year
2004. As a result, some of the agencies' performance contract measures
did align, while other measures did not. Each agency's performance
contract included statements conceptually linking performance goals
with strategic plans and departmentwide goals. For example, DISA's
fiscal year 2002 performance contract stated that "The performance
measures used in this contract directly support the goals and
objectives in the strategic plan," and DISA's strategic plan directly
related to DOD strategic goals and objectives. We found that all of
DODEA's performance measures aligned with the agency's strategic plan
goals and could be linked to departmentwide goals. With the exception
of the following example, all of DLA's fiscal year 2001 performance
contract measures aligned with the agency's strategic plan goals, but
not all aligned with departmentwide goals. DLA's contract lacked
performance measures that addressed its strategic plan goal to ensure
that the agency's workforce is able to deliver and sustain world-class
performance. Consequently, the performance contract did not provide DLA
and DOD officials with information to help assess recruiting needs or
workforce skills and training requirements that contribute to meeting
customer needs.
Furthermore, performance goals and measures used in the contracts we
reviewed did not cascade from the highest to the lowest levels of the
agencies to reinforce accountability throughout the organization.
Contracts Did Not Assign Accountability for Achieving Results to
Individuals at Lower Organizational Levels:
While defense agency performance contracts identified individuals
accountable for achieving performance results at the most senior levels
of DOD and the defense agencies, they did not identify individuals
responsible for tracking performance or making improvements at lower
organizational levels.
Defense Agency Task Force officials were directed to review the defense
agencies' performance contracts and make recommendations to the Deputy
Secretary of Defense on whether the performance contracts satisfied DOD
guidance. The Defense Management Council was directed to recommend ways
to reduce costs incurred by the defense agencies while improving
customer service. DOD officials told us that after the events of
September 11, 2001, OSD's attention on the performance contracts
temporarily diminished, less strict oversight occurred, and PA&E
officials did not hold agencies to meeting reporting requirements.
However, beginning in April 2002, the Deputy Secretary of Defense began
requiring the Under and Assistant Secretaries of Defense to provide
quarterly updates concerning the defense agencies' performance on key
performance contract measures and the corrective actions to be taken if
performance goals were not expected to be met. DOD officials were not
able to provide historical information concerning whether these reviews
occurred as frequently as required or whether corrective actions were
taken.
Although top leaders had oversight of the performance contracts, none
of the performance contracts we reviewed identified individuals, such
as business managers, responsible for achieving results at lower
organizational levels as a way to reinforce accountability for
achieving results and maintain focus on organizational priorities.
However, in a related internal management report called the Chief
Financial Executive 4th Quarter Fiscal Year 2002 Performance Contract
Report--which DISA used internally to report progress against
performance contract goals--DISA identified business managers
responsible for monitoring each performance measure.
Contracts Did Not Consistently Provide Information to Help Assess
Progress and Demonstrate Results:
Defense agency performance contracts did not consistently provide
information to help assess progress and demonstrate results. Although
the agencies' performance contracts included milestones or time frames
for completing specific initiatives, they differed in the degree to
which measures were clearly defined and outcome-oriented and the extent
to which they included trend data. For instance, we found that some
measures in DODEA's performance contract were not clearly defined. One
example is:
"improve student achievement and education quality consistent with
Presidential and national educational goals. Per pupil costs will not
increase more than 7% over the previous year."
This measure is actually two measures. Furthermore, the measure does
not define the specific presidential and national education goals--such
as the No Child Left Behind Act of 2001--against which the agency
measured its progress.[Footnote 6]
In addition, DODEA and DISA did not consistently report trend data for
each measure in their performance contracts to show progress made over
time, and DLA limited trend data to the prior year. For example, in its
fiscal year 2003 performance contract results, DODEA reported that it
had met goals for teacher certification and school accreditation, but
the agency did not identify the baseline or percentage of teachers
certified or schools accredited.
Contracts Did Not Provide a Comprehensive View of Organizational
Performance:
Each agency's performance contract included measures that addressed key
business lines as well as cost, quality, quantity, and timeliness of
services, but, with the exception of DODEA, did not provide as
comprehensive a view of agency performance as possible because the
measures did not address all drivers of organizational performance,
including customer, financial, and internal business processes, and
workforce learning and growth. Doing so could have helped officials
assess how well the agencies were positioned to perform in the future.
We found that each agency's most recent performance contract included
measures assessing aspects of each key business line. For example,
DLA's fiscal year 2001 contract included 26 measures that supported its
5 business lines, including Defense Automated Printing Service, Defense
Distribution, Defense National Stockpile Center, Defense Reutilization
and Marketing Service, and Supply (Energy and Non-Energy). DISA's
fiscal year 2002 contract included 29 measures that addressed 4
business lines, including Computing Services, Enterprise Acquisition
Services, Joint Warfighting and DOD-wide Enterprise Capabilities, and
Telecommunication Services. DODEA's fiscal year 2003 performance
contract included 15 measures that addressed its 2 business lines--
domestic and overseas education.
While measures addressed the agencies' key business lines as well as
cost, quality, and timeliness of services provided, they did not all
address all drivers of organizational performance. For example, the
contracts did not always reflect the different concerns of customers
and stakeholders. According to defense agency officials, PA&E officials
prescribed certain measures, although agency officials believed these
were not value added and did not align with agency strategic goals. For
instance, PA&E officials required DODEA officials to identify
Management Headquarters Costs--and specifically the "cost per school"-
-in DODEA's performance contract. Agency officials told us that this
measure had limited value because it provided aggregated data rather
than specific information for individual school's costs. Furthermore,
according to a former senior official in the Office of the Secretary of
Defense for Acquisition, Technology, and Logistics, the performance
contracts did not reflect the needs of the chiefs of staff of the
services and the Secretary of Defense, who were key customers and
stakeholders. This official believed that the measures were too
technical and "in the weeds" to help inform strategic management
decisions and were more appropriate for lower-level managers to use
when monitoring program performance.
Linkage between Resource Needs and Performance Outcomes Was Not Clearly
Established:
None of the contracts we reviewed identified the resources needed to
meet performance goals or explained how additional resources could have
contributed to achieving expected levels of performance. Such
information could have helped officials to better understand how trade-
offs among different investments--such as human capital, information
technology, and workforce training--affected performance outcomes and
drove costs. For instance, DLA established a fiscal year 2001 goal of
improving inventory accuracy by 4.2 percent over the previous year's
performance, but it did not provide additional data in its performance
contract that would have helped officials to determine the resources
needed to achieve this goal, if any. DOD officials stated that they
began to establish links between resource needs and performance
outcomes by requiring the defense agencies to submit their performance
contracts at the same time as their Program Objectives
Memorandum.[Footnote 7]
Quality of Data Was Not Discussed:
None of the performance contracts we reviewed discussed the quality of
performance data, including reliability, data sources, or data
verification and validation procedures. Consequently, when reviewing
the performance contracts, top officials in DLA, DISA, and DODEA may
not have had information needed to determine whether reported data and
performance results were accurate, reliable, or current enough to use
in decision making and that corrective actions would produce the
desired outcomes. For example, an October 2000 internal audit initiated
by the DISA Inspector General concluded that data reported in DISA's
1st Quarter Fiscal Year 2000 Chief Financial Executive Report contained
errors or bias and lacked sufficient supporting documentation.[Footnote
8] The Inspector General reported that these conditions occurred
because DISA had not established a systematic approach for collecting
and reporting performance data. The Inspector General concluded that,
as a result, reported data provided limited assurance that DISA had
valid and reliable information to manage agencywide performance. The
Inspector General concluded that DISA should establish a data
collection and reporting process and definitive procedures to ensure
performance information was sufficiently complete, accurate, and
consistent to document actual performance and support decision making.
Senior DISA officials told us that the agency did not initially
implement related recommendations because DISA discontinued use of
performance contracts. However, according to these officials, DISA is
currently taking steps to address these recommendations by setting the
expectation that senior officials will be personally accountable for
the quality of data included in DISA's balanced scorecard and
establishing internal focal points for data collection and reporting.
In addition, DISA officials are reviewing commercial products to track
data electronically.
Contextual Information Could Have Better Informed Decision Making:
The performance contracts provided some contextual information about
some external factors outside of the agencies' control that may have
impacted the achievement of performance goals and provided general
information about the agencies' mission, business lines, and customers.
However, the performance contracts did not always provide some
contextual information to help officials understand each agency's
accomplishments for that fiscal year; determine how measures related to
agency and departmentwide goals; assess agency progress; understand the
impact that external factors, such as contingency operations, could
have on the agency's ability to meet its performance targets; and
determine how resource needs should be prioritized. For example, DISA
reported in its fiscal year 2002 performance contract under the Joint
Warfighting and DOD-wide Enterprise Capabilities business line that it
planned to, "Successfully complete fielding of a DOD standard spectrum
management information system (Spectrum XXI) software version 3.0 by
December 2001." However, it is unclear how this linked specifically to
DISA's strategic plan or DOD's goals or what external factors could
have impacted DISA's intended performance.
The Plan and Scorecard We Reviewed Show Improvement, but Could Be
Strengthened by Including Certain Key Attributes:
In fiscal year 2003, DOD renamed the performance contracts as
performance plans and, based on the agencies' experiences using
performance contracts, revised the oversight and review process and
guidance. DOD required performance measures, beginning in fiscal year
2004, to align with agency and departmentwide goals and provide a more
comprehensive view of agency performance. However, the plan and
scorecard we reviewed could be stronger tools for informing officials
about agency performance if they incorporated some additional key
attributes associated with results-oriented management such as linking
resource needs to performance, discussing data quality, and providing
contextual information for understanding the measures and reported
performance.
DOD Strengthens Oversight Process and Requires Inclusion of Additional
Key Attributes, but Some Are Still Lacking:
Six years after implementation of the performance contracts, the
original intent--to assist senior leaders at the defense agencies and
within OSD to oversee agency performance and to encourage the agencies
to become more responsive to customer needs--remains unchanged.
However, beginning in fiscal year 2004, DOD renamed the performance
contracts as performance plans and gave the defense agencies the option
to use either performance plans or balanced scorecards. These tools
were to build on the performance contracts, and to include additional
attributes associated with results-oriented management that were
previously lacking under the contracts. These changes centered on (1)
revising the oversight and review process to strengthen accountability
for achieving results, (2) requiring performance measures to align with
agency strategic goals and DOD's risk management framework, and (3)
requiring the measures to be more comprehensive in scope to better
demonstrate results.
The defense agencies we reviewed are at different stages of developing
and implementing performance plans and balanced scorecards, and DOD has
required these tools to be in place for less than a year. Consequently,
it is too soon to determine how useful they will be for managing and
measuring agency performance. However, DLA's balanced scorecard and
DODEA's performance plan showed progress in incorporating many of the
attributes associated with results-oriented management that we have
identified, but because DOD did not require the defense agencies to
include all of the attributes discussed in this report, the plan and
scorecard we reviewed continue to lack certain information--such as the
relationship between resource needs and performance, discussions of
data quality, and some contextual information--that could strengthen
these tools for managing defense agency performance. Although we
obtained DISA's draft balanced scorecard dated September 2003, we did
not evaluate it against the key attributes discussed in this report
because it had not been finalized by the time we completed our review.
However, we do discuss DISA's plans for addressing several of the key
attributes in the balanced scorecard that the agency is developing.
Table 3 shows the extent to which DLA's balanced scorecard and DODEA's
performance plan incorporated certain key attributes associated with
results-oriented management. We based our analysis specifically on the
performance plan and balanced scorecard, although other agency internal
management documents may have addressed these attributes. We also did
not evaluate the usefulness or appropriateness of the measures
themselves.
Table 3: Extent To Which DLA's Scorecard and DODEA's Plan Incorporated
Certain Key Attributes Associated with Results-Oriented Management:
Aligns goals and measures with agency and departmentwide goals;
Attribute: Each of the agency's strategic plan goals is supported by
performance measures;
DLA FY 2004: Fully incorporated;
DODEA FY 2004: Partially incorporated.
Aligns goals and measures with agency and departmentwide goals;
Attribute: Measures align with departmentwide goals;
DLA FY 2004: Fully incorporated;
DODEA FY 2004: Fully incorporated.
Aligns goals and measures with agency and departmentwide goals;
Attribute: Goals and measures cascade from the corporate level to the
lowest level of the agency [A];
DLA FY 2004: Fully incorporated;
DODEA FY 2004: Not incorporated[A].
Assigns accountability for achieving results;
Attribute: Establishes a foundation to hold top leadership accountable
for achieving results;
DLA FY 2004: Fully incorporated;
DODEA FY 2004: Fully incorporated.
Assigns accountability for achieving results;
Attribute: Performance appraisals for top agency leadership reflect
performance goals and measures;
DLA FY 2004: Fully incorporated;
DODEA FY 2004: Fully incorporated.
Assigns accountability for achieving results;
Attribute: Identifies individuals accountable for achieving results;
DLA FY 2004: Fully incorporated;
DODEA FY 2004: Not incorporated.
Demonstrates results;
Attribute: Includes a combination of output-and outcome-oriented
measures;
DLA FY 2004: Fully incorporated;
DODEA FY 2004: Fully incorporated.
Demonstrates results;
Attribute: Measures are clearly defined;
DLA FY 2004: Not incorporated;
DODEA FY 2004: Partially incorporated.
Demonstrates results;
Attribute: Provides trend data to demonstrate progress against
targeted performance;
DLA FY 2004: Fully incorporated;
DODEA FY 2004: Partially incorporated.
Provides comprehensive view of agency performance;
Attribute: Measures address factors that drive organizational
performance (financial, customer, and internal business processes and
workforce learning and growth);
DLA FY 2004: Fully incorporated;
DODEA FY 2004: Fully incorporated.
Provides comprehensive view of agency performance;
Attribute: Measures address cost, quality, quantity, and timeliness;
DLA FY 2004: Fully incorporated;
DODEA FY 2004: Fully incorporated.
Provides comprehensive view of agency performance;
Attribute: Measures address key business lines;
DLA FY 2004: Fully incorporated;
DODEA FY 2004: Fully incorporated.
Provides comprehensive view of agency performance;
Attribute: Incorporates customer and stakeholder priorities;
DLA FY 2004: Partially incorporated;
DODEA FY 2004: Partially incorporated.
Links resource needs to performance;
Attribute: Measures identify resources (e.g., human capital and
information technology) needed to achieve performance goals;
DLA FY 2004: Not incorporated;
DODEA FY 2004: Not incorporated.
Discusses data quality;
Attribute: Discusses data verification and validation procedures,
including reliability;
DLA FY 2004: Not incorporated;
DODEA FY 2004: Not incorporated.
Discusses data quality;
Attribute: Identifies sources of data for each measure;
DLA FY 2004: Not incorporated;
DODEA FY 2004: Not incorporated.
Provides contextual information;
Attribute: Provides context or explanation for understanding measures
and reported performance;
DLA FY 2004: Not incorporated;
DODEA FY 2004: Partially incorporated.
Source: GAO analysis of DLA's, DISA's, and DODEA's performance contracts.
Notes: "Fully incorporated" indicates that all measures exhibited that
particular attribute. "Partially incorporated" indicates that some of
the measures, but not all, exhibited that attribute, whereas "not
incorporated" means that none of the measures did.
[A] Performance measures did not cascade to lower organizational
levels. However, DODEA officials told us that they shared strategic
goals and objectives from their strategic plans with all organizational
levels and made these available to customers and stakeholders.
Additional information about the extent to which DLA's balanced
scorecard and DODEA's performance plan include these attributes is
discussed on the following pages.
Varying Progress Made in Aligning Performance Goals and Measures with
Agency and Departmentwide Strategic Goals and Cascading These to Lower
Organizational Levels:
The balanced scorecard and performance plan we reviewed showed varying
progress in meeting DOD's requirement to align performance goals and
measures with agency strategic plan goals and DOD's risk management
framework beginning in fiscal year 2004. However, DLA and DODEA
fulfilled DOD's requirement that every organization's performance goals
and measures be consistent with those of its next higher and lower
organizations. DLA and DODEA have adopted different strategies for
cascading their scorecard and plan, respectively. DLA's corporate-level
balanced scorecard aligns with its strategic plan goals to transform
DLA's operations over the next 2 to 5 years to better support its
customers and departmentwide performance measures outlined in DOD's
risk management framework. DLA has also cascaded its corporate
scorecard to each of its Defense Supply Centers located in Richmond,
Virginia, Columbus, Ohio, and Philadelphia, Pennsylvania, as well as
other field activities including the Defense Energy Support Center,
Defense Distribution Center, and Defense Reutilization and Marketing
Service. Each center's or field activity's scorecard identifies "how
and to what extent" their strategies support DLA's corporate-level
balanced scorecard and is expected to reflect DLA's overall vision of
"Right Item, Right Time, Right Place, Right Price, Every Time—Best
Value Solutions for America's Warfighters." According to DLA officials,
in this way, DLA provides a consistent vision to its customers,
stakeholders, and employees. They told us that this process also allows
customers, stakeholders, and employees to understand how corporate-
level strategies are supported by functional, day-to-day measures.
Figure 1 depicts how DLA's corporate-level scorecard aligns with
departmentwide goals and measures and how the Defense Supply Center
Richmond's internal scorecard supports these goals. DLA officials told
us that the goals within the institutional risk area of DOD's risk
management framework cascade down to the Under Secretary of Defense for
Acquisition, Technology, and Logistics' balanced scorecard. This
scorecard has four strategies that cascade down and inform DLA's
scorecard. In turn, the four strategies in DLA's corporate-level
scorecard cascade down to its defense supply centers. Figure 1 depicts
specifically how the three strategies within the customer portion of
DLA's scorecard cascade down to the Defense Supply Center Richmond. The
supply center has developed four measures that it uses to assess its
performance in terms of customer service on a monthly basis.
Figure 1: Depiction of How DLA's May 2004 Scorecard Supports
Departmentwide Goals and Cascades to Lower Organizational Levels:
[See PDF for image]
[End of figure]
DODEA's fiscal year 2004 performance plan does not include goals and
measures that support all of its community strategic plan goals and
objectives such as (1) developing, promoting, and maintaining a network
of meaningful partnerships and alliances to enhance social, emotional,
and academic growth and (2) maximizing resources or developing safe,
well-managed, and disciplined environments conducive to learning.
However, DODEA has linked each performance measure in its performance
plan with the risk management framework area that it supports. For
example, DODEA linked the measure "In fiscal years 2004-2009, no less
than 95 percent of the Department of Defense Domestic Dependent
Elementary and Secondary Schools teachers will be professionally
certified in the subject area and grade level to which they are
assigned within three years of employment" with force management risk,
and specifically the departmentwide goal to maintain a quality
workforce. DODEA officials told us that they continue to communicate
these goals and performance measures to employees and customers
systemwide through their community strategic plan, and school
improvement plans reflect these goals, as relevant.
Oversight and Review Process Strengthened to Reinforce Accountability:
DOD has strengthened the accountability of OSD and top agency
leadership for achieving performance results through the development of
OSD-level scorecards and semiannual reviews of defense agency
performance, and the agencies we reviewed are taking steps to
strengthen accountability at lower organizational levels. However,
DODEA's performance plan does not identify specific individuals who are
responsible for making improvements. The Under and Assistant
Secretaries of Defense continue to have responsibility for reviewing
the defense agencies' performance plans or scorecards and providing
direct feedback. However, some have developed balanced scorecards--such
as the Under Secretary of Defense for Personnel and Readiness (USD/
P&R)--and have taken steps to incorporate data from quarterly reviews
of agency performance measures in their balanced scorecards while
continuing to provide feedback to agency officials on a regular basis.
For example, prior to approving DODEA's fiscal year 2004 performance
plan, the Deputy Under Secretary of Defense for Military Community and
Family Policy within USD/P&R asked the agency to incorporate updated
information on its customer satisfaction survey and to explain annual
variations in the cost-per-pupil index. OSD and agency officials also
review performance data during periodic internal meetings. For example,
figure 2 shows the high-level DODEA performance measures that officials
within USD/P&R monitor on a quarterly basis in their balanced scorecard
called Monitoring the Status of the Force. Part A: Balanced Scorecard.
USD/P&R officials are responsible for monitoring the force management
risk area, and within this area, performance strategies related to
"maintain workforce satisfaction." Specifically, these officials
monitor DODEA's progress in meeting three performance contract
measures.
Figure 2: Example of the Type of Information in DODEA's Performance
Plan That OSD Officials Monitor, as of May 2004:
[See PDF for image]
[End of figure]
Additional details about each performance measure are also available to
USD/P&R officials, as depicted in figure 3.
Figure 3: Detailed Data on DODEA's Pupil to Teacher Ratio as of May
2004:
[See PDF for image]
[End of figure]
DOD officials told us that the Office of the Under Secretary of Defense
for Acquisition, Technology, and Logistics that oversees DLA and the
Assistant Secretary of Defense for Network Information and Integration
that oversees DISA are developing balanced scorecards that are expected
to incorporate certain performance measures from DLA's and DISA's
balanced scorecards, respectively, in order to help maintain oversight
and monitor the agencies' progress toward achieving specific
performance goals.
DOD has also implemented semiannual performance reviews at which time
the defense agencies brief PA&E officials, respective Under and
Assistant Secretaries of Defense, and agency officials on progress to
date against performance goals. DOD began holding midyear reviews
during May 2004, and officials expect to complete these reviews by
August. A DOD official told us that in the future, the first cycle
reviewing the prior year's performance will occur between September and
December and the second cycle, the midyear review, will occur between
May and August. DOD officials expect results from these meetings to be
reflected in the agencies' fiscal year 2005 performance plans and
scorecards. During a May 2004 midyear review of DLA's balanced
scorecard, the Director, PA&E, recommended that the agency include more
specific information on its customer surveys, including the population
surveyed and the way customer feedback is obtained.
By assigning responsibility for each performance measure to specific
individuals, some defense agencies are reinforcing accountability for
achieving performance goals at all levels of the organization. For
example, in addition to implementing performance appraisals for Senior
Executive Service staff that are linked to strategic goals and
objectives against which ratings are assessed, DLA identified
individuals in its balanced scorecard and charged them with monitoring,
reporting on, and implementing corrective actions for each scorecard
measure. A senior DISA official told us that the agency recently
approved linking balanced scorecard outcomes with Senior Executive
Service members' and supervisory employees' performance appraisals
beginning in fiscal year 2005 in order to institutionalize its balanced
scorecard and improve accountability for results. DISA also is
reinforcing accountability by requiring its senior leaders (scorecard
owners), who are accountable for achieving results, to provide status
updates during corporate-board meetings of senior agency officials. In
contrast, DODEA's performance plan does not identify specific
individuals accountable for monitoring, reporting on, and implementing
corrective actions for each performance measure. However, DODEA
officials told us that the agency includes performance plan objectives
in each manager's performance appraisal, and both monetary and
nonmonetary awards are used as incentives to meet or exceed goals and
measures.
Agencies' Progress in Demonstrating Results Varies:
DLA's balanced scorecard and DODEA's performance plan showed varying
progress in the extent to which they included outcome-oriented measures
and trend data to help demonstrate results, and the clarity of
performance measures continues to be an issue. We determined that DLA's
2004 scorecard includes outcome-oriented measures and trend data that
help to demonstrate progress made. For example, one of DLA's outcome-
oriented financial measures--the achievement of accuracy in non-energy
sales forecasting--includes baseline and trend data for the 3 preceding
fiscal years. However, DLA's 2004 scorecard measures include terms that
are generally not clearly defined, as demonstrated by the measure
"Percentage of reengineered functionality implemented in BSM." In
contrast, DODEA's performance plan includes almost all of the same
measures as did its fiscal year 2003 performance contract, and these
measures are not always clearly defined or accompanied by trend data.
As we previously mentioned, DODEA does not clearly define the specific
presidential and national education goals against which it measures
progress. Furthermore, DODEA's performance plan does not consistently
include baseline information or trend data to help officials gauge
progress. DODEA did not indicate the percentage of teachers who are
certified in the subject area and grade level to which they are
assigned within 3 years of employment, although the agency set a goal
of no less than 95 percent. The plan also does not indicate the
percentage of schools that have full high-speed connectivity to the
Internet, although it establishes a goal of 100 percent.
Agencies' Scorecard and Plan Show Progress in Providing a Comprehensive
View of Agency Performance:
Our analysis showed that DLA's scorecard and DODEA's performance plan
included measures for key business lines that provide officials with
information about agency performance and, to varying degrees, address
the four drivers of organizational performance as well as cost,
quality, and timeliness of services provided, and that all of the
agencies we reviewed are taking steps to better reflect customers' and
stakeholders' priorities.
DLA organizes its corporate-level scorecard by drivers of performance-
-including learning and growth and customer, financial, and internal
business processes--and includes strategies and supporting measures
that are cascaded to its defense supply centers and field activities.
As would be expected, lower organizational level scorecards used by the
defense supply centers include performance measures that address the
key business lines for which they are responsible and provide more
specific assessments of cost, quality, timeliness, and customer
satisfaction. For example, the Defense Supply Center Richmond's
scorecard includes performance measures focusing on aviation weapons
systems and environmental logistics support, whereas the Defense Supply
Center Philadelphia's scorecard focuses on the provision of clothing,
textiles, and medical equipment, among other items. The learning and
growth quadrant of DLA's corporate scorecard contains a measure to
identify gaps between workforce competencies and skills required to
meet mission requirements for 100 percent of DLA positions by the end
of fiscal year 2007. The Defense Supply Center Richmond's scorecard
includes several measures that address employee certification,
including acquisition training certification to ensure that the supply
center's workforce is fully proficient in the acquisition process. This
measure, in turn, aligns with the departmentwide policy of establishing
education and training for each defense acquisition position.
Similarly, DODEA's plan includes measures that address its two key
business lines--domestic and overseas education--and are cascaded to
schools worldwide through its community strategic plan.
Unlike DLA's balanced scorecard, DODEA's performance plan does not
specifically categorize performance measures by the factors that drive
organizational performance, but we identified at least one measure or
initiative that can be linked with each organizational driver. For
example, DODEA's performance plan includes a measure to limit increases
in per pupil cost to no more than 7 percent over the prior fiscal year.
This measure addresses both financial and internal business processes
that affect organizational performance. DODEA addresses workforce
learning and growth by establishing a goal to certify no less than 95
percent of teachers in the subject and grade levels to which they are
assigned within 3 years of employment. Finally, DODEA includes measures
that address customer satisfaction such as students' performance and
parents' satisfaction with local schools.
Our analysis showed that the defense agencies we reviewed are also
taking steps to better address specific customer and stakeholder
concerns about performance. Both DODEA and DLA conduct periodic reviews
of agency performance and make adjustments to meet customers' and
stakeholders' priorities. For instance, DLA's balanced scorecard
Executive Board, comprised of DLA's Vice Director, designated leaders
for each area of organizational performance (quadrant leaders), the
Chief Information Officer, and a Lead Center representative, conducts
monthly reviews of enterprisewide operational measures. Each quarter,
DLA's Corporate Board, comprised of DLA's Director, Vice Director, and
the heads of DLA's major organizational components, meets with
commanders from the field to review the agency's performance from the
corporate level down to the supply centers. The defense supply centers
conduct weekly reviews to monitor progress, identify developing
problems, and take corrective actions. According to DLA officials, to
make information more timely, accessible, and relevant for measuring
and managing the agency, DLA posts the agency's balanced scorecard on
its internal Web site to allow all employees, including agency managers
and leaders, to share performance data. This Web site is also available
to all of DLA's military customers and stakeholders. DLA also continues
to obtain customer and stakeholder feedback through a variety of
mechanisms, such as customer surveys, reports from customer service
liaisons embedded with deployed military service units, and meetings
between the agency's Director and leaders of the military services.
Moreover, DLA is taking steps to address the unique concerns of each
military service through the development of performance-based
agreements that include specific performance measures that are based on
collaborative, direct negotiations between DLA and each military
service. These agreements are intended to create and sustain end-to-end
customer support by aligning DLA's resources with the logistics
solutions that will meet the services' needs and maximize their
effectiveness.
In addition, we determined that the defense combat support agencies--
such as DLA and DISA--are taking steps to incorporate additional
strategies and measures in their balanced scorecards that respond to
customers' changing needs during contingencies and war. For example,
DLA's scorecard assesses readiness by measuring the percentage of items
that the military services identify as requiring surge and sustainment
support (e.g., hardware items, clothing and textiles, and medical
supplies) to ensure appropriate inventory levels to accommodate short
production lead times. DLA officials told us that to reduce procurement
lead time and decrease delivery delays to the agency's primary
customer, the warfighter, the agency has developed 22 long-term
contracts, called strategic supplier alliances, with its major
contractors. According to these officials, having these agreements in
place enhances customer support through the inclusion of better
performance metrics, such as reduced administrative lead times and
increased stock availability. DISA officials told us that as a combat
support agency, DISA's mission is to support the warfighter and that
the strategies and performance measures DISA is developing focus on
wartime requirements but also are applicable to peacetime environments.
For example, DISA is developing measures that will support its customer
strategy to eliminate bandwidth constraints in response to customer
complaints that limited bandwidth is a problem, especially during
contingency operations. According to DISA officials, removing current
bandwidth limitations can provide DISA's customers with better and
timelier information during combat and peacetime.
Funding Not Yet Linked with Performance:
A major goal of DOD is to strategically link major decisions to program
and budget development, and the defense agencies we reviewed are in the
initial stages of attempting to do this. DOD has directed that the
Under Secretary of Defense, Comptroller, and the Director, PA&E, review
program performance and assess the degree to which budget estimates
sustain and improve programmatic results and, as appropriate, recommend
alternate solutions or funding adjustments to correct resource needs if
performance goals are not met. Senior DOD officials told us that this
continues to be a challenging undertaking. The defense agencies we
reviewed are using their performance plans and scorecards to bring
attention to priority management projects. Although these tools still
do not discuss resource implications, DODEA included goals and measures
from its fiscal year 2004 performance plan in its fiscal year 2004-2005
operation and maintenance, defensewide, biennial budget estimates. We
believe that inclusion of the agency's priorities can help officials
better consider appropriate funding levels and trade-offs.
Although DISA is still developing its balanced scorecard, senior agency
officials told us that the agency plans to use the scorecard to bring
visibility and management attention to priority projects in order to
fund them appropriately and to improve collaboration internally. These
officials told us that DISA is in the process of aligning its
investment and scorecard strategies and expects initiatives in its
scorecard to be assigned funding priority. To this end, DISA is
developing a Program Objectives Memorandum that aligns investments with
its scorecard. During corporate-board meetings when senior agency
officials review the agency's performance, DISA requires each measure
owner to discuss the context and status of initiatives, including the
need for additional resources. According to DISA officials, the agency
has already experienced some benefits when testing this approach. DISA
officials identified a disconnect between customer expectations,
performance levels, and funding for an initiative to develop a Web-
based portal to share information about DISA's business activities
internally. The goal of the portal is to help employees perform
critical operations faster and more effectively. Officials told us that
although internal customers identified this initiative as a priority,
the portal did not receive dedicated funding and, as a result, was
neither as robust as planned nor on target to meet development time
frames. DISA officials told us that inclusion of the project in its
balanced scorecard is expected to raise these challenges to the
attention of senior leaders to encourage corrective action and
appropriate funding levels.
Quality of Data Is Not Discussed:
The balanced scorecard and performance plan we reviewed do not provide
decision makers with confidence that performance data are credible
because they do not include specific information on the procedures to
verify and validate performance information. In addition, no
information on the sources of data is included. Thus, decision makers
may not have complete information needed to determine whether reported
data and performance results are accurate.
Some Contextual Information Is Lacking:
The scorecard and plan we reviewed did not include some contextual
information that could have helped to inform decision makers because
DOD guidance did not require the defense agencies to do so. However, a
sample measure from DISA's draft September 2003 balanced scorecard
depicts the type and amount of contextual information that agency
officials have considered incorporating.
DLA's balanced scorecard lacks information about the agency's mission
and customers that had been included in its performance contract. In
contrast, DODEA's performance plan provides general information about
the agency's mission, business lines, and customers. However, neither
agency clearly articulates how its performance strategies and measures
help achieve mission or departmentwide goals or improve customer
responsiveness. Also, information about external factors, such as
recent contingency operations, that can affect achievement of
performance goals is limited or absent. For example, DODEA's
performance plan includes a measure to maintain pupil to teacher ratios
for kindergarten through grade 12 between 18:1 and 24:1 during fiscal
years 2004-2009. However, DODEA's performance plan does not explain
that studies have demonstrated that lower pupil to teacher ratios
contribute to improved student performance or how this measure supports
presidential and national goals against which the agency evaluates
itself. Providing this contextual information could assist officials
with realizing how human capital-related strategies, such as staffing,
affect the agency's goals to continue to improve student performance in
reading, math, and languages, and help establish links between resource
needs and performance outcomes.
In September 2003, DISA developed a draft balanced scorecard with a
sample measure that depicts the type and amount of contextual
information that agency officials have considered incorporating. The
sample measure, as currently structured, incorporates most of the
attributes associated with results-oriented management that we have
discussed in this report, including contextual information. However,
the sample measure does not include some aspects of data quality such
as reliability, validity, and verification, and it is unclear whether
DISA plans to discuss external factors that could affect the
achievement of performance goals or the link between resource needs and
performance beyond funding problems. For example, as currently
structured, the draft scorecard would:
(1) align measures with agency and departmentwide goals. The scorecard
is expected to identify the quadrant of DISA's scorecard that the
measure supports (e.g., customer) and the Assistant Secretary of
Defense's balanced scorecard goal and risk management framework area
with which it aligns.
(2) assign accountability for achieving results. The scorecard is
anticipated to identify individuals responsible for monitoring and
reporting on performance and making corrections, as needed for each
measure.
(3) demonstrate results. The scorecard is expected to define the
measure and present trend data showing performance improvements over 4
fiscal years.
(4) discuss data quality. The scorecard is expected to identify data
sources but does not include information about the way data are
collected or the verification and validation procedures.
(5) provide contextual information. The scorecard is expected to
include historical data, and an assessment of the measure's status
compared to prior years, and identify any challenges that need
attention (e.g., hot issues) such as funding problems.
Figure 4 shows the content that DISA officials told us that they expect
to include and the way this information addresses the attributes we
have identified in this report.
Figure 4: Format of a Sample Measure from DISA's Draft September 2003
Balanced Scorecard:
[See PDF for image]
Notes: PSA=Principal Staff Assistant. TBD=To Be Determined.
[End of figure]
DOD Captures and Shares Lessons Learned:
OSD/PA&E officials have developed some mechanisms to help the defense
agencies capture and share experiences and lessons learned from
developing performance plans and balanced scorecards. Some defense
agencies, such as DLA and DISA, are sharing their experiences with each
other to leverage lessons learned.
OSD/PA&E has taken some steps to help the defense agencies share their
experiences and lessons learned with developing performance plans and
balanced scorecards, and DOD plans additional initiatives. In response
to our suggestions during this review, an OSD/PA&E official decided to
continue to hold semiannual roundtables during which defense agency
officials could voluntarily share their experiences developing and
using performance plans or balanced scorecards. OSD/PA&E officials also
told us that they plan to revise and update a departmentwide Web site
originally used to disseminate agencies' performance contracts and
related guidance and to incorporate this information, as well as
defense agencies' performance plans and scorecards, into DOD's
performance management Web site. The original site was not updated as
frequently as it could have been, and firewalls used to protect
information from unauthorized access precluded defense agencies with
domain names that did not end in ".mil" from gaining access. For
example, DODEA officials were not able to access the original Web site
because the agency's World Wide Web domain address ends in ".edu."
Consequently, DODEA could not leverage other agencies' contracts and
performance measures when developing its own. DODEA officials told us
that they were not even aware that the Web site existed, but that it
would have been helpful for identifying some performance measures.
In addition, some defense agencies that we reviewed are proactively
sharing lessons learned from developing and implementing balanced
scorecards. For example, DISA officials told us that they contacted DLA
to learn how it had developed and implemented its balanced scorecard.
In addition, officials at both agencies told us that they have or are
developing internal Web sites to facilitate performance measurement.
DISA's internal Web site serves as a collaborative forum and repository
for balanced scorecard material, and provides related guidance on
developing balanced scorecards.
Conclusions:
DOD has improved its process for measuring and managing defense agency
performance by taking steps to include certain attributes associated
with results-oriented management in defense agencies' performance plans
and scorecards that were not included in the performance contracts.
Specifically, changes made to DOD's process for overseeing and
reviewing defense agencies' balanced scorecards and performance plans
have the potential to further strengthen the accountability of top
leadership for overseeing the defense agencies and achieving business
transformation goals. While it is too early to determine the extent to
which these plans and scorecards will be fully useful, these tools
still do not incorporate some specific attributes that would strengthen
them. Until all performance plans and scorecards identify individuals
responsible for monitoring progress and taking corrective actions,
DOD's ability to ensure accountability and continuous assessment at
lower organizational levels is hindered. Furthermore, without
performance measures that are clearly defined and accompanied by trend
data and absent a discussion of data quality and contextual
information--such as information about the agency's missions, business
lines, and customers as well as external limitations that could affect
achievement of performance goals--senior OSD and agency leaders will
not have the necessary information for fully assessing the extent to
which progress has been made against existing performance targets and
determining realistic performance goals for future years. Finally,
without demonstrating the relationship between resource needs and
performance outcomes, DOD officials do not have all of the information
needed to make improvements and consider trade-offs among competing
priorities.
Recommendations for Executive Action:
To make performance plans and scorecards more informative and useful
for decision making, and further strengthen these tools' potential for
measuring and managing defense agency performance, we recommend that
the Secretary of Defense direct the Director, Program Analysis and
Evaluation, to revise guidance to require all defense agencies to
address the following five recommendations in their plans and
scorecards:
* identify individuals accountable for achieving results at lower
organizational levels;
* include measures that are clearly defined and include trend data for
at least the past fiscal year's performance to help assess progress;
* identify resources needed to achieve performance goals and inform
budget decisions;
* discuss data quality, including the reliability, validity, and
limitations of performance measures as well as data sources; and:
* provide contextual information to better understand how performance
measures support the agency's mission.
Agency Comments and Our Evaluation:
In written comments on a draft of this report, DOD concurred or
partially concurred with all of our recommendations. DOD's comments are
reprinted in their entirety in appendix V. DOD also provided technical
comments that we incorporated as appropriate.
Specifically, DOD concurred with our recommendations to revise guidance
to require defense agencies to include information on individuals
accountable for achieving results at lower organizational levels;
include measures that are clearly defined and include trend data for at
least the past fiscal year's performance; discuss data quality,
including the reliability, validity, and limitations of performance
measures as well as data sources; and provide contextual information to
better understand how performance measures support the agency's
mission. DOD stated that it would revise guidance to either implement
or clarify these recommendations.
DOD partially concurred with our recommendation to revise guidance to
require all defense agencies to identify resources needed to achieve
performance goals and inform budget decisions in their performance
plans or balanced scorecards. DOD stated that it may not be possible to
fully cost performance goals given the complex nature of defense
operations and that the appropriate process for proposing resource
alternatives is the Planning, Programming, Budgeting, and Execution
System. However, DOD did acknowledge that the defense agencies'
performance plans should be synchronized with this process by
demonstrating how Planning, Programming, Budgeting, and Execution
System decisions are being implemented and by providing performance
information and metrics that can be used to inform the department's
annual budget and execution review and the allocation of resources. We
believe DOD is moving in the right direction by trying to synchronize
defense agency performance management with the budget process. However,
we continue to believe that the defense agencies' performance plans and
balanced scorecards would be an appropriate vehicle to discuss resource
implications. Full cost information may not be available, but the
performance plans and balanced scorecards should discuss resource
implications and include available cost information. Inclusion of this
information would help decision makers understand the connection
between performance goals and funding levels and evaluate competing
needs at all phases of the planning and budgeting process.
We are sending copies of this report to other interested congressional
committees; the Secretary of Defense; the Secretaries of the Army, the
Navy, and the Air Force; the Commandant of the Marine Corps; and the
Director, Office of Management and Budget. We will make copies
available to others upon request. In addition, the report will be
available at no charge on the GAO Web site at http://www.gao.gov.
If you or your staff have any questions regarding this report, please
call Sharon Pickup at (202) 512-4300 or Patricia Dalton at (202) 512-
6806. Additional contacts and staff acknowledgments are provided in
appendix VI.
Signed by:
Sharon L. Pickup:
Director, Defense Capabilities and Management:
Signed by:
Patricia A. Dalton:
Director, Strategic Issues:
[End of section]
Appendix I: Scope and Methodology:
To meet our first two reporting objectives, we interviewed
knowledgeable Department of Defense (DOD), defense agency, and service
officials. Specifically, we interviewed officials in the offices of the
Under and Assistant Secretaries of Defense who act as principal staff
assistants for the three defense agencies reviewed. We also collected
data from and interviewed officials in the Office of the Secretary of
Defense, Program Analysis and Evaluation and the Office of the
Comptroller. We also reviewed our extensive prior work on results-
oriented management and identified seven attributes that contribute to
effective government performance. While these do not cover all
attributes associated with results-oriented management, we believe they
are the most important for accurately assessing the strengths and
weaknesses of programs and making improvements. These include (1)
aligning goals and measures with agency and departmentwide goals; (2)
assigning accountability for achieving results; (3) demonstrating
results; (4) providing a comprehensive view of agency performance by
addressing factors that drive organizational performance; (5) linking
resource needs to performance; (6) discussing data quality; and (7)
providing contextual information. We systematically analyzed the
measures contained in selected performance contracts, performance
plans, and balanced scorecards against these attributes. We focused our
analysis on the Defense Logistics Agency (DLA), the Defense Information
Systems Agency (DISA), and the Department of Defense Education Activity
(DODEA). We judgmentally selected these three defense agencies to
review based on (1) their experience developing and implementing
performance contracts, plans, and balanced scorecards; (2) different
types of missions; and (3) types of services and types of customers. We
analyzed the most recent performance contracts used by DLA, DISA, and
DODEA, and we limited our analysis of balanced scorecards and
performance plans to DLA's fiscal year 2004 balanced scorecard and
DODEA's fiscal year 2004 performance plan. Although we obtained DISA's
draft balanced scorecard dated September 2003, we did not evaluate it
against the attributes we discuss in this report because DISA had not
yet finalized its scorecard by the time we completed our review.
However, we did discuss DISA officials' plans for developing a balanced
scorecard. We did not evaluate the usefulness or appropriateness of the
measures contained in the contracts, plans, and balanced scorecards or
verify actual or reported performance. In addition to analyzing the
performance contracts, performance plans, and balanced scorecards, we
also reviewed the defense agencies' strategic plans and annual
performance contract reports. Further, we reviewed the cascaded-
balanced scorecards that are used at lower organizational levels,
specifically, the fiscal year 2004 scorecards for DLA's supply centers
(Richmond, Virginia, Philadelphia, Pennsylvania, and Columbus, Ohio)
and for its Defense Energy Support Center, Defense Distribution Center,
and Defense Reutilization and Marketing Service. We also reviewed
school improvement plans for three elementary and middle schools
located at Fort Bragg, North Carolina, to determine whether they
aligned with DODEA's community strategic plan and performance contract
goals and measures.
To assess the extent to which DOD has established mechanisms to capture
and share lessons learned from the implementation of performance plans
and balanced scorecards, we interviewed knowledgeable DOD, defense
agency, and service officials and reviewed pertinent documents.
We performed our work between July 2003 and May 2004 in accordance with
generally accepted government auditing standards. We obtained comments
on a draft of this report from DOD and incorporated comments where
appropriate.
[End of section]
Appendix II: Department of Defense's Risk Management Framework:
In 2001, the Department of Defense (DOD) established a risk management
framework to use in considering trade-offs among defense objectives and
resource constraints. The framework consists of four dimensions of risk
with supporting performance measures that are being cascaded to the
defense agencies and activities as well as departmentwide. Although not
depicted, financial considerations underlay each of DOD's risk areas.
Figure 5: DOD's Risk Management Framework:
[See PDF for image]
[End of figure]
[End of section]
Appendix III: Fiscal Year When Defense Agencies Implemented Their First
Performance Contract, Plan, or Scorecard:
This appendix shows the fiscal year during which each defense agency
and activity implemented its first performance contract or performance
plan. The defense agencies and activities that implemented performance
contracts later converted these to performance plans or scorecards. We
use the generic term "performance plan" in figures 6 and 7 to include
both performance plans and balanced scorecards.
Figure 6: Defense Agencies:
[See PDF for image]
[End of figure]
Figure 7: Defense Field Activities:
[See PDF for image]
[End of figure]
[End of section]
Appendix IV: Certain Key Attributes Associated with Results-Oriented
Management:
Establishing goals and measuring performance are essential to
successful results-oriented management practices. Measuring
performance allows organizations to track the progress they are making
toward their goals and gives managers critical information on which to
base decisions for improving their programs. Our body of work on
results-oriented management practices has identified key attributes of
success. This work indicates that agencies that are successful in
achieving business transformation goals, such as those being focused on
by the Department of Defense in its performance contracts and plans,
strive to establish practices and performance systems at all levels of
the agency that include the seven key attributes described in this
appendix.
* Aligns goals and measures with agency and departmentwide goals.
Performance goals and measures should align with an agency's long-term
strategic goals and mission as well as with higher-level departmentwide
priorities, with the relationship clearly articulated. Such linkage is
important in ensuring that agency efforts are properly aligned with
goals (and thus contribute to their accomplishment) and in assessing
progress toward achieving these goals. Goals and measures also should
cascade from the corporate level of the agency to the operational level
to provide managers and staff with a road map that shows how their day-
to-day activities contribute to achieving agency and departmentwide
performance goals. In addition, measures used at the lowest levels of
the agency to manage specific programs should directly relate to unit
results and upwards to the corporate level of the agency.
* Assigns accountability for achieving results. We have previously
reported that the single most important element of successful
management improvement initiatives is the demonstrated commitment of
top leaders in developing and directing reform efforts. Top leadership
must play a critical role in setting results-oriented goals and
quantifiable measures that are cascaded to lower organizational levels
and used to develop and reinforce accountability for achieving results,
maintain focus on the most pressing issues confronting the
organization, and sustain improvement programs and performance,
especially during times of leadership transition. One way to reinforce
accountability is through the use of employee performance appraisals
that reflect an organization's goals.
* Demonstrates results. A key element of any major program is its
ability to establish clearly defined goals and performance measures to
monitor and report progress to management, identify corrective actions,
set priorities, and allocate resources. Performance measures should
show an organization's progress toward achieving an intended level of
performance or results and include quantifiable targets or other
measurable values to help assess progress and trend data toward
accomplishing goals. Using a combination of output measures that assess
the actual level of work accomplished or services provided and outcome
measures that assess the actual results, effects, or impacts of a
program or service compared to the intended purpose can also help
determine progress made.
* Provides a comprehensive view of agency performance. For each key
business line, performance goals and measures should provide a
comprehensive view of performance, including customers' and
stakeholders' priorities. Goals and measures should address key
performance dimensions such as (1) factors that drive organizational
performance--including financial, customer, and internal business
processes, and workforce learning and growth--and (2) aspects of
customer satisfaction, including timeliness, quality, quantity, and
cost of services provided. Doing so can allow managers and other
stakeholders to assess accomplishments, make decisions, realign
processes, and assign accountability without having an excess of data
that could obscure rather than clarify performance issues.
* Links resource needs to performance. One of the ways that performance
management can be promoted is if this information becomes relevant for
(1) identifying resources (e.g., human capital, information technology,
and funding) needed to achieve performance goals; (2) measuring cost;
and (3) informing budget decisions. When resource allocation decisions
are linked to performance, decision makers can gain a better
understanding of the potential effect of budget increases and decreases
on results.
* Discusses data quality. A key factor that contributes to the
usefulness of performance data is the degree to which officials are
confident that the performance information is credible. Useful
practices for helping decision makers to assess the quality and value
of performance data include discussion of data sources and standards
and methods--including data collection and reliability, verification
and validation procedures, and proposals to review these procedures--to
assess the quality of performance data. Reliability refers to whether
standard procedures for collecting and calculating results can be
applied to measures so that they produce the same results if applied
repeatedly to the same situation. Validity is the extent to which data
adequately represent actual performance. Verification is the assessment
of data completeness, accuracy, consistency, timeliness, and related
quality control practices.
* Provides contextual information. Performance reporting systems should
include information to help clarify aspects of performance that are
difficult to quantify or to provide explanatory information such as
factors that were within or outside the control of the agency. This
information is critical to identifying and understanding the factors
that contributed to a particular result and can help officials measure,
assess, and evaluate the significance of underlying factors that may
affect reported performance. In addition, this information can provide
context for decision makers to establish funding priorities and adjust
performance targets and assess means and strategies for accomplishing
an organization's goals and objectives.
[End of section]
Appendix V: Comments from the Department of Defense:
OFFICE OF THE SECRETARY OF DEFENSE:
PROGRAM ANALYSIS AND EVALUATION:
1800 DEFENSE PENTAGON:
WASHINGTON, D.C. 20301-1800:
AUG 25 2004:
Ms. Sharon L. Pickup:
Director, Defense Capabilities and Management:
U.S. Government Accountability Office:
Washington, DC 20548:
Dear Ms. Pickup:
This is the Department of Defense (DoD) response to the GAO draft
report, "DEFENSE MANAGEMENT: Tools for Measuring and Managing Defense
Agency Performance Could Be Strengthened," dated July 26, 2004 (GAO
Code 350410/GAO-04-919).
The Department concurs with four of the recommendations and partially
concurs with one recommendation. Detailed comments on the GAO
recommendations and report are enclosed. Revised guidance will be
forthcoming and will either implement or clarify these recommendations.
Sincerely,
Signed by:
Kenneth J. Krieg:
Enclosure: As stated:
GAO DRAFT REPORT - DATED JULY 26, 2004 GAO CODE 350410/GAO-04-919:
"DEFENSE MANAGEMENT: TOOLS FOR MEASURING AND MANAGING DEFENSE AGENCY
PERFORMANCE COULD BE STRENGTHENED":
DEPARTMENT OF DEFENSE COMMENTS TO THE RECOMMENDATIONS:
RECOMMENDATION 1: The GAO recommended that the Secretary of Defense
direct the Director, Program Analysis and Evaluation, to revise
guidance to require all Defense agencies to include in their
performance plans and scorecards the identification of individuals
accountable for achieving results at lower organizational levels. (Page
35/Draft Report):
DoD RESPONSE: Concur. The Department agrees that the Defense Agencies
not currently including this information should do so in order to
strengthen their respective performance plans. This is in agreement
with the August 22, 2003 DoD guidance that the Defense Agencies should
use the Balanced Scorecard/Risk Management Framework prescribed in
Management Initiative Decision (MID) 901 as the guiding principles for
structuring their performance plans.
RECOMMENDATION 2: The GAO recommended that the Secretary of Defense
direct the Director, Program Analysis and Evaluation, to revise
guidance to require all Defense agencies to include in their
performance plans and scorecards measures that are clearly defined and
include trend data for at least the past fiscal year's performance to
help assess progress. (Page 35/Draft Report):
DoD RESPONSE: Concur. The Department agrees that the Defense Agencies
not currently including this information should do so in order to
strengthen their respective performance plans. This is in agreement
with the August 22, 2003 DoD guidance that the Defense Agencies should
use the Balanced Scorecard/Risk Management Framework prescribed in MID
901 as the guiding principles for structuring their performance plans.
RECOMMENDATION 3: The GAO recommended that the Secretary of Defense
direct the Director, Program Analysis and Evaluation, to revise
guidance to require all Defense agencies to include in their
performance plans and scorecards the identification of resources needed
to achieve performance goals and inform budget decisions. (Page 35/
Draft Report):
DoD RESPONSE: Partially Concur. Given the complex nature of defense
operations, it may not be possible for Defense Agencies to fully cost
performance goals. The authoritative guidance to propose resource
alternatives is the Planning, Programming, Budgeting, and Execution
System (PPBES). However, the Department agrees that the Defense Agency
plans should be in synchronization with this process by demonstrating
how PPBES decisions are being implemented, and by providing the
performance information and metrics that can be used as part of the
Department's annual budget and execution review to ascertain whether an
appropriate allocation of resources exists in current budgets. This is
in agreement with the December 20, 2002, DoD guidance on performance
and budget integration prescribed in MID 910, and the May 22, 2003 DoD
guidance for implementing a 2-year Planning, Programming, Budgeting,
and Execution Process prescribed in MID 913.
RECOMMENDATION 4: The GAO recommended that the Secretary of Defense
direct the Director, Program Analysis and Evaluation, to revise
guidance to require all Defense agencies to include in their
performance plans and scorecards a discussion on data quality including
the reliability, validity, and limitations of performance measures as
well as data sources.
(Page 35/Draft Report):
DoD RESPONSE: Concur. The Department agrees that the Defense Agencies
not currently including this information should do so in order to
strengthen their respective performance plans. This is in agreement
with the August 22, 2003 DoD guidance that the Defense Agencies should
use the Balanced Scorecard/Risk Management Framework prescribed in MID
901 as the guiding principles for structuring their performance plans.
RECOMMENDATION 5: The GAO recommended that the Secretary of Defense
direct the Director, Program Analysis and Evaluation, to revise
guidance to require all Defense agencies to include in their
performance plans and scorecards the contextual information to better
understand how performance measures support the agency's mission. (Page
35/Draft Report):
DoD RESPONSE: Concur. The Department agrees that the Defense Agencies
not currently including this information should do so in order to
strengthen their respective performance plans. This is in agreement
with the August 22, 2003 DoD guidance that the Defense Agencies should
use the Balanced Scorecard/Risk Management Framework prescribed in MID
901 as the guiding principles for structuring their performance plans.
[End of section]
Appendix VI: Key Contact and Staff Acknowledgments:
GAO Contact:
Marilyn K. Wasleski (202) 512-8436:
Acknowledgments:
Gabrielle M. Anderson, Krislin M. Bolling, Elizabeth H. Curda, Benjamin
T. Licht, William A. McKelligott, Elizabeth G. Mead, Marjorie L. Pratt,
and Robert K. Wild also made significant contributions to the report.
[End of section]
Related GAO Products:
Future Years Defense Program: Actions Needed to Improve Transparency of
DOD's Projected Resource Needs. GAO-04-514. Washington, D.C.: May 7,
2004.
Department of Defense: Further Actions Needed to Establish and
Implement a Framework for Successful Business Transformation. GAO-04-
626T. Washington, D.C.: March 31, 2004.
Results-Oriented Government: GPRA Has Established a Solid Foundation
for Achieving Greater Results. GAO-04-38. Washington, D.C.: March 10,
2004.
Satellite Communications: Strategic Approach Needed for DOD's
Procurement of Commercial Satellite Bandwidth. GAO-04-206. Washington,
D.C.: December 10, 2003.
Defense Inventory: Several Actions Needed to Further DLA's Efforts to
Mitigate Shortages of Critical Spare Parts. GAO-03-709. Washington,
D.C.: August 1, 2003.
Results-Oriented Cultures: Creating a Clear Linkage between Individual
Performance and Organizational Success. GAO-03-488. Washington, D.C.:
March 14, 2003.
Defense Management: New Management Reform Program Still Evolving. GAO-
03-58. Washington, D.C.: December 12, 2002.
Tax Administration: IRS Needs to Further Refine Its Tax Filing Season
Performance Measures. GAO-03-143. Washington, D.C.: November 22, 2002.
Performance Budgeting: Opportunities and Challenges. GAO-02-1106T.
Washington, D.C.: September 19, 2002.
Information Technology: Defense Information Systems Agency Can Improve
Investment Planning and Management Controls. GAO-02-50. Washington,
D.C.: March 15, 2002.
Defense Management: Actions Needed to Sustain Reform Initiatives and
Achieve Greater Results. GAO/NSIAD-00-72. Washington, D.C.: July 25,
2000.
Management Reform: Elements of Successful Improvement Initiatives GAO/
T-GGD-00-26. Washington, D.C.: October 15, 1999.
Performance Plans: Selected Approaches for Verification and Validation
of Agency Performance Information. GAO/GGD-99-139. Washington, D.C.:
July 30, 1999.
The Results Act: An Evaluator's Guide to Assessing Agency Annual
Performance Plans. GAO/GGD-10.1.20. Washington, D.C.: April 1, 1998.
Executive Guide: Effectively Implementing the Government Performance
and Results Act. GAO/GGD-96-118. Washington, D.C.: June 1, 1996.
FOOTNOTES
[1] Defense agencies and DOD field activities perform similar support
functions. However, the field activities are smaller and serve a more
limited portion of DOD than do defense agencies. For purposes of this
report, we use the term "defense agencies" to refer collectively to
defense agencies and field activities.
[2] For purposes of this report, we use the term "performance
contracts" to include both performance contracts as well as
accompanying annual performance contract reports that discuss agencies'
progress toward meeting performance contract goals.
[3] The balanced scorecard is a private-sector concept introduced by
Robert Kaplan and David Norton in 1992 to assess organizational
performance and is used by several government agencies.
[4] Some of these products include GAO, Results-Oriented Government:
GPRA Has Established a Solid Foundation for Achieving Greater Results,
GAO-04-38 (Washington, D.C.: Mar. 10, 2004); The Results Act: An
Evaluator's Guide to Assessing Agency Annual Performance Plans, GAO/
GGD-10.1.20 (Washington, D.C.: Apr. 1, 1998); and Executive Guide:
Effectively Implementing the Government Performance and Results Act,
GAO/GGD-96-118 (Washington, D.C.: June 1, 1996).
[5] S. Rep. No. 108-46, at 347 (2003).
[6] The No Child Left Behind Act of 2001, among other things, focused
attention on closing the achievement gaps among various groups of
students.
[7] Program Objectives Memorandum is the final product of the
programming process within DOD. The Program Objectives Memorandum
displays the resource allocation decisions of the DOD components in
response to, and in accordance with, Defense Planning Guidance.
[8] Defense Information Systems Agency Inspector General. Audit of
DISA's Performance Contract. Final Report. Project No. 2000-H-601.
(Washington, D.C.: October 24, 2000).
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
