Defense Management
Additional Actions Needed to Enhance DOD's Risk-Based Approach for Making Resource Decisions Gao ID: GAO-06-13 November 15, 2005The Department of Defense (DOD) is simultaneously conducting costly military operations and transforming its forces and business practices while it is also competing for resources in an increasingly constrained fiscal environment. As a result, GAO has advocated that DOD adopt a comprehensive threat or risk management approach as a framework for decision making. In its 2001 strategic plan, the Quadrennial Defense Review (QDR), DOD stated its intent to establish an approach--the risk management framework--to balance priorities against risk over time and monitor results against its strategic goals. GAO was asked to (1) assess the extent to which DOD has implemented the framework, including using it to make investment decisions, and (2) identify the most significant challenges DOD faces in implementing the framework, or a similar approach.
DOD has taken some positive steps to implement the framework, but additional actions are needed before DOD can show real and sustainable progress in using a risk-based and results-oriented approach to strategically allocate resources across the spectrum of its investment priorities. For example, DOD defined four risk areas, and developed performance goals and department-level measures, but it needs to, among other things, further develop and refine the measures so that they clearly demonstrate results and provide a well-rounded depiction of departmental performance. DOD's current strategic plan and goals also are not clearly linked to the framework's performance goals and measures, and linkages between the framework and budget are also unclear. While DOD officials stated that risk was considered during the fiscal year 2006 budget cycle, DOD's budget submission does not specifically discuss how DOD identified or assessed risks to establish DOD-wide investment priorities. Without better measures, clear linkages, and greater transparency, DOD will be unable to fully measure progress in achieving strategic goals or demonstrate to Congress and others how it considered risks, and made trade-off decisions, balancing needs and costs for weapon programs and other investment priorities. DOD faces four challenges that have affected the implementation of the framework. First, DOD's organizational culture resists department-level approaches to priority setting and investment decisions. Second, sustained leadership, adequate transparency, and appropriate accountability are lacking. Further, no one individual or office has been assigned overall responsibility or sufficient authority for the framework's implementation. DOD also has not developed implementation goals or timelines with which to establish accountability, or measure progress. Finally, integrating the risk management framework with decision support processes and related reform initiatives into a coherent, unified management approach for the department is a challenge that DOD plans to address during the 2005 QDR. However, GAO has concerns about DOD's ability to follow through on this integration, because of its limited success in implementing other management reforms. Unless DOD successfully addresses these challenges and effectively implements the framework, or a similar approach, it will likely continue to experience (1) a mismatch between programs and budgets, and (2) a proportional, rather than strategic, allocation of resources to the services.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-06-13, Defense Management: Additional Actions Needed to Enhance DOD's Risk-Based Approach for Making Resource Decisions
This is the accessible text file for GAO report number GAO-06-13
entitled 'Defense Management: Additional Actions Needed to Enhance
DOD's Risk-Based Approach for Making Resource Decisions' which was
released on November 16, 2005.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Subcommittee on Readiness and Management Support,
Committee on Armed Services, U.S. Senate:
United States Government Accountability Office:
GAO:
November 2005:
Defense Management:
Additional Actions Needed to Enhance DOD's Risk-Based Approach for
Making Resource Decisions:
GAO-06-13:
GAO Highlights:
Highlights of GAO-06-13, a report to the Subcommittee on Readiness and
Management Support, Committee on Armed Services, U.S. Senate:
Why GAO Did This Study:
The Department of Defense (DOD) is simultaneously conducting costly
military operations and transforming its forces and business practices
while it is also competing for resources in an increasingly constrained
fiscal environment. As a result, GAO has advocated that DOD adopt a
comprehensive threat or risk management approach as a framework for
decision making. In its 2001 strategic plan, the Quadrennial Defense
Review (QDR), DOD stated its intent to establish an approach”the risk
management framework”to balance priorities against risk over time and
monitor results against its strategic goals.
GAO was asked to (1) assess the extent to which DOD has implemented the
framework, including using it to make investment decisions, and (2)
identify the most significant challenges DOD faces in implementing the
framework, or a similar approach.
What GAO Found:
DOD has taken some positive steps to implement the framework, but
additional actions are needed before DOD can show real and sustainable
progress in using a risk-based and results-oriented approach to
strategically allocate resources across the spectrum of its investment
priorities. For example, DOD defined four risk areas, and developed
performance goals and department-level measures, but it needs to, among
other things, further develop and refine the measures so that they
clearly demonstrate results and provide a well-rounded depiction of
departmental performance. DOD‘s current strategic plan and goals also
are not clearly linked to the framework‘s performance goals and
measures, and linkages between the framework and budget are also
unclear. While DOD officials stated that risk was considered during the
fiscal year 2006 budget cycle, DOD‘s budget submission does not
specifically discuss how DOD identified or assessed risks to establish
DOD-wide investment priorities. Without better measures, clear
linkages, and greater transparency, DOD will be unable to fully measure
progress in achieving strategic goals or demonstrate to Congress and
others how it considered risks, and made trade-off decisions, balancing
needs and costs for weapon programs and other investment priorities.
DOD‘s Risk Management Framework:
Force Management Risk;
Definition: Challenge of sustaining personnel, infrastructure, and
equipment.
Operational Risk;
Definition: Challenge of deterring or defeating near-term threats.
Future Challenges Risk;
Definition: Challenge of dissuading, deterring, defeating longer-term
threats.
Institutional Risk;
Definition: Challenge of improving efficiency (includes financial
management).
Source: DOD.
[End of table]
DOD faces four challenges that have affected the implementation of the
framework. First, DOD‘s organizational culture resists department-level
approaches to priority setting and investment decisions. Second,
sustained leadership, adequate transparency, and appropriate
accountability are lacking. Further, no one individual or office has
been assigned overall responsibility or sufficient authority for the
framework‘s implementation. DOD also has not developed implementation
goals or timelines with which to establish accountability, or measure
progress. Finally, integrating the risk management framework with
decision support processes and related reform initiatives into a
coherent, unified management approach for the department is a challenge
that DOD plans to address during the 2005 QDR. However, GAO has
concerns about DOD‘s ability to follow through on this integration,
because of its limited success in implementing other management
reforms. Unless DOD successfully addresses these challenges and
effectively implements the framework, or a similar approach, it will
likely continue to experience (1) a mismatch between programs and
budgets, and (2) a proportional, rather than strategic, allocation of
resources to the services.
What GAO Recommends:
GAO recommends that DOD take various actions to increase its chances of
successfully implementing a risk-based approach for investment decision
making, such as developing results-oriented measures and assigning
clear leadership with appropriate accountability and authority to
implement the framework. DOD partially concurred with our
recommendations.
www.gao.gov/cgi-bin/getrpt?GAO-06-13.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Sharon Pickup at (202)
512-9619 or pickups@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
Despite Positive Steps, Additional Actions Needed to Fully Implement
the Risk Management Framework:
Cultural Resistance, Combined with the Lack of Leadership,
Implementation Goals, and Process Integration, Affects DOD's
Implementation of the Risk Management Framework:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Comments from the Department of Defense:
Appendix III: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Definitions and Examples of DOD Department-Level Measures (as
of November 2004):
Table 2: The Number of Activity and Performance Measures for Each
Quadrant:
Table 3: Military Service and Defense-Wide Percentage of the 2005 and
2006 Future Years Defense Programs:
Table 4: Select Initiatives to Improve Investment Decision Making:
Figures:
Figure 1: The Risk Management Cycle:
Figure 2: Comparison of the Balanced Scorecard and the Risk Management
Framework:
Abbreviations:
CBO: Congressional Budget Office:
CMO: chief management official:
DOD: Department of Defense:
FYDP: Future Years Defense Program:
GAO: Government Accountability Office:
GPRA: Government Performance and Results Act:
JCIDS: Joint Capabilities Integration and Development System:
OSD: Office of the Secretary of Defense:
PA&E: Program Analysis and Evaluation (PA&E):
PPBE: Planning, Programming, Budgeting, and Execution:
P&R: Personnel and Readiness:
PART: Program Assessment Rating Tool:
QDR: Quadrennial Defense Review:
United States Government Accountability Office:
Washington, DC 20548:
November 15, 2005:
The Honorable John Ensign:
Chairman:
The Honorable Daniel K. Akaka
Ranking Minority Member:
Subcommittee on Readiness and Management Support:
Committee on Armed Services:
United States Senate:
Among the 21st century challenges facing the Department of Defense
(DOD) and the nation as a whole are difficult decisions concerning how
to strike an affordable balance between current and future national
security needs and between national security and domestic
needs.[Footnote 1] For example, DOD is simultaneously maintaining a
high pace of military operations for combating terrorism and
transforming its military forces and business operations for the 21st
century while it is also competing for federal resources in an
increasingly fiscally constrained environment. We have advocated that
DOD--as well as the rest of the federal government--adopt a
comprehensive threat or risk management approach as a framework for
decision making.[Footnote 2] This approach would fully link strategic
goals to plans and budgets; assess the values and risks of various
courses of actions as a tool for reexamining defense programs, setting
priorities, and allocating resources; and use performance measures to
assess outcomes.
To its credit, DOD introduced a balanced scorecard for risk management,
commonly known as the risk management framework, in its strategic plan,
the 2001 Quadrennial Defense Review (QDR) report. The 2001 strategic
plan articulated the new administration's emphasis on transforming
military forces and defense business practices to meet the emerging
challenges facing our nation. DOD intended the framework to be used as
a management tool to focus DOD's efforts on implementing the defense
program as outlined in the strategic plan. In particular, DOD's senior
leadership intended the risk management framework to assist decision
makers in formulating top-down strategy, balancing investment
priorities against risk over time, measuring near-and midterm outputs
against strategic goals, and focusing on actual performance results.
According to DOD officials, the risk management framework also was
intended to increase transparency within the department over the
decision-making process. During the ongoing 2005 QDR, DOD plans to
refine the risk management framework.
You asked us to examine the status of DOD's efforts to adopt a risk-
based approach to decision making, given the emphasis that DOD was
placing on the risk management framework. In response, we (1) assessed
the extent to which DOD has implemented its risk management framework,
including the extent to which DOD has used the framework to make
investment decisions; and (2) identified the most significant
challenges DOD faced in implementing the risk management framework or a
similar risk-based and results-oriented management approach.
To assess the extent to which DOD has implemented the risk management
framework, we analyzed key documents, policy guidance, data, and
interview results, and compared the analysis to the principles for
managing risk and results identified in prior GAO reports. In addition,
we conducted interviews with DOD and service officials, and members of
the Joint Staff. We discussed the department's progress in implementing
the risk management framework with members of the Defense Business
Board. We also analyzed DOD's department-level performance goals and
measures that are associated with the risk management framework and
assessed how DOD reported that information externally. We did not
validate the appropriateness of the risk management framework's risk
quadrants or the procedures that DOD has in place to ascertain the
reliability of performance data and we also did not assess the basis
for DOD's investment decisions. To identify the most significant
challenges DOD faced in implementing the risk management framework, we
analyzed documents, data, and interview results, and compared the
results of this analysis to the key practices to assist mergers and
organizational transformation identified in prior GAO reports.[Footnote
3] A more detailed discussion of our scope and methodology is presented
in appendix I.
Our work was performed from October 2004 through September 2005 in
accordance with generally accepted government auditing standards.
Results in Brief:
DOD has taken positive steps toward implementing the risk management
framework; however, additional actions are needed before the framework
is fully implemented and DOD can demonstrate real and sustainable
progress in using a risk-based and results-oriented approach to
strategically allocate resources across the spectrum of its investment
priorities. For example, while DOD established four risk areas, or
quadrants, and developed performance goals and measures of two types--
activity measures (measures to track initiatives) and performance
measures--the majority of these measures do not provide sufficient
information to monitor performance against the risk quadrants' goals.
Specifically, and contrary to results-oriented management principles,
the risk management framework's measures (1) do not clearly demonstrate
results, (2) do not provide a well-rounded depiction of performance
across the department, and (3) are not being systemically monitored
across all quadrants, except for the force management quadrant. In
addition, the framework's performance goals and measures are not
clearly linked to DOD's current strategic plan and strategic goals.
Lacking measures that follow results-oriented management principles and
clear linkages to strategic goals, DOD may be unable to provide a clear
roadmap of how its activities at all levels contribute to meeting DOD's
strategic goals. Finally, although DOD officials stated that risk was
considered in the fiscal year 2006 budget cycle, the fiscal year 2006
budget submission does not include any specific information on how DOD
systematically identified or assessed departmental risks to establish
DOD-wide investment priorities. Therefore, the linkages between the
risk management framework and the budget are unclear. Without better
measures, clear linkages, and greater transparency, DOD will be unable
to fully measure progress in achieving strategic goals or demonstrate
to Congress and others how it considered risks and made trade-off
decisions, balancing needs and costs for weapon system programs and
other investment priorities.
DOD faces four key challenges that affect its ability to fully
implement the risk management framework, or a similar risk-based and
results-oriented management approach: (1) overcoming cultural
resistance to the transformational change represented by such an
approach in a department as massive, complex, and decentralized as DOD;
(2) maintaining sustained leadership and clear accountability for this
cultural transformation; (3) providing implementation goals and
timelines to gauge progress in transforming the culture; and (4)
integrating the risk management framework with decision support
processes and related reform initiatives into a coherent, unified
management approach for the department. Our prior work on results-
oriented management and organizational transformation and mergers has
shown that addressing these challenges is at the center of successful
change management efforts in leading organizations. DOD is having
difficulties implementing the framework because it has not addressed
these four challenges. With respect to the first challenge, DOD's size
and complexity result in a culture that makes developing department-
level approaches to priority setting and investment decision making
difficult. For example, the allocation of budgets on a proportional,
rather than a strategic, basis among the services is a long-standing
budgetary problem that we have reported about for years. Second, the
lack of sustained leadership and clear accountability for the
framework's implementation has resulted in a lack of emphasis and
understanding of its status and purpose within the department. Because
of the lack of sustained leadership for other management reform
efforts, we have supported legislation to create a chief management
official (CMO) at DOD to provide this leadership.[Footnote 4] Third,
DOD did not establish implementation goals or timelines with which to
establish accountability, measure progress, and build momentum.
Finally, integrating the risk management framework with other decision
support processes and related reform initiatives into a coherent,
unified management approach is a challenge that DOD intends to address
in the ongoing 2005 QDR. Illustrating this challenge, DOD is attempting
to implement the risk management framework while it is also shifting to
biennial budgeting and reforming defense planning. Our work has shown
that if risk-based and results-oriented management approaches are to be
successfully implemented, they must be integrated into the usual cycle
of agency decision making. Unless DOD addresses these challenges and
successfully implements the risk management framework, or a similar
approach, it may continue to experience (1) a mismatch between programs
and budgets, and (2) the proportional, rather than strategic,
allocation of resources to the services. Therefore, Congress may have
insufficient transparency into how DOD has identified and assessed
risks and made trade-offs in its investment decision making.
In this report, we recommend that DOD take various actions to increase
its chances of successfully implementing a risk-based approach for
investment decision making. This includes developing results-oriented
measures and assigning clear leadership with appropriate accountability
and authority to implement and sustain the risk management framework,
or a similar approach. In written comments on a draft of this report,
DOD partially concurred with our recommendations. DOD's comments and
our evaluation of them are on page 25 of this report.
Background:
In our report, High-Risk Series: An Update,[Footnote 5] we identified
agencies' lack of comprehensive risk management strategies as an
emerging challenge for the federal government. Increasingly limited
fiscal resources across the federal government, coupled with the
emerging requirements from the changing security environment, emphasize
the need for DOD to develop a risk-based strategic investment approach.
For this reason, we have advocated that DOD adopt a comprehensive risk
management approach for decision making.[Footnote 6] Furthermore, DOD
and other federal agencies are required by statute to develop a results-
oriented management approach to strategically allocate resources on the
basis of performance.[Footnote 7] The balanced scorecard--a concept to
balance an organization's focus across financial, customer, internal
business, and learning and growth management areas--is one approach for
developing results-oriented management that government agencies have
recently started to adopt.[Footnote 8] At the direction of the
Secretary of Defense, DOD developed a risk management framework that
DOD later aligned with its results-oriented management activities
through a DOD balanced scorecard.
Risk Management Is an Emerging 21st Century Challenge:
An emerging challenge for the federal government involves the need for
the completion of comprehensive national threat and risk assessments in
a variety of areas. For example, emerging requirements from the
changing security environment, coupled with increasingly limited fiscal
resources across the federal government, emphasize the need for
agencies to adopt a sound approach to establishing resource
decisions.[Footnote 9] We have advocated that the federal government,
including DOD, adopt a comprehensive threat or risk management approach
as a framework for decision making that fully links strategic goals to
plans and budgets, assesses values and risks of various courses of
actions as a tool for setting priorities and allocating resources, and
provides for the use of performance measures to assess outcomes. Based
on our review of the literature,[Footnote 10] as shown in figure 1, the
goal of risk management is to integrate systematic concern for risk
into the usual cycle of agency decision making and implementation.
Figure 1: The Risk Management Cycle:
[See PDF for image]
[End of figure]
A risk management cycle represents a series of analytical and
managerial steps, basically sequential, that can be used to assess
risk, evaluate alternatives for reducing risks, choose among those
alternatives, implement the alternatives, monitor their implementation,
and continually use new information to adjust and revise the
assessments and actions, as needed. Adoption of a risk management cycle
such as this can aid in assessing risk by determining which
vulnerabilities should be addressed, and how they should be addressed,
within available resources. For the purposes of this report, we focused
on the stages of the risk management cycle that involve DOD's actions
to set strategic goals and objectives, establish investment priorities
based on risk assessments, and implementation and monitoring.
Risk management's objectives are essentially the same as those of good
management, and they are consistent with the broad economy and
efficiency objectives of good government--namely, to provide better
outcomes for the same amount of money, or to provide the same outcomes
with less money. Therefore, risk management's objectives are also
compatible with those of the federal government's results-oriented
management approach, which was enacted in the Government Performance
and Results Act (GPRA) of 1993,[Footnote 11] and the balanced scorecard
approach. Congress enacted GPRA to focus the federal government on
achieving results through the creation of clear links between the
process of allocating scarce resources and an agency's strategic goals,
or the expected results to be achieved with those resources. Building
on GPRA's foundation, the current administration has taken steps to
strengthen the integration of budget, cost, and performance information
by including budget and performance integration as one of its
management initiatives under the umbrella of the President's Management
Agenda.[Footnote 12] The Budget and Performance Integration initiative
includes efforts such as the Program Assessment Rating Tool (PART),
improving outcome measures, and improving monitoring of program
performance.[Footnote 13] The balanced scorecard approach is a
management tool that some federal agencies have adopted to help them
translate the strategy set forth in a results-oriented management
approach into the operational objectives that drive both behavior and
performance. The balanced scorecard consists of four management areas
that organizations should focus on--financial, customer, internal
business, and learning and growth.
DOD's 2001 Strategic Plan Outlines a New Risk Management Framework:
DOD introduced the risk management framework in its strategic plan, the
2001 QDR report. The 2001 strategic plan articulated the new
administration's emphasis on transforming military forces and defense
business practices to meet the changing threats facing our nation. In
his guidance to the department for the 2001 QDR strategic planning
process, the Secretary of Defense stated the need for DOD to use a risk
mitigation approach for balancing force, resource, and modernization
requirements across defense planning timelines. This guidance also
stated that DOD must include the identification of output-based
measures to reduce inefficiencies through the department in any
approach to risk management. Building on the guidance, the 2001 QDR
outlined DOD's risk management framework. According to the QDR, the
framework would enable DOD to address the tension between preparing for
future threats and meeting the demands of the present with finite
resources. It was also intended to ensure that DOD was sized, shaped,
postured, committed, and managed with a view toward accomplishing the
strategic plan's defense policy goals.
DOD adapted the balanced scorecard concept to the risk management
framework by substituting the four dimensions of risk--force
management, operational, future challenges, and institutional--for the
scorecard's four management areas. The risk management framework was to
be a transformational tool that would provide a balanced perspective of
the organization's execution of strategy and ensure a top-down
approach. The 2002 policy guidance also designated four preliminary
performance goals for each of the four risk quadrants. In addition, the
guidance required that performance goals and measures were to be
cascaded to the services and defense agencies. Figure 2 shows a
comparison, as provided by DOD.
Figure 2: Comparison of the Balanced Scorecard and the Risk Management
Framework:
[See PDF for image]
[End of figure]
Despite Positive Steps, Additional Actions Needed to Fully Implement
the Risk Management Framework:
Despite positive steps, DOD needs to take additional actions before the
risk management framework is fully implemented and DOD can demonstrate
real and sustainable progress in using a risk-based and results-
oriented approach to strategically allocate resources across the
spectrum of its investment priorities. For example, DOD is still in the
process of developing department-level measures for the framework that
address results-based management principles, such as linking
performance information to strategic goals so that this information can
be used to monitor performance results and determine how well the
department is doing in achieving its strategy. Without more results-
oriented performance measures, DOD may be unable to provide the
services and other defense components with clear roadmaps of how their
activities contribute to meeting DOD's strategic goals. In addition,
the framework's performance goals and measures are not clearly linked
to DOD's current strategic plan and strategic goals. Furthermore, the
extent to which the risk management framework is linked to the budget
cycle is unclear. Without better measures, clear linkages, and greater
transparency, DOD will be unable to fully measure progress in achieving
strategic goals or demonstrate to Congress and others how it considered
risks and made trade-offs in making investment decisions.
Developing a Set of Measures That Can Be Used to Monitor Performance Is
a Work in Progress:
DOD has taken positive steps toward developing measures for each of the
performance goals under the framework's four risk quadrants; however,
developing a set of measures that can be used to monitor performance
results is still a work in progress. Based on GAO's prior work on
results-based management principles, we found that leading
organizations' performance measures are: (1) designed to demonstrate
results, or provide information on how well the organization is
achieving its goals; (2) limited to a vital few, and balanced across
priorities; and (3) used by management to improve performance.[Footnote
14] However, the set of measures DOD has developed for the risk
management framework do not adequately address these principles. While
DOD established four risk quadrants and developed performance goals and
measures of two types--activity measures (measures to track
initiatives) and performance measures--the majority of its measures do
not provide sufficient information to monitor performance against the
risk quadrants' goals.
First, DOD officials acknowledge that establishing department-level
measures for the framework that demonstrate results is still a work in
progress, as the majority of the risk management framework's measures
require further development or refinement. In fact, as shown in table
1, 44 of the 77 department-level measures for all four quadrants, or
over 50 percent, are activity measures. According to DOD sources,
activity measures are to result in a new performance measure, a new
baseline or benchmark, or define a new capability, rather than monitor
a specific annual performance target. Once these activities are
completed, DOD officials stated that the department will be better able
to monitor department-level performance against strategic goals.
However, our analysis found that the activity measures, as defined in
DOD's external reports, typically do not provide sufficient information
to monitor the department's progress in achieving the stated goal they
are to measure, such as developing a new performance measure or
baseline. The desired outcomes for activity measures generally state
that a task was or will be completed by a certain date but they do not
provide sufficient information on whether the activity is on schedule,
the interdependencies among tasks, or the contribution toward enhancing
the department's performance. Therefore, Congress and other external
stakeholders lack information and adequate assurances that DOD is
making progress in implementing a risk-based and results-oriented
management approach to making investment decisions.
Table 1: Definitions and Examples of DOD Department-Level Measures (as
of November 2004):
Type: Activity measures;
Number: 44[A];
Definition: Activity measures track developmental activities, are
usually qualitative, and track key milestones or events in lieu of a
specific annual performance target;
Examples: Deny enemy advantages and exploit weaknesses;
Description of desired outcome monitored by measure: Roadmap will be
complete by the end of fiscal year 2005.
Examples: Enhance homeland defense and consequence management;
Description of desired outcome monitored by measure: Strategy will be
complete by the first quarter of fiscal year 2005.
Type: Performance measures;
Number: 33[A];
Definition: Performance measures track current outputs and set
quantitative annual targets for performance that are measurable;
Examples: Reserve component enlisted recruiting quality;
Description of desired outcome monitored by measure: Target > 90% of
recruits holding high school diplomas; Actual 88% of recruits holding
high school diplomas.
Examples: Reduce customer wait time (in days);
Description of desired outcome monitored by measure: Target 15 days
from order to receipt for material goods; Actual 24 days from order to
receipt for material goods.
Source: GAO analysis of the Risk Management Framework's performance
measures.
[A] We have recoded five performance measures as activity measures as
these measures tracked milestones and events, which corresponds with
DOD's definition of an activity measure.
[End of table]
Second, DOD's department-level performance measures are still a work in
progress in that these measures do not provide a well-rounded depiction
of DOD's performance. In our previous work, we have found that
performance measurement efforts that are not balanced across priorities
may skew an agency's performance and keep its senior leadership from
seeing the whole picture.[Footnote 15] For example, in developing
department-level measures for the risk management framework, DOD
appears to have overemphasized its force management priorities at the
expense of operational risk. As illustrated in table 2, the operational
risk quadrant has no performance measures, while the force management
risk quadrant has a total of 36 measures, including 15 activity
measures and 21 performance measures.
Table 2: The Number of Activity and Performance Measures for Each
Quadrant:
Activity measures: Operational: 15;
Performance measures: Operational: 21;
Total measures: Operational: 36[A].
Force Management: Operational;
Activity measures: 9;
Performance measures: 0;
Total measures: 9.
Activity measures: 11;
Performance measures: 10;
Total measures: 21[B].
Force Management;
Activity measures: 9;
Performance measures: 2;
Total measures: 11[C].
Source: GAO analysis of DOD data.
[A] We have recoded two performance measures as activity measures as
these measures tracked milestones and events, which corresponds with
DOD's definition of an activity measure.
[B] We have recoded one performance measure as an activity measure as
this measure tracked milestones and events, which corresponds with
DOD's definition of an activity measure.
[C] We have recoded two performance measures as activity measures as
these measures tracked milestones and events, which corresponds with
DOD's definition of an activity measure.
[End of table]
In providing technical comments to a draft of this report, DOD objected
to our recoding of five department-level performance measures as
activity measures. We recoded these measures because they tracked
milestones and events, which corresponded to DOD's definition of an
activity measure. The measures we recoded addressed the following:
* a civilian human resources strategic plan,
* a military human resources strategic plan,
* monitor the status of defense technology objectives,
* strategic transformation appraisal, and:
* support acquisition excellence goals.
Finally, DOD officials indicated that DOD is systematically using
performance measures to monitor progress and improve performance for
only one risk quadrant, although individual measures under the other
three risk quadrants may be monitored. We have found that leading
organizations use performance information to improve organizational
performance and identify performance gaps, and to provide incentives
that reinforce a results-oriented management approach.[Footnote 16]
According to DOD officials, the force management quadrant is the only
quadrant that is managed by one individual and one office--the Under
Secretary of Defense for Personnel and Readiness and his office. These
officials stated that this situation is a critical factor in the
progress DOD has made in systematically monitoring performance across
the force management quadrant on a routine basis. For example,
officials stated that the Under Secretary of Defense personally leads
quarterly monitoring sessions on the force management quadrant's
performance. DOD officials also told us that the Under Secretary of
Defense for Personnel and Readiness has greatly facilitated this
monitoring by developing a centralized database to capture the
performance data used to track DOD's performance in meeting the
quadrant's goals. Unless all of the risk management framework's
quadrants are systematically monitored, implementation of the framework
may be hindered and the framework risks becoming a paper-driven,
compliance exercise. Indeed, one DOD official told us that he views the
risk management framework and its measures as a "reporting drill" and,
in addition, his office would not change its processes if DOD was to no
longer use the framework.
Cascading the Risk Management Framework's Goals and Measures Is an
Ongoing Effort:
DOD is still in the process of cascading the risk management
framework's goals and measures to the services. We have found that
leading organizations seek to establish clear hierarchies of goals and
measures that cascade down so that subordinate units have
straightforward roadmaps to demonstrate how their activities contribute
to meeting the organization's strategy.[Footnote 17] According to DOD
officials, all of the services are attempting to align their existing
performance measures with the department-level performance goals and
measures. However, service officials said that it is challenging to
cascade the department-level activity measures, because these measures
represent very broad initiatives that may not be applicable at all DOD
levels. Officials from one service said they have had to develop new
measures to align with the department-level measures, because they had
been assessing performance with fewer measures than the Office of the
Secretary of Defense had developed.
Developing a Strategic Plan with Clear Linkages between the Risk
Management Framework and Strategic Goals Is a Critical Next Step:
The risk management framework's performance goals and measures are not
clearly linked--a key principle of results-oriented management--to a
coherent strategic plan.[Footnote 18] The development of such a
strategic plan is a critical next step in using a risk-based and
results-oriented approach to making investment decisions. Without these
linkages, DOD cannot easily demonstrate how achievement of a
performance goal or measure contributes to the achievement of strategic
goals and ultimately the organization's mission. Our previous work
indicated that DOD's strategic plan, the 2001 QDR, did not provide a
sound foundation for the risk management framework.[Footnote 19] We
reported that the usefulness of the 2001 QDR was limited by the lack of
focus on longer-term threats and requirements for critical support
capabilities, and provided few insights into how future threats and
planned technical advances could affect future force requirements. In
turn, this lack of focus and insight limited the QDR's usefulness as a
foundation for fundamentally reassessing U.S. defense plans and
programs and for balancing resources across near-and midterm risks.
DOD officials indicated that DOD has not yet defined the linkages
between the risk management framework's performance goals and the
strategic goals in the 2001 QDR. Furthermore, the Defense Business
Board's official minutes for its July 28, 2005, meeting contained a
recommendation that the Secretary of Defense define department-level
objectives, which should then be cascaded down the department.[Footnote
20] In discussing the ongoing 2005 QDR, DOD stated that although the
department would continue its efforts to do so, establishing these
linkages was very challenging because of the size and scope of DOD's
operations. However, as suggested by the Defense Business Board and our
previous work, if DOD's strategic plan is to drive the department's
operations, a straightforward linkage is needed among strategic goals,
annual performance goals, and day-to-day activities.[Footnote 21] The
ongoing 2005 QDR offers DOD the opportunity to strengthen its strategic
planning.
Although Risk Considered, Linkages Between the Risk Management
Framework and Budget Are Unclear:
According to DOD officials, the department has begun to consider risk
in its investment decision making; however, the full extent to which
the framework's risk-based and results-oriented approach has been
linked to the fiscal year 2006 budget cycle is unclear. Our work
indicates that leading organizations link strategy to the budget
process through results-oriented management to evaluate potential
investments or initiatives.[Footnote 22]
DOD sources indicated that the department has begun to consider risk
during its usual cycle of investment decision making. For example,
according to DOD sources, the Secretary of Defense articulated broad
areas for increasing or decreasing risk under each quadrant in the
fiscal years 2006-2011 planning guidance, leaving it up to the defense
components to decide how to structure their investment decisions within
those broad areas consistent with the Secretary's risk guidance. In
addition, DOD officials stated that the framework has increased
awareness within the department on the need to balance risk over time.
For example, when DOD reduced the fiscal years 2006-2011 defense
program by $30 billion, DOD officials stated that the department did
not take the traditional budgetary approach of cutting each defense
component's budget by a certain percentage. Instead, DOD officials
stated that the Secretary of Defense used a collaborative approach with
service participation to discuss where to take the budget reductions
and how these cuts would affect risk, although DOD officials offered
various views on how extensively the framework was used to make those
decisions.
Second, DOD required that the services and other defense components
offset any funding increase in one area with a funding decrease in
another area for the fiscal years 2006-2007 budget submission.
According to DOD officials, risk--whether on the basis of "professional
judgment" or analysis--was considered in these deliberations. For
example, the Army's plan for fiscal years 2006-2023 articulated areas
for increasing risks so that it could decrease risk in the operational
risk dimension by investing in current capacity.
However, the fiscal year 2006 budget submission does not include any
specific information on how DOD systematically identified or assessed
departmental risks to establish DOD-wide investment priorities. For
example, the military services' share of the Future Years Defense
Program (FYDP) remained relatively unchanged from fiscal year 2005 to
fiscal year 2006 (see table 3),[Footnote 23] providing one indication
that the risk management framework may not yet be a useful tool for
balancing departmental risks across the services.
Table 3: Military Service and Defense-Wide Percentage of the 2005 and
2006 Future Years Defense Programs:
Department of the Army;
2005 Percentage of FYDP: 24.23;
2006 Percentage of FYDP: 24.63;
Percentage change by department: 0.40.
Department of the Navy;
2005 Percentage of FYDP: 29.75;
2006 Percentage of FYDP: 29.47;
Percentage change by department: -0.28.
Department of the Air Force;
2005 Percentage of FYDP: 29.80;
2006 Percentage of FYDP: 29.82;
Percentage change by department: 0.02.
Defense-wide;
2005 Percentage of FYDP: 16.22;
2006 Percentage of FYDP: 16.08;
Percentage change by department: -0.14.
Total;
2005 Percentage of FYDP: 100.00;
2006 Percentage of FYDP: 100.00.
Source: GAO analysis of DOD FYDP data.
Note: Totals may not add due to rounding.
[End of table]
DOD has reported on the risk management framework in the department's
GPRA and other reporting requirements. For example, the fiscal year
2004 Performance and Accountability Report describes what DOD is doing,
or plans to do, to define, measure, and monitor performance goals in
the four risk quadrants but does not discuss the implementation status
of the risk management framework. Furthermore, the fiscal year 2004
report, the most recent available, provided insufficient information to
assist Congress in overseeing how DOD plans to prioritize investment
decisions within or across the risk quadrants. Without more detailed
information, Congress may have insufficient transparency into how DOD
has identified and assessed risks and made trade-offs in its investment
decision making. In addition, we reported in May 2004 that
congressional visibility over investment decision making also was
limited by the absence of linkages between the risk management
framework and military capabilities planning and the FYDP.[Footnote 24]
Because the FYDP lacked these linkages, we concluded that decision
makers could not use it to determine how a proposed increase in
capability would affect the risk management framework.
Our work also has shown that the FYDP may understate the costs of
weapon system programs; therefore, DOD may be starting more programs
than it can afford. For example, our assessment of 54 major programs,
representing an investment of over $800 billion, found that the
majority of these programs were costing more and taking longer to
develop than planned.[Footnote 25] Problems occurred because of DOD's
overly optimistic planning assumptions about the long-term costs of
weapon system programs and its failure to capture early on the
requisite knowledge that is needed to efficiently and effectively
manage program risks. When DOD has too many programs competing for
funding and approves programs with low levels of knowledge, it is
accepting the attendant likely adverse cost and schedule risks. As a
result, it will probably get fewer quantities for the same investment
or face difficult choices on which investments it cannot afford to
pursue. The findings of our work suggest that having a departmentwide
investment strategy for weapon systems, to allocate resources across
investment priorities, would help reduce these risks.
Cultural Resistance, Combined with the Lack of Leadership,
Implementation Goals, and Process Integration, Affects DOD's
Implementation of the Risk Management Framework:
Four key challenges impede DOD's progress toward implementing the risk
management framework. The first implementation challenge facing DOD is
overcoming cultural resistance to change in a department as massive,
complex, and decentralized as DOD. The second challenge is the lack of
sustained leadership, and the third challenge is the absence of
implementation goals and timelines. These challenges relate to DOD's
failure to follow crucial transformational steps. The fourth challenge-
-integrating the risk management framework with decision support
processes and related reform initiatives, into a coherent, unified
management approach for the department--relates to key results-oriented
management practices. Unless DOD addresses these challenges and
successfully implements the risk management framework, or a similar
approach, it may continue to experience (1) a mismatch between programs
and budgets, and (2) the proportional, rather than strategic,
allocation of resources to the services.
Transforming DOD's Organizational Culture Is a Significant Challenge:
Transforming DOD's organizational culture--from a focus on inputs and
programs to strategically balancing investment risks and monitoring
outcomes across the department--through the implementation of the risk
management framework is a significant challenge for the department for
several reasons. First, as we noted in our 21st Century Challenges
report, to successfully transform, DOD needs to overcome the inertia of
various organizations, policies, and practices that became rooted in
the Cold War era.[Footnote 26] The department's expense, size, and
complexity, however, make overcoming this resistance and inertia
difficult. In fiscal year 2004, DOD reported that its operations
involved $1.2 trillion in assets, $1.7 trillion in liabilities, over
$605 billion in net cost of operations, and over 3.3 million military
and civilian personnel. For fiscal year 2005, DOD received
appropriations of about $417 billion. Moreover, execution of its
operations spans a wide range of defense organizations, including the
military services and their respective major commands and functional
activities, numerous large defense agencies and field activities, and
various combatant and joint operation commands, which are responsible
for military operations for specific geographic regions or theaters of
operations.
Second, DOD's highly decentralized management structure is another
contributing factor that makes cultural change difficult. Although
under the authority, direction, and control of the Secretary of
Defense, the military services have the legislative authority to
organize, equip, and train the nation's armed forces for combat under
Title 10 of the U.S. Code. Furthermore, Congress directly appropriates
funds to the services for programs and activities that support these
purposes. In the opinion of knowledgeable DOD officials, this
legislative authority has resulted in a culture that makes it difficult
to develop department-level, or joint, management approaches. For
example, the allocation of budgets on a proportional, rather than a
strategic basis, among the military services is a long-standing
budgetary problem that we have identified as a major management
challenge for the department.[Footnote 27] In addition, the Joint
Defense Capabilities Study, chartered by the Secretary of Defense in
March 2003, made the following observations on how DOD's organizational
culture does not reinforce a departmental or joint approach to
investment decision making and results management:[Footnote 28]
* DOD's bottom-up strategic planning process did not support early
senior leadership involvement and did not provide integrated
departmentwide objectives, priorities, and roles as a framework for
planning joint capabilities.
* Service-centric focus on programs and weapons platforms resulted in a
process that did not provide an accurate picture of joint needs, nor
did it provide a consistent view of priorities and acceptable risks
across the department.
* The resulting budget did not optimize capabilities at either the
department or the service level.
* Accountability and feedback focused on monetary input rather than
output; therefore, much of the information provided did not support the
senior leaders' decision making as it did not tell how well the
department was being resourced to meet current and future mission
requirements.
Lack of Sustained Leadership and Appropriate Accountability Has
Challenged DOD's Implementation of the Risk Management Framework:
The lack of sustained leadership attention and appropriate
accountability has challenged DOD's progress in implementing the risk
management framework. Our work has indicated that sustained leadership
is a key transformational, or change management, practice.[Footnote 29]
However, knowledgeable DOD officials indicated that DOD's senior
leadership did not provide sustained attention to the framework's
implementation. For example, a DOD official actively involved in the
framework's implementation stated that meetings with senior leadership
that were to provide oversight of the framework's implementation have
not been regularly scheduled. DOD officials indicated that as a result
of this lack of sustained leadership, DOD has not placed much emphasis
on implementing the risk management framework at the department level.
In addition, other DOD officials stated that changes in leadership have
made it difficult to implement the risk management framework or develop
performance measures. For example, since October 2004, DOD has
experienced turnover in the following senior level positions, including
the Deputy Secretary of Defense; the Under Secretary of Defense for
Acquisition, Technology and Logistics; and the Director of Program
Analysis and Evaluation (PA&E). Lacking sustained leadership attention,
DOD officials offered conflicting perspectives on the status of the
risk management framework with some officials suggesting that the
framework had been overtaken by other performance-based or risk-based
management initiatives while another suggested that the framework was
primarily a compliance exercise. DOD officials also held differing
perspectives on the purpose of the framework, including the beliefs
that it was developed to monitor the Secretary of Defense's priority
areas or that it was a programming and budgeting tool.
Implementation of the risk management framework has also been
challenged by the lack of clear lines of authority and appropriate
accountability. No single individual or organization has been given
overarching leadership responsibilities, authority, or the
accountability for achieving the framework's implementation. Instead,
the responsibility for various tasks and performance measures have been
spread among several organizations, including the Director, PA&E; the
Under Secretary of Defense for Personnel and Readiness (P&R); and the
Under Secretary of Defense, Comptroller/Chief Financial Officer.
We testified in April 2005 that as DOD embarks on large-scale change
initiatives, the complexity and long-term nature of these initiatives
require the development of an executive position capable of providing
strong and sustained leadership--over a number of years and various
administrations.[Footnote 30] For this reason, we have supported
legislation to create a CMO at DOD to provide such sustained
leadership.[Footnote 31] A CMO could also provide the leadership needed
to successfully develop a risk-based and results-oriented management
approach at DOD, such as the risk management framework.
Lack of Implementation Goals and Timelines Further Challenges DOD's
Implementation of Risk Management Framework:
Accountability for implementation of the risk management framework also
has been hindered by the absence of implementation goals and timelines
with which to gauge progress. As we have previously reported,
successful change management efforts use implementation goals and
timelines to pinpoint performance shortfalls and gaps, suggest
midcourse corrections, and build momentum by demonstrating
progress.[Footnote 32] However, DOD's limited guidance on the risk
management framework did not establish implementation goals and
timelines, nor did it require that implementation goals and timelines
be developed. According to knowledgeable DOD officials, DOD did not see
the need for implementation goals or timelines because the framework
was not meant to change processes or create new ones, but rather was a
management tool to improve upon investment decision-making processes.
Regardless of how DOD classifies the risk management framework, we have
found that implementation goals and timelines are essential to any
transformational change, such as that envisioned by the Secretary of
Defense with the risk management framework, because of the number of
years it can take to complete the change.[Footnote 33] Moreover, the
absence of implementation goals and timelines makes it difficult to
determine whether progress has been made in implementing the framework
over the last 2 ½ years, and whether DOD's revisiting of the framework
during the 2005 QDR represents an evolutionary progression or
implementation delays.
Integrating the Risk Management Framework with Decision Support
Processes and Related Reform Initiatives Is a Significant Challenge:
DOD faces a significant challenge integrating the risk management
framework with decision support processes for planning, programming,
and budgeting and with related reform initiatives into a coherent,
unified management approach. The goal of both risk management and
results-oriented management is to integrate the systematic concern for
risk and performance into the usual cycle of agency decision making and
implementation. DOD's challenge in meeting these goals is demonstrated
by the number of initiatives, as shown in table 4, that DOD has put in
place to improve investment decision making and manage performance
results. For example, both capabilities planning and the risk
management framework are to define risks and develop performance
measures but, according to DOD officials, the department is still
determining how to align capabilities planning with the risk management
framework. Other initiatives, including GPRA and PART, are also to
develop performance measures and DOD is still working on integrating
these initiatives with the risk management framework and individual
performance monitoring approaches of the services and other defense
components into a single, integrated system. In December 2002, the
Deputy Secretary of Defense issued a memorandum to correct this
situation by requiring the alignment of the risk management framework
and the President's Management Agenda with DOD's results-oriented
management activities, including those associated with GPRA.
Table 4: Select Initiatives to Improve Investment Decision Making:
Initiative: Two-Year Planning, Programming, Budgeting, and Execution
Process (PPBE);
Description: In 2003, DOD implemented a 2-year cycle for its strategic
planning, program development, and resource determination process. DOD
stated that this change was needed to integrate DOD's processes for
strategic planning, identification of needs for military capabilities,
systems development and acquisition, and program and budget
development. During the second year of the biennial budget, DOD is to
focus on budget execution and program performance.
Initiative: Enhanced Planning Process;
Description: In fiscal year 2004, DOD initiated a reform of defense
planning to make it more responsive and adaptive to the needs of senior
decision makers. The process is to result in fiscally constrained
guidance and priorities-- for military forces, modernization, readiness
and sustainability, and supporting business processes and
infrastructure activities--for program development. The enhanced
planning process is to integrate the outcomes of operational,
enterprise, and capabilities planning efforts in a document called the
Joint Programming Guidance. The Joint Programming Guidance is to
provide a link between planning and programming, and it is to provide
guidance to the DOD components for the development of their program
proposals.
Initiative: Capabilities Planning;
Description: The 2001 QDR announced a defense strategy built around the
concept of shifting to a "capabilities-based" approach to defense.
According to the 2001 QDR, while DOD cannot know with confidence what
nation, group of nations, or nonstate actor might pose a threat to U.S.
vital interests, it is possible to anticipate the capabilities an
adversary might employ. Capabilities planning is to provide a top-down,
competitive approach to weigh options against resource constraints
across a spectrum of challenges and to apportion risk against those
challenges. It is also to enable risk assessments and trade-off
decisions across DOD organizational stovepipes. The new concept
stresses joint solutions to problems, requires the identification of
risk trade-offs within and across mission areas, and treats uncertainty
explicitly.
Initiative: Program/Budget Framework Initiative;
Description: As part of the financial management enterprise initiatives
of the Business Management Modernization Program, this initiative is to
provide a foundation for a new program and budget data structure using
a common language that enables senior level DOD decision makers to
weigh options versus resource constraints across a spectrum of
challenges. The framework is to consist of a number of related data
transparency initiatives that span across all portions of the PPBE
process, including creating department-level definitions for the four
risk quadrants. One of the stated benefits is establishing an ability
to view programs and resources based on the risk management framework.
Initiative: Joint Capabilities Integration and Development System
(JCIDS);
Description: A system for the Joint Staff to assess gaps in military
joint warfighting capabilities and recommend solutions to resolve those
gaps. This system is replacing DOD's requirements- generation process
for major acquisitions in an effort to shift the focus to a more
capabilities-based approach for determining joint warfighting needs
rather than a threat-based approach focused on individual systems and
platforms. Under this system, boards comprised of high-level DOD
civilians and military officials are to identify future capabilities
needed around key functional concepts and areas, such as command and
control, force application, and battlespace awareness, and to make
trade-offs among air, space, land, and sea platforms in doing so.
Initiative: President's Management Agenda;
Description: The President's Management Agenda contains five
initiatives aimed at improving federal agency management and
performance: (1) strategic human capital management, (2) competitive
sourcing, (3) improved financial performance, (4) expand electronic
government, and (5) budget and performance integration. The President
cited our work on high-risk areas and major management challenges in
developing his initiatives, and implementation of the agenda has
reinforced the need to focus agencies' efforts on achieving key
management and performance improvements.
Initiative: Budget and Performance Integration;
Description: The budget and performance integration initiatives of the
President's Management Agenda include elements such as the PART used to
review programs, an emphasis on improving outcome measures, and
improving monitoring of program performance. PART is the central
element in the performance budgeting piece of the President's
Management Agenda. PART builds on GPRA by actively promoting the use of
results-oriented information to assess programs in the budget.
Source: GAO analysis.
[End of table]
We note that these reform initiatives address key business processes
within the department and that we have placed DOD's overall business
transformation on our list of federal programs and activities at high
risk of waste, fraud, abuse, and mismanagement.[Footnote 34]
The Under Secretary of Defense for Acquisition, Technology and
Logistics indicated that DOD plans to address the challenge associated
with the integration of DOD's planning, resourcing, and execution
processes and initiatives, including the risk management framework. The
Under Secretary stated that one task of the ongoing 2005 QDR was
"strategic process integration." The Under Secretary also stated that
the department is planning to provide a roadmap with performance goals
and timelines on how it will implement initiatives to improve strategic
process integration. This roadmap is to be submitted with the 2005 QDR
report to Congress in early 2006 with the fiscal year 2007 budget.
Conclusions:
DOD has made some progress in implementing the risk management
framework, including establishing risk quadrants and performance goals.
However, more work will be required for DOD to be able to put in place
a management tool, such as the risk management framework, to
strategically balance the allocation of resources across the spectrum
of its investment priorities against risk over time and to monitor
performance. The development of performance measures that clearly
demonstrate results and that are cascaded down throughout the
department would enable DOD to provide a clear roadmap of how its
activities at all levels contribute to meeting its strategic goals and
would assist the department in aligning the core processes and
resources of its four military services and multiple defense agencies
to better support a departmental or joint approach to national
security. Furthermore, the risk management framework cannot be fully
implemented until its performance goals are clearly linked to DOD's
strategic planning goals. Unless a cause and effect relationship can be
demonstrated between the department's performance measures and
strategic goals, the framework's usefulness as a tool for monitoring
DOD's execution of its strategic plan and identifying performance goals
will be severely restricted, if not eliminated. Furthermore, the fiscal
year 2006 budget submission does not provide sufficient information on
how DOD identified or assessed departmental risks to establish DOD-wide
investment priorities; thus, the linkages between the framework and the
budget are unclear. Without better measures, clear linkages, and
greater transparency, DOD will be unable to fully measure progress in
achieving strategic goals or demonstrate to Congress and others how it
considered risks and made trade-off decisions, balancing needs and
costs for weapon programs and other investment priorities.
The efforts of DOD's senior leadership to establish a risk-based and
results-oriented management approach have been impeded by some key
challenges. The lack of sustained leadership and clear lines of
accountability has hampered implementation of the risk management
framework and the establishment and achievement of implementation goals
and timelines. Strong and sustained leadership could enable DOD to
overcome resistance to change that exists in a department as massive
and complex as DOD. In addition, the establishment of implementation
goals and timelines could enable DOD to determine what progress has
been made in implementing the risk management framework. Furthermore,
the successful integration of the risk management framework into DOD's
investment decision-making processes, including recent reform
initiatives, could assist DOD in its overall transformation efforts.
Until DOD develops a risk-based and results-oriented management
approach for making investment decisions, it will likely continue to
experience a mismatch between programs and budgets, and the
proportional, rather than strategic, allocation of resources to the
services.
Recommendations for Executive Action:
To address the challenges associated with implementing the risk
management framework, or a similar risk-based management approach, we
recommend that the Secretary of Defense take the following four
actions:
* develop or refine department-level performance measures so that they
clearly demonstrate performance results and cascade those measures down
throughout the department,
* assign clear leadership with accountability and authority to
implement and sustain the risk management framework,
* develop implementation goals and timelines, and:
* demonstrate the integration of the risk management framework with
DOD's decision support processes and related reform initiatives to
improve investment decision making and manage performance results.
Agency Comments and Our Evaluation:
In written comments on a draft of this report, DOD partially concurred
with our four recommendations. DOD's written comments are reprinted in
their entirety in appendix II. DOD also provided technical comments,
which we incorporated as appropriate.
DOD partially concurred with our first recommendation. DOD stated that
it concurred with our recommendation that the Secretary of Defense
refine department-level performance measures so that they clearly
demonstrate results, but that it did not concur with the notion that
effectively cascading the risk management framework has been inhibited
by the current suite of performance measures. DOD noted that that a
number of defense components--including the Army, DOD Comptroller, the
Defense Logistics Agency, and the Defense Information Systems Agency--
have successfully cascaded departmentwide strategic goals and
implemented frameworks to measure their organization's performance. DOD
also believes that empowering the leadership at the component level to
develop measures, while ensuring strategic alignment, is the most
effective way of encouraging performance management and increasing its
utility. In our report, we acknowledge that DOD has taken positive
steps toward developing a performance monitoring system and cascading
the framework's goals and measures to defense components. However, our
recommendation addresses limitations in those measures that currently
hinder DOD's ability to use the risk management framework as a
management tool for aligning the components' performance goals and
measures with the risk management framework, or for strategic balancing
investment decisions across the risk quadrants. For example, the
majority of the risk management framework's measures are activity
measures, or initiatives, that do not monitor a specific annual
performance target, nor do these measures provide sufficient
information to determine whether the activity is on schedule or
contributes to enhancing the department's overall performance. Finally,
our recommendation is not intended to suggest that DOD not empower the
components to develop performance measures, but rather that DOD
establish a clear hierarchy of goals and measures that provide
straightforward roadmaps to demonstrate how the components' activities
contribute to meeting DOD's strategic goals.
DOD partially concurred with our second recommendation that the
Secretary of Defense assign clear leadership with accountability and
authority to implement and sustain the risk management framework. DOD
stated that, although it agrees that such leadership is key to any
successful performance management system, the department's senior
executives provide sufficient leadership and accountability for
implementing and sustaining the risk management framework. DOD also
stated that it did not agree that a new organization or bureaucratic
structure is needed to ensure successful implementation and sustainment
of the risk management framework. We agree that DOD has assigned
specific roles and responsibilities for goals and measures associated
with the risk management framework to various high-level DOD officials.
However, we based our recommendation on the fact that no single
individual, with appropriate authority, was held responsible for
ensuring that the risk management framework was implemented across the
department. Further, our recommendation does not propose that DOD set
up a new organization or bureaucratic structure, but, as stated in this
report, we continue to believe that one way to provide strong and
sustained leadership for change initiatives, such as the risk
management framework, over a number of years and various
administrations is to legislatively establish a CMO.
In partially concurring with our third recommendation to develop
implementation goals and timelines, DOD agreed that tracking progress
in implementing the risk management framework is a good management
practice. DOD stated that it has established goals and timelines for
the risk management framework that are unique to the individual
metrics, or measures, and that because the risk management framework
continually evolves over time, new metrics will be developed while
others may be retired. As we stated in the report, successful change
management efforts use implementation goals--such as, for example,
linking the risk management framework to the budget--and timelines for
meeting those goals, to pinpoint shortfalls and gaps, suggest midcourse
corrections, and build momentum by demonstrating progress. Therefore,
while DOD may continually refine the individual goals and measures
associated with the framework's risk quadrants, we believe that goals
and timelines for the overall implementation of the framework across
the department are essential for keeping this reform initiative on
track.
DOD partially concurred with our fourth recommendation that the
Secretary of Defense demonstrate the integration of the risk management
framework with DOD's decision support processes and related reform
initiatives to improve investment decision making and manage
performance results. DOD stated that the department is currently
studying ways to further integrate the risk management framework with
other decision support processes, but no single framework or decision
model can provide all the necessary information or flexibility needed
by the Secretary of Defense and his senior leadership team. We
recognize that DOD's senior leadership needs reliable information from
a variety of sources and flexibility to make decisions among
alternative actions or solutions. However, if the risk management
framework is to successfully serve as a management tool to assist
decision makers in formulating top-down strategy, balancing investment
priorities against risk over time, measuring near-and midterm outputs
against strategic goals, and focusing on actual performance results--as
intended by DOD's senior leadership--it is crucial that it be
successfully integrated with DOD's investment decision-making
processes, including recent reform initiatives.
We are sending copies of this report to interested congressional
committees; the Secretaries of Defense, Army, Navy, and Air Force; the
Commandant of the Marine Corps; and the Director, Office of Management
and Budget. We will also make copies available to others upon request.
In addition, the report will be available at no charge on GAO's Web
site at http://www/gao.gov.
If you or your staff have any questions about this report, please
contact me at (202) 512-9619 or pickups@gao.gov. Contact points for our
offices of Congressional Relations and Public Affairs may be found on
the last page of this report. GAO staff who made major contributions to
this report are listed in appendix III.
Signed by:
Sharon L. Pickup:
Director, Defense Capabilities and Management:
[End of section]
Appendix I: Scope and Methodology:
To assess to what extent the Department of Defense (DOD) has
implemented the risk management framework, we obtained and analyzed DOD
directives, briefings, and other documents that described the risk
management framework's purpose, implementation status, and performance
measures. We also obtained and analyzed DOD's 2001 Quadrennial Defense
Review and annual strategic planning and budget documents. Moreover, we
interviewed knowledgeable DOD and service officials involved with the
implementation of the risk management framework. Specifically, we
obtained testimonial evidence from officials representing the Office of
the Secretary of Defense (OSD) offices--such as Program Analysis and
Evaluation; Comptroller; Policy; Acquisition, Technology and Logistics;
and Personnel and Readiness--the Joint Staff, the military services,
and the Defense Business Board. To identify key risk-based and results-
oriented management principles, we reviewed our prior reports and other
relevant literature, including information on the balanced scorecard
concept. For example, we identified characteristics of results-oriented
performance measures. These characteristics focused on performance
measures that are (1) designed to demonstrate results by providing
information on how well the organization is achieving its goals; (2)
limited to a vital few, and balanced across priorities; and (3) used by
management to improve performance. As another example, risk-based and
results-oriented management principles indicate that leading
organizations seek to establish clear hierarchies of goals and measures
that cascade down so that subordinate units have straightforward
roadmaps to demonstrate how their activities contribute to meeting the
organization's strategy. We systematically analyzed and compared the
risk management framework's department-level performance measures with
these characteristics. However, we did not validate the procedures that
DOD has in place to ascertain the reliability of the data used to
support the performance measures. Regarding strategic planning, these
principles focused on (1) establishing clear linkages among strategic
planning goals, resources, performance goals and measures and (2)
integrating the consideration of risk into the usual cycle of agency
decision making and implementation. While these principles do not cover
all attributes associated with risk-based and results-oriented
management approaches, we believe that they are the most important ones
for assessing DOD's progress in implementing the risk management
framework.
To identify the most significant challenges, we reviewed our previous
work on change management principles. We then compared DOD's
implementation of the risk management framework to sound change
management principles and interviewed knowledgeable DOD officials about
the challenges that faced the department in implementing the risk
management framework. In addition, we reviewed our previous work to
determine to what extent deficiencies in DOD's overall business
transformation efforts might influence the implementation of the risk
management framework.
Our work was performed from October 2004 through September 2005 in
accordance with generally accepted government auditing standards.
[End of section]
Appendix II: Comments from the Department of Defense:
OFFICE OF THE SECRETARY OF DEFENSE:
PROGRAM ANALYSIS AND EVALUATION:
1600 DEFENSE PENTAGON:
WASHINGTON, DC 20301-1600:
NOV - 2 2005:
Ms. Sharon Pickup:
Director, Defense Capabilities and Management:
U.S. Government Accountability Office:
441 G Street, N.W.:
Washington, DC 20548:
Dear Ms. Pickup:
This is the Department of Defense (DoD) response to the GAO draft
report "DEFENSE MANAGEMENT: Additional Actions Needed to Enhance DoD's
Risk-Based Approach for Making Resource Decisions," dated October 4,
2005 (GAO Code 350611/GAO-06-13).
DoD has reviewed the report for technical accuracy and content and
partially concurs with the GAO's recommendations. The Department agrees
that it is vitally important to have a coherent management process to
set goals and objectives, measure performance, and respond rapidly to
changing world events. Furthermore, DoD is committed to continuously
refining its risk management framework to improve the linkages between
strategy, performance goals, and performance results. While GAO
recommends a number of actions to strengthen the risk management
framework, the Department believes that its approach to date reflects a
deliberate and tailored plan that has proven effective for managing
this enterprise.
Attached are specific comments on each recommendation, as well as
technical comments.
We appreciate the opportunity to comment on the draft report.
Signed by:
Stanley R. Szemborski:
VADM, USN:
Principal Deputy Director:
Enclosures: As Stated:
GAO DRAFT REPORT - DATED OCTOBER 4,2005:
GAO CODE 350611/GAO-06-13:
"DEFENSE MANAGEMENT: Additional Actions Needed to Enhance DoD's Risk-
Based Approach for Making Resource Decisions"
DEPARTMENT OF DEFENSE COMMENTS TO THE RECOMMENDATIONS:
RECOMMENDATION 1: The GAO recommended that the Secretary of Defense
develop or refine department-level performance measures so that they
clearly demonstrate performance results. The GAO further recommended
that the measures be cascaded through the Department. (p. 30/GAO Draft
Report):
DoD RESPONSE: Partially Concur. The Department concurs with the
recommendation to refine department-level performance measures. DoD has
worked continuously to refine both its performance measures and risk
management framework in order to clearly demonstrate performance
results and better link strategy to outcomes. In fact, one of the
primary outputs of many activity metrics are new, refined measures that
include the data necessary for quantitative evaluation. The Department
believes that the process of developing better, more refined
performance measures should continue, as no performance management
system is ever truly "complete."
However, DoD nonconcurs with the notion that effectively cascading the
risk framework has been inhibited by the current suite of performance
measures. DoD has demonstrated substantial progress in cascading
measures throughout the Department. Upon introducing the risk
management framework, the Department set out to ensure alignment across
components and Defense agencies by requiring that each organization's
performance plan or balanced scorecard: (t) reflect the balanced
scorecard risk quadrants; (2) reflect the Department's overall
performance objectives; and (3) align with and support the outcomes and
performance metrics of the next higher organization. A number of
Defense components-including the Army, DoD Comptroller, Defense
Logistics Agency (DLA), and Defense Information Systems Agency (DISA)-
have successfully cascaded Department-wide strategic goals and
implemented frameworks to measure their organization's performance. DoD
believes that empowering the leadership at the component level to
develop measures that are meaningful and relevant to one's own
organization, while ensuring strategic alignment, is the most effective
way of encouraging performance management and increasing its utility.
RECOMMENDATION 2: The GAO recommended that the Secretary of Defense
assign clear leadership with accountability and authority to implement
and sustain the risk management framework. (p. 30/GAO Draft Report):
DOD RESPONSE: Partially Concur. The Department agrees-that clear
leadership with accountability and authority are key characteristics of
any successful performance management system. However, contrary to the
GAO's assessment, the Department has taken a number of deliberate
actions to ensure proper accountability and authority. Therefore, DoD
does not agree that a new organization or bureaucratic structure is
needed to ensure successful implementation and/or sustainment of the
risk management framework.
DoD's senior executives, including Principal Staff Assistants and their
Performance Management Coordinators, provide sufficient leadership and
accountability for implementing and sustaining the risk management
framework. Specific roles and responsibilities include:
(1) the development of annual performance targets to support the risk
management framework; (2) data -collection; (3) verification and
validation of performance data; and (4) management of activities to
improve performance and achieve targets during, and following, the year
of execution. By empowering senior leaders throughout the Department to
measure organizational performance, and manage accordingly, the
Department has made great progress in ensuring accountability.
RECOMMENDATION 3: The GAO recommended that the Secretary of Defense
develop implementation goals and timelines. (p. 30/GAO Draft Report):
DoD RESPONSE: Partially Concur. DoD agrees that tracking progress in
implementing the framework is a good management practice. However, DoD
has taken the necessary steps to track implementation of the risk
management framework. For example, DoD has established goals and
timelines for the risk management framework that are unique to the
individual metrics. The Department has also established either a
performance metric or activity metric for every relevant area. Since
the risk management framework continually evolves over time to reflect
changes in strategy, priorities, and senior leadership interest, new
metrics will be developed while others may retired. Because individual
metric development occurs at different points in time, there is not one
single milestone or event that will mark completion of the risk
management framework as a whole.
RECOMMENDATION 4: The GAO recommended that the Secretary of Defense
demonstrate the integration of the risk management framework with DoD's
decision support processes and related reform initiatives to improve
investment decisionmaking and manage performance results. (p. 30/GAO
Draft Report):
DOD RESPONSE: Partially Concur. The Department agrees that integrating
the risk management framework with other decision support processes and
related reform initiatives is both a natural evolution of the risk
management framework's maturation effort and a beneficial output of the
process. As such, the Department is currently studying ways to further
integrate the risk management framework with other decision support
processes. However, the Secretary of Defense and his senior leadership
team must be able to make well-informed decisions in a variety of
circumstances. No single framework or decision model can provide all of
the necessary information and/or flexibility.
Technical Comments Draft GAO Report GAO-06-13, GAO Code 350611:
Issue: Page 2, paragraph 2. Sentence reads: "..we conducted interviews
with DoD and service officials, members of the Joint Staff, and members
of the Defense Business Board."
The Defense Business Board does not officially speak on the
Department's behalf.
Issue: Page 12, paragraph 2. The sentence reads: "44 of the 77
departmental-level measures .. are activity measures." Page 13, Table
1. Footnote describes a recoding of five performance measures to
activity measures.
DoD objects to recoding these metrics from performance metrics to
activity metrics.
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
Sharon Pickup, (202) 512-9619 or pickups@gao.gov:
Acknowledgments:
In addition to the contact named above, David Moser, Assistant
Director; Donna Byers; Gina Flacco; and Renee S. Brown made key
contributions to this report.
FOOTNOTES
[1] See GAO, 21st Century Challenges: Reexamining the Base of the
Federal Government, GAO-05-325SP (Washington, D.C.: February 2005) for
a comprehensive compendium of areas throughout the federal government
that could be considered for reexamination and review by Congress.
[2] GAO, High-Risk Series: An Update, GAO-05-207 (Washington, D.C.:
January 2005).
[3] GAO, Highlights of a GAO Forum: Mergers and Transformation: Lessons
Learned for a Department of Homeland Security and Other Federal
Agencies, GAO-03-293SP (Washington, D.C.: Nov. 14, 2002), and Results-
Oriented Cultures: Implementation Steps to Assist Mergers and
Organizational Transformations, GAO-03-669 (Washington, D.C.: July 2,
2003).
[4] S. 780, 109th Cong. §1 (2005).
[5] GAO-05-207.
[6] GAO-05-207.
[7] The Government Performance and Results Act of 1993 (Pub. L. No. 103-
62).
[8] The balanced scorecard approach was advocated by Professor Robert
Kaplan and Dr. David Norton in the November/December 1992 Harvard
Business Review.
[9] GAO-05-325SP.
[10] See for example, Committee of Sponsoring Organizations of the
Treadway Commission, Enterprise Risk Management--Integrated Framework:
Executive Summary (New York, N.Y.: September 2004).
[11] Pub. L. No. 103-62 (1993).
[12] The President's Management Agenda, by focusing on a number of
targeted areas, seeks to improve the performance management of the
federal government.
[13] For further information see: GAO, Performance Budgeting: PART
Focuses Attention on Program Performance, but More Can Be Done to
Engage Congress, GAO-06-28 (Washington, D.C.: Oct. 28, 2005);
Management Reform: Assessing the President's Management Agenda, GAO-05-
574T (Washington, D.C.: Apr. 21, 2005); Results-Oriented Government:
GPRA Has Established a Solid Foundation for Achieving Greater Results,
GAO-04-38 (Washington, D.C.: Mar. 10, 2004); and Performance Budgeting:
Observations on the Use of OMB's Program Assessment Rating Tool for the
Fiscal Year 2004 Budget, GAO-04-174 (Washington, D.C.: Jan. 30, 2004).
[14] GAO, Executive Guide: Effectively Implementing the Government
Performance and Results Act, GAO/GGD-96-118 (Washington, D.C.: June
1996) and Managing for Results: Enhancing Agency Use of Performance
Information for Management Decision Making, GAO-05-927 (Washington,
D.C.: Sept. 9, 2005).
[15] GAO/GGD-96-118.
[16] GAO/GGD-96-118 and GAO-05-927.
[17] GAO/GGD-96-118.
[18] GAO/GGD-96-118.
[19] GAO, Quadrennial Defense Review: Future Reviews Can Benefit from
Better Analysis and Changes in Timing and Scope, GAO-03-13 (Washington,
D.C.: Nov. 4, 2002).
[20] The Defense Business Board was established in 2001 by the
Secretary of Defense to provide DOD's senior leadership with leading-
edge, actionable advice on management improvements.
[21] GAO, Managing for Results: Critical Issues for Improving Agencies'
Strategic Plans, GAO/GGD-97-180 (Washington, D.C.: Sept. 16, 1997).
[22] See GAO/GGD-96-118.
[23] The Future Years Defense Program provides information on DOD's
current and planned outyear budget requests.
[24] GAO, Future Years Defense Program: Actions Needed to Improve
Transparency of DOD's Projected Resource Needs, GAO-04-514 (Washington,
D.C.: May 7, 2004).
[25] GAO, Defense Acquisitions: Assessments of Selected Major Weapon
Programs, GAO-05-301 (Washington, D.C.: Mar. 31, 2005).
[26] GAO-05-325SP.
[27] GAO-05-325SP.
[28] Joint Defense Capabilities Study Team, Joint Defense Capabilities
Study: Final Report (Washington, D.C.: December 2003).
[29] GAO, Management Reform: Elements of Successful Improvement
Initiatives, GAO/T-GGD-00-26 (Washington, D.C.: Oct. 15, 1999).
[30] GAO-05-520T and GAO-05-629T.
[31] S. 780, 109th Cong. §1 (2005).
[32] GAO-03-669.
[33] GAO-03-669.
[34] GAO-05-207.
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
