Defense Management

Additional Actions Needed to Enhance DOD's Risk-Based Approach for Making Resource Decisions Gao ID: GAO-06-13 November 15, 2005

The Department of Defense (DOD) is simultaneously conducting costly military operations and transforming its forces and business practices while it is also competing for resources in an increasingly constrained fiscal environment. As a result, GAO has advocated that DOD adopt a comprehensive threat or risk management approach as a framework for decision making. In its 2001 strategic plan, the Quadrennial Defense Review (QDR), DOD stated its intent to establish an approach--the risk management framework--to balance priorities against risk over time and monitor results against its strategic goals. GAO was asked to (1) assess the extent to which DOD has implemented the framework, including using it to make investment decisions, and (2) identify the most significant challenges DOD faces in implementing the framework, or a similar approach.

DOD has taken some positive steps to implement the framework, but additional actions are needed before DOD can show real and sustainable progress in using a risk-based and results-oriented approach to strategically allocate resources across the spectrum of its investment priorities. For example, DOD defined four risk areas, and developed performance goals and department-level measures, but it needs to, among other things, further develop and refine the measures so that they clearly demonstrate results and provide a well-rounded depiction of departmental performance. DOD's current strategic plan and goals also are not clearly linked to the framework's performance goals and measures, and linkages between the framework and budget are also unclear. While DOD officials stated that risk was considered during the fiscal year 2006 budget cycle, DOD's budget submission does not specifically discuss how DOD identified or assessed risks to establish DOD-wide investment priorities. Without better measures, clear linkages, and greater transparency, DOD will be unable to fully measure progress in achieving strategic goals or demonstrate to Congress and others how it considered risks, and made trade-off decisions, balancing needs and costs for weapon programs and other investment priorities. DOD faces four challenges that have affected the implementation of the framework. First, DOD's organizational culture resists department-level approaches to priority setting and investment decisions. Second, sustained leadership, adequate transparency, and appropriate accountability are lacking. Further, no one individual or office has been assigned overall responsibility or sufficient authority for the framework's implementation. DOD also has not developed implementation goals or timelines with which to establish accountability, or measure progress. Finally, integrating the risk management framework with decision support processes and related reform initiatives into a coherent, unified management approach for the department is a challenge that DOD plans to address during the 2005 QDR. However, GAO has concerns about DOD's ability to follow through on this integration, because of its limited success in implementing other management reforms. Unless DOD successfully addresses these challenges and effectively implements the framework, or a similar approach, it will likely continue to experience (1) a mismatch between programs and budgets, and (2) a proportional, rather than strategic, allocation of resources to the services.

Recommendations

Our recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.

Director: Team: Phone:


GAO-06-13, Defense Management: Additional Actions Needed to Enhance DOD's Risk-Based Approach for Making Resource Decisions This is the accessible text file for GAO report number GAO-06-13 entitled 'Defense Management: Additional Actions Needed to Enhance DOD's Risk-Based Approach for Making Resource Decisions' which was released on November 16, 2005. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to the Subcommittee on Readiness and Management Support, Committee on Armed Services, U.S. Senate: United States Government Accountability Office: GAO: November 2005: Defense Management: Additional Actions Needed to Enhance DOD's Risk-Based Approach for Making Resource Decisions: GAO-06-13: GAO Highlights: Highlights of GAO-06-13, a report to the Subcommittee on Readiness and Management Support, Committee on Armed Services, U.S. Senate: Why GAO Did This Study: The Department of Defense (DOD) is simultaneously conducting costly military operations and transforming its forces and business practices while it is also competing for resources in an increasingly constrained fiscal environment. As a result, GAO has advocated that DOD adopt a comprehensive threat or risk management approach as a framework for decision making. In its 2001 strategic plan, the Quadrennial Defense Review (QDR), DOD stated its intent to establish an approach”the risk management framework”to balance priorities against risk over time and monitor results against its strategic goals. GAO was asked to (1) assess the extent to which DOD has implemented the framework, including using it to make investment decisions, and (2) identify the most significant challenges DOD faces in implementing the framework, or a similar approach. What GAO Found: DOD has taken some positive steps to implement the framework, but additional actions are needed before DOD can show real and sustainable progress in using a risk-based and results-oriented approach to strategically allocate resources across the spectrum of its investment priorities. For example, DOD defined four risk areas, and developed performance goals and department-level measures, but it needs to, among other things, further develop and refine the measures so that they clearly demonstrate results and provide a well-rounded depiction of departmental performance. DOD‘s current strategic plan and goals also are not clearly linked to the framework‘s performance goals and measures, and linkages between the framework and budget are also unclear. While DOD officials stated that risk was considered during the fiscal year 2006 budget cycle, DOD‘s budget submission does not specifically discuss how DOD identified or assessed risks to establish DOD-wide investment priorities. Without better measures, clear linkages, and greater transparency, DOD will be unable to fully measure progress in achieving strategic goals or demonstrate to Congress and others how it considered risks, and made trade-off decisions, balancing needs and costs for weapon programs and other investment priorities. DOD‘s Risk Management Framework: Force Management Risk; Definition: Challenge of sustaining personnel, infrastructure, and equipment. Operational Risk; Definition: Challenge of deterring or defeating near-term threats. Future Challenges Risk; Definition: Challenge of dissuading, deterring, defeating longer-term threats. Institutional Risk; Definition: Challenge of improving efficiency (includes financial management). Source: DOD. [End of table] DOD faces four challenges that have affected the implementation of the framework. First, DOD‘s organizational culture resists department-level approaches to priority setting and investment decisions. Second, sustained leadership, adequate transparency, and appropriate accountability are lacking. Further, no one individual or office has been assigned overall responsibility or sufficient authority for the framework‘s implementation. DOD also has not developed implementation goals or timelines with which to establish accountability, or measure progress. Finally, integrating the risk management framework with decision support processes and related reform initiatives into a coherent, unified management approach for the department is a challenge that DOD plans to address during the 2005 QDR. However, GAO has concerns about DOD‘s ability to follow through on this integration, because of its limited success in implementing other management reforms. Unless DOD successfully addresses these challenges and effectively implements the framework, or a similar approach, it will likely continue to experience (1) a mismatch between programs and budgets, and (2) a proportional, rather than strategic, allocation of resources to the services. What GAO Recommends: GAO recommends that DOD take various actions to increase its chances of successfully implementing a risk-based approach for investment decision making, such as developing results-oriented measures and assigning clear leadership with appropriate accountability and authority to implement the framework. DOD partially concurred with our recommendations. www.gao.gov/cgi-bin/getrpt?GAO-06-13. To view the full product, including the scope and methodology, click on the link above. For more information, contact Sharon Pickup at (202) 512-9619 or pickups@gao.gov. [End of section] Contents: Letter: Results in Brief: Background: Despite Positive Steps, Additional Actions Needed to Fully Implement the Risk Management Framework: Cultural Resistance, Combined with the Lack of Leadership, Implementation Goals, and Process Integration, Affects DOD's Implementation of the Risk Management Framework: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Scope and Methodology: Appendix II: Comments from the Department of Defense: Appendix III: GAO Contact and Staff Acknowledgments: Tables: Table 1: Definitions and Examples of DOD Department-Level Measures (as of November 2004): Table 2: The Number of Activity and Performance Measures for Each Quadrant: Table 3: Military Service and Defense-Wide Percentage of the 2005 and 2006 Future Years Defense Programs: Table 4: Select Initiatives to Improve Investment Decision Making: Figures: Figure 1: The Risk Management Cycle: Figure 2: Comparison of the Balanced Scorecard and the Risk Management Framework: Abbreviations: CBO: Congressional Budget Office: CMO: chief management official: DOD: Department of Defense: FYDP: Future Years Defense Program: GAO: Government Accountability Office: GPRA: Government Performance and Results Act: JCIDS: Joint Capabilities Integration and Development System: OSD: Office of the Secretary of Defense: PA&E: Program Analysis and Evaluation (PA&E): PPBE: Planning, Programming, Budgeting, and Execution: P&R: Personnel and Readiness: PART: Program Assessment Rating Tool: QDR: Quadrennial Defense Review: United States Government Accountability Office: Washington, DC 20548: November 15, 2005: The Honorable John Ensign: Chairman: The Honorable Daniel K. Akaka Ranking Minority Member: Subcommittee on Readiness and Management Support: Committee on Armed Services: United States Senate: Among the 21st century challenges facing the Department of Defense (DOD) and the nation as a whole are difficult decisions concerning how to strike an affordable balance between current and future national security needs and between national security and domestic needs.[Footnote 1] For example, DOD is simultaneously maintaining a high pace of military operations for combating terrorism and transforming its military forces and business operations for the 21st century while it is also competing for federal resources in an increasingly fiscally constrained environment. We have advocated that DOD--as well as the rest of the federal government--adopt a comprehensive threat or risk management approach as a framework for decision making.[Footnote 2] This approach would fully link strategic goals to plans and budgets; assess the values and risks of various courses of actions as a tool for reexamining defense programs, setting priorities, and allocating resources; and use performance measures to assess outcomes. To its credit, DOD introduced a balanced scorecard for risk management, commonly known as the risk management framework, in its strategic plan, the 2001 Quadrennial Defense Review (QDR) report. The 2001 strategic plan articulated the new administration's emphasis on transforming military forces and defense business practices to meet the emerging challenges facing our nation. DOD intended the framework to be used as a management tool to focus DOD's efforts on implementing the defense program as outlined in the strategic plan. In particular, DOD's senior leadership intended the risk management framework to assist decision makers in formulating top-down strategy, balancing investment priorities against risk over time, measuring near-and midterm outputs against strategic goals, and focusing on actual performance results. According to DOD officials, the risk management framework also was intended to increase transparency within the department over the decision-making process. During the ongoing 2005 QDR, DOD plans to refine the risk management framework. You asked us to examine the status of DOD's efforts to adopt a risk- based approach to decision making, given the emphasis that DOD was placing on the risk management framework. In response, we (1) assessed the extent to which DOD has implemented its risk management framework, including the extent to which DOD has used the framework to make investment decisions; and (2) identified the most significant challenges DOD faced in implementing the risk management framework or a similar risk-based and results-oriented management approach. To assess the extent to which DOD has implemented the risk management framework, we analyzed key documents, policy guidance, data, and interview results, and compared the analysis to the principles for managing risk and results identified in prior GAO reports. In addition, we conducted interviews with DOD and service officials, and members of the Joint Staff. We discussed the department's progress in implementing the risk management framework with members of the Defense Business Board. We also analyzed DOD's department-level performance goals and measures that are associated with the risk management framework and assessed how DOD reported that information externally. We did not validate the appropriateness of the risk management framework's risk quadrants or the procedures that DOD has in place to ascertain the reliability of performance data and we also did not assess the basis for DOD's investment decisions. To identify the most significant challenges DOD faced in implementing the risk management framework, we analyzed documents, data, and interview results, and compared the results of this analysis to the key practices to assist mergers and organizational transformation identified in prior GAO reports.[Footnote 3] A more detailed discussion of our scope and methodology is presented in appendix I. Our work was performed from October 2004 through September 2005 in accordance with generally accepted government auditing standards. Results in Brief: DOD has taken positive steps toward implementing the risk management framework; however, additional actions are needed before the framework is fully implemented and DOD can demonstrate real and sustainable progress in using a risk-based and results-oriented approach to strategically allocate resources across the spectrum of its investment priorities. For example, while DOD established four risk areas, or quadrants, and developed performance goals and measures of two types-- activity measures (measures to track initiatives) and performance measures--the majority of these measures do not provide sufficient information to monitor performance against the risk quadrants' goals. Specifically, and contrary to results-oriented management principles, the risk management framework's measures (1) do not clearly demonstrate results, (2) do not provide a well-rounded depiction of performance across the department, and (3) are not being systemically monitored across all quadrants, except for the force management quadrant. In addition, the framework's performance goals and measures are not clearly linked to DOD's current strategic plan and strategic goals. Lacking measures that follow results-oriented management principles and clear linkages to strategic goals, DOD may be unable to provide a clear roadmap of how its activities at all levels contribute to meeting DOD's strategic goals. Finally, although DOD officials stated that risk was considered in the fiscal year 2006 budget cycle, the fiscal year 2006 budget submission does not include any specific information on how DOD systematically identified or assessed departmental risks to establish DOD-wide investment priorities. Therefore, the linkages between the risk management framework and the budget are unclear. Without better measures, clear linkages, and greater transparency, DOD will be unable to fully measure progress in achieving strategic goals or demonstrate to Congress and others how it considered risks and made trade-off decisions, balancing needs and costs for weapon system programs and other investment priorities. DOD faces four key challenges that affect its ability to fully implement the risk management framework, or a similar risk-based and results-oriented management approach: (1) overcoming cultural resistance to the transformational change represented by such an approach in a department as massive, complex, and decentralized as DOD; (2) maintaining sustained leadership and clear accountability for this cultural transformation; (3) providing implementation goals and timelines to gauge progress in transforming the culture; and (4) integrating the risk management framework with decision support processes and related reform initiatives into a coherent, unified management approach for the department. Our prior work on results- oriented management and organizational transformation and mergers has shown that addressing these challenges is at the center of successful change management efforts in leading organizations. DOD is having difficulties implementing the framework because it has not addressed these four challenges. With respect to the first challenge, DOD's size and complexity result in a culture that makes developing department- level approaches to priority setting and investment decision making difficult. For example, the allocation of budgets on a proportional, rather than a strategic, basis among the services is a long-standing budgetary problem that we have reported about for years. Second, the lack of sustained leadership and clear accountability for the framework's implementation has resulted in a lack of emphasis and understanding of its status and purpose within the department. Because of the lack of sustained leadership for other management reform efforts, we have supported legislation to create a chief management official (CMO) at DOD to provide this leadership.[Footnote 4] Third, DOD did not establish implementation goals or timelines with which to establish accountability, measure progress, and build momentum. Finally, integrating the risk management framework with other decision support processes and related reform initiatives into a coherent, unified management approach is a challenge that DOD intends to address in the ongoing 2005 QDR. Illustrating this challenge, DOD is attempting to implement the risk management framework while it is also shifting to biennial budgeting and reforming defense planning. Our work has shown that if risk-based and results-oriented management approaches are to be successfully implemented, they must be integrated into the usual cycle of agency decision making. Unless DOD addresses these challenges and successfully implements the risk management framework, or a similar approach, it may continue to experience (1) a mismatch between programs and budgets, and (2) the proportional, rather than strategic, allocation of resources to the services. Therefore, Congress may have insufficient transparency into how DOD has identified and assessed risks and made trade-offs in its investment decision making. In this report, we recommend that DOD take various actions to increase its chances of successfully implementing a risk-based approach for investment decision making. This includes developing results-oriented measures and assigning clear leadership with appropriate accountability and authority to implement and sustain the risk management framework, or a similar approach. In written comments on a draft of this report, DOD partially concurred with our recommendations. DOD's comments and our evaluation of them are on page 25 of this report. Background: In our report, High-Risk Series: An Update,[Footnote 5] we identified agencies' lack of comprehensive risk management strategies as an emerging challenge for the federal government. Increasingly limited fiscal resources across the federal government, coupled with the emerging requirements from the changing security environment, emphasize the need for DOD to develop a risk-based strategic investment approach. For this reason, we have advocated that DOD adopt a comprehensive risk management approach for decision making.[Footnote 6] Furthermore, DOD and other federal agencies are required by statute to develop a results- oriented management approach to strategically allocate resources on the basis of performance.[Footnote 7] The balanced scorecard--a concept to balance an organization's focus across financial, customer, internal business, and learning and growth management areas--is one approach for developing results-oriented management that government agencies have recently started to adopt.[Footnote 8] At the direction of the Secretary of Defense, DOD developed a risk management framework that DOD later aligned with its results-oriented management activities through a DOD balanced scorecard. Risk Management Is an Emerging 21st Century Challenge: An emerging challenge for the federal government involves the need for the completion of comprehensive national threat and risk assessments in a variety of areas. For example, emerging requirements from the changing security environment, coupled with increasingly limited fiscal resources across the federal government, emphasize the need for agencies to adopt a sound approach to establishing resource decisions.[Footnote 9] We have advocated that the federal government, including DOD, adopt a comprehensive threat or risk management approach as a framework for decision making that fully links strategic goals to plans and budgets, assesses values and risks of various courses of actions as a tool for setting priorities and allocating resources, and provides for the use of performance measures to assess outcomes. Based on our review of the literature,[Footnote 10] as shown in figure 1, the goal of risk management is to integrate systematic concern for risk into the usual cycle of agency decision making and implementation. Figure 1: The Risk Management Cycle: [See PDF for image] [End of figure] A risk management cycle represents a series of analytical and managerial steps, basically sequential, that can be used to assess risk, evaluate alternatives for reducing risks, choose among those alternatives, implement the alternatives, monitor their implementation, and continually use new information to adjust and revise the assessments and actions, as needed. Adoption of a risk management cycle such as this can aid in assessing risk by determining which vulnerabilities should be addressed, and how they should be addressed, within available resources. For the purposes of this report, we focused on the stages of the risk management cycle that involve DOD's actions to set strategic goals and objectives, establish investment priorities based on risk assessments, and implementation and monitoring. Risk management's objectives are essentially the same as those of good management, and they are consistent with the broad economy and efficiency objectives of good government--namely, to provide better outcomes for the same amount of money, or to provide the same outcomes with less money. Therefore, risk management's objectives are also compatible with those of the federal government's results-oriented management approach, which was enacted in the Government Performance and Results Act (GPRA) of 1993,[Footnote 11] and the balanced scorecard approach. Congress enacted GPRA to focus the federal government on achieving results through the creation of clear links between the process of allocating scarce resources and an agency's strategic goals, or the expected results to be achieved with those resources. Building on GPRA's foundation, the current administration has taken steps to strengthen the integration of budget, cost, and performance information by including budget and performance integration as one of its management initiatives under the umbrella of the President's Management Agenda.[Footnote 12] The Budget and Performance Integration initiative includes efforts such as the Program Assessment Rating Tool (PART), improving outcome measures, and improving monitoring of program performance.[Footnote 13] The balanced scorecard approach is a management tool that some federal agencies have adopted to help them translate the strategy set forth in a results-oriented management approach into the operational objectives that drive both behavior and performance. The balanced scorecard consists of four management areas that organizations should focus on--financial, customer, internal business, and learning and growth. DOD's 2001 Strategic Plan Outlines a New Risk Management Framework: DOD introduced the risk management framework in its strategic plan, the 2001 QDR report. The 2001 strategic plan articulated the new administration's emphasis on transforming military forces and defense business practices to meet the changing threats facing our nation. In his guidance to the department for the 2001 QDR strategic planning process, the Secretary of Defense stated the need for DOD to use a risk mitigation approach for balancing force, resource, and modernization requirements across defense planning timelines. This guidance also stated that DOD must include the identification of output-based measures to reduce inefficiencies through the department in any approach to risk management. Building on the guidance, the 2001 QDR outlined DOD's risk management framework. According to the QDR, the framework would enable DOD to address the tension between preparing for future threats and meeting the demands of the present with finite resources. It was also intended to ensure that DOD was sized, shaped, postured, committed, and managed with a view toward accomplishing the strategic plan's defense policy goals. DOD adapted the balanced scorecard concept to the risk management framework by substituting the four dimensions of risk--force management, operational, future challenges, and institutional--for the scorecard's four management areas. The risk management framework was to be a transformational tool that would provide a balanced perspective of the organization's execution of strategy and ensure a top-down approach. The 2002 policy guidance also designated four preliminary performance goals for each of the four risk quadrants. In addition, the guidance required that performance goals and measures were to be cascaded to the services and defense agencies. Figure 2 shows a comparison, as provided by DOD. Figure 2: Comparison of the Balanced Scorecard and the Risk Management Framework: [See PDF for image] [End of figure] Despite Positive Steps, Additional Actions Needed to Fully Implement the Risk Management Framework: Despite positive steps, DOD needs to take additional actions before the risk management framework is fully implemented and DOD can demonstrate real and sustainable progress in using a risk-based and results- oriented approach to strategically allocate resources across the spectrum of its investment priorities. For example, DOD is still in the process of developing department-level measures for the framework that address results-based management principles, such as linking performance information to strategic goals so that this information can be used to monitor performance results and determine how well the department is doing in achieving its strategy. Without more results- oriented performance measures, DOD may be unable to provide the services and other defense components with clear roadmaps of how their activities contribute to meeting DOD's strategic goals. In addition, the framework's performance goals and measures are not clearly linked to DOD's current strategic plan and strategic goals. Furthermore, the extent to which the risk management framework is linked to the budget cycle is unclear. Without better measures, clear linkages, and greater transparency, DOD will be unable to fully measure progress in achieving strategic goals or demonstrate to Congress and others how it considered risks and made trade-offs in making investment decisions. Developing a Set of Measures That Can Be Used to Monitor Performance Is a Work in Progress: DOD has taken positive steps toward developing measures for each of the performance goals under the framework's four risk quadrants; however, developing a set of measures that can be used to monitor performance results is still a work in progress. Based on GAO's prior work on results-based management principles, we found that leading organizations' performance measures are: (1) designed to demonstrate results, or provide information on how well the organization is achieving its goals; (2) limited to a vital few, and balanced across priorities; and (3) used by management to improve performance.[Footnote 14] However, the set of measures DOD has developed for the risk management framework do not adequately address these principles. While DOD established four risk quadrants and developed performance goals and measures of two types--activity measures (measures to track initiatives) and performance measures--the majority of its measures do not provide sufficient information to monitor performance against the risk quadrants' goals. First, DOD officials acknowledge that establishing department-level measures for the framework that demonstrate results is still a work in progress, as the majority of the risk management framework's measures require further development or refinement. In fact, as shown in table 1, 44 of the 77 department-level measures for all four quadrants, or over 50 percent, are activity measures. According to DOD sources, activity measures are to result in a new performance measure, a new baseline or benchmark, or define a new capability, rather than monitor a specific annual performance target. Once these activities are completed, DOD officials stated that the department will be better able to monitor department-level performance against strategic goals. However, our analysis found that the activity measures, as defined in DOD's external reports, typically do not provide sufficient information to monitor the department's progress in achieving the stated goal they are to measure, such as developing a new performance measure or baseline. The desired outcomes for activity measures generally state that a task was or will be completed by a certain date but they do not provide sufficient information on whether the activity is on schedule, the interdependencies among tasks, or the contribution toward enhancing the department's performance. Therefore, Congress and other external stakeholders lack information and adequate assurances that DOD is making progress in implementing a risk-based and results-oriented management approach to making investment decisions. Table 1: Definitions and Examples of DOD Department-Level Measures (as of November 2004): Type: Activity measures; Number: 44[A]; Definition: Activity measures track developmental activities, are usually qualitative, and track key milestones or events in lieu of a specific annual performance target; Examples: Deny enemy advantages and exploit weaknesses; Description of desired outcome monitored by measure: Roadmap will be complete by the end of fiscal year 2005. Examples: Enhance homeland defense and consequence management; Description of desired outcome monitored by measure: Strategy will be complete by the first quarter of fiscal year 2005. Type: Performance measures; Number: 33[A]; Definition: Performance measures track current outputs and set quantitative annual targets for performance that are measurable; Examples: Reserve component enlisted recruiting quality; Description of desired outcome monitored by measure: Target > 90% of recruits holding high school diplomas; Actual 88% of recruits holding high school diplomas. Examples: Reduce customer wait time (in days); Description of desired outcome monitored by measure: Target 15 days from order to receipt for material goods; Actual 24 days from order to receipt for material goods. Source: GAO analysis of the Risk Management Framework's performance measures. [A] We have recoded five performance measures as activity measures as these measures tracked milestones and events, which corresponds with DOD's definition of an activity measure. [End of table] Second, DOD's department-level performance measures are still a work in progress in that these measures do not provide a well-rounded depiction of DOD's performance. In our previous work, we have found that performance measurement efforts that are not balanced across priorities may skew an agency's performance and keep its senior leadership from seeing the whole picture.[Footnote 15] For example, in developing department-level measures for the risk management framework, DOD appears to have overemphasized its force management priorities at the expense of operational risk. As illustrated in table 2, the operational risk quadrant has no performance measures, while the force management risk quadrant has a total of 36 measures, including 15 activity measures and 21 performance measures. Table 2: The Number of Activity and Performance Measures for Each Quadrant: Activity measures: Operational: 15; Performance measures: Operational: 21; Total measures: Operational: 36[A]. Force Management: Operational; Activity measures: 9; Performance measures: 0; Total measures: 9. Activity measures: 11; Performance measures: 10; Total measures: 21[B]. Force Management; Activity measures: 9; Performance measures: 2; Total measures: 11[C]. Source: GAO analysis of DOD data. [A] We have recoded two performance measures as activity measures as these measures tracked milestones and events, which corresponds with DOD's definition of an activity measure. [B] We have recoded one performance measure as an activity measure as this measure tracked milestones and events, which corresponds with DOD's definition of an activity measure. [C] We have recoded two performance measures as activity measures as these measures tracked milestones and events, which corresponds with DOD's definition of an activity measure. [End of table] In providing technical comments to a draft of this report, DOD objected to our recoding of five department-level performance measures as activity measures. We recoded these measures because they tracked milestones and events, which corresponded to DOD's definition of an activity measure. The measures we recoded addressed the following: * a civilian human resources strategic plan, * a military human resources strategic plan, * monitor the status of defense technology objectives, * strategic transformation appraisal, and: * support acquisition excellence goals. Finally, DOD officials indicated that DOD is systematically using performance measures to monitor progress and improve performance for only one risk quadrant, although individual measures under the other three risk quadrants may be monitored. We have found that leading organizations use performance information to improve organizational performance and identify performance gaps, and to provide incentives that reinforce a results-oriented management approach.[Footnote 16] According to DOD officials, the force management quadrant is the only quadrant that is managed by one individual and one office--the Under Secretary of Defense for Personnel and Readiness and his office. These officials stated that this situation is a critical factor in the progress DOD has made in systematically monitoring performance across the force management quadrant on a routine basis. For example, officials stated that the Under Secretary of Defense personally leads quarterly monitoring sessions on the force management quadrant's performance. DOD officials also told us that the Under Secretary of Defense for Personnel and Readiness has greatly facilitated this monitoring by developing a centralized database to capture the performance data used to track DOD's performance in meeting the quadrant's goals. Unless all of the risk management framework's quadrants are systematically monitored, implementation of the framework may be hindered and the framework risks becoming a paper-driven, compliance exercise. Indeed, one DOD official told us that he views the risk management framework and its measures as a "reporting drill" and, in addition, his office would not change its processes if DOD was to no longer use the framework. Cascading the Risk Management Framework's Goals and Measures Is an Ongoing Effort: DOD is still in the process of cascading the risk management framework's goals and measures to the services. We have found that leading organizations seek to establish clear hierarchies of goals and measures that cascade down so that subordinate units have straightforward roadmaps to demonstrate how their activities contribute to meeting the organization's strategy.[Footnote 17] According to DOD officials, all of the services are attempting to align their existing performance measures with the department-level performance goals and measures. However, service officials said that it is challenging to cascade the department-level activity measures, because these measures represent very broad initiatives that may not be applicable at all DOD levels. Officials from one service said they have had to develop new measures to align with the department-level measures, because they had been assessing performance with fewer measures than the Office of the Secretary of Defense had developed. Developing a Strategic Plan with Clear Linkages between the Risk Management Framework and Strategic Goals Is a Critical Next Step: The risk management framework's performance goals and measures are not clearly linked--a key principle of results-oriented management--to a coherent strategic plan.[Footnote 18] The development of such a strategic plan is a critical next step in using a risk-based and results-oriented approach to making investment decisions. Without these linkages, DOD cannot easily demonstrate how achievement of a performance goal or measure contributes to the achievement of strategic goals and ultimately the organization's mission. Our previous work indicated that DOD's strategic plan, the 2001 QDR, did not provide a sound foundation for the risk management framework.[Footnote 19] We reported that the usefulness of the 2001 QDR was limited by the lack of focus on longer-term threats and requirements for critical support capabilities, and provided few insights into how future threats and planned technical advances could affect future force requirements. In turn, this lack of focus and insight limited the QDR's usefulness as a foundation for fundamentally reassessing U.S. defense plans and programs and for balancing resources across near-and midterm risks. DOD officials indicated that DOD has not yet defined the linkages between the risk management framework's performance goals and the strategic goals in the 2001 QDR. Furthermore, the Defense Business Board's official minutes for its July 28, 2005, meeting contained a recommendation that the Secretary of Defense define department-level objectives, which should then be cascaded down the department.[Footnote 20] In discussing the ongoing 2005 QDR, DOD stated that although the department would continue its efforts to do so, establishing these linkages was very challenging because of the size and scope of DOD's operations. However, as suggested by the Defense Business Board and our previous work, if DOD's strategic plan is to drive the department's operations, a straightforward linkage is needed among strategic goals, annual performance goals, and day-to-day activities.[Footnote 21] The ongoing 2005 QDR offers DOD the opportunity to strengthen its strategic planning. Although Risk Considered, Linkages Between the Risk Management Framework and Budget Are Unclear: According to DOD officials, the department has begun to consider risk in its investment decision making; however, the full extent to which the framework's risk-based and results-oriented approach has been linked to the fiscal year 2006 budget cycle is unclear. Our work indicates that leading organizations link strategy to the budget process through results-oriented management to evaluate potential investments or initiatives.[Footnote 22] DOD sources indicated that the department has begun to consider risk during its usual cycle of investment decision making. For example, according to DOD sources, the Secretary of Defense articulated broad areas for increasing or decreasing risk under each quadrant in the fiscal years 2006-2011 planning guidance, leaving it up to the defense components to decide how to structure their investment decisions within those broad areas consistent with the Secretary's risk guidance. In addition, DOD officials stated that the framework has increased awareness within the department on the need to balance risk over time. For example, when DOD reduced the fiscal years 2006-2011 defense program by $30 billion, DOD officials stated that the department did not take the traditional budgetary approach of cutting each defense component's budget by a certain percentage. Instead, DOD officials stated that the Secretary of Defense used a collaborative approach with service participation to discuss where to take the budget reductions and how these cuts would affect risk, although DOD officials offered various views on how extensively the framework was used to make those decisions. Second, DOD required that the services and other defense components offset any funding increase in one area with a funding decrease in another area for the fiscal years 2006-2007 budget submission. According to DOD officials, risk--whether on the basis of "professional judgment" or analysis--was considered in these deliberations. For example, the Army's plan for fiscal years 2006-2023 articulated areas for increasing risks so that it could decrease risk in the operational risk dimension by investing in current capacity. However, the fiscal year 2006 budget submission does not include any specific information on how DOD systematically identified or assessed departmental risks to establish DOD-wide investment priorities. For example, the military services' share of the Future Years Defense Program (FYDP) remained relatively unchanged from fiscal year 2005 to fiscal year 2006 (see table 3),[Footnote 23] providing one indication that the risk management framework may not yet be a useful tool for balancing departmental risks across the services. Table 3: Military Service and Defense-Wide Percentage of the 2005 and 2006 Future Years Defense Programs: Department of the Army; 2005 Percentage of FYDP: 24.23; 2006 Percentage of FYDP: 24.63; Percentage change by department: 0.40. Department of the Navy; 2005 Percentage of FYDP: 29.75; 2006 Percentage of FYDP: 29.47; Percentage change by department: -0.28. Department of the Air Force; 2005 Percentage of FYDP: 29.80; 2006 Percentage of FYDP: 29.82; Percentage change by department: 0.02. Defense-wide; 2005 Percentage of FYDP: 16.22; 2006 Percentage of FYDP: 16.08; Percentage change by department: -0.14. Total; 2005 Percentage of FYDP: 100.00; 2006 Percentage of FYDP: 100.00. Source: GAO analysis of DOD FYDP data. Note: Totals may not add due to rounding. [End of table] DOD has reported on the risk management framework in the department's GPRA and other reporting requirements. For example, the fiscal year 2004 Performance and Accountability Report describes what DOD is doing, or plans to do, to define, measure, and monitor performance goals in the four risk quadrants but does not discuss the implementation status of the risk management framework. Furthermore, the fiscal year 2004 report, the most recent available, provided insufficient information to assist Congress in overseeing how DOD plans to prioritize investment decisions within or across the risk quadrants. Without more detailed information, Congress may have insufficient transparency into how DOD has identified and assessed risks and made trade-offs in its investment decision making. In addition, we reported in May 2004 that congressional visibility over investment decision making also was limited by the absence of linkages between the risk management framework and military capabilities planning and the FYDP.[Footnote 24] Because the FYDP lacked these linkages, we concluded that decision makers could not use it to determine how a proposed increase in capability would affect the risk management framework. Our work also has shown that the FYDP may understate the costs of weapon system programs; therefore, DOD may be starting more programs than it can afford. For example, our assessment of 54 major programs, representing an investment of over $800 billion, found that the majority of these programs were costing more and taking longer to develop than planned.[Footnote 25] Problems occurred because of DOD's overly optimistic planning assumptions about the long-term costs of weapon system programs and its failure to capture early on the requisite knowledge that is needed to efficiently and effectively manage program risks. When DOD has too many programs competing for funding and approves programs with low levels of knowledge, it is accepting the attendant likely adverse cost and schedule risks. As a result, it will probably get fewer quantities for the same investment or face difficult choices on which investments it cannot afford to pursue. The findings of our work suggest that having a departmentwide investment strategy for weapon systems, to allocate resources across investment priorities, would help reduce these risks. Cultural Resistance, Combined with the Lack of Leadership, Implementation Goals, and Process Integration, Affects DOD's Implementation of the Risk Management Framework: Four key challenges impede DOD's progress toward implementing the risk management framework. The first implementation challenge facing DOD is overcoming cultural resistance to change in a department as massive, complex, and decentralized as DOD. The second challenge is the lack of sustained leadership, and the third challenge is the absence of implementation goals and timelines. These challenges relate to DOD's failure to follow crucial transformational steps. The fourth challenge- -integrating the risk management framework with decision support processes and related reform initiatives, into a coherent, unified management approach for the department--relates to key results-oriented management practices. Unless DOD addresses these challenges and successfully implements the risk management framework, or a similar approach, it may continue to experience (1) a mismatch between programs and budgets, and (2) the proportional, rather than strategic, allocation of resources to the services. Transforming DOD's Organizational Culture Is a Significant Challenge: Transforming DOD's organizational culture--from a focus on inputs and programs to strategically balancing investment risks and monitoring outcomes across the department--through the implementation of the risk management framework is a significant challenge for the department for several reasons. First, as we noted in our 21st Century Challenges report, to successfully transform, DOD needs to overcome the inertia of various organizations, policies, and practices that became rooted in the Cold War era.[Footnote 26] The department's expense, size, and complexity, however, make overcoming this resistance and inertia difficult. In fiscal year 2004, DOD reported that its operations involved $1.2 trillion in assets, $1.7 trillion in liabilities, over $605 billion in net cost of operations, and over 3.3 million military and civilian personnel. For fiscal year 2005, DOD received appropriations of about $417 billion. Moreover, execution of its operations spans a wide range of defense organizations, including the military services and their respective major commands and functional activities, numerous large defense agencies and field activities, and various combatant and joint operation commands, which are responsible for military operations for specific geographic regions or theaters of operations. Second, DOD's highly decentralized management structure is another contributing factor that makes cultural change difficult. Although under the authority, direction, and control of the Secretary of Defense, the military services have the legislative authority to organize, equip, and train the nation's armed forces for combat under Title 10 of the U.S. Code. Furthermore, Congress directly appropriates funds to the services for programs and activities that support these purposes. In the opinion of knowledgeable DOD officials, this legislative authority has resulted in a culture that makes it difficult to develop department-level, or joint, management approaches. For example, the allocation of budgets on a proportional, rather than a strategic basis, among the military services is a long-standing budgetary problem that we have identified as a major management challenge for the department.[Footnote 27] In addition, the Joint Defense Capabilities Study, chartered by the Secretary of Defense in March 2003, made the following observations on how DOD's organizational culture does not reinforce a departmental or joint approach to investment decision making and results management:[Footnote 28] * DOD's bottom-up strategic planning process did not support early senior leadership involvement and did not provide integrated departmentwide objectives, priorities, and roles as a framework for planning joint capabilities. * Service-centric focus on programs and weapons platforms resulted in a process that did not provide an accurate picture of joint needs, nor did it provide a consistent view of priorities and acceptable risks across the department. * The resulting budget did not optimize capabilities at either the department or the service level. * Accountability and feedback focused on monetary input rather than output; therefore, much of the information provided did not support the senior leaders' decision making as it did not tell how well the department was being resourced to meet current and future mission requirements. Lack of Sustained Leadership and Appropriate Accountability Has Challenged DOD's Implementation of the Risk Management Framework: The lack of sustained leadership attention and appropriate accountability has challenged DOD's progress in implementing the risk management framework. Our work has indicated that sustained leadership is a key transformational, or change management, practice.[Footnote 29] However, knowledgeable DOD officials indicated that DOD's senior leadership did not provide sustained attention to the framework's implementation. For example, a DOD official actively involved in the framework's implementation stated that meetings with senior leadership that were to provide oversight of the framework's implementation have not been regularly scheduled. DOD officials indicated that as a result of this lack of sustained leadership, DOD has not placed much emphasis on implementing the risk management framework at the department level. In addition, other DOD officials stated that changes in leadership have made it difficult to implement the risk management framework or develop performance measures. For example, since October 2004, DOD has experienced turnover in the following senior level positions, including the Deputy Secretary of Defense; the Under Secretary of Defense for Acquisition, Technology and Logistics; and the Director of Program Analysis and Evaluation (PA&E). Lacking sustained leadership attention, DOD officials offered conflicting perspectives on the status of the risk management framework with some officials suggesting that the framework had been overtaken by other performance-based or risk-based management initiatives while another suggested that the framework was primarily a compliance exercise. DOD officials also held differing perspectives on the purpose of the framework, including the beliefs that it was developed to monitor the Secretary of Defense's priority areas or that it was a programming and budgeting tool. Implementation of the risk management framework has also been challenged by the lack of clear lines of authority and appropriate accountability. No single individual or organization has been given overarching leadership responsibilities, authority, or the accountability for achieving the framework's implementation. Instead, the responsibility for various tasks and performance measures have been spread among several organizations, including the Director, PA&E; the Under Secretary of Defense for Personnel and Readiness (P&R); and the Under Secretary of Defense, Comptroller/Chief Financial Officer. We testified in April 2005 that as DOD embarks on large-scale change initiatives, the complexity and long-term nature of these initiatives require the development of an executive position capable of providing strong and sustained leadership--over a number of years and various administrations.[Footnote 30] For this reason, we have supported legislation to create a CMO at DOD to provide such sustained leadership.[Footnote 31] A CMO could also provide the leadership needed to successfully develop a risk-based and results-oriented management approach at DOD, such as the risk management framework. Lack of Implementation Goals and Timelines Further Challenges DOD's Implementation of Risk Management Framework: Accountability for implementation of the risk management framework also has been hindered by the absence of implementation goals and timelines with which to gauge progress. As we have previously reported, successful change management efforts use implementation goals and timelines to pinpoint performance shortfalls and gaps, suggest midcourse corrections, and build momentum by demonstrating progress.[Footnote 32] However, DOD's limited guidance on the risk management framework did not establish implementation goals and timelines, nor did it require that implementation goals and timelines be developed. According to knowledgeable DOD officials, DOD did not see the need for implementation goals or timelines because the framework was not meant to change processes or create new ones, but rather was a management tool to improve upon investment decision-making processes. Regardless of how DOD classifies the risk management framework, we have found that implementation goals and timelines are essential to any transformational change, such as that envisioned by the Secretary of Defense with the risk management framework, because of the number of years it can take to complete the change.[Footnote 33] Moreover, the absence of implementation goals and timelines makes it difficult to determine whether progress has been made in implementing the framework over the last 2 ½ years, and whether DOD's revisiting of the framework during the 2005 QDR represents an evolutionary progression or implementation delays. Integrating the Risk Management Framework with Decision Support Processes and Related Reform Initiatives Is a Significant Challenge: DOD faces a significant challenge integrating the risk management framework with decision support processes for planning, programming, and budgeting and with related reform initiatives into a coherent, unified management approach. The goal of both risk management and results-oriented management is to integrate the systematic concern for risk and performance into the usual cycle of agency decision making and implementation. DOD's challenge in meeting these goals is demonstrated by the number of initiatives, as shown in table 4, that DOD has put in place to improve investment decision making and manage performance results. For example, both capabilities planning and the risk management framework are to define risks and develop performance measures but, according to DOD officials, the department is still determining how to align capabilities planning with the risk management framework. Other initiatives, including GPRA and PART, are also to develop performance measures and DOD is still working on integrating these initiatives with the risk management framework and individual performance monitoring approaches of the services and other defense components into a single, integrated system. In December 2002, the Deputy Secretary of Defense issued a memorandum to correct this situation by requiring the alignment of the risk management framework and the President's Management Agenda with DOD's results-oriented management activities, including those associated with GPRA. Table 4: Select Initiatives to Improve Investment Decision Making: Initiative: Two-Year Planning, Programming, Budgeting, and Execution Process (PPBE); Description: In 2003, DOD implemented a 2-year cycle for its strategic planning, program development, and resource determination process. DOD stated that this change was needed to integrate DOD's processes for strategic planning, identification of needs for military capabilities, systems development and acquisition, and program and budget development. During the second year of the biennial budget, DOD is to focus on budget execution and program performance. Initiative: Enhanced Planning Process; Description: In fiscal year 2004, DOD initiated a reform of defense planning to make it more responsive and adaptive to the needs of senior decision makers. The process is to result in fiscally constrained guidance and priorities-- for military forces, modernization, readiness and sustainability, and supporting business processes and infrastructure activities--for program development. The enhanced planning process is to integrate the outcomes of operational, enterprise, and capabilities planning efforts in a document called the Joint Programming Guidance. The Joint Programming Guidance is to provide a link between planning and programming, and it is to provide guidance to the DOD components for the development of their program proposals. Initiative: Capabilities Planning; Description: The 2001 QDR announced a defense strategy built around the concept of shifting to a "capabilities-based" approach to defense. According to the 2001 QDR, while DOD cannot know with confidence what nation, group of nations, or nonstate actor might pose a threat to U.S. vital interests, it is possible to anticipate the capabilities an adversary might employ. Capabilities planning is to provide a top-down, competitive approach to weigh options against resource constraints across a spectrum of challenges and to apportion risk against those challenges. It is also to enable risk assessments and trade-off decisions across DOD organizational stovepipes. The new concept stresses joint solutions to problems, requires the identification of risk trade-offs within and across mission areas, and treats uncertainty explicitly. Initiative: Program/Budget Framework Initiative; Description: As part of the financial management enterprise initiatives of the Business Management Modernization Program, this initiative is to provide a foundation for a new program and budget data structure using a common language that enables senior level DOD decision makers to weigh options versus resource constraints across a spectrum of challenges. The framework is to consist of a number of related data transparency initiatives that span across all portions of the PPBE process, including creating department-level definitions for the four risk quadrants. One of the stated benefits is establishing an ability to view programs and resources based on the risk management framework. Initiative: Joint Capabilities Integration and Development System (JCIDS); Description: A system for the Joint Staff to assess gaps in military joint warfighting capabilities and recommend solutions to resolve those gaps. This system is replacing DOD's requirements- generation process for major acquisitions in an effort to shift the focus to a more capabilities-based approach for determining joint warfighting needs rather than a threat-based approach focused on individual systems and platforms. Under this system, boards comprised of high-level DOD civilians and military officials are to identify future capabilities needed around key functional concepts and areas, such as command and control, force application, and battlespace awareness, and to make trade-offs among air, space, land, and sea platforms in doing so. Initiative: President's Management Agenda; Description: The President's Management Agenda contains five initiatives aimed at improving federal agency management and performance: (1) strategic human capital management, (2) competitive sourcing, (3) improved financial performance, (4) expand electronic government, and (5) budget and performance integration. The President cited our work on high-risk areas and major management challenges in developing his initiatives, and implementation of the agenda has reinforced the need to focus agencies' efforts on achieving key management and performance improvements. Initiative: Budget and Performance Integration; Description: The budget and performance integration initiatives of the President's Management Agenda include elements such as the PART used to review programs, an emphasis on improving outcome measures, and improving monitoring of program performance. PART is the central element in the performance budgeting piece of the President's Management Agenda. PART builds on GPRA by actively promoting the use of results-oriented information to assess programs in the budget. Source: GAO analysis. [End of table] We note that these reform initiatives address key business processes within the department and that we have placed DOD's overall business transformation on our list of federal programs and activities at high risk of waste, fraud, abuse, and mismanagement.[Footnote 34] The Under Secretary of Defense for Acquisition, Technology and Logistics indicated that DOD plans to address the challenge associated with the integration of DOD's planning, resourcing, and execution processes and initiatives, including the risk management framework. The Under Secretary stated that one task of the ongoing 2005 QDR was "strategic process integration." The Under Secretary also stated that the department is planning to provide a roadmap with performance goals and timelines on how it will implement initiatives to improve strategic process integration. This roadmap is to be submitted with the 2005 QDR report to Congress in early 2006 with the fiscal year 2007 budget. Conclusions: DOD has made some progress in implementing the risk management framework, including establishing risk quadrants and performance goals. However, more work will be required for DOD to be able to put in place a management tool, such as the risk management framework, to strategically balance the allocation of resources across the spectrum of its investment priorities against risk over time and to monitor performance. The development of performance measures that clearly demonstrate results and that are cascaded down throughout the department would enable DOD to provide a clear roadmap of how its activities at all levels contribute to meeting its strategic goals and would assist the department in aligning the core processes and resources of its four military services and multiple defense agencies to better support a departmental or joint approach to national security. Furthermore, the risk management framework cannot be fully implemented until its performance goals are clearly linked to DOD's strategic planning goals. Unless a cause and effect relationship can be demonstrated between the department's performance measures and strategic goals, the framework's usefulness as a tool for monitoring DOD's execution of its strategic plan and identifying performance goals will be severely restricted, if not eliminated. Furthermore, the fiscal year 2006 budget submission does not provide sufficient information on how DOD identified or assessed departmental risks to establish DOD-wide investment priorities; thus, the linkages between the framework and the budget are unclear. Without better measures, clear linkages, and greater transparency, DOD will be unable to fully measure progress in achieving strategic goals or demonstrate to Congress and others how it considered risks and made trade-off decisions, balancing needs and costs for weapon programs and other investment priorities. The efforts of DOD's senior leadership to establish a risk-based and results-oriented management approach have been impeded by some key challenges. The lack of sustained leadership and clear lines of accountability has hampered implementation of the risk management framework and the establishment and achievement of implementation goals and timelines. Strong and sustained leadership could enable DOD to overcome resistance to change that exists in a department as massive and complex as DOD. In addition, the establishment of implementation goals and timelines could enable DOD to determine what progress has been made in implementing the risk management framework. Furthermore, the successful integration of the risk management framework into DOD's investment decision-making processes, including recent reform initiatives, could assist DOD in its overall transformation efforts. Until DOD develops a risk-based and results-oriented management approach for making investment decisions, it will likely continue to experience a mismatch between programs and budgets, and the proportional, rather than strategic, allocation of resources to the services. Recommendations for Executive Action: To address the challenges associated with implementing the risk management framework, or a similar risk-based management approach, we recommend that the Secretary of Defense take the following four actions: * develop or refine department-level performance measures so that they clearly demonstrate performance results and cascade those measures down throughout the department, * assign clear leadership with accountability and authority to implement and sustain the risk management framework, * develop implementation goals and timelines, and: * demonstrate the integration of the risk management framework with DOD's decision support processes and related reform initiatives to improve investment decision making and manage performance results. Agency Comments and Our Evaluation: In written comments on a draft of this report, DOD partially concurred with our four recommendations. DOD's written comments are reprinted in their entirety in appendix II. DOD also provided technical comments, which we incorporated as appropriate. DOD partially concurred with our first recommendation. DOD stated that it concurred with our recommendation that the Secretary of Defense refine department-level performance measures so that they clearly demonstrate results, but that it did not concur with the notion that effectively cascading the risk management framework has been inhibited by the current suite of performance measures. DOD noted that that a number of defense components--including the Army, DOD Comptroller, the Defense Logistics Agency, and the Defense Information Systems Agency-- have successfully cascaded departmentwide strategic goals and implemented frameworks to measure their organization's performance. DOD also believes that empowering the leadership at the component level to develop measures, while ensuring strategic alignment, is the most effective way of encouraging performance management and increasing its utility. In our report, we acknowledge that DOD has taken positive steps toward developing a performance monitoring system and cascading the framework's goals and measures to defense components. However, our recommendation addresses limitations in those measures that currently hinder DOD's ability to use the risk management framework as a management tool for aligning the components' performance goals and measures with the risk management framework, or for strategic balancing investment decisions across the risk quadrants. For example, the majority of the risk management framework's measures are activity measures, or initiatives, that do not monitor a specific annual performance target, nor do these measures provide sufficient information to determine whether the activity is on schedule or contributes to enhancing the department's overall performance. Finally, our recommendation is not intended to suggest that DOD not empower the components to develop performance measures, but rather that DOD establish a clear hierarchy of goals and measures that provide straightforward roadmaps to demonstrate how the components' activities contribute to meeting DOD's strategic goals. DOD partially concurred with our second recommendation that the Secretary of Defense assign clear leadership with accountability and authority to implement and sustain the risk management framework. DOD stated that, although it agrees that such leadership is key to any successful performance management system, the department's senior executives provide sufficient leadership and accountability for implementing and sustaining the risk management framework. DOD also stated that it did not agree that a new organization or bureaucratic structure is needed to ensure successful implementation and sustainment of the risk management framework. We agree that DOD has assigned specific roles and responsibilities for goals and measures associated with the risk management framework to various high-level DOD officials. However, we based our recommendation on the fact that no single individual, with appropriate authority, was held responsible for ensuring that the risk management framework was implemented across the department. Further, our recommendation does not propose that DOD set up a new organization or bureaucratic structure, but, as stated in this report, we continue to believe that one way to provide strong and sustained leadership for change initiatives, such as the risk management framework, over a number of years and various administrations is to legislatively establish a CMO. In partially concurring with our third recommendation to develop implementation goals and timelines, DOD agreed that tracking progress in implementing the risk management framework is a good management practice. DOD stated that it has established goals and timelines for the risk management framework that are unique to the individual metrics, or measures, and that because the risk management framework continually evolves over time, new metrics will be developed while others may be retired. As we stated in the report, successful change management efforts use implementation goals--such as, for example, linking the risk management framework to the budget--and timelines for meeting those goals, to pinpoint shortfalls and gaps, suggest midcourse corrections, and build momentum by demonstrating progress. Therefore, while DOD may continually refine the individual goals and measures associated with the framework's risk quadrants, we believe that goals and timelines for the overall implementation of the framework across the department are essential for keeping this reform initiative on track. DOD partially concurred with our fourth recommendation that the Secretary of Defense demonstrate the integration of the risk management framework with DOD's decision support processes and related reform initiatives to improve investment decision making and manage performance results. DOD stated that the department is currently studying ways to further integrate the risk management framework with other decision support processes, but no single framework or decision model can provide all the necessary information or flexibility needed by the Secretary of Defense and his senior leadership team. We recognize that DOD's senior leadership needs reliable information from a variety of sources and flexibility to make decisions among alternative actions or solutions. However, if the risk management framework is to successfully serve as a management tool to assist decision makers in formulating top-down strategy, balancing investment priorities against risk over time, measuring near-and midterm outputs against strategic goals, and focusing on actual performance results--as intended by DOD's senior leadership--it is crucial that it be successfully integrated with DOD's investment decision-making processes, including recent reform initiatives. We are sending copies of this report to interested congressional committees; the Secretaries of Defense, Army, Navy, and Air Force; the Commandant of the Marine Corps; and the Director, Office of Management and Budget. We will also make copies available to others upon request. In addition, the report will be available at no charge on GAO's Web site at http://www/gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-9619 or pickups@gao.gov. Contact points for our offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. Signed by: Sharon L. Pickup: Director, Defense Capabilities and Management: [End of section] Appendix I: Scope and Methodology: To assess to what extent the Department of Defense (DOD) has implemented the risk management framework, we obtained and analyzed DOD directives, briefings, and other documents that described the risk management framework's purpose, implementation status, and performance measures. We also obtained and analyzed DOD's 2001 Quadrennial Defense Review and annual strategic planning and budget documents. Moreover, we interviewed knowledgeable DOD and service officials involved with the implementation of the risk management framework. Specifically, we obtained testimonial evidence from officials representing the Office of the Secretary of Defense (OSD) offices--such as Program Analysis and Evaluation; Comptroller; Policy; Acquisition, Technology and Logistics; and Personnel and Readiness--the Joint Staff, the military services, and the Defense Business Board. To identify key risk-based and results- oriented management principles, we reviewed our prior reports and other relevant literature, including information on the balanced scorecard concept. For example, we identified characteristics of results-oriented performance measures. These characteristics focused on performance measures that are (1) designed to demonstrate results by providing information on how well the organization is achieving its goals; (2) limited to a vital few, and balanced across priorities; and (3) used by management to improve performance. As another example, risk-based and results-oriented management principles indicate that leading organizations seek to establish clear hierarchies of goals and measures that cascade down so that subordinate units have straightforward roadmaps to demonstrate how their activities contribute to meeting the organization's strategy. We systematically analyzed and compared the risk management framework's department-level performance measures with these characteristics. However, we did not validate the procedures that DOD has in place to ascertain the reliability of the data used to support the performance measures. Regarding strategic planning, these principles focused on (1) establishing clear linkages among strategic planning goals, resources, performance goals and measures and (2) integrating the consideration of risk into the usual cycle of agency decision making and implementation. While these principles do not cover all attributes associated with risk-based and results-oriented management approaches, we believe that they are the most important ones for assessing DOD's progress in implementing the risk management framework. To identify the most significant challenges, we reviewed our previous work on change management principles. We then compared DOD's implementation of the risk management framework to sound change management principles and interviewed knowledgeable DOD officials about the challenges that faced the department in implementing the risk management framework. In addition, we reviewed our previous work to determine to what extent deficiencies in DOD's overall business transformation efforts might influence the implementation of the risk management framework. Our work was performed from October 2004 through September 2005 in accordance with generally accepted government auditing standards. [End of section] Appendix II: Comments from the Department of Defense: OFFICE OF THE SECRETARY OF DEFENSE: PROGRAM ANALYSIS AND EVALUATION: 1600 DEFENSE PENTAGON: WASHINGTON, DC 20301-1600: NOV - 2 2005: Ms. Sharon Pickup: Director, Defense Capabilities and Management: U.S. Government Accountability Office: 441 G Street, N.W.: Washington, DC 20548: Dear Ms. Pickup: This is the Department of Defense (DoD) response to the GAO draft report "DEFENSE MANAGEMENT: Additional Actions Needed to Enhance DoD's Risk-Based Approach for Making Resource Decisions," dated October 4, 2005 (GAO Code 350611/GAO-06-13). DoD has reviewed the report for technical accuracy and content and partially concurs with the GAO's recommendations. The Department agrees that it is vitally important to have a coherent management process to set goals and objectives, measure performance, and respond rapidly to changing world events. Furthermore, DoD is committed to continuously refining its risk management framework to improve the linkages between strategy, performance goals, and performance results. While GAO recommends a number of actions to strengthen the risk management framework, the Department believes that its approach to date reflects a deliberate and tailored plan that has proven effective for managing this enterprise. Attached are specific comments on each recommendation, as well as technical comments. We appreciate the opportunity to comment on the draft report. Signed by: Stanley R. Szemborski: VADM, USN: Principal Deputy Director: Enclosures: As Stated: GAO DRAFT REPORT - DATED OCTOBER 4,2005: GAO CODE 350611/GAO-06-13: "DEFENSE MANAGEMENT: Additional Actions Needed to Enhance DoD's Risk- Based Approach for Making Resource Decisions" DEPARTMENT OF DEFENSE COMMENTS TO THE RECOMMENDATIONS: RECOMMENDATION 1: The GAO recommended that the Secretary of Defense develop or refine department-level performance measures so that they clearly demonstrate performance results. The GAO further recommended that the measures be cascaded through the Department. (p. 30/GAO Draft Report): DoD RESPONSE: Partially Concur. The Department concurs with the recommendation to refine department-level performance measures. DoD has worked continuously to refine both its performance measures and risk management framework in order to clearly demonstrate performance results and better link strategy to outcomes. In fact, one of the primary outputs of many activity metrics are new, refined measures that include the data necessary for quantitative evaluation. The Department believes that the process of developing better, more refined performance measures should continue, as no performance management system is ever truly "complete." However, DoD nonconcurs with the notion that effectively cascading the risk framework has been inhibited by the current suite of performance measures. DoD has demonstrated substantial progress in cascading measures throughout the Department. Upon introducing the risk management framework, the Department set out to ensure alignment across components and Defense agencies by requiring that each organization's performance plan or balanced scorecard: (t) reflect the balanced scorecard risk quadrants; (2) reflect the Department's overall performance objectives; and (3) align with and support the outcomes and performance metrics of the next higher organization. A number of Defense components-including the Army, DoD Comptroller, Defense Logistics Agency (DLA), and Defense Information Systems Agency (DISA)- have successfully cascaded Department-wide strategic goals and implemented frameworks to measure their organization's performance. DoD believes that empowering the leadership at the component level to develop measures that are meaningful and relevant to one's own organization, while ensuring strategic alignment, is the most effective way of encouraging performance management and increasing its utility. RECOMMENDATION 2: The GAO recommended that the Secretary of Defense assign clear leadership with accountability and authority to implement and sustain the risk management framework. (p. 30/GAO Draft Report): DOD RESPONSE: Partially Concur. The Department agrees-that clear leadership with accountability and authority are key characteristics of any successful performance management system. However, contrary to the GAO's assessment, the Department has taken a number of deliberate actions to ensure proper accountability and authority. Therefore, DoD does not agree that a new organization or bureaucratic structure is needed to ensure successful implementation and/or sustainment of the risk management framework. DoD's senior executives, including Principal Staff Assistants and their Performance Management Coordinators, provide sufficient leadership and accountability for implementing and sustaining the risk management framework. Specific roles and responsibilities include: (1) the development of annual performance targets to support the risk management framework; (2) data -collection; (3) verification and validation of performance data; and (4) management of activities to improve performance and achieve targets during, and following, the year of execution. By empowering senior leaders throughout the Department to measure organizational performance, and manage accordingly, the Department has made great progress in ensuring accountability. RECOMMENDATION 3: The GAO recommended that the Secretary of Defense develop implementation goals and timelines. (p. 30/GAO Draft Report): DoD RESPONSE: Partially Concur. DoD agrees that tracking progress in implementing the framework is a good management practice. However, DoD has taken the necessary steps to track implementation of the risk management framework. For example, DoD has established goals and timelines for the risk management framework that are unique to the individual metrics. The Department has also established either a performance metric or activity metric for every relevant area. Since the risk management framework continually evolves over time to reflect changes in strategy, priorities, and senior leadership interest, new metrics will be developed while others may retired. Because individual metric development occurs at different points in time, there is not one single milestone or event that will mark completion of the risk management framework as a whole. RECOMMENDATION 4: The GAO recommended that the Secretary of Defense demonstrate the integration of the risk management framework with DoD's decision support processes and related reform initiatives to improve investment decisionmaking and manage performance results. (p. 30/GAO Draft Report): DOD RESPONSE: Partially Concur. The Department agrees that integrating the risk management framework with other decision support processes and related reform initiatives is both a natural evolution of the risk management framework's maturation effort and a beneficial output of the process. As such, the Department is currently studying ways to further integrate the risk management framework with other decision support processes. However, the Secretary of Defense and his senior leadership team must be able to make well-informed decisions in a variety of circumstances. No single framework or decision model can provide all of the necessary information and/or flexibility. Technical Comments Draft GAO Report GAO-06-13, GAO Code 350611: Issue: Page 2, paragraph 2. Sentence reads: "..we conducted interviews with DoD and service officials, members of the Joint Staff, and members of the Defense Business Board." The Defense Business Board does not officially speak on the Department's behalf. Issue: Page 12, paragraph 2. The sentence reads: "44 of the 77 departmental-level measures .. are activity measures." Page 13, Table 1. Footnote describes a recoding of five performance measures to activity measures. DoD objects to recoding these metrics from performance metrics to activity metrics. [End of section] Appendix III: GAO Contact and Staff Acknowledgments: GAO Contact: Sharon Pickup, (202) 512-9619 or pickups@gao.gov: Acknowledgments: In addition to the contact named above, David Moser, Assistant Director; Donna Byers; Gina Flacco; and Renee S. Brown made key contributions to this report. FOOTNOTES [1] See GAO, 21st Century Challenges: Reexamining the Base of the Federal Government, GAO-05-325SP (Washington, D.C.: February 2005) for a comprehensive compendium of areas throughout the federal government that could be considered for reexamination and review by Congress. [2] GAO, High-Risk Series: An Update, GAO-05-207 (Washington, D.C.: January 2005). [3] GAO, Highlights of a GAO Forum: Mergers and Transformation: Lessons Learned for a Department of Homeland Security and Other Federal Agencies, GAO-03-293SP (Washington, D.C.: Nov. 14, 2002), and Results- Oriented Cultures: Implementation Steps to Assist Mergers and Organizational Transformations, GAO-03-669 (Washington, D.C.: July 2, 2003). [4] S. 780, 109th Cong. §1 (2005). [5] GAO-05-207. [6] GAO-05-207. [7] The Government Performance and Results Act of 1993 (Pub. L. No. 103- 62). [8] The balanced scorecard approach was advocated by Professor Robert Kaplan and Dr. David Norton in the November/December 1992 Harvard Business Review. [9] GAO-05-325SP. [10] See for example, Committee of Sponsoring Organizations of the Treadway Commission, Enterprise Risk Management--Integrated Framework: Executive Summary (New York, N.Y.: September 2004). [11] Pub. L. No. 103-62 (1993). [12] The President's Management Agenda, by focusing on a number of targeted areas, seeks to improve the performance management of the federal government. [13] For further information see: GAO, Performance Budgeting: PART Focuses Attention on Program Performance, but More Can Be Done to Engage Congress, GAO-06-28 (Washington, D.C.: Oct. 28, 2005); Management Reform: Assessing the President's Management Agenda, GAO-05- 574T (Washington, D.C.: Apr. 21, 2005); Results-Oriented Government: GPRA Has Established a Solid Foundation for Achieving Greater Results, GAO-04-38 (Washington, D.C.: Mar. 10, 2004); and Performance Budgeting: Observations on the Use of OMB's Program Assessment Rating Tool for the Fiscal Year 2004 Budget, GAO-04-174 (Washington, D.C.: Jan. 30, 2004). [14] GAO, Executive Guide: Effectively Implementing the Government Performance and Results Act, GAO/GGD-96-118 (Washington, D.C.: June 1996) and Managing for Results: Enhancing Agency Use of Performance Information for Management Decision Making, GAO-05-927 (Washington, D.C.: Sept. 9, 2005). [15] GAO/GGD-96-118. [16] GAO/GGD-96-118 and GAO-05-927. [17] GAO/GGD-96-118. [18] GAO/GGD-96-118. [19] GAO, Quadrennial Defense Review: Future Reviews Can Benefit from Better Analysis and Changes in Timing and Scope, GAO-03-13 (Washington, D.C.: Nov. 4, 2002). [20] The Defense Business Board was established in 2001 by the Secretary of Defense to provide DOD's senior leadership with leading- edge, actionable advice on management improvements. [21] GAO, Managing for Results: Critical Issues for Improving Agencies' Strategic Plans, GAO/GGD-97-180 (Washington, D.C.: Sept. 16, 1997). [22] See GAO/GGD-96-118. [23] The Future Years Defense Program provides information on DOD's current and planned outyear budget requests. [24] GAO, Future Years Defense Program: Actions Needed to Improve Transparency of DOD's Projected Resource Needs, GAO-04-514 (Washington, D.C.: May 7, 2004). [25] GAO, Defense Acquisitions: Assessments of Selected Major Weapon Programs, GAO-05-301 (Washington, D.C.: Mar. 31, 2005). [26] GAO-05-325SP. [27] GAO-05-325SP. [28] Joint Defense Capabilities Study Team, Joint Defense Capabilities Study: Final Report (Washington, D.C.: December 2003). [29] GAO, Management Reform: Elements of Successful Improvement Initiatives, GAO/T-GGD-00-26 (Washington, D.C.: Oct. 15, 1999). [30] GAO-05-520T and GAO-05-629T. [31] S. 780, 109th Cong. §1 (2005). [32] GAO-03-669. [33] GAO-03-669. [34] GAO-05-207. GAO's Mission: The Government Accountability Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO's Web site ( www.gao.gov ) contains abstracts and full-text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics. Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as "Today's Reports," on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select "Subscribe to e-mail alerts" under the "Order GAO Products" heading. Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. Government Accountability Office 441 G Street NW, Room LM Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov Automated answering system: (800) 424-5454 or (202) 512-7470: Public Affairs: Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, D.C. 20548:

The Justia Government Accountability Office site republishes public reports retrieved from the U.S. GAO These reports should not be considered official, and do not necessarily reflect the views of Justia.