Streamlining Government

Opportunities Exist to Strengthen OMB's Approach to Improving Efficiency Gao ID: GAO-10-394 May 7, 2010

Given record budget deficits and continuing fiscal pressures, the federal government must seek to deliver results more efficiently. The prior Administration sought to improve efficiency under the Program Assessment Rating Tool (PART) by requiring programs to have at least one efficiency measure and procedures for improving efficiency, and show annual efficiency gains. The current administration has also emphasized efficiency in some initiatives. GAO was asked to examine (1) the types of PART efficiency measures and the extent to which they included typical elements of an efficiency measure; (2) the extent to which selected programs showed gains and how they used efficiency measures for decision making; (3) the challenges selected programs faced in developing and using efficiency measures; and (4) other strategies that can be used to improve efficiency. GAO analyzed the 36 efficiency measures in 21 selected programs in 5 agencies and a generalizable sample from the other 1,355 measures governmentwide, reviewed documents and interviewed officials from selected programs, reviewed literature on efficiency, and interviewed experts.

Under PART, most programs developed an efficiency measure. However, according to GAO's analysis, 26 percent did not include both typical efficiency measure elements--an input (e.g., labor hours or costs) as well as an output or outcome (e.g., the product, service, or result produced). Most frequently missing was the input (69 percent). For example, a measure developed by the National Nuclear Safety Security Administration considered the number of information assets reviewed for certification without considering costs of review. This could result in measures that do not capture efficiency. GAO has previously recommended agencies improve cost information for decision making, but they are in various stages of implementation. However, alternative forms of measurement, such as reducing costly error rates, could still be useful. Of the efficiency measures GAO reviewed that had both typical elements, a similar number reported gains and losses. Officials for some programs stated that the efficiency measures reported for PART were useful, and described ways in which they used the data, such as to evaluate proposals from field units, lower the cost of a contract, or make decisions to shift production. Others did not find the efficiency measures useful because, for example, the program lacked control over key cost drivers, such as contractually required staffing levels, or because of concern that raising output could lower quality. Officials for all of the programs reviewed described challenges to developing and using program-level efficiency measures and performance measures in general. Challenges included interpreting outcome-level efficiency information, such as the cost of improving or maintaining the condition of watershed acres, when factors other than program funding, such as past impacts from mining, affected conditions as well; achieving required annual efficiency gains in cases where a program intervention takes years to implement; and inconsistent or limited guidance and technical assistance from the Office of Management and Budget (OMB) to agencies on how to measure efficiency. A variety of approaches have been used to improve efficiency, including governmentwide reviews, agency restructurings, process and technology improvements, and strategic spending approaches. The Administration has some initiatives along these lines, such as information technology and procurement reforms. The Government Performance and Results Act (GPRA) provides a framework for planning future efficiency gains while maintaining or improving effectiveness and quality of outputs or outcomes. OMB, as the focal point for management in the executive branch, provides guidance and supports information-sharing mechanisms, such as the Performance Improvement Council, which could also be used to create a more strategic and crosscutting focus on agency efforts to improve efficiency. OMB has not clearly indicated whether programs should continue measuring efficiency nor has it emphasized efficiency in its GPRA guidance to agencies.

Recommendations

Our recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.

Director: Bernice Steinhardt Team: Government Accountability Office: Strategic Issues Phone: (202) 512-6543


GAO-10-394, Streamlining Government: Opportunities Exist to Strengthen OMB's Approach to Improving Efficiency This is the accessible text file for GAO report number GAO-10-394 entitled 'Streamlining Government: Opportunities Exist to Strengthen OMB's Approach to Improving Efficiency' which was released on June 7, 2010. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to Congressional Requesters: United States Government Accountability Office: GAO: May 2010: Streamlining Government: Opportunities Exist to Strengthen OMB's Approach to Improving Efficiency: GAO-10-394: GAO Highlights: Highlights of GAO-10-394, a report to congressional requesters. Why GAO Did This Study: Given record budget deficits and continuing fiscal pressures, the federal government must seek to deliver results more efficiently. The prior Administration sought to improve efficiency under the Program Assessment Rating Tool (PART) by requiring programs to have at least one efficiency measure and procedures for improving efficiency, and show annual efficiency gains. The current administration has also emphasized efficiency in some initiatives. GAO was asked to examine (1) the types of PART efficiency measures and the extent to which they included typical elements of an efficiency measure; (2) the extent to which selected programs showed gains and how they used efficiency measures for decision making; (3) the challenges selected programs faced in developing and using efficiency measures; and (4) other strategies that can be used to improve efficiency. GAO analyzed the 36 efficiency measures in 21 selected programs in 5 agencies and a generalizable sample from the other 1,355 measures governmentwide, reviewed documents and interviewed officials from selected programs, reviewed literature on efficiency, and interviewed experts. What GAO Found: Under PART, most programs developed an efficiency measure. However, according to GAO‘s analysis, 26 percent did not include both typical efficiency measure elements”an input (e.g., labor hours or costs) as well as an output or outcome (e.g., the product, service, or result produced). Most frequently missing was the input (69 percent). For example, a measure developed by the National Nuclear Safety Security Administration considered the number of information assets reviewed for certification without considering costs of review. This could result in measures that do not capture efficiency. GAO has previously recommended agencies improve cost information for decision making, but they are in various stages of implementation. However, alternative forms of measurement, such as reducing costly error rates, could still be useful. Of the efficiency measures GAO reviewed that had both typical elements, a similar number reported gains and losses. Officials for some programs stated that the efficiency measures reported for PART were useful, and described ways in which they used the data, such as to evaluate proposals from field units, lower the cost of a contract, or make decisions to shift production. Others did not find the efficiency measures useful because, for example, the program lacked control over key cost drivers, such as contractually required staffing levels, or because of concern that raising output could lower quality. Officials for all of the programs reviewed described challenges to developing and using program-level efficiency measures and performance measures in general. Challenges included interpreting outcome-level efficiency information, such as the cost of improving or maintaining the condition of watershed acres, when factors other than program funding, such as past impacts from mining, affected conditions as well; achieving required annual efficiency gains in cases where a program intervention takes years to implement; and inconsistent or limited guidance and technical assistance from the Office of Management and Budget (OMB) to agencies on how to measure efficiency. A variety of approaches have been used to improve efficiency, including governmentwide reviews, agency restructurings, process and technology improvements, and strategic spending approaches. The Administration has some initiatives along these lines, such as information technology and procurement reforms. The Government Performance and Results Act (GPRA) provides a framework for planning future efficiency gains while maintaining or improving effectiveness and quality of outputs or outcomes. OMB, as the focal point for management in the executive branch, provides guidance and supports information-sharing mechanisms, such as the Performance Improvement Council, which could also be used to create a more strategic and crosscutting focus on agency efforts to improve efficiency. OMB has not clearly indicated whether programs should continue measuring efficiency nor has it emphasized efficiency in its GPRA guidance to agencies. What GAO Recommends: GAO recommends that OMB evolve toward a broader approach with its guidance and support to improve efficiency at government-wide, agency, and program levels. OMB concurred with our recommendations. View [hyperlink, http://www.gao.gov/products/GAO-10-394] or key components. For more information, contact Bernice Steinhardt at (202) 512-6543 or steinhardtb@gao.gov. [End of section] Contents: Letter: Background: Most Programs Developed an Efficiency Measure for PART, but Only about Half Clearly Included Typical Elements of an Efficiency Measure: Programs Showed Mixed Results in Terms of Improvements in Efficiency and Use of Efficiency Measures for Decision Making: Program Officials Reported Challenges to Developing and Using Efficiency Measures: Using GPRA as a Framework, a Broader Array of Strategies Can Be Used to Seek Improvements in Efficiency: Conclusions: Recommendations for Executive Action: Agency Comments: Appendix I: Objectives, Scope, and Methodology: Appendix II: Departments, Selected Program Assessment Rating Tool Program (PART) Names, and Summary of Programs: Appendix III: Department, PART Program Name, and Number of Efficiency Measures, Fiscal Year 2009 Funding Level, PART Program Type, and Efficiency Measure(s) for Selected Programs: Appendix IV: Comments from the Department of the Interior: Appendix V: GAO Contact and Staff Acknowledgments: Tables: Table 1: Examples of Efficiency Measures and Whether They Capture Efficiency: Table 2: Gains/Losses and Reported Use for Selected Programs' Efficiency Measures: Figures: Figure 1: Extent to Which 36 Efficiency Measures from Selected Programs Contained the Two Typical Elements of an Efficiency Measure and Other Attributes: Figure 2: Estimated Percentage of Efficiency Measures That Contained the Two Typical Elements of an Efficiency Measure and Other Attributes: Figure 3: Number of Individual Returns and IRS Staff Years for Individual Paper and Electronic Processing, Fiscal Years 1999-2010: Abbreviations: APQC: American Productivity and Quality Center: ATO: Air Traffic Organization: BPR: Business Process Reengineering: BRAC: Base Realignment and Closure: CFO: Chief Financial Officer: DOD: U.S. Department of Defense: EMDS: Ecosystem Management Decision Support: FAA: Federal Aviation Administration: FSA: Federal Student Aid: FTE: full-time equivalent: GPRA: Government Performance and Results Act of 1993: HM Treasury: Her Majesty's Treasury: IRS: Internal Revenue Service: IT: information technology: MCA: managerial cost accounting: NAO: National Audit Office: NHTSA: National Highway Traffic Safety Administration: NSLP: National School Lunch Program: OMB: Office of Management and Budget: OSHA: Occupational Safety and Health Administration: PART: Program Assessment Rating Tool: PMA: President's Management Agenda: SAVE: Securing Americans Value and Efficiency: SEA: State Education Agency: SFFAS: Statement of Federal Financial Accounting Standards: UK: United Kingdom: USDA: U.S. Department of Agriculture: VA: U.S. Department of Veterans Affairs: VERA: Veterans Equitable Resource Allocation: [End of section] United States Government Accountability Office: Washington, DC 20548: May 7, 2010: The Honorable Thomas R. Carper: Chairman: The Honorable John McCain: Ranking Member: Subcommittee on Federal Financial Management, Government Information, Federal Services, and International Security: Committee on Homeland Security and Governmental Affairs: United States Senate: The Honorable Tom Coburn: United States Senate: Weaknesses in the economy and financial markets--and the government's response to them--have contributed to recent increases in federal deficits, which reached a record level in fiscal year 2009. While a lot of attention has been given to the recent fiscal deterioration, the federal government faces even larger fiscal challenges, driven by certain factors, such as health care cost growth and demographic trends, which will persist long after the return of financial stability and economic growth. Given the magnitude of these challenges, the federal government must identify ways to operate and deliver results more efficiently as well as more effectively. In response to these fiscal challenges, the current Administration has emphasized the importance of reducing spending and improving government efficiency in recent initiatives. These initiatives have included: the Office of Management and Budget's (OMB) requirement for agencies to submit alternative targets for discretionary funding levels for fiscal year 2011 budget submissions that involved freeze and reduction scenarios, including the identification of 126 program terminations, reductions, and other areas of savings identified which, if enacted or implemented, could save approximately $23 billion; [Footnote 1] contracting and workforce reforms designed to save at least $40 billion a year; information technology management improvements designed to improve efficiency; and holding a contest to seek ideas from federal employees on how to increase efficiency and savings.[Footnote 2] Recently, the President also established a management advisory board to provide advice and recommendations on, among other things, improving the productivity of federal operations. [Footnote 3] At the same time, several broader government reform efforts over the past 17 years have also included a focus on improving efficiency. The Government Performance and Results Act of 1993 (GPRA),[Footnote 4] which Congress enacted in part to improve federal program effectiveness and accountability and enhance congressional decision making, was created partly to address waste and inefficiency in federal programs.[Footnote 5] The President's Management Agenda (PMA) [Footnote 6] and Program Assessment Rating Tool (PART)[Footnote 7] initiatives of the previous presidential administration emphasized improving government efficiency with specific requirements for agencies to develop program-level efficiency measures and show annual improvements in efficiency. Analysis of the experiences of federal agencies in developing and using efficiency measures under the PMA and PART initiatives, as well as identification of additional strategic and crosscutting approaches used by government, nongovernment, and business organizations to seek improvements in efficiency, could be helpful to agencies as they attempt to improve efficiency of programs. In response to your request, this report examines (1) the types of efficiency measures reported through PART for agency programs overall, and particularly for selected programs in five selected agencies, focusing on the extent to which they included typical elements of an efficiency measure, (2) for selected programs, the extent to which programs reporting efficiency measures through PART have shown efficiency gains and how programs have used efficiency measures for decision making, (3) for selected programs, the types of challenges to developing and using efficiency measures they have faced, and (4) other strategies that can be used to improve efficiency. Based on our review of the literature,[Footnote 8] an efficiency measure is typically defined as the ratio of two elements: a program's inputs (such as costs or hours worked by employees), to its outputs or outcomes. Outputs can be defined as the amount of products or services delivered by a program. Outcomes can be defined as the desired results of a program, such as events, occurrences, or changes in conditions, behaviors, or attitudes. In some literature, the inverse ratio of outcomes or outputs to inputs is referred to as a "productivity" measure,[Footnote 9] but for purposes of this report, we refer to either form of the ratio as an efficiency measure. It should be noted that an improvement in efficiency can be achieved by maintaining quantity or quality of outputs or outcomes while reducing costs, as well as by improving the quantity or quality of outputs or outcomes while maintaining (or reducing) costs. Thus an improvement in efficiency need not involve a reduction of costs. OMB initially described an efficiency measure as the ratio of a program's outcomes or outputs to inputs in the 2004 PART guidance. In the December 2007 PART guidance, OMB termed this type of ratio an "input productivity measure," and indicated that such measures could provide a useful approach for identifying efficiency measures. In the guidance, OMB also identified erroneous conclusions that can result from the use of simple output-input ratios to track changes over time in efficiency for programs that do not produce the same or similar outputs repetitively. OMB also identified challenges facing efforts to measure efficiency in research and development programs and construction of special purpose infrastructure projects. OMB broadened the discussion of efficiency measures in the revised guidance and proposed alternative approaches to tracking efficiency changes for such programs, such as meeting project cost, schedule, and performance goals. To address our objectives, we analyzed all 1,396 PART efficiency measures associated with 937 programs in a database provided by OMB. We conducted more detailed analysis of the 36 efficiency measures for 21 selected programs,[Footnote 10] as well as a random sample of 100 efficiency measures from all remaining programs. This sample was designed to enable us to generalize our analysis to the remaining efficiency measures for PART.[Footnote 11] We selected the 21 specific programs for review from five departments--the U.S. Departments of Agriculture, Education, the Interior, Labor, and Transportation. These departments were selected to represent variety in the extent to which they had developed managerial cost accounting systems as identified by our prior work, based on an assumption that the status of a department's cost accounting systems could affect the availability of cost information and thus the development of efficiency measures. [Footnote 12] We selected the 21 specific programs to represent a diverse array of functions and operations within the federal government, primarily focusing on the PART program type.[Footnote 13] Additional criteria were that the selected programs had relatively large fiscal year 2009 funding levels,[Footnote 14] and variety in the number of efficiency measures associated with the programs. In addition, we reviewed program documents, OMB documents, including PART assessments, and agency Web sites. We conducted a literature review as well as expert interviews to identify the elements of a typical efficiency measure, and to identify alternative approaches to improving efficiency. We interviewed officials from OMB and from the 21 selected programs, as well as officials from the five departments who were knowledgeable about performance measurement and financial systems for the departments. See appendix I for a more detailed discussion of our scope and methodology. We conducted the major portion of this performance audit from September 2008 to May 2010 in accordance with generally accepted government auditing standards.[Footnote 15] Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background: Congress enacted GPRA in part to inform congressional decision making by providing objective information on the relative effectiveness and efficiency of federal programs and spending. In addition to requiring executive agencies to develop strategic and annual performance plans, and measure and report on progress toward goals, GPRA also emphasized efficiency. According to the statute, GPRA was intended, among other things, to address problems of waste and inefficiency in federal programs, and to improve congressional decision making by providing objective information on the relative efficiency and effectiveness of federal programs and spending.[Footnote 16] OMB plays an important role in the management of the federal government's performance, and specifically GPRA implementation. Part of OMB's overall mission is to ensure that agency plans and reports are consistent with the President's budget and administration policies. OMB is responsible for receiving and reviewing agencies' strategic plans, annual performance plans, and annual performance reports. To improve the quality and consistency of these documents, OMB issues annual guidance to agencies for their preparation, including guidelines on format, required elements, and submission deadlines.[Footnote 17] In addition, GPRA requires OMB to prepare the overall governmentwide performance plan, based on agencies' annual performance plan submissions. The PMA and PART of the prior administration also included an emphasis on improving government efficiency, with requirements for agencies to develop program-level efficiency measures and show annual improvements in efficiency. In August 2001, the Bush Administration launched the PMA with the stated purpose of ensuring that resources entrusted to the federal government were well managed and wisely used. OMB developed criteria called "standards of success" to measure progress in five management initiatives under the PMA, as well as a scorecard to track agency progress under each initiative. Criteria to receive and maintain the highest rating score (green status) for the performance improvement initiative included that an agency's annual budget and performance documents include at least one efficiency measure for each program and that program performance and efficiency improvements be identified each year.[Footnote 18] PART, which was launched in 2002 as a component of the PMA, included assessment of the extent to which programs were tracking progress toward and achieving efficiency improvements. PART consisted of a set of questions developed to assess various types of federal executive branch programs, and addressed four aspects of a program: purpose and design, strategic planning, program management, and program results/ accountability. While there were references to efficiency in several different sections of the 2007 and 2008 PART guidance, two PART questions focused specifically on development of program-level efficiency measures with annual targets for improvement:[Footnote 19] * "Does the program have procedures (e.g., competitive sourcing/cost comparisons, information technology (IT) improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?" - In order to receive a "yes" response for this question, a program was to have regular procedures in place to achieve efficiencies and cost effectiveness, and had to have at least one efficiency measure with baseline and targets. Evidence could include efficiency measures, competitive sourcing plans, IT improvement plans designed to produce tangible productivity and efficiency gains, or IT business cases that documented how particular projects improved efficiency. * "Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?" - In order to receive a "yes" response for this question, a program had to demonstrate improved efficiency or cost effectiveness over the prior year, including meeting its efficiency target(s) in the question above. Most Programs Developed an Efficiency Measure for PART, but Only about Half Clearly Included Typical Elements of an Efficiency Measure: About 90 percent of all programs that received a PART assessment, including those in our selected review, developed at least one performance measure as an efficiency measure.[Footnote 20] However, we found that about half of the approved measures either did not contain typical elements of an efficiency measure, or were unclear. As table 1 below indicates, we analyzed a sample of the efficiency measures that were developed for PART, and, to the extent possible, placed them into one of the three categories shown in the table. (In some cases, the available information on the measure was insufficient for us to place it into one of the three categories, so we labeled these measures as "unclear.") Table 1: Examples of Efficiency Measures and Whether They Capture Efficiency: Type of measure: Input ÷ Output/outcome; Example: Cost per job created[A]; Does measure capture efficiency?: Yes. Type of measure: (Missing input) ÷ Output/outcome; Example: Annual number of information assets reviewed for certification and accreditation[B]; Does measure capture efficiency?: No; Measure indicates whether more or less is being produced, but not whether more or fewer resources are being used. Type of measure: Input ÷ (Missing output/outcome); Example: Administrative cost as a percentage of total program costs[C]; Does measure capture efficiency?: No; Measure indicates whether administrative costs change relative to total cost, but not whether more or fewer outputs or outcomes are being produced. Source: GAO analysis of OMB PART efficiency measures. [A] This efficiency measure was identified in response to the PART assessment for the Delta Regional Authority. [B] This efficiency measure was identified in response to the PART assessment for the Department of Energy National Nuclear Security Administration: Safeguards and Security program. [C] This efficiency measure was identified in response to the PART assessment for the Department of Energy Building Technologies program. [End of table] As figures 1 and 2 below illustrate, our analysis of the 36 efficiency measures from our selected programs and a random sample of the remaining efficiency measures indicates that about half of the efficiency measures contained typical elements by including both an input and an output or outcome. As illustrated in figure 1, for the 21 selected programs (listed in appendix II), we determined that 58 percent of the efficiency measures included both elements and 42 percent did not. In its guidance to programs, OMB stated that, although both output and outcome-oriented efficiency measures were acceptable, outcome efficiency measures were preferred. Because we obtained more in-depth information on the selected programs' measures, we further analyzed whether those that included both elements were output-or outcome-oriented and found most to be output-oriented. Figure 1: Extent to Which 36 Efficiency Measures from Selected Programs Contained the Two Typical Elements of an Efficiency Measure and Other Attributes: [Refer to PDF for image: series of pie-charts] Percentages of efficiency measures containing typical elements and other attributes (n=36): Included both typical elements (21): 58%; * Percentage with output or outcome (n=21): Output (13): 62%; Outcome (8): 38%; * Percentage with cost or time as the input (n=21): Cost (18): 86%; Time (3): 14%. Did not include a typical element (15): 42%; * Percentage missing a typical element (n=15): Input missing (13): 87%; Output or outcome missing (2): 13%. Source: GAO analysis of OMB PART data. Note: The typical elements of an efficiency measure include (1) an input and (2) an output or outcome. [End of figure] Figure 2 summarizes estimates for the remaining 1,355 efficiency measures, based on a random sample of 100 of those measures. We estimate that 48 percent of the measures included both elements, [Footnote 21] 26 percent did not, and the remaining 26 percent were unclear.[Footnote 22] Of those that did not contain both elements, the missing element was most often an input. Figure 2: Estimated Percentage of Efficiency Measures That Contained the Two Typical Elements of an Efficiency Measure and Other Attributes: [Refer to PDF for image: series of pie-charts] Percentages of efficiency measures containing typical elements and other attributes (n=100): Included both typical elements (48): 48%; * Percentage with cost or time as the input (n=48): Cost (45): 94%; Time (2): 4%; Time and cost (1): 2%. Did not include a typical element (26): 26%; * Percentage missing a typical element (n=26): Input missing (18): 69%; Output or outcome missing (8): 31%; Unclear (26): 26%. Source: GAO analysis of sample of 100 efficiency measures taken from OMB PART data. Note: The two typical elements of an efficiency measure include (1) an input and (2) an output or outcome. Estimates based on all 100 sampled efficiency measures have a 95 percent confidence interval of +/-10 percentage points. Estimates based on smaller samples of 48 and 26 above have 95 percent confidence intervals of +/-12 and +/-22 percentage points, respectively. [End of figure] In general, as indicated in table 1, the absence of these typical elements can result in measures that do not truly capture efficiency. Nevertheless, some of the information captured in these measures could still be of value to program officials for helping improve efficiency. For example, one measure from our selected programs--average time to correct/mitigate higher priority operations and maintenance deficiencies at certain facilities in the Bureau of Reclamation--did not contain an input element[Footnote 23]. However, program officials told us this was an important measure because it helped them prioritize which ongoing preventive maintenance projects they should repair first by categorizing repairs needed according to the likely costs of delaying the repairs. For example, a category 1 deficiency should normally be repaired immediately (within 3 to 6 months) to avoid escalating the cost of repair; a category 2 deficiency should be repaired in a few years. In contrast, a category 3 deficiency is normally repaired only if there is time and funding remaining after repairing category 1 and 2 deficiencies. In another example, the National School Lunch Program (NSLP) used a measure which was labeled an efficiency measure, but which did not have the typical ratio of inputs to outputs or outcomes. Instead, the measure focused on reducing the error rate in making program payments. Program officials characterized the measure as a process measure, rather than an output or outcome-based efficiency measure. An official said that out of $7 billion in total program payments, errors worth $2 billion occur in terms of under and over payments, for a net cost to the program of $1 billion. An official said that if they were able to reduce overall overpayments due to various types of error, it could save millions of dollars. Officials said this measure has been important in helping them take corrective actions to reduce the number of payments made in error. Among the selected programs, for the efficiency measures that contained an input, the type of information used to express the input varied in terms of both availability for use and completeness. Most of the efficiency measures we reviewed captured inputs in terms of cost, but a few used the amount of staff resources or time spent to produce an output or outcome as a proxy for cost. For example, the Department of Labor Energy Employees Occupational Illness Compensation program's efficiency measure was the average number of decisions per full-time equivalent (FTE), which we determined used information on work hours as estimated by FTEs as the input.[Footnote 24] While FTE information is often readily available and can be a useful proxy for cost, it does not necessarily reflect total cost because, for example, it would neither distinguish between higher and lower cost FTEs, nor would it include other costs, such as contractors, training, equipment, or facilities. In addition, dollar cost information can vary in how completely it captures the cost of producing outputs or outcomes. "Cost" generally can be thought of as the value of resources that have been, or must be, used or sacrificed to attain a particular objective,[Footnote 25] which, in the case of an efficiency measure, would be a unit of output or outcome. "Full cost" is generally viewed as including both direct costs (costs that can be specifically identified with a cost object, such as an output) and indirect costs (costs of resources that are jointly or commonly used to produce two or more types of outputs but are not specifically identifiable with any of the outputs).[Footnote 26] Managerial cost accounting (MCA) information can provide a more complete picture of the cost involved in producing program outputs or outcomes by recognizing resources when they are used and determining the full cost of producing government goods and services, including both direct and indirect costs. According to the Statement of Federal Financial Accounting Standards No. 4 (SFFAS 4), Managerial Cost Accounting Concepts and Standards for the Federal Government, which sets forth the fundamental elements for MCA in government agencies, [Footnote 27] costs may be measured, analyzed, and reported in many ways and can vary depending upon the circumstances and purpose for which the measurement is to be used. Our analysis of the cost information used by the selected programs showed that most of the measures used budgetary information, such as appropriations or obligations, for the cost element.[Footnote 28] Of the 18 efficiency measures from our selected programs that had both typical elements, and had cost as the input, 14 measures (78 percent) used a form of budgetary information. We have previously reported that using budgetary information, such as appropriations or obligations, may not completely capture the full cost of producing program outputs or outcomes because of differing time frames and account structures.[Footnote 29] With regard to timing, appropriations provide agencies legal authority to obligate funds for a given fiscal year or beyond. Consequently, agency outlays (payments against obligations for goods and services received) representing the resources used to produce a program's outputs or outcomes in a given year may flow from obligations made in a prior year's appropriation. Therefore a given year's appropriations or obligations may not represent the resources actually used to produce a program's outputs or outcomes in that year. With regard to account structures, appropriations accounts developed over the last 200 years were oriented in different ways in response to specific needs. For example, some appropriations accounts reflect items of expense, such as salaries or construction, while others reflect organizations, processes, or programs. Further, program-oriented account structures may cover multiple programs or may exclude some indirect resources used by the programs. Though budgetary information may not completely cover the cost of producing program outputs or outcomes, several program officials said it was the most complete information available to them and best met the needs of Congress. For example, the Department of Labor Job Corps program, which used budgetary information in its efficiency measure, divided its request in the fiscal year 2010 Job Corps Congressional Budget Justification into three categories: operations, construction, and administration. However, the program's efficiency measure--cost per participant in the Job Corps program--was based entirely on the operations category, which encompassed 92 percent of the program's fiscal year 2010 request, meaning the measure did not capture the remaining 8 percent of construction-or administration-related costs that were also associated with program participation. A study commissioned by the Job Corps recommended that all direct costs associated with Job Corps appropriations be included in the measure if full costs were to be determined. This would include actual expenditures (i.e., outlays rather than appropriations or obligations) for Job Corps appropriations provided for operations, construction, [Footnote 30] and direct administrative costs.[Footnote 31] Program officials indicated they did not believe including the additional costs would provide useful information because there were relatively few opportunities to find efficiencies in the construction or administration categories. Additionally, a Department of the Interior Wildland Fire Management budget official told us that while they had access to more complete cost data, this information was not necessarily accurate or easy to obtain because it had to be collected from five different entities with different cost accounting systems.[Footnote 32] They also preferred to use budgetary information because it helped to justify their appropriations request to Congress. Program officials noted that each of their three efficiency measures was based on obligations data.[Footnote 33] Relative to time or budgetary information, some agencies have sought to develop more complete cost information by using MCA systems capable of accumulating and analyzing both financial and nonfinancial data in order to determine, among other things, the unit cost of producing program outputs or outcomes. Such systems are also capable of recognizing resources when they are used and determining the full cost of producing government goods and services, including both direct and indirect costs. However, in earlier work we found that only 3 of the 10 Chief Financial Officer (CFO) Act agencies we reviewed had implemented MCA systems entitywide: Interior, the Social Security Administration, and Labor.[Footnote 34] Transportation had made significant progress in implementing MCA entitywide and three agencies--Agriculture, Health and Human Services, and Housing and Urban Development--planned to implement MCA systems when upgrading their overall financial management systems. The three remaining agencies we reviewed-- Education, the Treasury, and Veterans Affairs--had no plans to implement MCA departmentwide,[Footnote 35] although Veterans Affairs was initiating a review to explore opportunities to do so. Consequently, we recommended that individual agencies commence or improve the development of entity-wide MCA systems as a fundamental component of their financial management system, as required by SFFAS 4 and the Federal Financial Management Improvement Act of 1996.[Footnote 36] For this report, of the five agencies we reviewed, we selected three-- Interior, Labor, and Transportation--because we previously reported they had either implemented MCA systems entitywide, or were planning to do so. Nevertheless, we did not find widespread use of MCA system data for the efficiency measures we reviewed either in these agencies or in the other two agencies--Education and Agriculture--that did not have entitywide MCA systems. Of the 18 efficiency measures from our selected programs that included typical elements, four measures (22 percent) used a distinct MCA system to determine costs. Those programs that relied on MCA data produced outputs, such as the Student Aid Administration program (student aid disbursements), the Federal Aviation Administration's (FAA) Air Traffic Organization Terminal (take offs and landing operations) and Technical (maintenance and modernization of equipment needed to provide air traffic services) programs, and the Department of the Interior's Fisheries program (pounds of trout per dollar). In addition, legislation was enacted in the 1990s, which resulted in both Federal Student Aid (FSA) and FAA developing MCA systems to improve performance.[Footnote 37],[Footnote 38] Of the remaining 14 efficiency measures, officials from several of those programs told us they used budgetary information because they either did not have access to an MCA system, the system they could access produced poor data, or the information would not be useful for congressional decision making. For example, the Department of Education did not have a departmentwide MCA system, though it is now considering creating such a system in response to a prior recommendation we made.[Footnote 39] Also, officials with the Department of Transportation CFO office told us that the department had taken a decentralized approach in which some of their operating administrations--such as the FAA and Federal Transit Administration--had developed and were using their own MCA system. In addition, although the Department of Labor's CFO had developed an MCA system and made it available to its agencies and programs, officials from the five Department of Labor programs we reviewed indicated that they did not use it for their efficiency measures because, in their opinions, the system was either not useful, not sufficiently developed for their needs, did not capture all the program's costs, or captured a different type of funding than was used for the efficiency measure. Finally as indicated previously, a Department of the Interior Wildland Fire budget official told us that cost information for their program was neither easy to access nor was it as useful for budget justification purposes. Programs Showed Mixed Results in Terms of Improvements in Efficiency and Use of Efficiency Measures for Decision Making: The selected programs that had measures with both elements of a typical efficiency measure reported mixed results under PART in terms of gains and losses in efficiency. As previously indicated in figure 2, 21 of the 36 efficiency measures developed by the programs selected for our review had both of the elements of a typical efficiency measure. As can be seen in table 2, 8 of the 21 efficiency measures (representing seven different programs), showed an improvement in efficiency between the baseline and most current year. Ten of the efficiency measures (representing seven programs) showed a decrease in efficiency over the reported periods. Three measures (representing two programs) had only baseline data. Table 2: Gains/Losses and Reported Use for Selected Programs' Efficiency Measures: Department: Agriculture; Program: Plant & Animal Health Monitoring; Reported use of efficiency measure(s): Used; Efficiency measures: Value of damage prevented or mitigated by the monitoring and surveillance programs per dollar spent; Efficiency: Gain. Department: Education; Program: Smaller Learning Communities; Reported use of efficiency measure(s): Did not use[A]; Efficiency measures: FY 03 Cohort: Cost (in dollars) per student demonstrating proficiency or advanced skills in reading; Efficiency: Gain. Department: Education; Program: Smaller Learning Communities; Reported use of efficiency measure(s): Did not use[A]; Efficiency measures: FY 03 Cohort: Cost (in dollars) per student demonstrating proficiency or advanced skills in mathematics; Efficiency: Gain. Department: Education; Program: Smaller Learning Communities; Reported use of efficiency measure(s): Did not use[A]; Efficiency measures: FY 04 Cohort: Cost (in dollars) per student demonstrating proficiency or advanced skills in reading; Efficiency: Loss. Department: Education; Program: Smaller Learning Communities; Reported use of efficiency measure(s): Did not use[A]; Efficiency measures: FY 04 Cohort: Cost (in dollars) per student demonstrating proficiency or advanced skills in mathematics; Efficiency: Loss. Department: Education; Program: Smaller Learning Communities; Reported use of efficiency measure(s): Did not use[A]; Efficiency measures: FY 05 Cohort: Cost (in dollars) per student demonstrating proficiency or advanced skills in reading; Efficiency: Baseline data only. Department: Education; Program: Smaller Learning Communities; Reported use of efficiency measure(s): Did not use[A]; Efficiency measures: FY 05 Cohort: Cost (in dollars) per student demonstrating proficiency or advanced skills in mathematics; Efficiency: Baseline data only. Department: Education; Program: Student Aid Administration; Reported use of efficiency measure(s): Used; Efficiency measures: Direct administrative unit costs for origination and disbursement of student aid; Efficiency: Gain. Department: Interior; Program: Fish and Wildlife Services Fisheries; Reported use of efficiency measure(s): Used; Efficiency measures: Pounds/dollar of healthy rainbow trout produced for recreation; Efficiency: Loss. Department: Interior; Program: Wildland Fire Management; Reported use of efficiency measure(s): Used; Efficiency measures: Number of acres treated in the wildland-urban interface per million dollars gross investment; Efficiency: Gain. Department: Interior; Program: Wildland Fire Management; Reported use of efficiency measure(s): [Empty]; Efficiency measures: Number of acres treated outside the wildland- urban interface per million dollars gross investment; Efficiency: Loss. Department: Interior; Program: Wildland Fire Management; Reported use of efficiency measure(s): [Empty]; Efficiency measures: Number of acres in fire regimes 1, 2, or 3 moved to a better condition class per million dollars of gross investment; Efficiency: Loss. Department: Labor; Program: Energy Employees Occupational Illness Compensation; Reported use of efficiency measure(s): Did not use; Efficiency measures: Average number of decisions per full-time equivalent; Efficiency: Gain. Department: Labor; Program: Job Corps; Reported use of efficiency measure(s): Did not use; Efficiency measures: Cost per participant; Efficiency: Loss. Department: Labor; Program: Occupational Safety & Health Administration; Reported use of efficiency measure(s): Did not use; Efficiency measures: Inspections per Compliance Safety & Health Officer; Efficiency: Loss. Department: Labor; Program: Unemployment Insurance Administration State Grants; Reported use of efficiency measure(s): Did not use; Efficiency measures: Number of timely and accurate initial benefit claims per $1,000 of inflation-adjusted base grant funds; Efficiency: Gain. Department: Labor; Program: Workforce Investment Act-Migrant & Seasonal Farmworkers; Reported use of efficiency measure(s): Did not use; Efficiency measures: Cost per participant; Efficiency: Loss. Department: Transportation; Program: FAA Air Traffic Organization-Technical Operations; Reported use of efficiency measure(s): Did not use; Efficiency measures: Unit cost for providing ATO-technical operations services; Efficiency: Gain. Department: Transportation; Program: FAA Air Traffic Organization-Terminal Programs; Reported use of efficiency measure(s): Used; Efficiency measures: Unit cost for providing terminal services; Efficiency: Loss. Department: Transportation; Program: FAA Air Traffic Organization-Terminal Programs; Reported use of efficiency measure(s): Used; Efficiency measures: Productivity rate at service delivery points; Efficiency: Loss. Department: Transportation; Program: National Highway Traffic Safety Administration-Operations & Research; Reported use of efficiency measure(s): Did not use; Efficiency measures: Average costs incurred to complete a defect investigation; Efficiency: Baseline data only. Department: Total number of efficiency measures; Efficiency measures: 21; Net gain: 8; Net loss: 10; Baseline data only: 3. Source: GAO analysis of OMB and agency data and agency officials. Notes: Table excludes measures missing typical elements of an efficiency measure. We determined the net change in efficiency over time by comparing the latest year's actual data to the baseline. Some programs had only one year of reported data for making comparisons, while other programs had multiple years of reported data. Reporting the net change over a several year period may obscure interim annual gains or losses in reported efficiency. [A] Agency officials indicated they initially used the efficiency data collected to explore whether there might be some relationship between costs per student and either uses of funds or number of grade levels served, and determined that the data were not of sufficient quality to permit that analysis. We concluded the information was therefore not useful for decisionmaking. [End of table] We have previously reported that agencies can use performance information to make various types of management decisions to improve programs and results.[Footnote 40] The same is true for performance measures that track efficiency--managers need to use the information to help them identify actions needed to bring about improved efficiency. Our review of selected programs that had measures with both elements of a typical efficiency measure found variety in terms of whether officials reported using efficiency measures. We also found no clear relationship between efficiency gains or losses and whether program officials reported using or not using efficiency measures. Officials from three of the seven programs that reported efficiency gains described using their efficiency measures, while officials for three additional programs with efficiency gains said they did not use the efficiency measures. Officials for the other program with efficiency gains reported mixed pictures, saying they did not use the efficiency measure but found some value in the measure or its components. A similar mix was found among programs that reported net losses in efficiency, with officials for three programs using the efficiency measures and officials for four programs not using them. One example of a program that showed a net gain over time for its efficiency measure and for which officials reported using the data was the Department of Education's Student Aid Administration program. Reducing costs was one of the primary objectives of the program. Their efficiency measure--direct administrative unit costs to originate and disburse student loans and Pell Grants--showed a gain in efficiency from 2006 to 2008. The agency provides federal assistance to eligible students by partnering with postsecondary schools, financial institutions, and guaranty agencies (state and nonprofit agencies that guarantee loans against default). Program officials told us they used information from this measure to establish targets for reduced unit costs for their lending transactions. For example, they reported using the data to negotiate a lower cost for the origination of direct student loans by a sole-source contractor.[Footnote 41] FSA used a contractor to originate the loans made directly to students. The contract allowed for a certain quantity of loan originations for a set price, up to a maximum number of loans each year. According to program officials, the sharp reduction in credit availability due to the financial crisis beginning in 2008 led to an increase in demand for FSA direct loans. FSA had projected that demand for direct student loans in the 4th quarter of fiscal year 2009 would exceed the contract maximum by 3 million loans. The contractor proposed a price of $8.9 million for the additional loans, arguing that the added volume would require higher infrastructure costs associated with greater call center capacity. FSA officials told us they analyzed historical data for their efficiency measure and found that the unit cost to originate loans decreased as volume increased. They used this analysis to challenge the contractor's bid and succeeded in lowering the agreed price to $4.9 million. Officials reported that legislation, federal cost accounting standards, and our previous recommendations all contributed to pressure to track unit costs and try to lower administrative costs. Consequently, the agency had developed a number of cost models, which facilitated their developing the efficiency measure for PART. The Department of the Interior's Fisheries program provides an example in which the efficiency measure showed a net loss but officials said they used the efficiency measure data to make management decisions. The efficiency measure tracked the efficiency (pounds per dollar) of producing healthy rainbow trout for recreation. For the first 4 years examined, fiscal years 2004 through 2007, the efficiency measure varied slightly, indicating that overall efficiency was relatively stable. For fiscal year 2008, however, the measure fell, indicating a significant drop in efficiency. Officials attributed this drop to a 31 percent increase in feed, energy, and utility costs that was experienced throughout the country in 2008 and was beyond their control. Several fishery stations reported 40 percent increases in feed costs in just 1 year. Officials told us that having information about the decline in efficiency was valuable because it led individual stations to look for opportunities to lower other costs of production that were within their control. For example, program managers said they used their efficiency measure data to help them decide to phase out the production of inefficient (more costly) strains of trout. In addition, they said they used the measure to help manage the losses resulting from diseased trout that could not be sold by shifting production from one fishery to another that did not have a problem with disease. Officials said they thought it was easier for programs that directly produced products or provided services to develop and use efficiency measures. They said they had a relatively easy time of developing their efficiency measure because they directly produce a product (i.e., rainbow trout). The Department of the Interior's Wildland Fire Management Program reported mixed efficiency results. Of their three efficiency measures, two showed a net loss and one showed a net gain. Even though the results were mixed, officials said they used the data to establish ranges of acceptable cost estimates for contract or grant proposals and to identify outliers. Officials said their efficiency measures, which tracked numbers of wildland acres treated or moved to a better condition class (to reduce the likelihood of wildland fires) per million dollars, enabled them to identify unusually high or low costs when evaluating proposals from field units for funding treatments. They could identify a proposal that did not fall within the normal range of prior projects in terms of costs, do further analysis, and ask for explanations from field staff to better understand why the proposal was outside the norm. Program officials also said they used a tool called Ecosystem Management Decision Support (EMDS) to help prioritize projects and allocate funding for future years. They said EMDS takes into account various factors, including past performance and efficiency. For example, fuel treatments that demonstrated greater efficiency would be given higher priority for funding under EMDS, other factors being equal. While FAA's Air Traffic Organization Technical Operations program's efficiency measure showed a net gain, officials said they did not use it to make major decisions. ATO Technical Operations is responsible for maintaining and modernizing equipment needed in the national airspace system to deliver air traffic services. It fields, repairs, and maintains a huge network of complex equipment, including radars, instrument landing systems, radio beacons, runway lighting, and computer systems. The efficiency measure, unit cost for providing ATO Technical Operations services, is the "total labor obligations for the Technical Operations' Service Unit" divided by the total hours of operational availability (or equipment "uptime"). Officials said the measure was used as a baselining effort, and no decisions have been made as a result. Officials explained that they cannot significantly influence labor costs because of a labor agreement that requires ATO to maintain 6,100 direct employees. Officials said they have used data for the denominator of the efficiency measure, on the hours of operational availability. Equipment needs to be available continuously, and currently is about 99.7 percent of the time. Officials said they have not done the marginal cost analysis to determine whether it would be cost-effective to try to increase equipment uptime, but they have broken the data down by location and looked for outliers and tried to address impediments to operational availability at certain locations. They also said that while they have not used the efficiency measure to make any management decisions, it has been valuable in helping to orient staff to think about costs of operations and how to go about looking for efficiency improvements. Lastly, the Department of Labor's Occupational Safety and Health Administration (OSHA) program reported a net loss for the efficiency measure and told us they did not use the data. Officials said the current efficiency measure--inspections per Compliance Safety and Health Officer--was only a "back room calculation" and was not something they promoted or used to make decisions within the organization. They said they did not evaluate the performance of staff based on the number of inspections they conducted, because doing so could lead to a perverse effect of rushing through inspections in order to complete them more quickly, resulting in poorer quality inspections. In addition, officials said they did not believe anyone used the OSHA efficiency measure other than for reporting purposes. Program Officials Reported Challenges to Developing and Using Efficiency Measures: Officials from all of the selected programs we reviewed identified one or more challenges related to developing or using efficiency measures. The challenges cited were not new; we have reported on similar types of challenges in our prior work on PART and performance measurement issues in general.[Footnote 42] Challenges related to OMB's guidance and technical assistance for efficiency measures specifically included: a program definition that did not correspond well to program operations; an emphasis on developing outcome-oriented efficiency measures; achieving required annual improvement targets for efficiency; and inconsistencies and limitations in OMB's guidance and technical assistance. In addition, officials described the difficulty of trying to compare the relative efficiency of programs (or units within programs) that have significantly different objectives, activities, or cost data. Developing Efficiency Measures Based on a Program Definition That Did Not Correspond Well to Operations: We previously reported that determining the appropriate program or unit of analysis for a PART assessment was not always obvious, and what OMB determined was useful did not necessarily match agency organization or planning elements.[Footnote 43] We found that OMB sometimes aggregated separate programs into one for the purposes of a PART assessment, and in other cases disaggregated programs. Aggregating programs sometimes made it difficult to create a limited, but comprehensive, set of performance measures for programs with multiple missions, and agency officials noted that difficulties could arise when unrelated programs and programs with uneven success levels were combined for PART. At the same time, disaggregating a program too narrowly could distort its relationship to other programs involved in achieving a common goal, and sometimes ignored the interdependence of programs by artificially isolating programs from the larger contexts in which they operated. While OMB, in response to one of our recommendations, expanded PART guidance on how a unit of analysis was to be determined, problems related to defining programs for PART remained. An OMB staff member acknowledged to us that OMB often combined what agencies considered and managed as separate programs in order to identify a program for PART. According to some program officials, the way in which OMB grouped their activities into a program for the PART assessment was not useful, and so the resulting program-level efficiency measure developed for PART was not useful. Officials from the National Highway Traffic Safety Administration (NHTSA) within the Department of Transportation told us that the way OMB and the department defined their program for the PART assessment was a key challenge to developing a useful efficiency measure. Officials said that NHTSA's mission and operations are organized along two major programmatic lines: highway and motor vehicle safety. In contrast, for purposes of PART and development of the required efficiency measures, NHTSA was organized into two programs that received separate PART assessments: Operations and Research, and Grant Management. As a consequence, officials said the efficiency measure developed for the Operations and Research program was not meaningful. They said they were revising their efficiency measures and planned to develop one for each of the programmatic areas. Emphasis on Developing Outcome-Oriented Efficiency Measures: In previous work, we identified challenges involved in developing useful results-or outcome-oriented performance measures for some programs, such as those geared toward long-term health outcomes and research and development.[Footnote 44] We reported that many of the outcomes for which federal programs are responsible are part of a broader effort involving federal, state, local, nonprofit, and private partners, and that it is often difficult to isolate a particular program's contribution to an outcome.[Footnote 45] However, we also reported on how selected agencies that had limited control over the achievement of their intended objectives addressed the challenge by employing various strategies, such as including intermediate outcomes within their direct control along with far-reaching or end outcomes. [Footnote 46] In a previous review of PART, we reported that OMB had taken steps to clarify PART guidance on using outcome and output performance measures, and had accepted administrative efficiency measures instead of outcome-level efficiency measures for some programs.[Footnote 47] However, we also reported that agencies had mixed success in reaching agreement with OMB in these areas. As mentioned above, of the 21 measures from selected programs that had typical elements of an efficiency measure, 13 contained outputs, and 8 contained outcomes. While OMB's PART guidance described efficiency measures as including both outcome-and output-level impacts, it stated that the best efficiency measures captured outcomes. Further, program officials told us that OMB pressed some programs to have efficiency measures that captured outcomes instead of outputs. Similar to findings from our prior work, some officials we interviewed for this review said it was difficult for their programs to interpret outcome-level efficiency measure information, because factors other than program funding affected the outcome of the program. For example, the purpose of the Forest Service's Watershed program is to restore, enhance, and maintain watershed conditions, including soil, water, air, and forest and rangeland vegetation within the national forests and grasslands. Management of these physical and biological resources provides a foundation for healthy, viable ecosystems.[Footnote 48] The Watershed program received a "Results Not Demonstrated" rating from the OMB 2006 PART assessment process because it lacked long-term, outcome-based performance and efficiency measures to track the performance of land management activities on national forest and nonfederal watersheds, or demonstrated water quality improvement over time. Basically, the Forest Service was unable to track how watershed projects were prioritized, identify the benefits associated with restoration projects, and determine whether those projects improved watershed condition. Officials said they had previously proposed the unit cost of watershed improvement projects as an efficiency measure under PART, but OMB rejected it partly because it was an output-rather than an outcome-level measure. According to Forest Service documents, factors beyond its control affect watershed conditions, and it is difficult to demonstrate the impact of program activities on watersheds and try to determine the most cost-effective way to improve the outcome. The agency's ability to improve the condition of watersheds depends on many factors, including what percentage of the land affecting the watershed is privately owned as opposed to owned by the Forest Service and past impacts--for example, an official said that lands that were previously mined may be more difficult to restore. Officials said that the cost of trying to improve some watersheds would exceed available funds, and in some cases passive restoration, or doing nothing and letting natural processes return, could improve conditions as rapidly as any program interventions could. Forest Service officials said they reached agreement with OMB to develop an outcome-oriented efficiency measure based on the cost of improving or maintaining the condition of watershed acres. According to a 2008 report prepared by the Forest Service,[Footnote 49] in order to be able to relate costs to outcomes, program officials explained that they will need to develop a consistent approach for assessing watershed condition and a system that would enable them to track changes in watershed conditions and relate these changes to Forest Service management activities. Following implementation of this approach, the agency would be able to track improvements in program outcomes and relate changes to cost. Achieving Required Annual Improvement Targets for Efficiency: OMB's PMA and PART guidance required programs to set annual improvement targets for their efficiency measures. We previously reported that in some programs, long-term outcomes are expected to occur over time through multiple steps, and that it can take years to observe program results. For these programs, it can be difficult to identify performance measures that will provide information on annual progress toward program results.[Footnote 50] Along these lines, some program officials we interviewed told us it was not reasonable to expect annual improvements in efficiency for some programs because it might take several years for an increase in efficiency to be realized as a result of some intervention or investment, or because a technological advance might result in a one- time cost savings that would not continue to be achieved over time. For example, the Plant and Animal Health Monitoring and Surveillance programs of Agriculture's Animal and Plant Health Inspection Service, which protects the health and value of agriculture and natural resources through early detection of pest and disease outbreaks, had an efficiency measure that tracked the value of damage prevented or mitigated by the program per dollar spent. Program officials told us that it was difficult to show improvements in efficiency every year. They said that as a science-based program, it took time to develop new technologies that improved efficiency, and the effect might be a one- time improvement in efficiency that would not result in continued additional efficiency gains over time. Similarly, officials from the Department of the Interior's Endangered Species program stated that the timeframe needed to achieve results in terms of conservation and recovery of an endangered species is longer than an annual or even 5- year timeframe. They said it is difficult to associate additional funding with a defined outcome in a given year. Officials from the Department of Labor's Center for Program Planning and Results acknowledged that their office and OMB strongly encouraged agencies and programs to show annual improvements for efficiency measures, which led to some friction in setting targets for out-years for some programs. They said that pressure to show annual improvements in efficiency resulted in some programs revising targets for the efficiency measures every year because they could not achieve the annual targets. An official said that there was a lot of focus on numerical annual targets for efficiency measures, and because some programs cannot realistically see improvements in efficiency in a 1- year time period, monitoring trends would be better. Inconsistent or Limited OMB Guidance and Technical Assistance: As we previously reported, OMB staff had to exercise judgment in interpreting and applying the PART tool to complex federal programs, and were not fully consistent in interpreting the guidance.[Footnote 51] In prior reviews of PART, we identified instances in which OMB staff inconsistently defined appropriate measures, in terms of outcomes versus outputs, for programs. We reported that some program officials said that OMB staff used different standards to define measures as outcome oriented. We also reported that OMB took steps to try to encourage consistent application of PART in evaluating government programs, including pilot testing the assessment instrument, clarifying guidance, conducting consistency reviews, and making improvements to guidance based on experience.[Footnote 52] OMB also issued examples of efficiency measures it identified as exemplary [Footnote 53] and expanded the guidance on efficiency measures. [Footnote 54] While officials for some programs we interviewed told us that OMB assistance and feedback under PART were valuable in developing useful efficiency measures, officials for other programs cited inconsistencies and limitations in OMB's PART guidance and technical assistance that made the development of acceptable and useful efficiency measures more challenging. For example, officials for Agriculture's Plant and Animal Health Monitoring programs said they worked with the department and OMB representatives to discuss efficiency measures and obtain feedback on proposed measures. Officials said feedback obtained was useful and allowed them to consider options they had not previously identified, and in some cases they incorporated the advice. Officials said that the efficiency measure tracking the value of damage prevented and mitigated per program dollar spent was a direct result of an OMB recommendation. [Footnote 55] However, officials for other programs said that PART guidance and OMB technical assistance and feedback provided to programs on efficiency measures were insufficient or inconsistent. For example, officials for the Department of the Interior's Endangered Species program, which lacked an efficiency measure that had been approved by OMB, said they believed that OMB's review of proposed efficiency measures was inconsistent. Officials said that OMB rejected a proposed output-level efficiency measure for the Endangered Species program and pushed for an outcome-level measure, but approved a similar measure for another program in a different federal department. Similarly, officials for the Forest Service Watershed program in Agriculture, which did not have any of its proposed efficiency measures accepted by OMB for the PART assessment, stated that lack of consistency on OMB's part in defining acceptable efficiency measures complicated the process for them. They said OMB rejected a measure they proposed, but approved a similar measure for another agency. Further, officials for OSHA in the Department of Labor indicated that they worked with two OMB analysts who were not as familiar with their agency as the current analyst and created rework. Overall, they did not believe the process they undertook with OMB to develop an efficiency measure was fruitful. Comparing Efficiency across or within Programs When Program Objectives, Activities, or Cost Data Differ: Officials we interviewed from the Department of Education's Office of Federal Student Aid indicated that they eventually wanted to use data for the Student Aid Administration program's efficiency measure (direct administrative unit costs for origination and disbursement of student aid), to compare the costs of similar activities performed by different contractors. However, we previously reported that challenges can result from the difficult but potentially useful process of comparing the costs of programs related to similar goals.[Footnote 56] We have also reported that in order to effectively compare a program to alternative strategies for achieving the same goals, comprehensive data on the program and comparable data on alternatives need to be available.[Footnote 57] In our prior work on human services programs, we reported that OMB officials recognized that programs are different and it may not be possible to compare costs across programs, especially when costs are defined differently due to programmatic differences.[Footnote 58] Officials from some selected programs we reviewed questioned whether it was reasonable to use efficiency measures for comparative analysis of performance across programs when the objectives, activities, or costs of the programs differed significantly. For example, an official from the Department of Labor's Job Corps program said it was not appropriate to compare their program's performance to that of other department employment and training programs in terms of the efficiency measure, which tracked cost (appropriations) per participant. According to the program's PART assessment, the program's purpose is to assist eligible disadvantaged youth (ages 16-24) who need and can benefit from intensive education and training services to become more employable, responsible, and productive citizens. Participants have characteristics, such as being a school dropout, homeless, or in need of intensive counseling to help them participate successfully in school or hold a job, that are barriers to employment. Program officials said that Job Corps is quite different from other employment and training programs run by the department because it involves removing participants from a negative environment and placing them in a totally different, primarily residential, environment. Such a model involves higher operating costs associated with providing participants intensive services in a residential setting for up to 2 years, which would make it appear less efficient when compared to nonresidential programs.[Footnote 59],[Footnote 60] As another example, officials for the Endangered Species program at the Department of the Interior questioned whether it made sense to try to compare the efficiency of efforts to protect different species. The program works with states, tribes, other federal agencies, nongovernmental organizations, academia, and private landowners to promote the conservation and prevent extinction of over 1,300 endangered or threatened species. As noted in the program's strategic plan,[Footnote 61] each species has inherent biological constraints which create challenges to its recovery. Officials told us that they work with vastly different species in different regions, many factors affect the complexity of their work, and each case is unique. We previously reported that species are ranked by priority, but rankings do not reflect how much funding is needed to protect a species. [Footnote 62] Officials told us that the cost of an intervention, such as building a fence, could be much cheaper for one species in a particular region than for another species in a different location. The head of the department's Office of Planning and Performance Management in the Office of the Secretary said that because the effort to save some species is so much more complicated and expensive than for others, it is not meaningful to simply compare the "cost per unit" or efficiency of saving different species without considering other factors such as the time frame involved, and the scope and level of treatment needed. For example, he suggested that it was not reasonable to try to compare the cost of saving the polar bear to the cost of saving a species of plant.[Footnote 63] Using GPRA as a Framework, a Broader Array of Strategies Can Be Used to Seek Improvements in Efficiency: As stated above, OMB's approach to improving the efficiency of federal programs under PMA and PART focused on requiring individual programs to develop efficiency measures, identify procedures to achieve efficiencies, and achieve annual gains in efficiency. In prior reports, we concluded that PART's focus on program-level assessments could not substitute for GPRA's focus on thematic goals and department- and governmentwide crosscutting comparisons.[Footnote 64] Through our review of literature, we identified a variety of strategic and crosscutting approaches that government, nongovernment, and business organizations have used in their efforts to improve efficiency. For example, the United Kingdom and some state governments provide some important insights into such governmentwide efficiency efforts. These approaches share a common theme that performance can be maintained or even improved while reducing unnecessary costs associated with outmoded or wasteful operations, processes, and purchases. These approaches to efficiency improvement differ from OMB's approach under PMA/PART in that they can be applied at government-or agencywide levels in addition to being applied within specific programs. Officials from some selected programs provided examples of additional efforts they were undertaking to improve efficiency, some of which can be aligned with these broader approaches we identified in the literature. Broadening the application of these approaches beyond the program level could help to identify even greater opportunities for improvements in the efficiency of federal government operations. GPRA's planning and reporting requirements can provide a framework for agencies to take a more strategic approach to improving federal government efficiency. Governmentwide Reviews Can Help Identify and Develop Strategies to Improve Efficiency: Governmentwide reviews have been conducted in the United Kingdom (UK) and by some state governments in the U.S. to help identify and implement strategic approaches to improve efficiency. Such reviews have been ordered by executive leadership to address a wide range of government activity. Reviews have been broad in scope, and initiatives undertaken to improve efficiency have been crosscutting and could be applied across processes, services, and organizations rather than just at the program level as required for federal agencies under OMB's PART approach. In the UK in 2004, Her Majesty's (HM) Treasury published a first of its kind, government-wide efficiency review that examined government processes, identified opportunities for cutting costs and improving services, and developed proposals to deliver sustainable efficiencies in the use of resources within both central and local government. The review focused on improving government efficiency in areas such as procurement, funding, regulation, citizen services, and administration. The efficiency review proposed strategies to improve efficiency that were adopted by HM Treasury in the UK's 2004 budget. HM Treasury actively supported departments in their individual efficiency programs. HM Treasury negotiated efficiency goals with each department and created a centralized efficiency team managed by the Office of Government Commerce to help departments achieve efficiency gains. HM Treasury brought in outside expertise, including senior figures from the private and public sector, to support and work with departments. Additional specialist change agents were employed to assist departments with trying to achieve efficiency improvements in areas such as e-government, human resources, IT, finance, construction, and commodity procurement. Change agents addressed problems created by highly fragmented markets that crossed departmental boundaries. To assist departments in financing efficiency improvement programs, HM Treasury created a £300 million Efficiency Challenge Fund that provided departments with matching funds for efficiency improvement programs. Funds were approved based on objective criteria such as the ratio of expected savings to matching funds, probability of achieving savings, evidence that alternative funds were not available, and progress in delivering efficiency gains. In a final review of the completed efficiency program in November 2008, HM Treasury reported that the program led to £26.5 billion in annual efficiency gains (60 percent of which were direct cost savings while the remainder represented increased levels of public service rather than immediate cash savings). These final results have not been audited, although portions of earlier reported efficiency gains were reviewed by the UK National Audit Office (NAO) with mixed results. In 2007, more than halfway through implementing the efficiency program, the NAO reviewed a sample of the reported efficiency gains and found that some had a significant risk of inaccuracy. Nevertheless, NAO concluded at the time that of the £13.3 billion ($21.2 billion) reported gains, 26 percent (£3.5 billion ($5.6 billion)) fairly represented efficiencies achieved, 51 percent (£6.7 billion ($10.7 billion)) appeared to represent improvements in efficiency but had associated measurement issues and uncertainty, and 23 percent (£3.1 billion ($4.9 billion)) had potential to represent improvements in efficiency, but the measures used either had not yet demonstrated efficiency or the reported gains could be substantially incorrect. NAO cited measurement problems arising from longstanding weaknesses in departments' data systems and from trying to measure savings in areas with complex relationships between inputs and outputs. Despite the caveats identified by NAO in trying to verify the reported efficiency gains, NAO reported that "the efficiency program made important contributions and there is now a greater focus on efficiency among senior staff." In the U.S., several state governments initiated a variety of governmentwide reviews. For example, Arizona initiated an efficiency review in 2003 to try to find ways to improve customer service, reduce cost, and eliminate duplication while drawing heavily on internal state resources and experts in state government to manage the effort. The Arizona review investigated potential savings in 12 statewide, or crosscutting, issues that affected multiple agencies and offered the greatest potential for efficiency savings. In 2004, California initiated an ongoing review, the California Performance Review, with four major components: executive branch reorganization, program performance assessment and budgeting, improved services and productivity, and acquisition reform. Iowa Excellence is another governmentwide effort designed to improve customer service and cut costs in state government. Iowa agencies examined their performance using Malcolm Baldrige National Quality Program criteria. The state governmentwide review efforts share these beneficial features: serving as an effective method of cost-saving analysis, helping with prioritizing services to citizens, and providing a targeted goal for the administration of state governments that may contribute to improved government efficiency and effectiveness. Restructuring Outmoded Government Organizations and Operations Can Contribute to Improvements in Efficiency: Solving the daunting fiscal challenges facing the nation will require rethinking the base of existing federal spending and tax programs, policies, and activities by reviewing their results and testing their continued relevance and relative priority for a changing society. Such a reexamination offers the prospect of addressing emerging needs by weeding out programs and policies that are outdated or ineffective. Those programs and policies that remain relevant could be updated and modernized by improving their targeting and efficiency through such actions as redesigning allocation and cost-sharing provisions, consolidating facilities and programs, and streamlining and reengineering operations and processes.[Footnote 65] While significant efficiency gains can be achieved by restructuring outmoded government organizations and operations to better meet current needs, we have reported that such restructurings can be immensely complex and politically charged.[Footnote 66] All key players must be involved in the process--Congress, the President, affected executive branch agencies, their employees and unions, and other interested parties, including the public. The fundamental restructuring of the health care system for veterans in the mid-1990s and the Department of Defense (DOD) Base Realignment and Closure (BRAC) process demonstrate the significant efficiencies that can result from reexamining the base of federal programs. U.S. Department of Veterans Affairs (VA) Health Care: In the mid-1990s, the U.S. Department of Veterans Affairs (VA), recognizing that its health care system was inefficient and in need of reform, followed the lead of private sector health care providers and began reorganizing its system to improve efficiency and access.[Footnote 67] In 1995, VA introduced substantial operational and structural changes in its health care system to improve the quality, efficiency of, and access to care by reducing its historical reliance on inpatient care. VA shifted its focus from a bed-based, inpatient system emphasizing specialty care to one emphasizing primary care provided on an outpatient basis. To support VA's restructuring efforts, Congress enacted legislation in October 1996 that eliminated several restrictions on veterans' eligibility for VA outpatient care, which allowed VA to serve more patients. VA also phased in a new national resource allocation method, the Veterans Equitable Resource Allocation (VERA) system, as part of a broader effort to provide incentives for networks and medical centers to improve efficiency and serve more veterans. Networks that increased their patient workload compared with other networks gained resources under VERA; those whose patient workloads decreased compared with other networks lost resources. As we reported, VA recognized that VERA networks were responsible for fostering change, eliminating duplicative services, and encouraging cooperation among medical facilities. We reported that increased efficiency resulting from increased outpatient care, staff reductions and reassignments, and integrations at the medical centers resulted in savings. For example, from fiscal year 1996 to 1998, the VA reduced staff by approximately 16,114 (8 percent), resulting in estimated annual savings of $897 million. In some cases, however, improvements in efficiency did not save money because hospitals reinvested funds to enhance or offer new services. Base Realignment and Closures: The military base realignment and closure experience provides another example of the efficiencies that can be gained by reexamining outmoded government structures and operations to meet current operating needs. In the late 1980s, changes in the national security environment resulted in a defense infrastructure with more bases than DOD needed. To enable DOD to close unneeded bases and realign other bases, Congress enacted legislation that instituted BRAC rounds in 1988, 1991, 1993, 1995, and 2005. A special commission established for the 1988 round made realignment and closure recommendations to the Senate and House Committees on the Armed Services. For the succeeding rounds, special BRAC Commissions were set up, as required by legislation, to make specific recommendations to the President, who in turn sent the commissions' recommendations to Congress. While the statutory requirements vary across the BRAC rounds, those in the 2005 round stipulate that closure and realignment decisions must be based upon selection criteria, a current force structure plan, and infrastructure inventory developed by the Secretary of Defense. Further, the selection criteria were required to be publicized in the Federal Register to solicit public comments on the criteria before they were finalized. A clear authorization was mandated by Congress involving both the executive and legislative branches of government while recognizing and involving those affected by the government's actions. With the completion of the recommended actions for the first four BRAC rounds by 2001, DOD had significantly reduced its domestic infrastructure through the realignment and closure of hundreds of bases and had reportedly generated billions in net savings or cost avoidances during the process. While DOD's focus for the four BRAC rounds through 1995 was largely on eliminating excess capacity, the Secretary of Defense at the outset of the BRAC 2005 round--the fifth such round taken on by the department-- indicated its intent to reshape DOD's installations and realign DOD forces to meet defense needs for the next 20 years and eliminate excess physical capacity--the operation, sustainment, and recapitalization of which diverts resources from defense capability. Both DOD and the BRAC Commission reported that their primary consideration in making recommendations for the BRAC 2005 round was military value, which includes considerations such as an installation's current and future mission capabilities. As such, many of the BRAC 2005 recommendations involve complex realignments that reflect operational capacity to maximize warfighting capability and efficiency. We have reported that the fifth round, BRAC 2005, will be the biggest, most complex, and costliest BRAC round ever, in part because, unlike previous rounds, the Secretary of Defense viewed the 2005 round as an opportunity not only to achieve savings but also to assist in transforming the department. For example, DOD is consolidating facilities and programs through a BRAC action to relocate five training centers from across the United States into a single medical education and training center at one installation. Although anticipated savings resulting from implementing BRAC 2005 recommendations, which the department could use for other defense programs, remain an important consideration in justifying the need for this round, our calculations using DOD's fiscal year 2010 BRAC budget estimates have shown that estimated savings DOD expects to generate over the 20-year period ending in 2025 have declined from the BRAC Commission's estimate of $36 billion to $10.9 billion in constant fiscal year 2005 dollars.[Footnote 68] Process Improvement Methods and Technology Improvements Can Increase Efficiency: Process improvement methods can increase product quality and decrease costs, resulting in improved efficiency.[Footnote 69] Process improvement methods can involve examining processes and systems to identify and correct costly errors, bottlenecks, or duplicative processes while maintaining or improving the quality of outputs. There are numerous process methods that use different tools and techniques. For example, Six Sigma is a data-driven approach based on the idea of eliminating defects and errors that contribute to losses of time, money, opportunities, or business. The main idea behind Six Sigma is to measure the defects in a process and then devise solutions to eliminate them, helping an organization approach a high quality level. Another method is Business Process Reengineering (BPR), which redesigns the way work is done to better support the organization's mission and reduce costs. Reengineering starts with a high-level assessment of the organization's mission, strategic goals, and customers. As a result of the strategic assessment, BPR identifies, analyzes, and redesigns an organization's core business processes with the aim of achieving dramatic improvements in critical performance measures, such as cost, quality, service, and speed. A 2009 study conducted by the American Productivity and Quality Center (APQC)[Footnote 70] identified a variety of methods, including Six Sigma and Business Process Re-engineering, which have been used by organizations to focus on process improvement.[Footnote 71] The study included a survey of 281 small-to-large-sized enterprises with annual gross revenue of $4.2 trillion to identify current process-focused practices and learn about process effectiveness. Survey respondents identified various efficiency related improvements resulting from their process improvement approaches, such as streamlined processes, improved customer satisfaction, quality improvements, and improved decision making. In relation to process improvement, modernizing processes through investments in technology can generate efficiency gains. Our prior work indicates that the federal government can help streamline processes and potentially reduce long-term costs by facilitating technology enhancements.[Footnote 72] For example, as shown in figure 3, growth in electronic filing has allowed the Internal Revenue Service (IRS) to reduce staff years used to process paper tax returns. As electronic filing increased between fiscal years 1999 and 2006, IRS reduced the number of staff years devoted to total tax return processing by 34 percent.[Footnote 73] We have also reported that processing is more accurate and costs are lower to IRS as a result of electronic filing--IRS saves $2.71 for every return that is filed electronically instead of on paper. Figure 3: Number of Individual Returns and IRS Staff Years for Individual Paper and Electronic Processing, Fiscal Years 1999-2010: [Refer to PDF for image: combined stacked vertical bar and multiple line graph] Fiscal year: 1999; Staff years devoted to paper filing: 4,384; Staff years devoted to electronic filing: 280; Paper returns processed: 96.6 million; Electronic returns processed: 29.3 million. Fiscal year: 2000; Staff years devoted to paper filing: 4,108; Staff years devoted to electronic filing: 274; Paper returns processed: 92.9 million; Electronic returns processed: 35.4 million. Fiscal year: 2001; Staff years devoted to paper filing: 4,290; Staff years devoted to electronic filing: 302; Paper returns processed: 90.2 million; Electronic returns processed: 40.2 million. Fiscal year: 2002; Staff years devoted to paper filing: 4,207; Staff years devoted to electronic filing: 265; Paper returns processed: 84.6 million; Electronic returns processed: 46.8 million. Fiscal year: 2003; Staff years devoted to paper filing: 3,613; Staff years devoted to electronic filing: 226; Paper returns processed: 78.3 million; Electronic returns processed: 54.6 million. Fiscal year: 2004; Staff years devoted to paper filing: 3,281; Staff years devoted to electronic filing: 253; Paper returns processed: 70.2 million; Electronic returns processed: 61.3 million. Fiscal year: 2005; Staff years devoted to paper filing: 3,060; Staff years devoted to electronic filing: 208; Paper returns processed: 65.3 million; Electronic returns processed: 68.3 million. Fiscal year: 2006; Staff years devoted to paper filing: 2,815; Staff years devoted to electronic filing: 155; Paper returns processed: 61.9 million; Electronic returns processed: 72.8 million. Fiscal year: 2007; Staff years devoted to paper filing: 2,701; Staff years devoted to electronic filing: 152; Paper returns processed: 59.4 million; Electronic returns processed: 79.6 million. Fiscal year: 2008; Staff years devoted to paper filing: 2,509; Staff years devoted to electronic filing: 185; Paper returns processed: 51.6 million; Electronic returns processed: 89.3 million. Fiscal year: 2009[A]; Staff years devoted to paper filing: 2,183; Staff years devoted to electronic filing: 187; Paper returns processed: 46.9 million; Electronic returns processed: 93.3 million. Fiscal year: 2010[A]; Staff years devoted to paper filing: 1,981; Staff years devoted to electronic filing: 197; Paper returns processed: 42.6 million; Electronic returns processed: 98.3 million. Source: GAO analysis of IRS data. [A] Fiscal years 2009 and 2010 are IRS projections. [End of figure] The President's 2011 Budget described a variety of initiatives the administration intends to undertake to streamline existing IT infrastructure, improve the management of IT investments, and leverage new IT to improve the efficiency and effectiveness of federal government operations.[Footnote 74] In June 2009, the U.S. Chief Information Officer (CIO) launched the IT Dashboard, which allows the American people to monitor IT investments across the federal government. The IT Dashboard displays performance data on nearly 800 investments that agencies classify as major. The performance data used to track the 800 major IT investments include schedule, cost, and the agency CIO's assessment of the risk of the investment's ability to accomplish its goals. Beginning in January 2010, the U.S. CIO began holding TechStat Accountability Sessions--face-to-face, evidence-based reviews of IT programs, undertaken with OMB and agency leadership, to improve overall performance. According to the U.S. CIO's Web site on TechStat, in some cases this review process is leading to projects being eliminated. The administration has also indicated it intends to: * consolidate data centers to reduce costs and increase efficiency; * pursue "cloud computing," which will enable agencies to share information technology services and software rather than purchase or develop their own; * continue to pursue various "e-government" initiatives, which are expected to deliver services more efficiently both within across agency lines; and: * employ federal enterprise architectures and supporting segment architectures to streamline processes and modernize services, in many cases across agency lines. In addition to these IT initiatives, the Administration has also placed emphasis on reducing errors in payments. Executive Order 13520, signed in November 2009,[Footnote 75] requires, among other things, publishing information about improper payments on the Internet, including targets for reduction and recovery, and assigning a senior official to be accountable for reducing and recovering improper payments at relevant agencies. The executive order also lays out steps intended to lead to enhanced accountability of contractors and incentives and accountability provisions for state and local governments for reducing improper payments. Consistent with OMB's PART guidance for programs to identify procedures to improve efficiency, officials from several of the selected programs we reviewed said they had modernized information technology to reduce costs and improve services.[Footnote 76] Officials from the Department of Labor's Job Corps program said they reduced Federal Telecommunication Costs through the use of voice over Internet protocol and other improvements in technology, while expanding the use of video conferencing and e-learning to improve customer service. As a result of these efforts, officials reported cutting communication costs by $1 million. Officials for the Department of the Interior's Endangered Species program said they used information technology to reduce errors due to hand entry of data. They said that by eliminating manual entry of data, errors were reduced, which resulted in more accurate information and increased efficiency. Such methods are consistent with PART guidance to identify procedures, such as information technology improvements, to improve efficiency. However, the program-level focus of the PART process would not necessarily lead to an examination of efficiency improvements to be gained by improving the processes and systems outside a program's purview. Government processes and systems can involve multiple programs within and across federal agencies. For example, we previously reviewed the cost of administering seven key human services programs and found that the federal government may help balance administrative cost savings with program effectiveness and integrity by simplifying policies and facilitating technology improvements.[Footnote 77] Simplifying policies--especially those related to eligibility determination processes and federal funding structures--could save resources, improve productivity, and help staff focus more time on performing essential program activities. By helping states facilitate technology enhancements across programs, the federal government can help streamline processes and potentially reduce long- term costs. As another example, we have reported that the federal agencies that share responsibility for detecting and preventing seafood fraud [Footnote 78]--the Department of Homeland Security's Customs and Border Protection, the Department of Commerce's National Marine Fisheries Service, and the Department of Health and Human Services' Food and Drug Administration--have not taken advantage of opportunities to share information that could benefit each agency's efforts to detect and prevent seafood fraud, nor have they identified similar and sometimes overlapping activities that could be better coordinated to use limited resources more efficiently. For example, each agency has its own laboratory capability for determining seafood species and uses different methodologies for creating standards for species identification. The result is that neither the laboratories nor the data developed in them are shared. A Strategic Approach to Spending Can Be Used to Reduce Input Costs and Improve Efficiency: We have recommended that agencies take a strategic approach to spending that involves a range of activities--from using "spend analysis" to develop a better picture of what an agency is spending on goods and services, to taking an organization-wide approach for procuring goods and services.[Footnote 79] We found that private sector companies have adopted these activities to help leverage their buying power, reduce costs, and better manage suppliers of goods and services. By strategically managing costs, government can improve efficiency in the same way as private sector organizations examined in our prior work.[Footnote 80] "Spend analysis" is a tool that provides information about how much is being spent for goods and services, identifies buyers and suppliers, and helps identify opportunities to leverage buying power to save money and improve performance. To obtain this information, organizations use a number of practices involving automating, extracting, supplementing, organizing, and analyzing procurement data. Organizations then use these data to institute a series of structural, process, and role changes aimed at moving away from a fragmented procurement process to a more efficient and effective process in which managers make decisions on an organizationwide basis. Spend analysis allows for the creation of lower-cost consolidated contracts at the local, regional, or global level. As part of a strategic procurement effort, spend analysis allows companies to monitor trends in small and minority-owned business supplier participation to try to address the proper balance between small and minority business utilization, in addition to pursuing equally important corporate financial savings goals for strategic sourcing. Spend analysis is an important component of the administration's plans to improve government procurement. Along these lines, OMB issued memoranda in July and October of 2009 instructing agencies to increase competition for new contracts.[Footnote 81] The administration also set a net savings target of $40 billion to be achieved by agencies through improved contracting practices in fiscal year 2010 and 2011. The October memorandum provided agencies guidelines for increasing competition for contracts and structuring contracts to achieve the best results at the least cost to the taxpayer. Specifically, the memorandum recommends the use of spend analysis to identify the agency's largest spending categories, analyze and compare levels of competition achieved by different organizations within the agency, determine if more successful practices may exist for obtaining greater marketplace competition for a given spending category. Among the programs we reviewed, officials from the Job Corps program reported that they achieved improvements in efficiency by using some elements of a strategic spending approach. For example, Job Corps officials indicated that the program has avoided approximately $1 million in utility costs by purchasing energy from utilities using competitive bids in deregulated markets. When an area of the country became deregulated, the program would analyze the utility prices and quantities of electricity or natural gas used by the Job Corps centers in the area. If prices in the deregulated market looked favorable, the energy contracts for the centers would be placed out for bid to all eligible energy suppliers. Job Corps would select the bid with the best price and terms and set up a contract to purchase energy from them for a fixed period of time (usually 1 or 2 years). When the contracts came to an end, the process would be repeated. If the prices on the deregulated market were not favorable at that time, then the centers could revert back to the local utilities for their energy. Job Corps also conducted energy audits to identify problem areas and propose solutions to reduce energy costs at facilities where energy usage was above the benchmark. Job Corps reportedly reduced energy costs through investments in energy saving projects, training of staff and students to control energy use, and using an online system to review and analyze billing and procurement of energy in deregulated markets.[Footnote 82] GPRA Could Provide a Framework for Structuring a More Strategic Approach to Improving Government Efficiency: The administration has not clearly indicated whether it will continue to emphasize measuring efficiency at the program level as it did under PART. Rather, in describing its approach to performance and management in the President's budget,[Footnote 83] the Administration stated that GPRA and PART increased the production of measurements in many agencies, resulting in the availability of better measures than previously existed; however, these initial successes have not led to increased use. To encourage senior leaders to deliver results against the most important priorities, the administration tasked agencies with identifying and committing to a limited number of priority goals, generally three to eight, with high value to the public. The goals were to have ambitious, but realistic, targets to achieve within 18 to 24 months without need for new resources or legislation, and well- defined, outcome-based measures of progress. Further, in the coming year, the Administration will ask agency leaders to carry out a similar priority-setting exercise with top managers of their bureaus to set bureau-level goals and align those goals, as appropriate, with agencywide priority goals. These efforts are not distinct from the goal-setting and measurement expectations set forth in GPRA, but rather reflect an intention to translate GPRA from a reporting exercise to a performance improving practice across the federal government. By making agencies' top leaders responsible for specific goals that they themselves have named as most important, the Administration has stated that it hopes to dramatically improve accountability and the chances that government will deliver results on what matters most. To complement the renewed focus on achieving priority outcomes, the Administration has also proposed increased funding to conduct program evaluations to determine whether and how selected programs are contributing to desired outcomes. The Administration intends to take a three-tiered approach to funding new program initiatives. First, more money is proposed for promoting the adoption of programs and practices that generate results backed up by strong evidence. Second, for an additional group of programs with some supportive evidence but not as much, additional resources are allocated on the condition that the programs will be rigorously evaluated going forward. Third, the approach encourages agencies to innovate and to test ideas with strong potential--ideas supported by preliminary research findings or reasonable hypotheses. We have previously reported on how program evaluations can contribute to more useful and informative performance reports through assisting program managers in developing valid and reliable performance reporting and filling gaps in needed program information, such as establishing program impact and reasons for observed performance and addressing policy questions that extend beyond or across program borders.[Footnote 84] In addition to program evaluations that determine program impact or outcomes, we have identified cost-effectiveness analysis as a means to assess the cost of meeting a single goal or objective, which can be used to identify the least costly alternative for meeting that goal. In addition cost-benefit analysis aims to identify all relevant costs and benefits, usually expressed in dollar terms.[Footnote 85] Given the challenges program managers we interviewed cited in developing and using outcome-based efficiency measures, such evaluations might fill gaps in understanding the cost of achieving outcomes and allow for cost comparisons across alternative program strategies intended to produce the same results. GPRA's focus on strategic planning, development of long-term goals, and accountability for results provides a framework that Congress, OMB, and executive branch agencies could use to promote and apply various approaches to achieving efficiency gains in federal agencies. Congress enacted GPRA in part to address waste and inefficiency in federal programs. Agencies could use strategic plans as a vehicle for identifying longer-term efficiency improvement goals and strategies for achieving them. They could use annual performance plans to describe performance goals designed to contribute to longer-term efficiency goals, and annual performance and accountability reports to monitor progress toward achieving annual or longer-term efficiency goals. GPRA could provide a framework that would balance efforts to improve efficiency with overall improvements in outcomes. GPRA was intended to provide a balanced picture of performance that focused on effectiveness as well as efficiency. Officials from some selected programs identified a risk that focusing on reducing costs to improve efficiency could potentially have negative effects on the quantity or quality of outputs or outcomes. For example, officials for the Smaller Learning Communities program at the Department of Education said their outcome-level efficiency measures, which tracked the cost per student demonstrating proficiency or advanced skills in math or reading, could result in unintended negative consequences such as providing motivation for grantees to cut costs by lowering teacher salaries, lower proficiency standards so that more students would be classified as proficient, or engage in "creaming" (focus only on those students most likely to achieve gains). OMB's PART guidance included recognition that efforts to improve efficiency can involve risk to quality, outcomes, or other factors such as customer satisfaction. The PART guidance included as an example how reducing processing time to be more efficient could result in increased error rates. OMB recommended that programs assess risks associated with efficiency improvement efforts and develop risk management plans if needed. Similarly, in the United Kingdom's governmentwide efficiency program, departments could only report improvements in efficiency if they could also demonstrate that the quality of public services was not adversely affected by the reforms.[Footnote 86] Under GPRA, agencies' plans and performance measures are expected to strike difficult balances among competing demands, including program outcomes, cost, service quality, customer satisfaction, and other stakeholder concerns. Therefore agencies could mitigate the risk to program outcomes and quality associated with taking a narrow cost-cutting approach by developing GPRA goals, strategies, and performance measures that clearly balance these competing demands. We have previously reported that OMB could use the provision of GPRA that calls for OMB to develop a governmentwide performance plan to address critical federal performance and management issues, including redundancy and other inefficiencies in how we do business. It could also provide a framework for any restructuring efforts.[Footnote 87] This provision has not been fully implemented, however. OMB issued the first and only such plan in February 1998 for fiscal year 1999. Further, as the focal point for overall management in the executive branch, OMB could provide guidance and management and reporting tools to increase federal agencies' focus on efficiency improvements. OMB's main vehicle for providing guidance on the development of agency strategic plans and performance plans and reports, OMB Circular A-11, Section 6 (Preparation and Submission of Strategic Plans, Annual Performance Plans, and Annual Program Performance Reports), makes no reference to establishing long-term goals for efficiency gains or describing strategies for how performance outcomes can be achieved more efficiently. References to efficiency in the guidance primarily pertain to the inclusion of program-level efficiency measures in agency budget justifications. OMB could also support mechanisms to share information and encourage agency efforts to improve efficiency. OMB has previously developed or contributed to mechanisms for sharing information and encouraging improvements to federal programs in the past, such as Web sites to share information, highlight success, and identify best practices for initiatives.[Footnote 88] For example, www.results.gov had information on best practices related to PMA initiatives, and www.expectmore.gov provided information on PART assessments and improvement plans. OMB's own Web site contained information and examples of what it considered to be high-quality PART performance measures; discussion papers on measurement topics, such as how to effectively measure what a program is trying to prevent; and strategies to address some of the challenges of measuring the results of research and development programs. OMB recently launched a collaborative wiki page which is intended to provide an online forum for federal managers to share lessons learned and leading practices for using performance information to drive decisionmaking.[Footnote 89] OMB has sponsored various management councils, such as the President's Management Council and the Performance Improvement Council, which include representatives of agencies and serve as forums for information sharing among agencies and with OMB. We have also reported that OMB has hosted standing working groups and committees comprised of agency and OMB staff, and has hosted workshops to address important issues and identify and share best practices. For example, OMB helped form a subgroup among agency officials responsible for the PMA budget and performance integration initiative to share lessons learned and discuss strategies to address challenges of developing efficiency measures in the grant context. Conclusions: The prior Administration's approach to improving efficiency under PMA and PART focused on measuring and achieving efficiency gains at the program level. The approach involved requiring each program to develop at least one efficiency measure and demonstrate annual gains in efficiency, as well as to have regular procedures in place for achieving improvements in efficiencies. Although most programs that received a PART assessment developed an efficiency measure, not all of these measures included both elements of a typical efficiency measure- -an input as well as an output or outcome. The absence of these typical elements can result in measures that do not truly capture efficiency. Nevertheless, other forms of measures intended to improve efficiency, such as those focused on reducing costly error rates, could still provide useful information. Officials for some selected programs we reviewed indicated that the efficiency measures reported for PART were useful and described ways in which they used data for efficiency measures, such as to evaluate proposals from field units, lower the cost of a contract, or make decisions to shift production. Other officials we interviewed did not find the measures useful for decision making. Officials for all of the programs described challenges to developing and using efficiency measures that were similar to challenges we previously reported on in prior work on PART and performance measures in general. For example, in one case the way OMB defined the program boundaries did not line up well with how managers ran the activities, which resulted in measures that were not useful for decision making. Some program officials indicated it was not always feasible to meet the requirement to demonstrate annual gains in efficiency, given that improvement could take multiple years to achieve. Some officials cited inconsistencies and limitations in the guidance and technical support from OMB on how to develop and use efficiency measures. OMB has not clarified whether programs should continue to collect and use efficiency measure data established for PART. Such clarification is necessary to help guide any refinements, as needed, to the current process, as well as broader issues. While tracking efficiency at the program level can be useful, this approach can miss opportunities to seek efficiencies on a larger scale, such as efforts that cross traditional program and agency boundaries. The experiences of private and public sector entities in implementing strategic and crosscutting approaches to improving efficiency can provide insights for federal agencies. For example, process improvement and modernization of systems can be undertaken both within and across organizational boundaries to increase quality, reduce waste, and lower costs. Analyzing spending and procurement strategies to leverage buying power and improve performance can identify opportunities to reduce the cost of producing agency outputs and outcomes. Broader, governmentwide reviews and analysis of restructuring opportunities that involve a wider scope of government activity can be used to identify strategic, crosscutting approaches to improving efficiency that emphasize the need to maintain or improve other key dimensions of performance. Such approaches have the potential to yield significant gains in efficiency that would be difficult to achieve by individual programs working in isolation. The current Administration has begun to identify some important opportunities for crosscutting efficiencies in its proposed information technology initiatives and procurement reforms and has tasked agencies with establishing agency cost reduction goals and asked federal employees to submit their suggestions for cost savings. Efforts to improve efficiency can take multiple years to accomplish and can require changes in strategy and collaboration within and across organizational lines. Furthermore, efficiency can only be improved if other performance dimensions, such as the quality or quantity of agency outputs and outcomes, are maintained or improved as resources are reduced; or conversely, if quality and quantity of outputs/outcomes are improved with a given level of resources. The Administration has signaled its intent to make greater use of program evaluation to determine which programs are producing desired results. Program evaluations can also be used to determine the cost of achieving outcomes, an approach that could aid in identifying the most cost-effective program designs. Continuing to build on the experiences and lessons learned from prior initiatives, with a concerted focus on specific levels of governments-- governmentwide, agency, and program--could help to identify, introduce, and sustain additional efficiency gains on a more systematic and systemic basis at these same levels. The planning and reporting requirements of GPRA could serve as a framework for developing agency or across-agency strategies for improving efficiency and tracking results. By implementing the governmentwide performance plan provision of GPRA, OMB could provide further impetus to identifying efficiency goals to be achieved by consolidating operations or restructuring programs on a governmentwide basis. Further, OMB's A-11 guidance on preparing agency strategic and performance plans could place greater emphasis on improvements in efficiency. OMB has multiple management groups and information-sharing mechanisms, including a new wiki, which could be used to identify and share successful approaches to improving efficiency, whether applied at the program or other levels of government. Recommendations for Executive Action: We recommend that the Director of OMB take the following four actions: * Evolve toward a broader approach that emphasizes identifying and pursuing strategies and opportunities to improve efficiency at each of the governmentwide, agency, and program levels. - At the governmentwide level, OMB should look for additional opportunities to consolidate or restructure duplicative or inefficient operations that cut across agency lines. One vehicle for doing this is the GPRA-required governmentwide performance plan. - At the agency level, OMB should clarify its A-11 guidance to agencies on establishing efficiency goals and strategies in their agency-level GPRA strategic and performance plans, and reporting on the results achieved in performance reports. Guidance should stress the importance of looking for efficiencies across as well as within components and programs and maintaining or improving key dimensions of performance such as effectiveness, quality, or customer satisfaction, while also striving for efficiency gains. - At the program level, OMB should clarify whether agencies are to continue developing and using program-level efficiency measures. If so, OMB should provide enhanced guidance and technical support to agencies that addresses how to develop and use efficiency measures to improve efficiency and mitigate the challenges we identified. * Collect and disseminate information on strategies and lessons learned from successful efforts to improve efficiency by federal agencies, other governments, and the private sector. Possible vehicles for collection and dissemination of this information include good practices guides, workshops, Web sites, wikis, and management councils, such as the President's Management Council and the Performance Improvement Council. Agency Comments: We provided a draft of this report for review to OMB and the Departments of Agriculture, Education, the Interior, Labor, and Transportation. In oral comments, OMB representatives indicated that OMB concurred with our recommendations, adding that they thought the report will be useful as they revise their guidance to agencies on how to address efficiency improvements. OMB also provided technical comments which we incorporated where appropriate. In their written comments (see appendix IV), Interior also concurred with our recommendations, but urged caution with regard to the recommendation that OMB provide additional guidance on the use of efficiency measures by agencies and programs. In particular, Interior cautioned against inviting standardized direction that would have agencies comparing efficiency across and within programs, considering the inherent differences in scope, complexity, and quality of outputs and outcomes. Interior indicated it seeks maximum flexibility for federal managers in using efficiency measures when they make sense and can be used to drive to the desired goals for the program. The Departments of Education and Labor provided technical comments, which we incorporated where appropriate. The Departments of Agriculture and Transportation did not provide comments. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution of it until 30 days from the date of this letter. At that time, we will send copies of this report to the appropriate congressional committees; the Secretaries of Agriculture, Education, the Interior, Labor, and Transportation; the Director of OMB; and other interested parties. The report will also be available at no charge on the GAO Web site at [hyperlink, http://www.gao.gov]. If you or your staff have any questions regarding this report, please contact me at (202) 512-6543 or steinhardtb@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V. Signed by: Bernice Steinhardt: Director, Strategic Issues: [End of section] Appendix I: Objectives, Scope, and Methodology: The objectives of our review were to examine: (1) the types of efficiency measures reported through the Program Assessment Rating Tool (PART) for agency programs overall, and particularly for selected programs in five selected agencies, focusing on the extent to which they included typical elements of an efficiency measure; (2) for selected programs, the extent to which programs reporting efficiency measures through PART have shown efficiency gains and how programs have used efficiency measures for decision making; (3) for selected programs, the types of challenges to developing and using efficiency measures they have faced; and (4) other strategies that can be used to improve efficiency. To address these objectives, we selected five departments from those on which we had reported in 2007 concerning implementation of a managerial cost accounting system (MCA).[Footnote 90] Because we wanted to include agencies with variety in the types of cost data available, we selected some departments that had--and some that had not--developed an MCA system. The Departments of the Interior, Labor and Transportation were selected because these were the only departments out of the 10 agencies we reviewed at the time that had implemented--or had made significant progress in implementing--MCA departmentwide.[Footnote 91] To compare and contrast findings from these departments, we selected two other departments that had not implemented an MCA system. The United States Department of Agriculture was selected because the department indicated in our 2007 report that it planned to implement an MCA system the next time it upgraded its financial management system. The Department of Education was selected because it indicated it had no plans to implement an entitywide MCA system. After choosing the departments, we selected 21 programs to review from the set of all programs that had a received a PART assessment by the Office of Management and Budget (OMB).[Footnote 92] PART was developed to assess and improve program performance so that the federal government could achieve better results. According to OMB, a PART review helped identify a program's strengths and weaknesses to inform funding and management decisions aimed at making the program more effective. A PART review included program-level performance information and efficiency measures for the programs.[Footnote 93] The PART data we received from OMB contained 1,396 efficiency measures which were associated with 937 programs that received a PART assessment. Within the five departments, we selected the 21 specific programs for review to represent a diverse array of functions and operations within the federal government, as indicated by the PART program type.[Footnote 94] Of the seven PART program types, we selected five for inclusion in this study, excluding research and development and credit.[Footnote 95] Additional criteria were that the selected programs have relatively large fiscal year 2008 funding levels, and variety in the number of efficiency measures associated with the programs. For the first objective regarding the extent to which efficiency measures included typical necessary elements, we first identified the elements and developed a definition by conducting a literature review as well as expert interviews. We then performed various degrees of analysis on (1) all efficiency measures for all programs represented in the PART database, (2) all of the measures for our selected programs, and (3) a random sample of 100 efficiency measures taken from the PART database. The following describes the analysis we conducted on each of these three populations: * Analysis on the complete PART database: The analysis we conducted on all PART efficiency measures resulted in a set of summary statistics, such as the fiscal year 2008 total funding by PART program type, the mean amount of funding each program received within the program types, the number of programs for each PART program type, the number of programs that had between zero and eight efficiency measures, and the number of programs in each selected department by PART program type. * Analysis of PART measures selected with certainty from 21 programs in five departments: For the 21 programs we selected, we conducted a more detailed analysis on the 36 associated efficiency measures. [Footnote 96] However, any findings based on this analysis cannot be generalized beyond these particular measures. We performed a content analysis review of these measures, which was based upon the PART efficiency measure data; our review of applicable documents concerning the measures and programs, such as the programs' PART assessments; and interviewing program officials to discuss the measures and programs. For each of these measures, we identified whether certain attributes were present, and the documents we reviewed and interviews we conducted aided in this effort at times. The fields from the PART database we used to assess each efficiency measure were the agency and program name, the text for each efficiency measure and, when present, the more detailed efficiency measure explanation. Using this information, we determined whether each of the measures included the program's inputs (such as cost or hours worked by employees) as well as its outputs or outcomes. When we identified a measure as having an output or outcome element, we distinguished between the two. We also analyzed whether there was either a time or cost attribute to each measure. For each of these attributes, the potential answers were "Yes," "No," or "Unclear."[Footnote 97] To determine whether an efficiency measure had these attributes, we defined each term for this particular exercise. We defined an input as a resource, such as cost or employee time, used to produce outputs or outcomes. We defined outputs as the amount of products and services delivered by a program. We defined outcomes as the desired results of a program, such as events, occurrences or changes in conditions, behaviors or attitudes. We defined a measure to have an attribute of time or cost when the measure appeared to include some type of attribute of time (e.g., "hours worked by employees," "per month," "annually," or "within three months,") or cost, respectively. We conducted our coding by having three team members independently code each of the 36 efficiency measures without each knowing how the other two coders assessed each measure. Afterward, the three coders discussed and reconciled any differences and reached agreement in all incidents. Finally, we determined whether the cost element was based on budgetary information or MCA information. * Analysis of a random sample from the PART database: This analysis involved selecting a random sample of 100 efficiency measures from the remaining 1,355 efficiency measures in the PART database.[Footnote 98] Estimates based on the sample can be generalized to estimate characteristics of the remaining population of 1,355 efficiency measures. Because we followed a probability procedure based on random selections, our sample is only one of a large number of samples that we might have drawn. Since each sample could have provided different estimates, we express our confidence in the precision of our particular sample's results as a 95 percent confidence interval (e.g., plus or minus 10 percentage points). This is the interval that would contain the actual population value for 95 percent of the samples we could have drawn. As a result, we are 95 percent confident that each of the confidence intervals in this report will include the true values in the study population. Unless otherwise noted, all percentage estimates have 95 percent confidence intervals of within plus or minus 10 percentage points of the estimate itself. The analysis we conducted on these measures was similar to the analysis we conducted for the selected programs, meaning we analyzed and determined if each measure had an input, output or outcome, time or cost attribute and used the same definition and coding procedures. However, because we did not have in-depth information from interviews or program documents concerning these measures, in some cases we were unable to conclude whether certain efficiency measures included necessary elements and consequently, classified about a quarter of the sample as unclear. Also, because of the lack of detailed information on the measures, we could not distinguish between outputs and outcomes expressed for these measures. Simultaneously with the content analysis of the efficiency measures, for the second and third objectives, on how selected agencies/programs used efficiency measures and the extent to which they reported efficiency gains, and what challenges or constraints to developing and using efficiency measures they faced, we reviewed program Web sites, PART assessments, other documents provided by program officials, and interviewed program officials identified by the departments as knowledgeable about the particular program and its efficiency measure(s). These interviews consisted of asking agency officials a similar set of questions with topics such as how the efficiency measure(s) was developed and used, associated challenges, and alternative methods for evaluating efficiency. For the two programs that did not have any efficiency measures in PART, we asked questions such as whether they had other efficiency-related measures they tracked internally which were unrelated to PART, whether there had been prior attempts to develop an efficiency measure, and whether they had experienced specific challenges to developing and using efficiency measures. In addition to interviewing program officials, we also interviewed at least one official in each of the five departments who was responsible for performance measurement at the departmentwide level. These interviews also had a similar set of questions and were specific to departmentwide performance measurement issues, such as whether the department had its own guidelines or guidance pertaining to developing and using efficiency measures, how results for program- level efficiency measures get reported within the agency, and how program efficiency measures were used. Also at the department level, we interviewed officials associated with each of the five departments' Chief Financial Officer (CFO) offices, asking questions about the role the CFOs office played, if any, in developing efficiency measures for programs and inquiring about the development and use of a managerial cost accounting system. In addition to interviewing department and program officials, we interviewed OMB officials on several occasions about the approach to efficiency under PART and discussed, among other topics, the training and guidance OMB provided, and any lessons learned from the agencies' efforts to develop and use efficiency measures. OMB also provided us with documents detailing the history of the PART program. Finally, to determine whether a selected program's efficiency measure indicated a gain or loss, we reviewed the efficiency measure data that were reported in the program's PART assessment and subtracted the initial year of data from the latest year available. To verify the accuracy of the data, we asked program officials to confirm the data and when necessary, to provide us with the most recent data. To address the fourth objective regarding the approaches agencies can employ to improve efficiency, we interviewed program officials for the selected programs to learn about the approaches they use to evaluate efficiency and also conducted a two-stage literature review to determine alternative approaches. The first stage of the literature review consisted of examining GAO publications, Congressional Research Service reports, the Internet, and various databases for general information on strategic approaches to efficiency. We also participated in a business process management research report with the American Productivity and Quality Center (APQC),[Footnote 99] studying how organizations maintain quality across processes and products as well as meet customer requirements in the face of pressure to cut costs. Using information derived from the first literature review and the APQC report, we identified the broad set of approaches to improving efficiency. In our literature search, we looked for examples and ideas that used a broad array of strategies to seek improvements or affect efficiency from prior reports we have published and what other institutions that have done work on the subject. For this objective, we refer to 18 different pieces of literature from our comprehensive literature search. In conducting the literature review, we did not attempt to identify all potential alternative approaches that could lead to efficiency improvements but focused on approaches that appeared consistent with the broad definition of efficiency improvement that was used in this report. Furthermore, in addition to the interviews with program officials and the literature review, we interviewed experts on performance and efficiency measures, who discussed definitions, uses, and insights of efficiency measures. Among the experts, we interviewed officials in the United Kingdom's National Audit Office, which assessed the reliability of the efficiency gains reported by United Kingdom agencies as part of the United Kingdom's 2004 government-wide efficiency review. We also interviewed officials with the Office of the Auditor General of Canada, which is conducting a study on ways to improve the efficiency of that country's tax administration system. [End of section] Appendix IIP: Departments, Selected Program Assessment Rating Tool Program (PART) Names, and Summary of Programs: Department and PART program name: Department of Agriculture:Forest Service: Watershed; Program summary: Restore, enhance, and maintain watershed conditions including soil, water, air, and forest and rangeland vegetation within the national forests and grasslands. Management of these physical and biological resources provides a foundation for healthy, viable ecosystems. Department and PART program name: Department of Agriculture:National School Lunch Program; Program summary: Provides nutritionally balanced, low-cost or free lunches for public and nonprofit private schools. The program seeks to safeguard the health and well-being of the nation's children and support domestic agricultural production. Department and PART program name: Department of Agriculture:Plant and Animal Health Monitoring Programs; Program summary: Assists in protecting plant and animal resources from pests and diseases through ongoing monitoring and surveillance. Provides rapid detection, analysis, and reporting of pests and diseases to minimize potential losses. Department and PART program name: Department of Education: 21st Century Community Learning Centers; Program summary: Awards formula grants to state education agencies which, in turn, manage statewide competitions and award subgrants to local education agencies and community-based organizations. These grants support the creation of community learning centers that provide academic enrichment opportunities during nonschool hours for children, particularly students who attend high-poverty and low-performing schools. This program focuses on enrichment in core academic subjects, extracurricular enrichment, as well as literacy and other educational services to the families of participating children. Department and PART program name: Department of Education: Smaller Learning Communities; Program summary: Provides competitive grants to local education agencies to increase academic achievement in large high schools through the creation of smaller, more personalized learning environments. Department and PART program name: Department of Education: Student Aid Administration; Program summary: Provides financial assistance to postsecondary students and their families through administering federal student aid grants and loans. Department and PART program name: Department of the Interior: Bureau of Reclamation Water Management--Operation and Maintenance; Program summary: Ensures the operation and maintenance of reclamation facilities, delivers water to irrigators and municipal users, and provides storage to help mitigate flooding. The program also addresses issues such as water conservation, runoff from irrigated fields, and project financial management. Department and PART program name: Department of the Interior: Wildland Fire Management; Program summary: Manages and extinguishes fires on Department of the Interior lands and on other lands under fire protection agreements. The three largest program activities are fire preparedness, fire suppression, and hazardous fuels reduction (i.e., removal of small trees and brush that exacerbate fire risks). Department and PART program name: Department of the Interior: Fish and Wildlife Service--Endangered Species; Program summary: Protects threatened or endangered species and conserves their habitats. Lists species needing protection, consults on federal projects, awards grants, and works with partners on recovery actions. Department and PART program name: Department of the Interior: Fish and Wildlife Service--Fisheries; Program summary: Works to conserve and restore native aquatic species populations and their habitat and support recreational fishing. Department and PART program name: Department of the Interior: Office of Surface Mining--State Managed Abandoned Coal Mine Land Reclamation; Program summary: Reclaims and restores land and water degraded by coal mining activities conducted before 1977. Reclamation fees on current coal production fund the program, which has expanded to provide oversight over the 23 states and three Indian Tribes that carry out the program. Department and PART program name: Department of Labor: Energy Employees Occupational Illness Compensation Program; Program summary: Serves those who have contracted illness due to exposure to toxic substances or radiation while working at nuclear weapons and related covered facilities. Provides lump-sum compensation and health benefits to eligible Department of Energy nuclear weapons workers, or the survivors of such workers. Department and PART program name: Department of Labor: Job Corps; Program summary: Provides intensive education and training services to disadvantaged youth ages 16-24. These services are intended to help eligible youth obtain jobs, seek further education, or enter the military. The program serves approximately 60,000 youth nationwide through 122 centers, most of which are residential. Department and PART program name: Department of Labor: Occupational Safety and Health Administration; Program summary: Works to ensure, for every working person in the nation, safe and healthful working conditions. Implements the Occupational Safety and Health Act of 1970 by setting and enforcing standards, outreach and education, cooperative programs and compliance assistance. Department and PART program name: Department of Labor: Unemployment Insurance Administration State Grants; Program summary: Assists states in operating their unemployment insurance programs, which provide temporary income support to unemployed workers. States determine eligibility for benefits, which are financed through state-levied taxes. The Department of Labor funds the administrative expenses of these state programs. Department and PART program name: Department of Labor: Workforce Investment Act--Migrant and Seasonal Farmworkers; Program summary: Provides competitive grants to fund training, employment, and other services to help economically disadvantaged farmworkers and their families. Through these services, the program seeks to help them achieve economic self-sufficiency by strengthening their ability to gain stable employment. Department and PART program name: Department of Transportation: Federal Aviation Administration (FAA) Air Traffic Organization-- Terminal Programs; Program summary: Provides air traffic control services to guide aircraft in and out of airports across the country. Department and PART program name: Department of Transportation: FAA Air Traffic Organization--Technical Operations; Program summary: Maintains and modernizes equipment needed in the national airspace system to deliver air traffic services. It fields, repairs, and maintains a network of complex equipment, including radars, instrument landing systems, radio beacons, runway lighting, and computer systems. Department and PART program name: Department of Transportation: Federal Transit Administration New Starts; Program summary: Provides financial support for locally planned and operated public transit through competitive, discretionary capital investment grant transit projects including commuter rail, light rail, heavy rail, bus rapid transit, trolleys and ferries. Department and PART program name: Department of Transportation: Highway Infrastructure; Program summary: Provides financial grants and technical assistance to states to construct, maintain, and improve the performance of the nation's highway system in accordance with federal policy goals. Department and PART program name: Department of Transportation: National Highway Traffic Safety Administration--Operations and Research; Program summary: Advances highway safety through research and regulations concerning vehicle technologies and human behavior. Focuses on researching vehicle and behavioral safety countermeasures, issuing vehicle safety regulations, and investigating vehicle defects. Source: GAO analysis of selected PART assessments. [End of table] [End of section] Appendix III: Department, PART Program Name, and Number of Efficiency Measures, Fiscal Year 2009 Funding Level, PART Program Type, and Efficiency Measure(s) for Selected Programs: Department: Agriculture; PART program name and number of efficiency measures: Forest Service: Watershed (0); Fiscal year 2009 funding level (dollars in millions): $812; PART program type: Direct federal; Efficiency measure: None. Department: Agriculture; PART program name and number of efficiency measures: National School Lunch Program (NSLP) (3); Fiscal year 2009 funding level (dollars in millions): $8,517; PART program type: Block/formula grant; Efficiency measure: Dollars lost to error in the National School Lunch Program. Efficiency measure: Rate of verified applications not supported by adequate income documentation. Efficiency measure: Rate of administrative error in NSLP eligibility determination. Department: Agriculture; PART program name and number of efficiency measures: Plant and Animal Health Monitoring Programs (2); Fiscal year 2009 funding level (dollars in millions): $330; PART program type: Regulatory; Efficiency measure: Value of damage prevented or mitigated by the monitoring and surveillance programs per dollar spent. Efficiency measure: Improved efficiency through the use of targeted samplings versus the use of random sampling. Department: Education; PART program name and number of efficiency measures: 21st Century Community Learning Centers (3); Fiscal year 2009 funding level (dollars in millions): $1,081; PART program type: Block/formula grant; Efficiency measure: The average number of days it takes the department to submit the final monitoring report to a State Education Agency (SEA) after the conclusion of a site visit. Efficiency measure: The average number of weeks a state takes to resolve compliance findings in a monitoring visit report. Efficiency measure: The percentage of SEAs that submit complete data on 21st century program performance measures by the deadline. Department: Education; PART program name and number of efficiency measures: Smaller Learning Communities (6); Fiscal year 2009 funding level (dollars in millions): $80; PART program type: Competitive grant; Efficiency measure: Fiscal year 2003 cohort: Cost (in dollars) per student demonstrating proficiency or advanced skills in reading. Efficiency measure: Fiscal year 2003 cohort: Cost (in dollars) per student demonstrating proficiency or advanced skills in mathematics. Efficiency measure: Fiscal year 2004 cohort: Cost (in dollars) per student demonstrating proficiency or advanced skills in reading. Efficiency measure: Fiscal year 2004 cohort: Cost (in dollars) per student demonstrating proficiency or advanced skills in mathematics. Efficiency measure: Fiscal year 2005 cohort: Cost (in dollars) per student demonstrating proficiency or advanced skills in reading. Efficiency measure: Fiscal year 2005 cohort: Cost (in dollars) per student demonstrating proficiency or advanced skills in mathematics. Department: Education; PART program name and number of efficiency measures: Student Aid Administration (1); Fiscal year 2009 funding level (dollars in millions): $753; PART program type: Capital and service acquisition; Efficiency measure: Direct administrative unit costs for origination and disbursement of student aid. Department: Interior; PART program name and number of efficiency measures: Bureau of Reclamation Water Management--Operation and Maintenance (1); Fiscal year 2009 funding level (dollars in millions): $308; PART program type: Capital and service acquisition; Efficiency measure: Average time to correct/mitigate higher priority operations and maintenance deficiencies of reserved works facilities. Department: Interior; PART program name and number of efficiency measures: Fish and Wildlife Service--Endangered Species (0); Fiscal year 2009 funding level (dollars in millions): $277; PART program type: Regulatory; Efficiency measure: None. Department: Interior; PART program name and number of efficiency measures: Fish and Wildlife Service--Fisheries (1); Fiscal year 2009 funding level (dollars in millions): $126; PART program type: Competitive grant; Efficiency measure: Pounds/dollar of healthy rainbow trout produced for recreation. Department: Interior; PART program name and number of efficiency measures: Office of Surface Mining--State Managed Abandoned Coal Mine Land Reclamation (2); Fiscal year 2009 funding level (dollars in millions): $477; PART program type: Block/formula grant; Efficiency measure: Percentage of declared emergencies abated within 6 months. Efficiency measure: Provide appropriate grant funding within 60 days of a complete grant application. Department: Interior; PART program name and number of efficiency measures: Wildland Fire Management (3); Fiscal year 2009 funding level (dollars in millions): $859; PART program type: Direct federal; Efficiency measure: Number of acres treated in the wildland-urban interface per million dollars of gross investment. Efficiency measure: Number of acres treated outside the wildland-urban interface per million dollars gross investment. Efficiency measure: Number of acres in fire regimes 1, 2, or 3 moved to a better condition class per million dollars of gross investment. Department: Labor; PART program name and number of efficiency measures: Energy Employees Occupational Illness Compensation Program (1); Fiscal year 2009 funding level (dollars in millions): $1,161; PART program type: Direct federal; Efficiency measure: Average number of decisions per full-time equivalent. Department: Labor; PART program name and number of efficiency measures: Job Corps (1); Fiscal year 2009 funding level (dollars in millions): $1,611; PART program type: Capital and service acquisition; Efficiency measure: Cost per participant. Department: Labor; PART program name and number of efficiency measures: Occupational Safety and Health Administration (1); Fiscal year 2009 funding level (dollars in millions): $503; PART program type: Regulatory; Efficiency measure: Inspections per Compliance Safety and Health Officer. Department: Labor; PART program name and number of efficiency measures: Unemployment Insurance Administration State Grants (1); Fiscal year 2009 funding level (dollars in millions): $3,498; PART program type: Block/formula grant; Efficiency measure: Number of timely and accurate initial benefit payments claims per $1,000 of inflation-adjusted base grant funds. Department: Labor; PART program name and number of efficiency measures: Workforce Investment Act--Migrant and Seasonal Farmworkers (1); Fiscal year 2009 funding level (dollars in millions): $83; PART program type: Competitive grant; Efficiency measure: Cost per participant. Department: Transportation; PART program name and number of efficiency measures: Federal Aviation Administration (FAA) Air Traffic Organization--Technical Operations (2); Fiscal year 2009 funding level (dollars in millions): $2,650; PART program type: Direct federal; Efficiency measure: ATO-Technical Operations staffing ratio. Efficiency measure: Unit cost for providing ATO-Technical Operations services. Department: Transportation; PART program name and number of efficiency measures: FAA Air Traffic Organization--Terminal Programs (2)[A]; Fiscal year 2009 funding level (dollars in millions): $2,199; PART program type: Direct federal; Efficiency measure: Unit cost for providing terminal services. Efficiency measure: Productivity rate at service delivery points. Department: Transportation; PART program name and number of efficiency measures: Federal Transit Administration New Starts (1); Fiscal year 2009 funding level (dollars in millions): $1,569; PART program type: Competitive grant; Efficiency measure: Percent of projects under full funding grant agreements that have current total cost estimates that do not exceed baseline cost by more than 5 percent. Department: Transportation; PART program name and number of efficiency measures: Highway Infrastructure (3); Fiscal year 2009 funding level (dollars in millions): $41,325; PART program type: Block/formula grant; Efficiency measure: Percent of major federally funded transportation infrastructure projects with less than 2 percent annual growth in the project completion milestone. Efficiency measure: Median time to complete an Environmental Impact Statement. Efficiency measure: Percent of major federally funded transportation infrastructure projects with less than 2 percent annual growth in cost estimates. Department: Transportation; PART program name and number of efficiency measures: National Highway Traffic Safety Administration--Operations and Research (1); Fiscal year 2009 funding level (dollars in millions): $232; PART program type: Regulatory; Efficiency measure: Average costs incurred to complete a defect investigation. Source: GAO analysis of OMB's Program Assessment Rating Tool. [A] During the course of our review, FAA Air Traffic Organization-- Terminal Programs, changed the status of one of its PART measures (ATO Terminal staffing ratio) from an "output" measure to an "efficiency" measure. Therefore, we did not include this measure in our review. [End of table] [End of section] Appendix IV: Comments from the Department of the Interior: United States Department of the Interior: Office Of The Secretary: Washington, DC 20240: April 20, 2011: Ms. Bernice Steinhardt: Director, Strategic Issues: U.S. Government Accountability Office: 441 G Street, N.W. Washington, D.C. 20548: Dear Ms. Steinhardt: Thank you for providing the Department of the Interior the opportunity to review and comment on the draft Government Accountability Office Report entitled, Streamlining Government: Opportunities Exist to Strengthen OMB's Approach to Improving Efficiency, (GA0-10-394). The Department does concur with many aspects of the recommendations made in the report. First, we agree that OMB is in a position to determine government-wide opportunities for efficiency that involve and could benefit multiple agencies. Using input from agencies on what has worked and how that can be applied across the Federal community would be value added. The councils and work groups that OMB hosts are very useful in that regard and we are finding the PIC to be a source of good guidance and best practices. With regard to additional guidance and direction on the use of efficiency measures in agencies and programs, we would urge caution. GAO observed that efficiency measurements should be balanced with considerations of quality, outcomes, and other factors such as customer satisfaction. Rather than ask for additional guidance on efficiency measures that may not be sufficiently focused on outcomes, we believe that OMB's initiative to strengthen and infuse a program evaluation capability in Federal agencies is just what is needed. We caution inviting standardized direction that would have us comparing efficiency across and within programs, considering the inherent differences in scope, complexity, and quality of outputs and outcomes. Rather we seek maximum flexibility for Federal managers in using efficiency measures when they make sense and can be used to drive to the desired goals for the program. As demonstrated by your study, not all programs have the same capacity for improved efficiency and some are more challenging to evaluate. We appreciate having the opportunity to comment. If you have questions or need additional information, please contact Dr. Richard Beck, Director, Office of Planning and Performance Management, at (202) 208- 1818. Sincerely, Signed by: Rhea Suh: Assistant Secretary: Policy, Management and Budget: [End of section] Appendix V: GAO Contact and Staff Acknowledgments: GAO Contact: Bernice Steinhardt, (202) 512-6543 or steinhardtb@gao.gov: Staff Acknowledgments: In addition to the individual named above, Elizabeth Curda, Assistant Director; Charlesetta Bailey; James Cook; Anne Inserra; Eric Knudson; Ricardo Sanchez; and Jeremy Williams made key contributions to the report. Cynthia Grant; Peter Grinnell; Carol Henn; Donna Miller; A.J. Stephens; Jay Smale; Jessica Thomsen; and John Warner also provided significant assistance. [End of section] Footnotes: [1] OMB, Memorandum for the Heads of Departments and Agencies (M-09- 20) on Planning for the President's Fiscal Year 2011 Budget and Performance Plans (Washington, D.C.: June 11, 2009). [2] According to OMB, federal employees submitted over 38,000 ideas to the President's SAVE (Securing Americans Value and Efficiency) Award contest, which was launched in September 2009. The winner's idea is supposed to be included in the 2011budget, and the employee who submitted it will be invited to meet the President. [3] Executive Order 13538, Establishing the President's Management Advisory Board, 75 Fed. Reg. 20,895 (April 19, 2010). [4] Pub. L. No. 103-62, 107 Stat. 285 (Aug. 3, 1993). [5] In addition to GPRA, executive agencies are subject to other general requirements related to efficiency. For example, agencies are required to implement and maintain systems of internal controls which are, in part, to assure effective and efficient operations. 31 U.S.C. § 3512(c); GAO, Standards for Internal Control in the Federal Government, [hyperlink, http://www.gao.gov/products/AIMD-00-21.3.1] (Washington, D.C.: November 1999). In addition, federal agencies must develop and maintain accounting and financial management systems that, consistent with OMB policies, provide for the systematic measurement of agency performance, among other things. 31 U.S.C. §§ 503(b), 902(a)(3)(D)(iv). [6] The PMA, which was first announced in 2001, consisted of five governmentwide management priorities, including budget and performance integration, strategic management of human capital, expanded electronic government, improved financial performance, and competitive sourcing. [7] OMB described PART, which was created in 2002, as a diagnostic tool meant to provide a consistent approach to evaluating federal programs as part of the executive budget formulation process. [8] See, for example, Harry P. Hatry, Performance Measurement: Getting Results, Second Edition (Baltimore, MD: The Urban Institute Press, 2007). [9] See, for example, GAO, Tax Administration: IRS Can Improve Its Productivity Measures by Using Alternative Methods, [hyperlink, http://www.gao.gov/products/GAO-05-671] (Washington, D.C.: July 11, 2005). [10] In addition to these 36 efficiency measures, there were a total of five additional efficiency measures included in the PART data we received from OMB for three of our selected programs. However, officials from each of these programs told us these five efficiency measures were no longer associated with PART, so we excluded them from our analysis. Further, one of the selected programs from the Department of Transportation, the Federal Aviation Administration (FAA) Air Traffic Organization (ATO)--Terminal Programs, changed the status of one of its PART measures (ATO Terminal Staffing Ratio) from an "output" measure to an "efficiency" measure. We did not include this measure in our review of efficiency measures for the selected programs. [11] Percentage estimates based on this sample have 95 percent confidence intervals of within +/-10 percentage points of the estimate itself, unless otherwise noted. See Appendix I for more information on sampling methodology. [12] GAO, Managerial Cost Accounting Practices: Implementation and Use Vary Widely across 10 Federal Agencies, [hyperlink, http://www.gao.gov/products/GAO-07-679] (Washington, D.C.: July 20, 2007). [13] PART classified programs as one of seven types: direct federal, competitive grant, block/formula grant, research and development, capital assets and acquisition, credit, and regulatory. We excluded research and development programs from our sample of selected programs based on the findings of a 2008 study by The National Academies which raised questions about the feasibility of developing valid outcome- based efficiency measures for federal research programs (Evaluating Research Efficiency in the U.S. Environmental Protection Agency, Committee on Evaluating the Efficiency of Research and Development Programs at the U.S. Environmental Protection Agency, The National Academies). We excluded credit programs from our sample of selected programs because of the relatively small number of these programs in the selected departments. [14] Fiscal year 2009 funding for the selected programs ranged from approximately $80 million to over $41 billion. [15] Work on the engagement was originally started in October 2006, but subsequently suspended before resuming in September 2008. [16] GPRA, §§ 2(a)(1), 2(b)(5). [17] GAO, Results-Oriented Government: GPRA Has Established a Solid Foundation for Achieving Greater Results, [hyperlink, http://www.gao.gov/products/GAO-04-38] (Washington, D.C.: Mar. 10, 2004). [18] OMB's PMA standards included references to additional approaches to improving efficiency, such as competitive sourcing and business process reengineering for commercial services management, developing business cases for major systems investments, and using earned value management to plan, execute, and manage major information technology (IT) investments. [19] Additional references to efficiency in OMB's PART assessment tool included language in the section on program purpose and design, which asked if the program design was free of flaws that would limit efficiency, with a requirement for "there — to be no strong evidence" that another approach or mechanism would be more efficient. For capital assets and service acquisition programs, PART questions (in the strategic planning section) included assessing whether credible analysis of alternatives had been conducted, to determine whether the agency was investing in something that provided the best value to the government. For regulatory programs, there was a specific question in the program results section asking whether the goals were achieved at the least incremental societal cost and whether the program maximized net benefits, to determine whether the program met its goals in the most efficient way possible. [20] Two of our selected programs--U.S. Department of Agriculture's Forest Service Watershed and Department of the Interior's Endangered Species--did not have any efficiency measures in PART, but officials from both of these programs told us they had proposed efficiency measures to OMB that had been rejected, and that they were developing new efficiency measures and had been in consultation with OMB seeking approval. [21] This sample enables us to generalize our analysis to the remaining efficiency measures for PART. These percentage estimates have 95 percent confidence intervals of within +/-10 percentage points of the estimate itself. Appendix I contains additional information on the sampling methodology. [22] We characterized a measure as "unclear" when it was ambiguous as to whether or not both elements (input plus output or outcome) were present, based on our analysis of how the measure was written and the accompanying explanation. [23] We did not consider "average time" as expressed in this measure to be an input because it tracked the number of calendar years that have passed, not the amount of work hours needed to correct/mitigate higher priority operations and maintenance deficiencies (which are outputs). [24] Full-time equivalent employment is the basic measure of levels of employment used in the budget. It is the total number of hours worked divided by the total number of compensable hours in a fiscal year. For example, in fiscal year 2009 an FTE represented 2,088 hours (8 hours per day for 261 days). [25] GAO, Performance Budgeting: Efforts to Restructure Budgets to Better Align Resources with Performance, [hyperlink, http://www.gao.gov/products/GAO-05-117SP] (Washington, D.C.: February 2005). [26] According to Statement of Federal Financial Accounting Standards No. 4, Managerial Cost Accounting Concepts and Standards for the Federal Government, examples of direct costs include: salaries and other benefits for employees who work directly on the output, materials and supplies used in the work, office space, and equipment and facilities that are used exclusively to produce the output; examples of indirect costs include: general administrative services; general research and technical support; security; rent; and operations and maintenance costs for building, equipment, and utilities. [27] The five standards in SFFAS 4 require government agencies to (1) accumulate and report the costs of activities on a regular basis for management information purposes; (2) establish responsibility segments, and measure and report the costs of each segment's outputs and calculate the unit cost of each output; (3) determine and report the full costs of government goods and services, including direct and indirect costs; (4) recognize the costs of goods and services provided by other federal entities; and (5) use and consistently follow costing methodologies or cost finding techniques most appropriate to the segment's operating environment to accumulate and assign costs to outputs. [28] Appropriations are a form of budget authority to incur obligations and to make payments from the Treasury for specified purposes. Obligations are a definite commitment that creates a legal liability of the government for the payment of goods and services ordered or received, or a legal duty on the part of the United States that could mature into a legal liability by virtue of actions on the part of the other party beyond the control of the United States. [29] [hyperlink, http://www.gao.gov/products/GAO-05-117SP]. [30] The study recommended including the annual depreciation amount for its property, plant, and equipment rather than the funds appropriated for construction for a given year. [31] Hei Tech Services, Inc., Job Corps Cost Measure: Selecting a Cost Measure to Assess Program Results (Dec. 1, 2008). [32] According to this official, the Department of the Interior is in the process of transitioning to a common business platform financial system, Financial Business Management Systems, but not all entities within the department have adopted the common system yet. [33] Two of the three measures concern the number of acres treated inside and outside the wildland-urban interface per million dollar gross investment. The third measure concerns the number of acres in fire regimes 1, 2, or 3 moved to a better condition class per million dollars of gross investment. [34] [hyperlink, http://www.gao.gov/products/GAO-07-679]. [35] Although Education did not have a departmentwide MCA system, as indicated below, Federal Student Aid (FSA) within Education had its own MCA system. [36] 31 U.S.C. § 3512 note. [37] The Higher Education Amendments of 1998, which amended the Higher Education Act of 1965, established a performance-based organization for the delivery of federal student financial assistance, after which Federal Student Aid, the one Department of Education program office with an operational MCA system, independently developed its MCA system. Pub. L. No. 105-244, title I, § 101(a), 112 Stat. 1581, 1604- 610 (Oct. 7, 1998), codified at 20 U.S.C. § 1018. PBOs are discrete units, led by a Chief Operating Officer, that commit to clear objectives, specific measurable goals, customer service standards, and targets for improved performance, see GAO-06-653T. [38] The Federal Aviation Reauthorization Act of 1996 required that FAA develop a cost accounting system that accurately reflects the investment, operating and overhead costs, revenues, and other financial measurement and reporting aspects of its operations. Pub. L. No. 104-264, § 276(a)(2), 110 Stat. 3213, 3248 (Oct. 9, 1996), codified at 49 U.S.C. § 45303(e). In addition, in 1997, the National Civil Aviation Review Commission (the "Mineta Commission") recommended that FAA establish a cost accounting system to support the objective of FAA operating in a more performance-based, business-like manner. [39] GAO, Managerial Cost Accounting Practices: Departments of Education, Transportation, and the Treasury, [hyperlink, http://www.gao.gov/products/GAO-06-301R] (Washington, D.C.: Dec. 19, 2005). [40] GAO, Managing for Results: Enhancing Agency Use of Performance Information for Management Decision Making, [hyperlink, http://www.gao.gov/products/GAO-05-927] (Washington, D.C.: Sept. 9, 2005). [41] A sole-source contract is a contract award without competition from other companies. Such contracts are used in instances in which only one source is deemed able to provide the service or product needed at the time. Without the pressure of competing bids to keep prices in check, having information on costs is critical to negotiating the terms of such contracts. [42] GAO, The Government Performance and Results Act: 1997 Governmentwide Implementation Will Be Uneven, [hyperlink, http://www.gao.gov/products/GAO/GGD-97-109] (Washington, D.C.: June 2, 1997); Managing for Results: Efforts to Strengthen the Link Between Resources and Results at the Administration for Children and Families, [hyperlink, http://www.gao.gov/products/GAO-03-9] (Washington, D.C.: Dec. 10, 2002); Performance Budgeting: Observations on the Use of OMB's Program Assessment Rating Tool for the Fiscal Year 2004 Budget, [hyperlink, http://www.gao.gov/products/GAO-04-174] (Washington, D.C.: Jan. 30, 2004); and Performance Budgeting: PART Focuses Attention on Program Performance, but More Can Be Done to Engage Congress, [hyperlink, http://www.gao.gov/products/GAO-06-28] (Washington, D.C.: Oct. 28, 2005). [43] [hyperlink, http://www.gao.gov/products/GAO-04-174], [hyperlink, http://www.gao.gov/products/GAO-06-28]. [44] [hyperlink, http://www.gao.gov/products/GAO-06-28], [hyperlink, http://www.gao.gov/products/GAO/GGD-97-109]. [45] [hyperlink, http://www.gao.gov/products/GAO-04-174], [hyperlink, http://www.gao.gov/products/GAO-03-9]. [46] GAO, Managing for Results: Measuring Program Results That Are Under Limited Federal Control, [hyperlink, http://www.gao.gov/products/GAO/GGD-99-16] (Washington, D.C.: Dec. 11, 1998). [47] [hyperlink, http://www.gao.gov/products/GAO-06-28]. [48] The Forest Service has clear authority to manage a broad spectrum of watershed activities on the national forests and to encourage the long-term stewardship of non-industrial private forestlands which contribute significantly to the health and productivity of the nation's watersheds. The Watershed program as delineated for the PART assessment encompassed the functional watershed program in the Watershed, Fish, Wildlife, Air, and Rare Plants Staff (WFW) and all Forest Service activities that contributed to improved watershed condition (e.g., vegetation management, reforestation, range management, wildlife and fisheries improvements, road decommissioning, etc.). It included at least 17 specific budget line items linked to meeting the goal of improving watershed condition from the Forest Service's Strategic Plan for Fiscal Years 2004-2008. [49] USDA Forest Service, Conceptual Framework for Determining and Tracking Changes in Watershed Condition on Lands Managed, revised February 13, 2008. [50] [hyperlink, http://www.gao.gov/products/GAO-06-28]. [51] [hyperlink, http://www.gao.gov/products/GAO-04-174]. [52] [hyperlink, http://www.gao.gov/products/GAO-04-174], [hyperlink, http://www.gao.gov/products/GAO-06-28]. [53] OMB, Examples of Performance Measures. [54] OMB, Program Assessment Rating Tool Guidance No. 2007-07: Guidance to Improve the Quality of PART Performance and Efficiency Goal (Dec. 12, 2007); and OMB, Program Assessment Rating Tool Guidance No. 2007-03: Guidance to Improve the Consistency of 2007 PART Assessments (May 15, 2007). [55] The measure showed an improvement in efficiency between 2007 and 2008, the only 2 years for which data were available. [56] [hyperlink, http://www.gao.gov/products/GAO-06-28]. [57] GAO, Program Evaluation: Improving the Flow of Information to the Congress, [hyperlink, http://www.gao.gov/products/GAO/PEMD-95-1] (Washington, D.C.: Jan. 30, 1995). [58] GAO, Human Service Programs: Demonstration Projects Could Identify Ways to Simplify Policies and Facilitate Technology Enhancements to Reduce Administrative Costs, [hyperlink, http://www.gao.gov/products/GAO-06-942] (Washington, D.C.: Sept. 19, 2006). [59] The Job Corps program hired a contractor to propose an alternative efficiency measure to try to capture the unique outcomes of the program. The contractor study proposed an outcome-level efficiency measure ("cost per successful program outcome"), but cautioned against comparison with other programs because estimates for other programs might not reflect full costs, and because comparisons could be misleading if program objectives were not identical. Hei Tech Services, Inc., Job Corps Cost Measure: Selecting a Cost Measure to Assess Program Results (Dec. 1, 2008). [60] In a prior review of PART, Labor officials told us that participants could remain in the Job Corps program for up to 2 years, which they considered adequate time to complete education or vocational training, and which generally resulted in higher wages, according to studies. However, they said that since costs per participant increased the longer a student remained in the program, Job Corps appeared less efficient compared with other job training programs. [hyperlink, http://www.gao.gov/products/GAO-06-28]. [61] The U.S. Fish and Wildlife Service, The Endangered Species Program's Strategic Plan, Draft (Sept. 19, 2008). [62] GAO, Endangered Species: Fish and Wildlife Service Generally Focuses Recovery Funding on High Priority Species, but Needs to Periodically Assess Its Funding Decisions, [hyperlink, http://www.gao.gov/products/GAO-05-211] (Washington, D.C.: Apr. 6, 2005). [63] As noted above, the Endangered Species program did not have an efficiency measure that was approved by OMB for PART. However, program officials said they used an efficiency measure internally: the average time to complete a 5-year review. (A 5-year review is a period analysis of a species' status conducted to ensure that the listing classification of a species as threatened or endangered is accurate.) [64] [hyperlink, http://www.gao.gov/products/GAO-06-28]. [65] GAO, 21st Century Challenges: Reexamining the Base of the Federal Government, [hyperlink, http://www.gao.gov/products/GAO-05-325SP] (Washington, D.C.: Feb.1, 2005). [66] [hyperlink, http://www.gao.gov/products/GAO-03-1168T]. [67] GAO, VA Health Care: Status of Efforts to Improve Efficiency and Access, [hyperlink, http://www.gao.gov/products/GAO/HEHS-98-48] (Washington, D.C.: Feb. 6, 1998). [68] GAO, Military Base Realignments and Closures: Estimated Costs Have Increased While Savings Estimates Have Decreased Since Fiscal Year 2009, [hyperlink, http://www.gao.gov/products/GAO-10-98R] (Washington: D.C.: Nov. 13, 2009). [69] GAO, DOD Information Technology: Software and Systems Process Improvement Programs vary in Use of Best Practices, [hyperlink, http://www.gao.gov/products/GAO-01-116] (Washington, D.C.: Mar. 30, 2001). [70] APQC is a nonprofit worldwide leader in process and performance improvement with members from government, nongovernment, and business organizations. [71] APQC, Operating Tactics in Tough Times: Reduce Costs and Retain Customers - Business Process Management Research (Houston, TX: Aug. 11, 2009). Some of the other methodologies covered in the report include Baldrige National Quality Program, Kaizen, ISO 9001, and LEAN. [72] GAO, Human Service Programs: Demonstration Projects Could Identify Ways to Simplify Policies and Facilitate Technology Enhancements to Reduce Administrative Costs, [hyperlink, http://www.gao.gov/products/GAO-06-942] (Washington, D.C.: Nov. 15, 2006). [73] GAO, Tax Administration: Most Filing Season Services Continue to Improve, but Opportunities Exist for Additional Savings, [hyperlink, http://www.gao.gov/products/GAO-07-27] (Washington, D.C.: Nov. 15, 2006). [74] OMB, Analytical Perspectives, Budget of the United States Government, Fiscal Year 2011 (Washington D.C.: February 2010). [75] Executive Order 13520, Reducing Improper Payments, 74 Fed. Reg. 13,520 (Nov. 20, 2009). [76] Some of the programs' modernization efforts were launched before PART. [77] The seven programs were Adoption Assistance, Child Care and Development Fund, Child Support Enforcement, food stamps, Foster Care, Temporary Assistance for Needy Families, and Unemployment Insurance. GAO, Human Service Programs: Demonstration Projects Could Identify Ways to Simplify Policies and Facilitate Technology Enhancements to Reduce Administrative Costs, [hyperlink, http://www.gao.gov/products/GAO-06-942] (Washington, D.C.: Sept. 19, 2006). [78] Seafood fraud occurs when seafood products are mislabeled for financial gain. See GAO, Seafood Fraud: FDA Program Changes and Better Collaboration among Key Federal Agencies Could Improve Detection and Prevention, [hyperlink, http://www.gao.gov/products/GAO-09-258] (Washington, D.C.: Feb. 19, 2009). [79] GAO, Best Practices: Using Spend Analysis to Help Agencies Take a More Strategic Approach to Procurement, [hyperlink, http://www.gao.gov/products/GAO-04-870] (Washington, D.C.: Sept. 16, 2004). [80] Between 2000 and 2003, prior GAO work studied procurement best practices of 11 companies--Bausch & Lomb; Brunswick Corporation; ChevronTexaco; Delta Air Lines; Dell; Dun & Bradstreet Corporation; Electronic Data Systems Corporation; Exxon Mobil Corporation; Hasbro, Inc.; International Business Machines; and Merrill Lynch & Co., Inc. See [hyperlink, http://www.gao.gov/products/GAO-04-870]. [81] Memorandum from Peter R. Orszag, Director, OMB, for the Heads of Departments and Agencies, Subject: Improving Government Acquisition (July 29, 2009). Memorandum from Lesley A. Field, Deputy Administrator, OMB, for Chief Acquisition Officers, Senior Procurement Executives, Subject: Increasing Competition and Structuring Contracts for the Best Results (Oct. 27, 2009). [82] Officials also reported using energywatchdog.com to receive a rebate of approximately $520,000 in fiscal year 2006 for overcharged utility costs at Job Corps centers. [83] OMB, Analytical Perspectives, Budget of the United States Government, Fiscal Year 2011 (Washington D.C.: February 2010). [84] GAO, Program Evaluation: Studies Helped Agencies Measure or Explain Program Performance, [hyperlink, http://www.gao.gov/products/GAO/GGD-00-204] (Washington, D.C.: Sept. 29, 2000). [85] GAO, Performance Measurement and Evaluation: Definitions and Relationships, [hyperlink, http://www.gao.gov/products/GAO-05-739SP] (Washington, D.C.: May 2005). [86] NAO, The Efficiency Programme: A Second Review of Progress (London, U.K.: Feb. 8, 2007). [87] [hyperlink, http://www.gao.gov/products/GAO-04-38]. [88] GAO, Grants Management: Enhancing Performance Accountability Provisions Could Lead to Better Results, [hyperlink, http://www.gao.gov/products/GAO-06-1046] (Washington, D.C.: Sept. 29, 2006). [89] [hyperlink, http://www.gao.gov/products/GAO-09-1011T]. [90] GAO, Managerial Cost Accounting Practices: Implementation and Use Vary Widely across 10 Federal Agencies, [hyperlink, http://www.gao.gov/products/GAO-07-679] (Washington, D.C.: July 20, 2007). [91] Alternatively, we could have selected the Social Security Administration, but chose to limit our review to cabinet-level departments. [92] The PART assessment years for the programs we selected ranged from 2003 to 2008. [93] OMB provided us with a database containing information on all programs that had received PART assessments and said the data were current as of January 14, 2009. We assessed the reliability of the OMB data and found that they were sufficiently reliable for purposes of this engagement. [94] All programs were considered to be direct federal and were assessed using 25 basic questions that comprised the direct federal PART. If a program delivered goods and services using one of the mechanisms captured in the other six PART types (competitive grant, block/formula grant, research and development, capital assets and acquisition, credit, or regulatory), it was assessed with additional specific questions tailored to the program type. [95] We excluded research and development programs from our sample of selected programs based on the findings of a 2008 study by The National Academies which raised questions about the feasibility of developing valid outcome-based efficiency measures for federal research programs (Evaluating Research Efficiency in the U.S. Environmental Protection Agency, Committee on Evaluating the Efficiency of Research and Development Programs at the U.S. Environmental Protection Agency, The National Academies). We excluded credit programs from our sample of selected programs because of the difficulty in making generalizations about such programs due to the relatively small number of these programs in the selected departments. [96] In addition to these 36 efficiency measures, there were a total of 5 additional efficiency measures included in the PART data we received from OMB for three of our selected programs. However, officials from each of these programs told us these 5 efficiency measures were no longer associated with PART, so we excluded them from our analysis. Further, one of the selected programs, the Department of Transportation's Federal Aviation Administration Air Traffic Organization (ATO)Terminal, changed one of its measures in PART, ATO- Terminal staffing ratio, from an "output" to an "efficiency" measure after our initial interview. As a result, we did not include this measure in our review. [97] When a measure was coded "No" for output/outcome, we coded the output or outcome type "N/A." [98] We excluded the 36 specifically selected efficiency measures from this population and the 5 efficiency measures which were included in PART but which program officials said should not be. [99] APQC, Operating Tactics in Tough Times: Reduce Costs and Retain Customers (Houston, TX: Aug. 11, 2009). [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO‘s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO‘s Web site, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548:

The Justia Government Accountability Office site republishes public reports retrieved from the U.S. GAO These reports should not be considered official, and do not necessarily reflect the views of Justia.