Performance Budgeting

Observations on the Use of OMB's Program Assessment Rating Tool for the Fiscal Year 2004 Budget Gao ID: GAO-04-174 January 30, 2004

The Office of Management and Budget's (OMB) Program Assessment Rating Tool (PART) is meant to provide a consistent approach to evaluating federal programs during budget formulation. To better understand its potential, congressional requesters asked GAO to examine (1) how PART changed OMB's fiscal year 2004 budget decisionmaking process, (2) PART's relationship to the Government Performance and Results Act of 1993 (GPRA), and (3) PART's strengths and weaknesses as an evaluation tool.

PART helped structure OMB's use of performance information for its internal program and budget analysis, made the use of this information more transparent, and stimulated agency interest in budget and performance integration. OMB and agency staff said this helped OMB staff with varying levels of experience focus on similar issues. Our analysis confirmed that one of PART's major impacts was its ability to highlight OMB's recommended changes in program management and design. Much of PART's potential value lies in the related program recommendations, but realizing these benefits requires sustained attention to implementation and oversight to determine if desired results are achieved. OMB needs to be cognizant of this as it considers capacity and workload issues in PART. There are inherent challenges in assigning a single rating to programs having multiple purposes and goals. OMB devoted considerable effort to promoting consistent ratings, but challenges remain in addressing inconsistencies among OMB staff, such as interpreting PART guidance and defining acceptable measures. Limited credible evidence on results also constrained OMB's ability to rate program effectiveness, as evidenced by the almost 50 percent of programs rated "results not demonstrated." PART is not well integrated with GPRA--the current statutory framework for strategic planning and reporting. By using the PART process to review and sometimes replace GPRA goals and measures, OMB is substituting its judgment for a wide range of stakeholder interests. The PART/GPRA tension was further highlighted by challenges in defining a unit of analysis useful for both program-level budget analysis and agency planning purposes. Although PART can stimulate discussion on program-specific measurement issues, it cannot substitute for GPRA's focus on thematic goals and department- and governmentwide crosscutting comparisons. Moreover, PART does not currently evaluate similar programs together to facilitate trade-offs or make relative comparisons. PART clearly must serve the President's interests. However, the many actors whose input is critical to decisions will not likely use performance information unless they feel it is credible and reflects a consensus on goals. It will be important for OMB to discuss timely with Congress the focus of PART assessments and clarify the results and limitations of PART and the underlying performance information. A more systematic congressional approach to providing its perspective on performance issues and goals could facilitate OMB's understanding of congressional priorities and thus increase PART's usefulness in budget deliberations.

Recommendations

Our recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.

Director: Team: Phone:


GAO-04-174, Performance Budgeting: Observations on the Use of OMB's Program Assessment Rating Tool for the Fiscal Year 2004 Budget This is the accessible text file for GAO report number GAO-04-174 entitled 'Performance Budgeting: Observations on the Use of OMB's Program Assessment Rating Tool for the Fiscal Year 2004 Budget' which was released on January 30, 2004. This text file was formatted by the U.S. General Accounting Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to Congressional Requesters: January 2004: PERFORMANCE BUDGETING: Observations on the Use of OMB's Program Assessment Rating Tool for the Fiscal Year 2004 Budget: [Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-174] GAO-04-174: GAO Highlights: Highlights of GAO-04-174, a report to congressional requesters Why GAO Did This Study: The Office of Management and Budget‘s (OMB) Program Assessment Rating Tool (PART) is meant to provide a consistent approach to evaluating federal programs during budget formulation. To better understand its potential, congressional requesters asked GAO to examine (1) how PART changed OMB‘s fiscal year 2004 budget decision-making process, (2) PART‘s relationship to the Government Performance and Results Act of 1993 (GPRA), and (3) PART‘s strengths and weaknesses as an evaluation tool. What GAO Found: PART helped structure OMB‘s use of performance information for its internal program and budget analysis, made the use of this information more transparent, and stimulated agency interest in budget and performance integration. OMB and agency staff said this helped OMB staff with varying levels of experience focus on similar issues. Our analysis confirmed that one of PART‘s major impacts was its ability to highlight OMB‘s recommended changes in program management and design. Much of PART‘s potential value lies in the related program recommendations, but realizing these benefits requires sustained attention to implementation and oversight to determine if desired results are achieved. OMB needs to be cognizant of this as it considers capacity and workload issues in PART. There are inherent challenges in assigning a single rating to programs having multiple purposes and goals. OMB devoted considerable effort to promoting consistent ratings, but challenges remain in addressing inconsistencies among OMB staff, such as interpreting PART guidance and defining acceptable measures. Limited credible evidence on results also constrained OMB‘s ability to rate program effectiveness, as evidenced by the almost 50 percent of programs rated ’results not demonstrated.“ PART is not well integrated with GPRA”the current statutory framework for strategic planning and reporting. By using the PART process to review and sometimes replace GPRA goals and measures, OMB is substituting its judgment for a wide range of stakeholder interests. The PART/GPRA tension was further highlighted by challenges in defining a unit of analysis useful for both program-level budget analysis and agency planning purposes. Although PART can stimulate discussion on program-specific measurement issues, it cannot substitute for GPRA‘s focus on thematic goals and department- and governmentwide crosscutting comparisons. Moreover, PART does not currently evaluate similar programs together to facilitate trade-offs or make relative comparisons. PART clearly must serve the President‘s interests. However, the many actors whose input is critical to decisions will not likely use performance information unless they feel it is credible and reflects a consensus on goals. It will be important for OMB to discuss timely with Congress the focus of PART assessments and clarify the results and limitations of PART and the underlying performance information. A more systematic congressional approach to providing its perspective on performance issues and goals could facilitate OMB‘s understanding of congressional priorities and thus increase PART‘s usefulness in budget deliberations. What GAO Recommends: GAO recommends that OMB (1) address the capacity demands of PART, (2) strengthen PART guidance, (3) address evaluation information availability and scope issues, (4) focus program selection on crosscutting comparisons and critical operations, (5) broaden the dialogue with congressional stakeholders, and (6) articulate and implement a complementary relationship between PART and GPRA. OMB generally agreed with our findings, conclusions, and recommendations and stated that it is already taking actions to address many of our recommendations. GAO also suggests that Congress consider the need for a structured approach to articulating its perspective and oversight agenda on performance goals and priorities for key programs. www.gao.gov/cgi-bin/getrpt?GAO-04-174. To view the full product, including the scope and methodology, click on the link above. For more information, contact Paul Posner at (202) 512-9573 or posnerp@gao.gov. [End of section] Contents: Letter: Results in Brief: Background: OMB Used the PART to Systematically Assess Program Performance and Make Results Known, but Follow-up on PART Recommendations Is Uncertain: Despite OMB's Considerable Efforts to Create a Credible Evaluation Tool, PART Assessments Require Judgment and Were Constrained by Data Limitations: Subjective Terms and a Restrictive Format Contributed to Subjective and Inconsistent Responses: There Were Inconsistencies in Defining Acceptable Measures and in Logically Responding to Question "Pairs": Disagreements on Performance Information Led to Creation of a "Results Not Demonstrated" Category: The Fiscal Year 2004 PART Process Was a Parallel, Competing Approach to GPRA's Performance Management Framework: Defining a "Unit of Analysis" That Is Useful for Program-Level Budget Analysis and Agency Planning Purposes Presents Challenges: Conclusions and General Observations: Matter for Congressional Consideration: Recommendations for Executive Action: Agency Comments: Appendixes: Appendix I: Scope and Methodology: Appendix II: The Fiscal Year 2004 PART and Differences Between the Fiscal Year 2004 and 2005 PARTs: Section I: Program Purpose & Design (Yes, No, N/A): Section II: Strategic Planning (Yes, No, N/A): Section III: Program Management (Yes, No, N/A): Section IV: Program Results (Yes, Large Extent, Small Extent, No): Appendix III: Development of PART: Fiscal Year 2003: Fiscal Year 2004: Fiscal Year 2005: Appendix IV: Comments from the Office of Management and Budget: Appendix V: GAO Contacts and Staff Acknowledgments: GAO Contacts: Acknowledgments: Tables: Table 1: Overview of Sections of PART Questions: Table 2: Overview of PART Program Types: Table 3: The Effect of Overall PART Score on Proposed Funding Changes (Discretionary Programs): Table 4: The Effect of Overall PART Score on Proposed Funding Changes (Small Discretionary Programs): Table 5: The Effect of Overall PART Score on Proposed Funding Changes (Medium-Size Discretionary Programs): Table 6: The Effect of Overall PART Score on Proposed Funding Changes (Large Discretionary Programs): Table 7: The Effect of PART Component Scores on Proposed Funding Changes (All Discretionary Programs): Table 8: The Effect of PART Component Scores on Proposed Funding Changes (Small Discretionary Programs): Table 9: Side-by-Side of the Fiscal Year 2005 PART and the Fiscal Year 2004 PART Questions: Figures: Figure 1: Fiscal Year 2004 PART Recommendations: Figure 2: Number of Discretionary PART Programs by Rating and Funding Result, Fiscal Years 2003-2004: Figure 3: The PART Process and Budget Formulation Timelines: Letter January 30, 2004: The Honorable George V. Voinovich: Chairman: Subcommittee on Oversight of Government Management, the Federal Workforce and the District of Columbia: Committee on Governmental Affairs: United States Senate: The Honorable Todd R. Platts: Chairman: Subcommittee on Government Efficiency and Financial Management: Committee on Government Reform: House of Representatives: The Honorable Sam Brownback: United States Senate: The Honorable Todd Tiahrt: House of Representatives: Since the 1950s, the federal government has attempted several governmentwide initiatives designed to better align spending decisions with expected performance--what is often commonly referred to as "performance budgeting." Consensus exists that prior efforts-- including the Hoover Commission, the Planning-Programming-Budgeting- System (PPBS), Management by Objectives, and Zero-Based Budgeting (ZBB)--failed to significantly shift the focus of the federal budget process from its long-standing concentration on the items of government spending to the results of its programs. In the 1990s, Congress and the executive branch laid out a statutory and management framework that provides the foundation for strengthening government performance and accountability, with the Government Performance and Results Act of 1993[Footnote 1] (GPRA) as its centerpiece. GPRA is designed to inform congressional and executive decision making by providing objective information on the relative effectiveness and efficiency of federal programs and spending. A key purpose of the act is to create closer and clearer links between the process of allocating scarce resources: and the expected results to be achieved with those resources. This type of integration is critical, as we have learned from prior initiatives that failed in part because they did not prove to be relevant to budget decision makers in the executive branch or Congress.[Footnote 2] GPRA requires not only a connection to the structures used in congressional budget presentations but also consultation between the executive and legislative branches on agency strategic plans, which gives Congress an oversight stake in GPRA's success.[Footnote 3] In its overall structure, focus, and approach GPRA incorporates two critical lessons learned from previous reforms. First, any approach designed to link plans and budgets--that is, to link the responsibility of the executive to define strategies and approaches with the legislative "power of the purse"--must explicitly involve both branches of our government. PPBS and ZBB failed in part because performance plans and measures were developed in isolation from congressional oversight and resource allocation processes. Second, the concept of performance budgeting has and likely will continue to evolve. Thus, no single definition of performance budgeting encompasses the range of past and present needs and interests of federal decision makers. The need for multiple definitions reflects the differences in the roles various participants play in the budget process. And, given the complexity and breadth of the federal budget process, performance budgeting must encompass a variety of perspectives in its efforts to link resources with results. This administration has made the integration of performance and budget information one of five governmentwide management priorities under its President's Management Agenda (PMA).[Footnote 4] A central element in this initiative is the Office of Management and Budget's (OMB) Program Assessment Rating Tool (PART) that OMB describes as a diagnostic tool meant to provide a consistent approach to evaluating federal programs as part of the: executive budget formulation process. The PART is the latest iteration of 50 years of federal performance budgeting initiatives. It applies 25 questions to all "programs"[Footnote 5] under four broad topics: (1) program purpose and design, (2) strategic planning, (3) program management, and (4) program results (i.e., whether a program is meeting its long-term and annual goals) as well as additional questions that are specific to one of seven mechanisms or approaches used to deliver the program.[Footnote 6] To better understand the PART's potential as a mechanism for assessing program goals and results, you asked us to examine (1) how the PART changed OMB's decision-making process in developing the President's fiscal year 2004 budget request; (2) the PART's relationship to the GPRA planning process and reporting requirements; and (3) the PART's strengths and weaknesses as an evaluation tool, including how OMB ensured that the PART was applied consistently. To respond to your request, we reviewed OMB materials on the development and implementation of the PART as well as the results produced by the PART assessments. To assess consistency of the PART's application, we performed analyses of OMB data from the PART program summary and assessment worksheets for each of the 234 programs OMB reviewed for fiscal year 2004, including a statistical analysis of the relationship between the PART scores and funding levels in the President's Budget. We also identified several sets of similar programs that we examined more closely to determine if comparable or disparate criteria were applied in producing the PART results for these clusters of programs. We reviewed 28 programs in nine clusters covering food safety, water supply, military equipment procurement, provision of health care, statistical agencies, block grants to assist vulnerable populations, energy research programs, wildland fire management, and disability compensation. We also interviewed OMB officials regarding their experiences with the PART in the fiscal year 2004 budget process. As part of our examination of the usefulness of the PART as an evaluation tool and also to obtain agency perspectives on the relationship between PART and GPRA, we interviewed department and agency officials, including senior managers, and program, planning, and budget staffs at (1) the Department of Health and Human Services (HHS), (2) the Department of Energy (DOE), and (3) the Department of the Interior (DOI). We selected these three departments because they had a variety of program types (e.g., block/formula grants, competitive grants, direct federal, and research and development) that were subject to the PART and could provide a broad-based perspective on how the PART was applied to different programs. With the exception of our summary analyses of all 234 programs, the information obtained from OMB and agency officials and our review of selected programs is not generalizable to the PART process for all 234 programs. However, the consistency and frequency with which similar issues were raised by OMB and agency officials suggest that our review reliably captures several significant and salient aspects of the PART as a budget and evaluation tool. Our review focused on the fiscal year 2004 PART process. We conducted our work from May 2003 through October 2003 in accordance with generally accepted government auditing standards. Detailed information on our scope and methodology appears in appendix I. OMB provided written comments on a draft of this report that are reprinted in appendix IV. Results in Brief: The PART has helped to structure and discipline OMB's use of performance information for its internal program analysis and budget review, made the use of this information more transparent, and stimulated agency interest in budget and performance integration. Both OMB and agency staff noted that this helped ensure that OMB staff with varying levels of experience focused on the same issues, fostering a more disciplined approach to discussing program performance with agencies. Several agency officials also told us that the PART was a catalyst for bringing agency budget, planning, and program staff together since none could fully respond to the PART questionnaire alone. Our analysis confirmed that one of the PART's major impacts was its ability to highlight OMB's recommended changes in program management and design. Over 80 percent of the recommendations made for the 234 programs assessed for the fiscal year 2004 budget process were for improvements in program design, assessment, and program management; less than 20 percent were related to funding issues. As OMB and others recognize, performance is not the only factor in funding decisions. Determining priorities--including funding priorities--is a function of competing values and interests. Although OMB generally proposed to increase funding for programs that received ratings of "effective" or "moderately effective" and proposed to cut funding for those programs that were rated "ineffective," our review confirmed OMB's statements that funding decisions were not applied mechanistically. That is, for some programs rated "effective" or "moderately effective" OMB recommended funding decreases, while for several programs judged to be "ineffective" OMB recommended additional funding in the President's budget request with which to implement changes. Much of the potential value of the PART lies in the related program recommendations and associated improvements, but realization of these benefits will require sustained attention to implementation and oversight in order to determine if the desired results are being achieved. Such attention and oversight takes time, and OMB needs to be cognizant of this as it considers the capacity and workload issues in the PART. Currently OMB plans to assess an additional 20 percent of all federal programs annually. Each year, the number of recommendations from previous years' evaluations will grow--and a system for monitoring their implementation will become more critical. OMB encouraged its Resource Management Offices (RMO) to consider many factors in selecting programs for the fiscal year 2004 PART assessments, such as continuing presidential initiatives and programs up for reauthorization. While all programs would eventually be reviewed over the 5-year period, selecting related programs for review in a given year would enable decision makers to analyze the relative efficacy of similar programs in meeting common or similar outcomes. We recommend that OMB centrally monitor and report on agency implementation and progress on PART recommendations to provide a governmentwide picture of progress and a consolidated view of OMB's workload in this area. In addition, to target scarce analytic resources and to focus decision makers' attention on the most pressing policy issues, we recommend that OMB reconsider plans for 100 percent coverage of federal programs by targeting PART assessments based on such factors as the relative priorities, costs, and risks associated with related clusters of programs and activities. We further recommend that OMB select for review in the same year related or similar programs or activities to facilitate such comparisons and trade-offs. Developing a credible evidence-based rating tool to provide bottom-line ratings for programs was a major impetus in developing the PART. However, inherent challenges exist in assigning a single "rating" to programs that often have multiple purposes and goals. Despite the considerable time and effort OMB has devoted to promoting consistent application of the PART, the tool is a work in progress. Additional guidance and considerable revisions are needed to meet OMB's goal of an objective, evidence-based assessment tool. In addition to difficulties with the tool itself--such as subjective terminology and a restrictive yes/no format--providing flexibility to assess multidimensional programs with multiple purposes and goals often implemented through multiple actors has led to a reliance on OMB staff judgments to apply general principles to specific cases. OMB staff were not fully consistent in interpreting the guidance for complex PART questions and in defining acceptable measures. In addition, the limited availability of credible evidence on program results also constrained OMB staff's ability to use the PART to rate programs' effectiveness. Almost 50 percent of the 234 programs assessed for fiscal year 2004 received a rating of "results not demonstrated" because OMB decided that program performance information, performance measures, or both were insufficient or inadequate. OMB, recognizing many of the limitations with the PART, modified the PART for fiscal year 2005 based on lessons learned during the fiscal year 2004 process, but issues remain. We therefore recommend that OMB continue to improve the PART guidance by (1) clarifying when output versus outcome measures are acceptable and (2) better defining an "independent, quality evaluation." We further recommend that OMB both clarify its expectations regarding the nature, timing, and amount of evaluation information it wants from agencies for the purposes of the PART and consider using internal agency evaluations as evidence on a case-by-case basis. The PART is not well integrated with GPRA--the current statutory framework for strategic planning and reporting. According to OMB officials, GPRA plans were organized at too high a level to be meaningful for program-level budget decision making. To provide decision makers with program-specific, outcome-based performance data useful for executive budget formulation, OMB has stated its intention to modify GPRA goals and measures with those developed under the PART. As a result, OMB's judgment about appropriate goals and measures is substituted for GPRA judgments based on a community of stakeholder interests. Agency officials we spoke with expressed confusion about the relationship between GPRA requirements and the PART process. Many view PART's program-by-program focus and the substitution of program measures as detrimental to their GPRA planning and reporting processes. OMB's effort to influence program goals is further evident in recent OMB Circular A-11 guidance[Footnote 7] that clearly requires each agency to submit a performance budget for fiscal year 2005, which will replace the annual GPRA performance plan. The tension between PART and GPRA was further highlighted by the challenges in defining a unit of analysis that is useful both for program-level budget analysis and agency planning purposes. Although the PART reviews indicated to OMB that GPRA measures are often not sufficient to help it make judgments about programs, the different units of analysis used in these two performance initiatives contributed to this outcome. For the PART, OMB created units of analysis that tied to discrete funding levels by both disaggregating and aggregating certain programs. In some cases, disaggregating programs for the PART reviews ignored the interdependency of programs by artificially isolating them from the larger contexts in which they operate. Conversely, in other cases in which OMB aggregated programs with diverse missions and outcomes for the PART reviews, it became difficult to settle on a single measure (or set of measures) that accurately captured the multiple missions of these diverse components. Both of these "unit of analysis" issues contributed to the lack of available planning and performance information. Although the PART can stimulate discussion on program-specific performance measurement issues, it is not a substitute for GPRA's strategic, longer-term focus on thematic goals and department-and governmentwide crosscutting comparisons. GPRA is a broad legislative framework that was designed to be consultative with Congress and other stakeholders and allows for varying uses of performance information, while the PART applies evaluation information to support decisions and program reviews during the executive budget formulation process. Moreover, GPRA can anchor the review of programs by providing an overall strategic context for programs' contributions toward agency goals. We therefore recommend that OMB seek to achieve the greatest benefit from both GPRA and PART by articulating and implementing an integrated, complementary relationship between the two. We further recommend that OMB continue to improve the PART guidance by expanding the discussion of how programs--also known as "units of analysis"--are determined, including recognizing the trade-offs, implications, or both of such determinations. As part of the President's budget preparation, the PART clearly must serve the President's interests. However, experience suggests that efforts to integrate budget and performance are promoted when Congress and other key stakeholders have confidence in the credibility of the analysis and the process used. It is unlikely that the broad range of players whose input is critical to decisions will use performance information unless they believe it is relevant, credible, reliable, and reflective of a consensus about performance goals among a community of interested parties. Similarly, the measures used to demonstrate progress toward a goal, no matter how worthwhile, cannot appear to serve a single set of interests without potentially discouraging use of this information by others. We therefore recommend that OMB attempt to build on the strengths of GPRA and PART by seeking to communicate early in the PART process with congressional appropriators and authorizers about what performance issues and information are most important to them in evaluating programs. Furthermore, while Congress has a number of opportunities to provide its perspective on performance issues and goals through its authorization, oversight, and appropriations processes, we suggest that Congress consider the need for a more structured approach for sharing with the executive branch its perspective on governmentwide performance matters, including its views on performance goals and outcomes for key programs and the oversight agenda. In commenting on a draft of this report, OMB generally agreed with our findings, conclusions, and recommendations. OMB outlined actions it is taking to address many of our recommendations, including refining the process for monitoring agencies' progress in implementing the PART recommendations, seeking opportunities for dialogue with Congress on agencies' performance, and continuing to improve executive branch implementation of GPRA plans and reports. OMB also suggested some technical changes throughout the report that we have incorporated as appropriate. OMB's comments appear in appendix IV. We also received technical comments on excerpts of the draft provided to the Departments of the Interior, Energy, and Health and Human Services, which are incorporated as appropriate. Background: The current administration has taken several steps to strengthen and further performance-resource linkages for which GPRA laid the groundwork. Central to the budget and performance integration initiative, the PART is meant to strengthen the process for assessing the effectiveness of programs by making that process more robust, transparent, and systematic. As noted above, the PART is a series of diagnostic questions designed to provide a consistent approach to rating federal programs. (See app. II for a reproduction of the PART.) Drawing on available performance and evaluation information, the questionnaire attempts to determine the strengths and weaknesses of federal programs with a particular focus on individual program results. The PART asks, for example, whether a program's long-term goals are specific, ambitious, and focused on outcomes, and whether annual goals demonstrate progress toward achieving long-term goals. It is designed to be evidence based, drawing on a wide array of information, including authorizing legislation, GPRA strategic plans and performance plans and reports, financial statements, inspector general and GAO reports, and independent program evaluations. PART questions are divided into four sections; each section is given a specific weight in determining the final numerical rating for a program. Table 1 shows an overview of the four PART sections and the weights OMB assigned. Table 1: Overview of Sections of PART Questions: Section: I. Program Purpose and Design; Description: To assess whether; * the purpose is clear, and; * the program design makes sense; Weight: 20%. Section: II. Strategic Planning; Description: To assess whether the agency sets valid programmatic; * annual goals, and; * long-term goals; Weight: 10%. Section: III. Program Management; Description: To rate agency management of the program, including; * financial oversight, and; * program improvement efforts; Weight: 20%. Section: IV. Program Results/Accountability; Description: To rate program performance on goals reviewed in; * the strategic planning section, and; * through other evaluations; Weight: 50%. Source: GAO analysis of the Budget of the United States Government, Fiscal Year 2004, Performance and Management Assessments (Washington, D.C.: February 2003). [End of table] In addition, each PART program is assessed according to one of seven approaches to service delivery. Table 2 provides an overview of these program types and the number and percentage of programs covered by each type in the fiscal year 2004 President's Budget performance assessments. Table 2: Overview of PART Program Types: Program type: 1. Direct federal; Description: Programs in which support and services are provided primarily by federal employees; Number/ percentage of programs[A]: 67; 29%. Program type: 2. Block/formula grant; Description: Programs that distribute funds to state, local, and tribal governments and other entities by formula or block grant; Number/percentage of programs[A]: 41; 18%. Program type: 3. Competitive grant; Description: Programs that distribute funds to state, local, and tribal governments, organizations, individuals, and other entities through a competitive process; Number/percentage of programs[A]: 37; 16%. Program type: 4. Capital assets and service acquisition; Description: Programs in which the primary means to achieve goals is the development and acquisition of capital assets (such as land, structures, equipment, and intellectual property) or the purchase of services (such as maintenance and information technology) from a commercial source; Number/percentage of programs[A]: 34; 15%. Program type: 5. Research and development; Description: Programs that focus on creating knowledge or applying it toward the creation of systems, devices, methods, materials, or technologies; Number/ percentage of programs[A]: 32; 14%. Program type: 6. Regulatory-based; Description: Programs that employ regulatory action to achieve program and agency goals through rule making that implements, interprets, or prescribes law or policy, or describes procedure or practice requirements. These programs issue significant regulations, which are subject to OMB review; Number/ percentage of programs[A]: 15; 6%. Program type: 7. Credit; Description: Programs that provide support through loans, loan guarantees, and direct credit; Number/percentage of programs[A]: 4; 2%. Program type: 8. Mixed[B]; Description: Programs that contain elements of different program types; Number/percentage of programs[A]: 4; 2%. Source: GAO summary and analysis of the Budget of the United States Government, Fiscal Year 2004, Performance and Management Assessments (Washington, D.C.: February 2003). [A] Percentages do not add to 100 percent due to rounding. [B] OMB noted that in rare cases, drawing questions from two of the seven PART program types--that is, creation of a "mixed" program type- -yields a more informative assessment. [End of table] During the fiscal year 2004 budget cycle, OMB applied the PART to 234 programs (about 20 percent of the fiscal year 2004 President's Budget request to Congress[Footnote 8]), and gave each program one of four overall ratings: (1) "effective," (2) "moderately effective," (3) "adequate," or (4) "ineffective" based on program design, strategic planning, management, and results. A fifth rating, "results not demonstrated," was given--independent of a program's numerical score-- if OMB decided that a program's performance information, performance measures, or both were insufficient or inadequate. The administration plans to assess an additional 20 percent of the budget each year until the entire executive branch has been reviewed. For more information on the development of the PART, see appendix III. OMB Used the PART to Systematically Assess Program Performance and Make Results Known, but Follow-up on PART Recommendations Is Uncertain: The PART clarified OMB's use of performance information in its budget decision-making process and stimulated new interest in budget and performance integration. OMB generally proposed budget increases for programs that received ratings of "effective" or "moderately effective" and decreased funding requests for those programs that were rated "ineffective," but there were clear exceptions. Moreover, the more important role of the PART was not in making resource decisions but in its support for recommendations to improve program design, assessment, and management. OMB's ability to use the PART to identify and address future program improvements and measure progress--a major purpose of the PART--is predicated on its ability to oversee the implementation of PART recommendations. However, it is not clear that OMB has a centralized system to oversee the implementation of such recommendations or evaluate their effectiveness. The PART Made Budget and Performance Integration at OMB More Transparent: The PART helped structure and discipline the use of performance information in the budget process and made the use of such information more transparent throughout the executive branch. According to OMB senior officials and many of the examiners and branch chiefs, the PART lent structure to a process that had previously been informal and gave OMB staff a systematic way of asking performance-related questions. Both agency and OMB staff noted that this helped ensure that OMB staff with varying levels of experience focused on the same issues, fostering a more disciplined approach to discussing performance within OMB and with agencies. Agency officials told us that by encouraging more communication between departments and OMB, the PART helps illuminate both how OMB makes budget decisions and how OMB staff think about program management. The PART also provided a framework for raising performance issues during the OMB Director's Reviews. OMB managers and staff reported that it led to richer discussions on what a program should be achieving, whether the program was performing effectively, and how program performance could be improved. Agencies also reported that the PART process expanded the dialogue between program, planning, and budget staffs, and stimulated interest in budget and performance integration. Several agency officials stated that the PART worksheets were a catalyst for bringing staffs together since none could fully respond to the questionnaire alone. OMB and agency officials agreed that the PART led to more interactions between OMB and agency program and planning staff and, in turn, increased program managers' awareness of and involvement in the budget process. According to OMB and several agency officials, the PART process--that is, responding to the PART questionnaire--involved staff outside of the performance management area. Additionally, both agency and OMB officials said that the attention given to programs that were not routinely reviewed was a positive benefit of the PART process. Use of Performance Information Was Evident in OMB's Recommendations: OMB senior officials told us that one of the PART's most notable impacts was its ability to highlight OMB's recommended changes in program management and design. As shown in figure 1, we found that 82 percent of PART recommendations addressed program assessment, design, and management issues; only 18 percent of the recommendations had a direct link to funding matters.[Footnote 9] Figure 1: Fiscal Year 2004 PART Recommendations: [See PDF for image] [End of figure] The majority of recommendations relate to changes that go well beyond funding consideration for one budget cycle. For example, OMB and HHS officials agree that the Foster Care program as it is currently designed does not provide appropriate incentives for the permanent placement of children; the program financially rewards states for keeping children in foster care instead of the original intent of providing temporary, safe, and appropriate homes for abused or neglected children until children can be returned to their families or other permanent arrangements can be made. The PART assessment provided support for OMB's recommendation that legislation be introduced that would create an option for states to participate in an alternate financing program that would "better meet the needs of each state's foster care population.": Performance information included in the PART for the Department of Labor's (DOL) Community Service Employment for Older Americans program helped to shape OMB's recommendation to increase competition for the grants. OMB concluded that although the Older Americans Act of 2000 amendments[Footnote 10] authorize competition for grants in cases in which grantees repeatedly fail to perform, the programs' 10 national grantees have historically been the sole recipients of grant funds regardless of performance. OMB recommended that DOL award national grants competitively to strengthen service delivery and open the door to new grantees. As OMB and others recognize, performance is not the only factor in funding decisions. Determining priorities--including funding priorities--is a function of competing values and interests. As seen in figure 2, we found that PART scores were generally positively related to proposed funding changes in discretionary programs but not in a mechanistic way. In other words, PART scores did not automatically determine funding changes. OMB proposed funding increases for most of the programs rated "effective" or "moderately effective" and proposed funding decreases for most of the programs rated "ineffective," but there were clear exceptions. Programs rated as "results not demonstrated"--which reflected a range of PART scores--had mixed results. Figure 2: Number of Discretionary PART Programs by Rating and Funding Result, Fiscal Years 2003-2004: [See PDF for image] Note: Discretionary programs refer to those programs with budgetary resources provided in appropriation acts. Because Congress controls spending for mandatory programs--generally entitlement programs such as food stamps, Medicare, and veterans' pensions--indirectly rather than directly through the appropriations process, we excluded them from our analysis. Of the 234 programs, we could not classify 11 as being either predominantly mandatory or discretionary; these programs are excluded from our analysis as well, and are listed in appendix I. [End of figure] A large portion of the variability in proposed budget changes could not be explained by the quantitative measures reported by the PART. Regressions of PART scores never explained more than about 15 percent of the proposed budget changes. For only the one-third of discretionary programs with the smallest budgets, we found that the composite PART scores had a modest but statistically significant effect on proposed budget changes (measured in percentage change) between fiscal years 2003 and 2004. For a fuller discussion of the statistical methods used, see appendix I. The relationship between performance levels and budget decisions was not one-dimensional. For example, OMB rated the Department of Defense's Basic Research program as "effective," but recommended a reduction in congressionally earmarked projects that it stated did not meet the program's merit review process. OMB also recommended reducing funding for DOE's International Nuclear Materials Protection and Cooperation program (rated "effective") because difficulties in obtaining international agreements had resulted in the availability of sufficient unobligated balances[Footnote 11] to make new funding unnecessary. However, OMB sometimes proposed funding increases for programs that were rated "ineffective" to implement improvement plans that had been developed, such as the Internal Revenue Service's new Earned Income Tax Credit compliance initiatives and DOE's revised environmental cleanup plans for its Environmental Management (Cleanup) program. Capacity Issues Could Affect OMB's Ability to Use the PART to Drive Program Improvements: OMB has said that a major purpose of the PART is to focus on program improvements and measure progress. Effectively implementing PART recommendations aimed at program improvements will require sustained attention and sufficient oversight of agencies to ensure that the recommendations are producing desirable results. However, each year, the number of recommendations from previous years' evaluations will grow. Currently, OMB plans to assess an additional 20 percent of all federal programs annually such that all programs would eventually be reviewed over a 5-year period. OMB encouraged its RMOs to consider a variety of factors in selecting programs for the fiscal year 2004 PART assessments, including continuing presidential initiatives and programs up for reauthorization. Strengthening the focus on selecting related programs for review in a given year would enable decision makers to analyze the relative efficacy of similar programs in meeting common or similar outcomes. As our work has shown, unfocused and uncoordinated programs waste scarce funds, confuse and frustrate program customers, and limit overall program effectiveness. Therefore it is prudent to highlight crosscutting program efforts and clearly relate and address the contributions of alternative federal strategies toward meeting similar goals. Although OMB has created a template for agencies to report on the status of their recommendations and has reported that agencies are implementing their PART recommendations, OMB has no central system for monitoring agency progress or evaluating the effectiveness of changes. While RMOs are responsible for overseeing agency progress, OMB senior managers will not have a comprehensive governmentwide picture of progress on the implementation of PART recommendations, nor will they have a complete picture of OMB's workload in this area. As OMB has recognized, following through on the recommendations is essential for improving program performance and ensuring accountability. Senior OMB managers readily recognized the increased workload the PART placed on examiners--in one public forum we attended, a senior OMB official described many examiners as being very concerned about the additional workload. However, OMB expects the workload to decline as OMB and agency staff become more familiar with the PART tool and process, and as issues with the timing of the PART reviews are resolved. Agency officials told us that originally, there was no formal guidance for reassessing PART programs--it varied by RMO. When issued, OMB's formal PART guidance limited reassessments to (1) updating the status/implementation of recommendations from the fiscal year 2004 PART and (2) revisiting specific questions for which new evidence exists. OMB expected that in most reassessments, only those questions in which change could be demonstrated would be "reopened." OMB officials acknowledged that this formal guidance is at least partly due to resource constraints. OMB staff were divided on whether the PART assessments made an appreciable difference in time spent on its budget review process. Many of those we spoke with told us that their workloads during the traditional budget season have always been heavy and that PART did not add significantly to their work, especially since the PART generally formalized a process already taking place. Those who did acknowledge workload concerns said that they were surprised at the amount of time it was taking to reassess programs. In fact, more than one OMB official told us that reassessing programs was taking almost as long as brand- new assessments, despite the fact that OMB scaled back the scope of these reassessments. Despite OMB's Considerable Efforts to Create a Credible Evaluation Tool, PART Assessments Require Judgment and Were Constrained by Data Limitations: OMB went to great lengths to encourage consistent application of the PART in the evaluation of government programs, including pilot testing the instrument, issuing detailed guidance, and conducting consistency reviews. However, while the instrument can undoubtedly be improved, any tool that is sophisticated enough to take into account the complexity of the U.S. government will always require OMB staff to exercise interpretation and judgment. Providing flexibility to assess multidimensional programs with multiple purposes and impacts has led to a reliance on OMB staff judgments to apply general principles to specific cases. Accordingly, OMB staff were not fully consistent in interpreting complex questions about agency goals and results. In addition, the limited availability of credible evidence on program results also constrained OMB's ability to use the PART to rate programs' effectiveness. Inherent Performance Measurement Challenges Make It Difficult to Meaningfully Interpret a Bottom-Line Rating: OMB published a single, bottom-line rating for the PART results as well as individual section scores, which are potentially more useful for identifying information gaps and program weaknesses. For example, one program that was rated "adequate" overall got high scores for purpose (80 percent) and planning (100 percent), but did poorly in being able to show results (39 percent) and in program management (46 percent). Thus, the individual section ratings provided a better understanding of areas needing improvement than the overall rating alone. Bottom-line ratings inevitably force choices on what best exemplifies a program's mission--even when a program has multiple goals--and encourages a determination of the effectiveness of the program even when performance data are unavailable, the quality of those data is uneven, or they convey a mixed message on performance. Many of the outcomes for which federal programs are responsible are part of a broader effort involving federal, state, local, nonprofit, and private partners. We have previously reported that it is often difficult to isolate a particular program's contribution to an outcome and especially so when it involves third parties.[Footnote 12] This was reinforced by the results of the fiscal year 2004 PART reviews. One of the patterns that OMB identified in its ratings was that grant programs received lower than average ratings. To OMB this suggested the need for greater effort by agencies to make grantees accountable for achieving overall program results. However, grant structure and design play a role in how federal agencies are able to hold third parties responsible and complicate the process of identifying the individual contributions of a federal program with multiple partners. In particular, block grants present implementation challenges, especially in those instances in which national goals are not compatible with state and local priorities. OMB Employed Numerous Tools and Techniques to Promote and Improve Consistent Application of the PART: OMB went to great lengths to encourage consistent application of the PART in the evaluation of government programs. These efforts included (1) testing the PART in selected agencies before use in the fiscal year 2004 assessment, (2) issuing detailed guidance and worksheets for use by PART teams, (3) making the Performance Evaluation Team (PET) available to answer PART implementation questions, (4) establishing an Interagency Review Panel (IRP) to review consistency of PART evaluations, and (5) making improvements to the fiscal year 2005 process and guidance based upon the fiscal year 2004 experience. OMB conducted a pilot test of the PART and released a draft of the PART questionnaire for public comment prior to its use for the fiscal year 2004 budget cycle. During Spring Review in 2002, OMB and agency staff piloted the draft PART on 67 programs. The PART was also shared with and commented on by the Performance Measurement Advisory Council and other external groups. According to OMB, the results of the Spring Review and feedback from external groups were used to revise the draft version of the PART to lessen subjectivity and increase the consistency of reviews. OMB issued detailed guidance to help OMB and agency staff consistently apply the PART and created electronic "templates" or worksheets to aid in completing PART assessments. This guidance explains the purpose of each question and describes the evidence required to support a "yes" or "no" answer. In order to account for different types of programs, several questions tailored to the seven program types were added to the PART (primarily in Section III--Program Management). While the PART guidance cannot be expected to cover every situation, the instructions established general standards for PART evaluations. PET addressed in "real time" questions and issues that OMB staff that were completing the PART evaluations repeatedly raised. PET consisted of examiners drawn from across the OMB organization representing a variety of programmatic knowledge and experiences. It served as a sounding board for OMB staff and a source for sharing experiences, issues, and useful approaches and also provided training to OMB and agency staff on the process. For example, in one OMB branch, staff were grappling with how to apply the PART to a set of block grants. They went through the instrument with the PET member from their RMO and continued to consult with that individual throughout the process. OMB also formed IRP, which consisted of both OMB and agency officials, to conduct a consistency check of the PART reviews and to review formal appeals of the process or results for particular questions. During the fiscal year 2004 budget process, IRP conducted a consistency review of 10 percent of the PART evaluations using a subset of the PART questions that OMB staff identified as being the most subjective or difficult to interpret. IRP also reviewed formal agency appeals to determine whether there was consistent treatment of similar situations. As an Evaluation Tool, the PART Has Weaknesses in Its Design and, as a Result, Its Implementation: Despite the considerable time and effort OMB has devoted to promoting consistent application of the PART, difficulties both with the tool itself (such as subjective terminology and a restrictive yes/no format) and with implementing the tool (including inconsistencies in defining acceptable measures and contradictory answers to "pairs" of related questions) aggravated the general performance measurement challenges described earlier. Subjective Terms and a Restrictive Format Contributed to Subjective and Inconsistent Responses: Many PART questions contain subjective terms that are open to interpretation. Examples include terminology such as "ambitious" in describing sought-after performance measures. Because the appropriateness of a performance measure depends on the program's purpose, and because program purposes can vary immensely, an ambitious goal for one program might be unrealistic for a similar but more narrowly defined program. Some agency officials claimed that having multiple statutory goals disadvantaged their programs. Without further guidance, subjective terminology can influence program ratings by permitting OMB staff's views about a program's purpose to affect assessments of the program's design and achievements. Although OMB employed a yes/no format for the PART because OMB believes it aided standardization, the format resulted in oversimplified answers to some questions. OMB received comments on the yes/no format in conducting the PART pilot. Some parties liked the certainty and forced choice of yes/no. Others felt the format did not adequately distinguish between the performance of various programs, especially in the results section (originally in the yes/no format). In response to these concerns, OMB revised the PART in the spring of 2002 to include four response choices in the results section (adding "small extent" and "large extent" to the original two choices "yes" and "no"), while retaining the dichotomous yes/no format in the other three sections. OMB acknowledged that a "yes" response should be definite and reflect a very high standard of performance, and that it would more likely be difficult to justify a "yes" answer than a "no" answer. Nonetheless, agency officials have commented that the yes/no format is a crude reflection of reality, in which progress in planning, management, or results is more likely to resemble a continuum than an on/off switch. Moreover, the yes/no format was particularly troublesome for questions containing multiple criteria for a "yes" answer. As discussed previously, we conducted an in-depth analysis of PART assessments for 28 related programs in nine clusters and compared the responses to related questions. That analysis showed six instances in which some OMB staff gave a "yes" answer for successfully achieving some but not all of the multiple criteria, while others gave a "no" answer when presented with a similar situation. For example, Section II, Question 1, asks, "Does the program have a limited number of specific, ambitious, long-term performance goals that focus on outcomes and meaningfully reflect the purpose of the program?" The PART defines successful long-term goals by multiple, distinct characteristics (program has long-term goals, time frames by which the goals are to be achieved, etc.), but does not clarify whether a program can receive a "yes" if each of the characteristics is met, or if most of the characteristics are met. This contributed to a number of inconsistencies across program reviews. For example, OMB judged DOI's Water Reuse and Recycling program "no" on this question, noting that although DOI set a long-term goal of 500,000 acre-feet per year of reclaimed water, it failed to establish a time frame for when it would reach the target. However, OMB judged the Department of Agriculture's and DOI's Wildland Fire programs "yes" on this question even though the programs' long-term goals of improved conditions in high-priority forest acres are not accompanied by specific time frames. In another example, OMB accepted DOD's recently established long-term strategic goals for medical training and provision of health care even though it did not yet have measures or targets for those goals. By breaking out targets and ambitious time frames separately from the question of annual goals, agencies have an opportunity to get credit for progress made. There Were Inconsistencies in Defining Acceptable Measures and in Logically Responding to Question "Pairs": In particular, our analysis of the nine program clusters revealed three instances in which OMB staff inconsistently defined appropriate measures--outcome versus output--for programs. Officials also told us that OMB staff used different standards to define measures as outcome oriented. This may reflect, in part, the complexity of and relationship between expected program benefits. Outcomes are generally defined as the results of outputs--products and services--delivered by a program. But in some programs, long-term outcomes are expected to occur over time through multiple steps. In these cases, short-term outcomes-- immediate changes in knowledge and awareness--might be expected to lead to intermediate outcomes--behavioral changes in the future--and eventually result in long-term outcomes--benefits to the public. In the employment and training area, OMB accepted short-term outcomes, such as obtaining high school diplomas or employment, as a proxy for long-term goals for the HHS Refugee Assistance program, which aims to help refugees attain economic self-sufficiency as soon as possible after they arrive. However, OMB did not accept the same employment rate measure as a proxy for long-term goals for the Department of Education's Vocational Rehabilitation program because it had not set long-term targets beyond a couple of years. In other words, although neither program contained long-term outcomes, such as participants gaining economic self-sufficiency, OMB accepted short-term outcomes in one instance but not the other. Similarly, OMB gave credit for output measures of claims processing (time, accuracy, and productivity) as a proxy for long-term goals for the Social Security Administration's Disability Insurance program, but did not accept the same output measures for the Veterans Disability Compensation program. OMB took steps to address this issue for fiscal year 2005. We also found that three "question pairs" on the PART worksheets are linked, yet in two of the three "pairs," a disconnect appeared in how OMB staff responded to these questions for a given program.[Footnote 13] For example, 29 of the 90 programs (32 percent) judged as lacking "independent and quality evaluations of sufficient scope conducted on a regular basis" (Section II, Question 5) were also judged as having "independent and quality evaluations that indicated the program is effective and achieving results" (Section IV, Question 5). There is a logical inconsistency in these two responses. In another instance, there was no linkage between the questions that examine whether a program has annual goals that demonstrate progress toward achieving long-term goals and whether the program actually achieves its annual goals. For example, 15 of the 75 programs (20 percent) judged not to have adequate annual performance goals (Section II, Question 2) were nevertheless credited for having made progress on their annual performance goals (Section IV, Question 2). However, the guidance for the latter question clearly indicates that a program must receive a "no" if it received a "no" on the existence of annual goals (Section II, Question 2). It seems that some raters held programs to a higher standard for the quality of goals than for progress on them. The Lack of Performance Information Creates Challenges in Effectively Measuring Program Performance: According to OMB, 115 out of 234 programs (49 percent) lacked "specific, ambitious, long-term performance goals that focus on outcomes" (Section II, Question 1). In addition, OMB found that 90 out of 234 programs (38 percent) lacked sufficient "independent, quality evaluations" (Section II, Question 5). While the validity of these assessments may be subject to interpretation and debate, our previous work[Footnote 14] has raised concerns about the capacity of federal agencies to produce evaluations of program effectiveness. The lack of evaluations may in part be driven by how OMB defined an "independent and quality evaluation." To be independent, nonbiased parties with no conflict of interest would conduct the evaluation, but agency officials felt that OMB staff started from the default position that agency-sponsored evaluations are, by definition, biased. However, our detailed review of 28 PART worksheets found only 7 instances in which OMB explicitly noted its rejection of evaluations: 1 for being too old, 3 for not being independent (of the 3, 1 was an internal agency review and 2 were conducted by industry groups), and the remaining 3 for not assessing program results. OMB officials have acknowledged that this issue was a point of friction with agencies and that beyond GAO, inspectors general, and other government reports that were automatically presumed to be independent, the independence standard was considered on a case-by-case basis. In these case-by-case situations, OMB staff told us that they looked for some degree of detachment and objectivity in the evaluations. For example, in the case of one DOE-sponsored evaluation, the OMB examiner attended the meetings of the review group that conducted the evaluation in order to see firsthand what sorts of questions the committee posed to the department officials. In OMB's estimation, there was clear independence. While OMB changed the fiscal year 2005 guidance to recognize evaluations contracted out to third parties and agency program evaluation offices as possibly being sufficiently independent, the new guidance generally prohibits evaluations conducted by the program itself from being considered "independent.": Other reasons evaluation data may be limited include (1) constraints on federal agencies' ability to influence program outcomes and reliance on states and others for data for programs for which responsibility has devolved to the states and (2) the lack of a statutory mandate or dedicated funds for evaluation, which agency officials told us can hamper efforts to conduct studies or to improve administrative data collection. As we have previously noted, program evaluations can take many forms and agencies may obtain evaluations in a variety of ways.[Footnote 15] Some evaluations simply analyze routinely collected program administrative data; others involve special surveys. The type of evaluation can greatly affect evaluation cost. Net impact evaluations compare outcomes for program participants to those of a randomly assigned control group and are designed for situations in which external factors are also known to influence those outcomes. However, the adequacy of an evaluation design can only be determined relative to the circumstances of the program being evaluated. In addition, agencies can obtain evaluations by having program or other agency staff collect and analyze the data, by conducting the work jointly with program partners (such as state agencies), or by hiring contract firms to do so. Our survey of 81 federal agency offices conducting evaluations in 1995 of program results found they were most commonly located in administrative offices at a major subdivision level or in program offices (43 and 30 percent, respectively). Overall, they reported conducting 51 percent of their studies in-house, while 34 percent were contracted out. Depending on the sensitivity of the study questions, agencies can conduct credible internal evaluations by adopting procedures to ensure the reliability and validity of data collection and analysis. Disagreements on Performance Information Led to Creation of a "Results Not Demonstrated" Category: During the PART process OMB created an additional rating category, "results not demonstrated," which was applied to programs regardless of their score if OMB decided that one or both of two conditions pertained: (1) OMB and the agency could not reach agreement on long- term and annual performance measures and (2) there was inadequate performance information. Almost 50 percent of the 234 programs assessed for fiscal year 2004 received this rating of "results not demonstrated," ranging from high-scoring programs such as the Consumer Product Safety Commission (83) to low-scoring programs such as the Department of Veterans Affairs Disability Compensation program (15). OMB officials said that this rating was given to programs when agreement could not be reached on long-term and annual performance measures and was applied regardless of the program's PART score. Our own review found that OMB generally assigned the "results not demonstrated" rating as described above.[Footnote 16] It is important for users of the PART information to interpret the "results not demonstrated" designation as "unknown effectiveness" rather than as meaning the program is "ineffective." Having evidence of poor results is not the same as lacking evidence of effectiveness. Because the PART guidance sets very high standards for obtaining a "yes," a "no" answer can mean either that a program did not meet the standards, or that there is no evidence on whether it met the standards. In some readily measured areas, lack of evidence of an action may indicate that the standard probably was not met. However, because effectiveness is often not readily observed, lack of evidence on program effectiveness cannot be automatically interpreted as meaning that a program is ineffective. Furthermore, an agency might have results for goals established under GPRA, but if OMB and the agency could not reach agreement on new or revised goals or measures, then OMB gave a program the rating "results not demonstrated.": Changes to the PART and Related Guidance for Fiscal Year 2005 Are Meant to Address Previously Identified Problems: OMB, recognizing many of the issues we have just discussed, made modifications to the PART instrument and guidance in time for the fiscal year 2005 process. OMB said these changes were based upon lessons learned during the fiscal year 2004 process and input from a variety of sources, such as PET, IRP, and agency officials, although we were unable to determine which changes resulted from which recommendations. Although the PART as used for fiscal year 2005 is very similar to that for fiscal year 2004, several questions were added, dropped, merged with other questions, or divided into two questions. For example, a research and development question used in the fiscal year 2004 PART that received "not applicable" answers in 13 out of the 32 cases in which it was applied was dropped from the fiscal year 2005 PART. According to OMB officials, several of the multicriteria questions were split into separate questions in order to reduce inconsistency, as described earlier in this report. Appendix II provides more complete information on the guidance changes between fiscal years 2004 and 2005. To complement the fiscal year 2005 PART guidance and offer strategies for addressing common performance measurement challenges, many of which were encountered during the fiscal year 2004 process, OMB released a separate document, titled Performance Measurement Challenges and Strategies, which was the result of a workshop in which agencies participated and identified measurement challenges and shared best practices and possible work-arounds. Instead of reestablishing IRP (which included both agency and OMB representatives) for the fiscal year 2005 process, OMB officials told us that PET (which included only OMB representatives) would conduct a consistency review of 25 percent of all PART evaluations, with at least one consistency check per OMB branch. OMB also told us that it has asked the National Academy of Public Administration (NAPA) to review PET's consistency review for the fiscal year 2005 process; the scope and results of that review were not available to us during our audit work.[Footnote 17] OMB senior officials cited resources, timing, and the differing needs of the fiscal year 2004 and 2005 PART processes as reasons for dropping the IRP review. The absence of agency participation in this important phase of the PART could hamper ensuring crucial transparency and credibility. The Fiscal Year 2004 PART Process Was a Parallel, Competing Approach to GPRA's Performance Management Framework: The PART was designed for and is used in the executive branch budget preparation and review process; as such, the goals and measures used in the PART must meet OMB's needs. However, GPRA--the current statutory framework for strategic planning and reporting--is a broader process involving the development of strategic and performance goals and objectives to be reported in strategic and annual plans. OMB's desire to collect performance data that better align with budget decision units means that the fiscal year 2004 PART process was a parallel competing structure to the GPRA framework. Although OMB acknowledges that GPRA was the starting point for the PART, as we explain below, the emphasis is shifting such that over time the performance measures developed for the PART and used in the budget process may come to drive agencies' strategic planning processes. Agencies told us that in some cases, OMB is replacing PART goals and measures for those of GPRA. Effective for fiscal year 2005, OMB's Circular A-11 guidance states that performance budgets are to replace GPRA's annual performance plans. Agencies see the change as detrimental to planning and reporting under GPRA and as a resource drain since they have to respond to both GPRA and PART requirements. Some agency officials told us that although the PART can stimulate discussion on program-specific performance measurement issues, it is not a substitute for GPRA's outcome-oriented, strategic look at thematic goals and departmentwide program comparisons. Moreover, while the PART does not eliminate the departmental strategic plans created under GPRA, many OMB and agency officials told us that the PART is being used to shape the strategic plans. OMB's Efforts to Link Performance Information with the Budget Often Conflict with Agencies' GPRA Planning Efforts: OMB guidance and officials made clear that GPRA goals, measures, and reports needed to be modified to provide decision makers with program- specific, outcome-based performance data that better aligned with the budget presentation in the President's Budget. According to OMB, such changes were needed because performance reporting under GPRA had evolved into a process separate from budget decision making, with GPRA plans organized at too high a level to be meaningful for program-level budget analysis and management review. Furthermore, according to OMB officials, GPRA plans had too many performance measures, which made it difficult to determine an agency's priorities. However, as some officials pointed out, the cumulative effect of adding new PART measures to GPRA plans may actually increase the number of measures overall; both agency and OMB officials recognize that this is contrary to goals issued by an OMB official previously responsible for the PART, indicating his desire to reduce the number of GPRA measures by at least 25 percent in at least 70 percent of federal departments.[Footnote 18] As a result of these sometimes-conflicting perspectives, agency officials said that responding to both PART and GPRA requirements increased their workloads and was a drain on staff resources. OMB's most recent Circular A-11 guidance clearly requires that each agency submit a performance budget for fiscal year 2005 and that this should replace the annual GPRA performance plan.[Footnote 19] These performance budgets are to include information from the PART assessments, where available, including all performance goals used in the assessment of program performance done under the PART process. Until all programs have been assessed using the PART, the performance budget will also include performance goals for agency programs that have not yet been assessed using the PART. OMB's movement from GPRA to PART is further evident in the fiscal year 2005 PART guidance stating that while existing GPRA performance goals may be a starting point during the development of PART performance goals, the GPRA goals in agency GPRA documents are to be revised significantly, as needed, to reflect OMB's instructions for developing the PART performance goals. Lastly, this same guidance states that GPRA plans should be revised to include any new performance measures used in the PART and unnecessary measures should be deleted from GPRA plans. OMB's interest in developing more useful program goals is further evident in its PART recommendations. Almost half of the fiscal year 2004 PART recommendations related to performance assessment-- developing outcome goals and measures; cost or efficiency measures; and increasing the tracking/monitoring of data, improving the tracking/ monitoring of data, or both. GPRA was generally the starting point for PART discussions about goals and measures, and many agency officials told us that OMB used the PART to modify agencies' existing GPRA goals and measures. Agency officials reported that the discussions about goals and measures were one of the main areas of contention during the PART process. At the same time, agency officials acknowledged that (1) sometimes OMB staff accepted current GPRA measures and (2) sometimes the new PART measures and goals were improvements over the old GPRA measures--the PART measures were more aggressive, more outcome- oriented, more targeted, or all of the above. Defining a "Unit of Analysis" That Is Useful for Program-Level Budget Analysis and Agency Planning Purposes Presents Challenges: The appropriate unit of analysis or "program" is not always obvious. What OMB determined was useful for a PART assessment did not necessarily match agency organization or planning elements. Although the units of analysis varied across the PART assessments, OMB's guidance stated that they should be linked to a recognized funding level in the budget. In some cases, OMB aggregated separate programs for the purposes of the PART, while in other cases it disaggregated programs. Aggregating programs to tie them to discrete funding levels sometimes made it difficult to create a limited, but comprehensive, set of measures for programs with multiple missions. Disaggregating programs sometimes ignored the interdependence of programs by artificially isolating programs from the larger contexts in which they operate. Both contributed to the lack of available planning and performance information. For example, aggregating rural water supply projects as a single unit of analysis may have been a logical choice for reviewing related activities, but it created problems in identifying planning and performance information useful for the PART since these projects are separately administered. In another case, HHS officials told us that the PART program Substance Abuse Treatment Programs of Regional and National Significance is an amalgamation of activities funded in a single budget line, not an actual program. They said it was a challenge to make these activities look as if they functioned as a single program. Disaggregating a program too narrowly can create problems by distorting its relationship to other programs involved in achieving a common goal. For example, agency officials described a homeless program in which outreach workers help homeless persons with emergency needs and refer them to other agencies for housing and needed services. They said that their OMB counterparts suggested that the program adopt long-term outcome measures indicating number of persons housed. Agency officials argued that chronically homeless people require many services and that this federal program often supports only some of the services needed at the initial stages of intervention. The federal program, therefore, could contribute to, but not be primarily responsible for, affecting late stages of the intervention process, like housing status. These issues reveal some of the unresolved tensions between the President's budget and performance initiative--a detailed budget perspective--and GPRA--a more strategic planning view. In particular, agency officials are concerned with problems in trying to respond to both and overwhelmingly agreed that the PART required a large amount of agency resources to complete. Moreover, some agency officials said that the PART (a program-specific review) is not well suited to one of the key purposes of strategic plans--to convey agencywide, long-term goals and objectives for all major functions and operations. In addition, the time horizons are different for the two initiatives--PART assessments focus on program accomplishments to date while GPRA strategic planning is long-term and prospective in nature. Changes Made to GPRA in the PART Process Create Uncertainty About Opportunities for Substantive Input by Interested Parties and Congressional Stakeholders: As noted above, PART goals and measures must meet OMB's needs, while GPRA is a broader process involving the development of strategic and performance goals and objectives to be reported in strategic and annual plans. As a phased reform, GPRA required development of the planning framework first, but also explicitly encouraged links to the budget.[Footnote 20] Our work has shown that under GPRA agencies have made significant progress.[Footnote 21] Additionally, GPRA requires agencies to consult with Congress and solicit the views of other stakeholders as they develop their strategic plans.[Footnote 22] We have previously reported[Footnote 23] that stakeholder involvement appears critical for getting consensus on goals and measures. Stakeholder involvement can be particularly important for federal agencies because they operate in a complex political environment in which legislative mandates are often broadly stated and some stakeholders may strongly disagree about the agency's mission and goals. The relationship between the PART and its process and the broader GPRA strategic planning process is still evolving. Some tension between the level of stakeholder involvement in the development of performance measures in the GPRA strategic planning process and the process of developing performance measures for the PART is inevitable. Compared to the relatively open-ended GPRA process any budget formulation process is likely to seem closed. An agency's communication with stakeholders, including Congress, about goals and measures created or modified during the formulation of the President's budget is likely to be less than during the development of the agency's own strategic or performance plan. Since different stakeholders have different needs and no one set of goals and measures can serve all purposes, the PART can complement GPRA but should not replace it. Although these tensions between the need for internal deliberations and broader consultations are inevitable, if the PART is to be accepted as a credible element in the development of the President's budget proposal, congressional understanding and acceptance of the tool and its analysis will be important. In order for performance information to more fully inform resource allocations, decision makers must also feel comfortable with the appropriateness and accuracy of the performance information and measures associated with these goals. It is unlikely that decision makers will use performance information unless they believe it is credible and reliable and reflects a consensus about performance goals among a community of interested parties. Similarly, the measures used to demonstrate progress toward a goal, no matter how worthwhile, cannot serve the interests of a single stakeholder or purpose without potentially discouraging use of this information by others. While it is still too soon to know whether OMB-directed measures will satisfy the needs of other stakeholders and GPRA's broader planning purposes, several appropriations subcommittees have stated, in their appropriations hearings, the need to link the PART with congressional oversight. For example, the House Committee on Appropriations, Subcommittee on the Department of the Interior and Related Agencies notes that while it supports the PMA, the costs of initiatives associated with it have generally not been requested in annual budget justifications or through reprogramming procedures.[Footnote 24] The Subcommittee, therefore, has been unable to evaluate the costs, benefits, and effectiveness of these initiatives or to weigh the priority that these initiatives should receive as compared with ongoing programs funded in the Interior Appropriations bill. Similarly, the House Report on Treasury and Transportation Appropriations included a statement in support of the PART, but noted that the administration's efforts must be linked with the oversight of Congress to maximize the utility of the PART process, and that if the administration treats as privileged or confidential the details of its rating process, it is less likely that Congress will use those results in deciding which programs to fund. Moreover, the Subcommittee said it expects OMB to involve the House and Senate Committees on Appropriations in the development of the PART ratings at all stages in the process.[Footnote 25] While Congress has a number of opportunities to provide its perspective on performance issues and performance goals, such as when it establishes or reauthorizes a new program, during the annual appropriations process, and in its oversight of federal operations, opportunities exist for Congress to more systematically articulate performance goals and outcomes for key programs of major concern and to allow for timely congressional input in the selection of the PART programs to be assessed. Conclusions and General Observations: OMB, through its development and use of the PART, has more explicitly infused performance information into the budget formulation process; increased the attention paid to evaluation and performance information; and ultimately, we hope, increased the value of this information to decision makers and other stakeholders. By linking performance information to the budget process, OMB has provided agencies with a powerful incentive for improving data quality and availability. The level of effort and involvement by senior OMB officials and staff clearly signals the importance of this strategy in meeting the priorities outlined in the PMA. OMB should be credited with opening up for scrutiny--and potential criticism--its review of key areas of federal program performance and then making its assessments available to a potentially wider audience through its Web site. While the PART clearly serves the needs of OMB in budget formulation, questions remain about whether it serves the needs of other key stakeholders. The PART could be strengthened to enhance its credibility and prospects for sustainability by such actions as (1) improving agencies' and OMB's capacity to cope with the demands of the PART, (2) strengthening the PART guidance, (3) expanding the base of credible performance information by strategically focusing evaluation resources, (4) selecting programs for assessment to facilitate crosscutting comparisons and trade-offs, (5) broadening the dialogue with congressional stakeholders, and (6) articulating and implementing a complementary relationship between PART and GPRA. OMB's ambitious schedule for assessing all federal programs by the fiscal year 2008 President's Budget will require a tremendous commitment of OMB's and agencies' resources. Implementation of the PART recommendations will be a longer-term and potentially more significant result of the PART process than the scores and ratings. No less important will be OMB's involvement both in encouraging agency progress and in signaling its continuing commitment to improving program management and results through the PART. OMB has created a template by which agencies report on the status of the recommendations and left follow-up on the recommendations to each RMO. However, there is no single focal point for evaluating progress and the results of agency efforts governmentwide; without this it will be difficult for OMB to judge the efficacy of the PART and to know whether the increased workload and trade-offs made with other activities is a good investment of OMB and agency resources. The goal of the PART is to evaluate programs systematically, consistently, and transparently, but in practice, the tool requires OMB staff to use independent judgment in interpreting the guidance and in making yes or no decisions for what are often complex federal programs. These difficulties are compounded by poor or partial program performance data. Therefore, it is not surprising that we found inconsistencies in our analysis of the fiscal year 2004 PART assessments. Recognizing the inherent limitations of any tool to provide a single performance answer or judgment on complex federal programs with multiple goals, continued improvements in the PART guidance, with examples throughout, can nonetheless help encourage a higher level of consistency as well as transparency. The PART requires more performance and evaluation information than agencies currently have, as demonstrated by the fact that OMB rated over 50 percent of the programs for fiscal year 2004 as "results not demonstrated" because they "did not have adequate performance goals" or "had not yet collected data to provide evidence of results." In the past, we too have noted limitations in the quality of agency performance and evaluation information and in agency capacity to produce rigorous evaluations of program effectiveness. Furthermore, our work has shown that few agencies deployed the rigorous research methods required to attribute changes in underlying outcomes to program activities. However, program evaluation information often requires large amounts of agency resources to produce, and the agency and OMB may not agree on what is important to measure, particularly when a set of measures cannot serve multiple purposes. Agreement on what are a department or agency's critical, high-risk programs and how best to evaluate them could help leverage limited resources and help determine what are the most important program evaluation data to collect. Federal programs are designed and implemented in dynamic environments where competing program priorities and stakeholders' needs must be balanced continually and new needs must be addressed. GPRA is a broad legislative framework that was designed to be consultative with Congress and other stakeholders and allows for varying uses of performance information, while the PART applies evaluation information to support decisions and program reviews during the executive budget formulation process. While the PART reflects the administration's management principles and the priority given to using performance information in OMB's decision-making process, its focus on program- level assessments cannot substitute for the inclusive, crosscutting strategic planning required by GPRA. Moreover, GPRA can anchor the review of programs by providing an overall strategic context for programs' contributions toward agency goals. Although PART and GPRA serve different needs, a strategy for integrating the two could help strengthen both. Opportunities exist to develop a more strategic approach to the selection and prioritization of areas to be assessed under the PART process. Targeting PART assessments based on such factors as the relative priorities, costs, and risks associated with related clusters of programs and activities could not only help ration scarce analytic resources but could also focus decision makers' attention on the most pressing policy and program issues. Moreover, such an approach could facilitate the use of PART assessments to review the relative contributions of similar programs to common or crosscutting goals and outcomes. As part of the President's budget preparation, the PART clearly must serve the President's interests. However, it is unlikely that the broad range of actors whose input is critical to decisions will use performance information unless they believe it is credible and reliable and reflects a consensus about performance goals among a community of interested parties. Similarly, the measures used to demonstrate progress toward a goal, no matter how worthwhile, cannot appear to serve a single set of interests without potentially discouraging use of this information by others. If the President or OMB wants the PART and its results to be considered in the congressional debate, it will be important for OMB to (1) involve congressional stakeholders early in providing input on the focus of the assessments; (2) clarify any significant limitations in the assessments as well as the underlying performance information; and (3) initiate discussions with key congressional committees about how they can best take advantage of and leverage PART information in authorizations, appropriations, and oversight processes. As we have previously reported, effective congressional oversight can help improve federal performance by examining the program structures agencies use to deliver products and services to ensure that the best, most cost-effective mix of strategies is in place to meet agency and national goals. While Congress has a number of opportunities to provide its perspective on performance issues and performance goals, such as when it establishes or reauthorizes a new program, during the annual appropriations process, and in its oversight of federal operations, a more systematic approach could allow Congress to better articulate performance goals and outcomes for key programs of major concern. Such an approach could also facilitate OMB's understanding of congressional priorities and concerns and, as a result, increase the usefulness of the PART in budget deliberations. Matter for Congressional Consideration: In order to facilitate an understanding of congressional priorities and concerns, we suggest that Congress consider the need for a strategy that could include (1) establishing a vehicle for communicating performance goals and measures for key congressional priorities and concerns; (2) developing a more structured oversight agenda to permit a more coordinated congressional perspective on crosscutting programs and policies; and (3) using such an agenda to inform its authorization, oversight, and appropriations processes. Recommendations for Executive Action: We have seven recommendations to OMB for building on and improving the first year's experience with the PART and its process. We recommend that the Director of OMB take the following actions: * Centrally monitor agency implementation and progress on PART recommendations and report such progress in OMB's budget submission to Congress. Governmentwide councils may be effective vehicles for assisting OMB in these efforts. * Continue to improve the PART guidance by (1) expanding the discussion of how the unit of analysis is to be determined to include trade-offs made when defining a unit of analysis, implications of how the unit of analysis is defined, or both; (2) clarifying when output versus outcome measures are acceptable; and (3) better defining an "independent, quality evaluation.": * Clarify OMB's expectations to agencies regarding the allocation of scarce evaluation resources among programs, the timing of such evaluations, as well as the evaluation strategies it wants for the purposes of the PART, and consider using internal agency evaluations as evidence on a case-by-case basis--whether conducted by agencies, contractors, or other parties. * Reconsider plans for 100 percent coverage of federal programs and, instead, target for review a significant percentage of major and meaningful government programs based on such factors as the relative priorities, costs, and risks associated with related clusters of programs and activities. * Maximize the opportunity to review similar programs or activities in the same year to facilitate comparisons and trade-offs. * Attempt to generate, early in the PART process, an ongoing, meaningful dialogue with congressional appropriations, authorization, and oversight committees about what they consider to be the most important performance issues and program areas warranting review. * Seek to achieve the greatest benefit from both GPRA and PART by articulating and implementing an integrated, complementary relationship between the two. Agency Comments: We provided a draft of this report to OMB for its review and comment. OMB generally agreed with our findings, conclusions, and recommendations. In addition, OMB outlined actions it is taking to address many of our recommendations, including refining the process for monitoring agencies' progress in implementing the PART recommendations, seeking opportunities for dialogue with Congress on agencies' performance, and continuing to improve executive branch implementation of GPRA plans and reports. OMB officials provided a number of technical comments and clarifications, which we incorporated as appropriate to ensure the accuracy of our report. OMB's comments appear in appendix IV. We also received technical comments on excerpts of the draft provided to the Departments of the Interior, Energy, and Health and Human Services. Comments received from the Departments of Energy and the Interior were incorporated as appropriate. The Department of Health and Human Services had no comments. OMB noted that performance information gleaned from the PART process has not only informed budget decisions but has also helped direct program management, identified opportunities to improve program design, and promoted accountability. We agree. As shown in figure 1 in our report, we found that 82 percent of PART recommendations addressed program assessment, design, and management issues; only 18 percent of the recommendations had a direct link to funding matters. We are sending copies of this report to the Director of OMB, appropriate congressional committees, and other interested members of Congress. We will also make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at [Hyperlink, http://www.gao.gov] http://www.gao.gov. If you or your staff have questions about this report, please contact Paul Posner at (202) 512-9573 or [Hyperlink, posnerp@gao.gov] posnerp@gao.gov. An additional contact and key contributors to this report are listed in appendix V. Signed by: David M. Walker Comptroller General of the United States: [End of section] Appendixes: Appendix I: Scope and Methodology: To address the objectives in this report, we reviewed Office of Management and Budget (OMB) materials and presentations on the development and implementation of the Program Assessment Rating Tool (PART) as well as the results of the PART assessments. Our review of materials included instructions for using PART, OMB's testimony concerning PART, and public remarks made by OMB officials at relevant conferences and training. We also reviewed PART-related information on OMB's Web site, including the OMB worksheets used to support the assessments, and attended OMB's PART training for the fiscal year 2004 process. For this report, we focused on the process and final results of the fiscal year 2004 PART process, but also looked at the initial stages of the fiscal year 2005 process. We compared the PART guidance for both years and asked agency and OMB staff to discuss generally the differences between the 2 fiscal years. We did not review the final results for the fiscal year 2005 PART, which are embargoed until the publication of the President's fiscal year 2005 budget request. For the same reasons, we did not review the results of any reassessments conducted for fiscal year 2005 on programs originally assessed for fiscal year 2004. This report presents the experiences of staff from the three departments and OMB officials who we interviewed. We did not directly observe the PART process (for either year) in operation nor did we independently verify the PART assessments as posted on OMB's Web site or the program or financial information contained in the documents provided as evidence for the PART assessments. We did, however, take several steps to ensure that we reliably downloaded and combined the PART summaries and worksheets with our budget and recommendation classifications. Our steps included (1) having the computer programs we used to create and process our consolidated dataset verified by a second programmer; (2) having transcribed data elements from all programs checked back to source files; and (3) having selected, computer-processed data elements checked back to source files for a random sample of programs and also for specific programs identified in our analyses. To better understand the universe of programs OMB assessed for fiscal year 2004, we developed overall profiles of PART results and examined relationships between such characteristics as type of program, type of recommendation, overall rating, total PART score, and answers for each question on PART. This review enabled us to generally confirm some information previously reported by OMB, for example, that PART scores do not automatically determine proposed funding and that grant programs scored lower overall than other types of programs. It also allowed us to select a sample of programs for more in-depth review, and this sample was used to determine which OMB and agency officials we interviewed. To gain a better understanding of the PART process at both OMB and agencies, to inform our examination of the usefulness of PART as an evaluation tool, and to obtain various perspectives on the relationship between PART and GPRA, we interviewed officials at OMB and three selected departments. At OMB, we interviewed a range of staff, such as associate directors, deputy assistant directors, branch chiefs, and examiners. Specifically, we interviewed staff in two Resource Management Offices (RMO). In the Human Resources Programs RMO, we spoke with staff from the Health Division and the Education and Human Resources Division. In the Natural Resources, Energy and Science RMO we interviewed staff from the Energy and Interior Branches. In addition, we obtained the views of two groups within OMB that were convened specifically for the PART process: the Performance Evaluation Team (PET) and the Interagency Review Panel (IRP). The IRP included agency officials in addition to staff from OMB. The three departments for which we reviewed the PART process were the Department of Energy (DOE), the Department of Health and Human Services (HHS), and the Department of the Interior (DOI). We selected these three departments based on our data analysis of program types. The departments selected and their agencies had a variety of program types (e.g., block/formula grants, competitive grants, direct federal, and research and development) that were subject to PART and could provide us with a broad-based perspective on how PART was applied to different programs employing diverse tools of government. We also chose these three departments because they had programs under PART review within the two RMOs at OMB where we did more extensive interviewing, thus enabling us to develop a more in-depth understanding of how the PART process operated for a subset of programs. We used this information to complement our broader profiling of all 234 programs assessed. Within DOE we studied the experiences of the Office of Science, the Office of Energy Efficiency and Renewable Energy, and the Office of Fossil Energy. Within HHS, we studied the experiences of the Administration for Children and Families, the Health Resources and Services Administration, and the Substance Abuse and Mental Health Services Administration. Within DOI, we studied the experiences of the Bureau of Land Management, the Bureau of Indian Affairs, and the National Park Service. We interviewed planning, budget, and program staff within each of the nine agencies as well as those at the department level. We also reviewed relevant supporting materials provided by these departments in conjunction with these interviews. To allow us to describe how PART was used in fiscal year 2004 to influence changes in future performance, we created a consolidated dataset in which we classified recommendations OMB made by three areas in need of improvement: (1) program design, (2) program management, and (3) program assessment. A fourth category was created for those recommendations that involved funding issues. We created a consolidated dataset of information from our analysis of recommendations and selected information from the PART program summary page and worksheet for each program.[Footnote 26] In addition, for approximately 95 percent of the programs, we identified whether the basis for program funding was mandatory or discretionary. It was important to separate discretionary and mandatory programs in our review of PART's potential influence on the President's budget proposals because funding for mandatory programs is determined through authorizations, not through the annual appropriations process. Of the 234 programs that OMB assessed for fiscal year 2004, we identified 27 mandatory programs and 196 discretionary, but could not categorize 11 programs as solely mandatory or discretionary because they were too mixed to classify.[Footnote 27] For discretionary programs, we explored the relationship between PART results and proposed budget changes in a series of regression analyses.[Footnote 28] Using statistical analysis, we found that PART scores influenced proposed funding changes for discretionary programs; however, a large amount of variability in these changes remains unexplained. We examined proposed funding changes between fiscal years 2003 and 2004 (measured by percentage change) and the relationship to PART scores for the programs assessed in the fiscal year 2004 President's Budget. These scores are the weighted sums of scores for four PART categories: Program Purpose and Design, Strategic Planning, Program Management, and Program Results and Accountability. The corresponding weights assigned by OMB are 0.2, 0.1, 0.2, and 0.5, respectively.[Footnote 29] Tables in this appendix report regression results obtained using the method of least squares with heteroskedasticity-corrected standard errors.[Footnote 30] The same estimation method is used throughout this analysis. Overall PART scores have a positive and statistically significant effect on discretionary program funding. The programs evaluated by OMB include both mandatory and discretionary programs. Regression results for mandatory programs showed--as expected--no relationship between PART scores and the level of funding in the President's Budget proposal. Assessment ratings, however, can potentially affect the funding for discretionary programs either in the President's Budget proposal or in congressional deliberations on spending bills.[Footnote 31] Table 3 reports the regression results for discretionary programs. Table 3: The Effect of Overall PART Score on Proposed Funding Changes (Discretionary Programs): Variable: Overall PART score; Coefficient estimate: 0.536; Robust standard error: 0.159; t-Statistic: 3.38; P-value: 0.001. Variable: Constant; Coefficient estimate: -25.671; Robust standard error: 8.682; t-Statistic: -2.96; P-value: 0.003. Source: GAO analysis of OMB data. Notes: R-squared = 0.058, Prob-F = 0.001, N = 196. Originally we identified 197 discretionary programs. However, no fiscal year 2004 budget estimate is reported for the Disclosed Worker Assistance program due to grant consolidation at the Department of Labor. (Budget of the United States Government, Fiscal Year 2004, Performance and Management Assessments (Washington, D.C.: February 2003), 191.) This reduced the number of discretionary programs to 196. [End of table] The estimated coefficient of the overall score is positive and significant. These results show that the aggregate PART score has a positive and statistically significant effect on the proposed change in discretionary programs' budget, suggesting that programs with better scores are more likely to receive larger proposed budget increases. To examine the effect of program size on our results, we divided all programs equally into three groups--small, medium, and large--based on their fiscal year 2003 funding estimate. Regressions similar to those reported in table 3 were then performed for discretionary programs in each group. The results, reported in tables 4, 5, and 6 suggest that the statistically significant effect of overall scores on budget outcomes exists only for the smaller programs. The estimated coefficient of the overall score for large programs, which is significant but only at the 10 percent level, reflects an outlier. [Footnote 32] Once this outlier is dropped, the estimated coefficient becomes statistically insignificant. Table 4: The Effect of Overall PART Score on Proposed Funding Changes (Small Discretionary Programs): Variable: Overall PART score; Coefficient estimate: 1.074; Robust standard error: 0.404; t-Statistic: 2.66; P-value: 0.010. Variable: Constant; Coefficient estimate: -50.523; Robust standard error: 21.155; t-Statistic: -2.39; P-value: 0.020. Source: GAO analysis of OMB data. Note: R-squared = 0.092, Prob-F = 0.01, N = 71. [End of table] Table 5: The Effect of Overall PART Score on Proposed Funding Changes (Medium-Size Discretionary Programs): Variable: Overall PART score; Coefficient estimate: 0.306; Robust standard error: 0.188; t-Statistic: 1.62; P-value: 0.109. Variable: Constant; Coefficient estimate: -17.984; Robust standard error: 12.480; t-Statistic: -1.44; P-value: 0.154. Source: GAO analysis of OMB data. Note: R-squared = 0.039, Prob-F = 0.109, N = 67. [End of table] Table 6: The Effect of Overall PART Score on Proposed Funding Changes (Large Discretionary Programs): Variable: Overall PART score; Coefficient estimate: 0.194; Robust standard error: 0.109; t-Statistic: 1.77; P-value: 0.082. Variable: Constant; Coefficient estimate: -8.216; Robust standard error: 7.778; t-Statistic: -1.06; P-value: 0.295. Source: GAO analysis of OMB data. Note: R-squared = 0.057, Prob-F = 0.082, N = 58. [End of table] The statistical analysis suggests that among the four components of the PART questionnaire, program purpose, management, and results have statistically significant effects on proposed funding changes, but the effects of program purpose and results are more robust across the estimated models. The overall score is a weighted average of four components: Program Purpose and Design, Strategic Planning, Program Management, and Program Results and Accountability.[Footnote 33] To identify which of the four components contribute to the significant relationship observed here, we examined the effect of each on proposed changes in programs' funding levels. Tables 7 and 8 show estimates from regressions of the proposed funding change on purpose, planning, management, and results scores for all discretionary programs as well as small discretionary programs alone. Table 7: The Effect of PART Component Scores on Proposed Funding Changes (All Discretionary Programs): Variable: Purpose; Coefficient estimate: 0.325; Robust standard error: 0.127; t-Statistic: 2.56; P-value: 0.011. Variable: Plan; Coefficient estimate: -0.259; Robust standard error: 0.199; t-Statistic: -1.30; P-value: 0.194. Variable: Management; Coefficient estimate: 0.191; Robust standard error: 0.117; t-Statistic: 1.63; P-value: 0.105. Variable: Results; Coefficient estimate: 0.363; Robust standard error: 0.205; t-Statistic: 1.77; P-value: 0.078. Variable: Constant; Coefficient estimate: -33.096; Robust standard error: 14.136; t-Statistic: -2.34; P-value: 0.020. Source: GAO analysis of OMB data. Note: R-squared = 0.087, Prob-F = 0.003, N = 196. [End of table] Table 8: The Effect of PART Component Scores on Proposed Funding Changes (Small Discretionary Programs): Variable: Purpose; Coefficient estimate: 0.223; Robust standard error: 0.274; t-Statistic: 0.81; P-value: 0.419. Variable: Plan; Coefficient estimate: -0.671; Robust standard error: 0.543; t-Statistic: -1.24; P-value: 0.221. Variable: Management; Coefficient estimate: 0.547; Robust standard error: 0.304; t-Statistic: 1.80; P-value: 0.077. Variable: Results; Coefficient estimate: 0.956; Robust standard error: 0.534; t-Statistic: 1.79; P-value: 0.078. Variable: Constant; Coefficient estimate: -42.455; Robust standard error: 34.800; t-Statistic: -1.22; P-value: 0.227. Source: GAO analysis of OMB data. Note: R-squared = 0.149, Prob-F = 0.043, N = 71. [End of table] These results suggest that among the four components, program purpose, management, and results are more likely to affect the proposed budget changes for discretionary programs. When all discretionary programs are included, the estimated coefficients are positive and significant for results (at the 10 percent level) and purpose. When only the small discretionary programs are included, the estimated coefficients are positive and significant for both management and results (at the 10 percent level). We also estimated the above regression for medium and large programs, but coefficient estimates were not statistically significant, except for the estimated coefficient of purpose for medium programs. PART scores explain at most about 15 percent of the proposed funding changes, leaving a large portion of the variability in proposed funding changes unexplained. This suggests that most of the variance is due to institutional factors, program specifics, and other unquantifiable factors. The coefficient of determination (or R2) is used to measure the proportion of the total variation in the regression's dependent variable that is explained by the variation in the regressors (independent variables).[Footnote 34] The maximum value of this measure across all estimated regressions is about 15 percent. Similar analyses were carried out for changes in the proposed budget for fiscal year 2004 and congressionally appropriated amounts in fiscal year 2002. Results were qualitatively similar to those reported here. To assess the strengths and weaknesses of PART as an evaluation tool and the consistency with which it was applied, we analyzed data from all 234 programs that OMB reviewed using PART for fiscal year 2004. As part of our examination of the consistency with which PART was applied to programs, we also focused on a subset of programs to assess the way in which certain measurement issues were addressed across those programs. The issues were selected from those identified in interviews with officials from the selected agencies described above and our own review of the PART program summaries and worksheets. Measurement issues included acceptance of output versus outcome measures of annual and long-term goals, types of studies accepted as program evaluations, acknowledgment of related programs, and justifications for judging a PART question as "not applicable." Programs were selected that formed clusters, each addressing a similar goal or shared a structural similarity pertinent to performance measurement, to examine whether PART assessment issues were handled similarly across programs when expected. We reviewed the worksheets and compared the treatment of assessment issues across specific questions within and across programs in a cluster to identify potential inconsistencies in how the tool was applied. We reviewed a total of 28 programs in nine clusters. The nine clusters are food safety, water supply, military equipment procurement, provision of health care, statistical agencies, block grants to assist vulnerable populations, energy research programs, wildland fire management, and disability compensation. With the exception of our summary analyses of all 234 programs, the information obtained from OMB and agency interviews, related material, and review of selected programs is not generalizable to the PART process for all 234 programs reviewed in fiscal year 2004. We conducted our review from May through October 2003 in accordance with generally accepted government auditing standards. [End of section] Appendix II: The Fiscal Year 2004 PART and Differences Between the Fiscal Year 2004 and 2005 PARTs: Below we have reproduced OMB's fiscal year 2004 PART instrument. We have also included the comparison of fiscal year 2004 and fiscal year 2005 PART questions that appeared in the fiscal year 2005 PART guidance (see table 9). Section I: Program Purpose & Design (Yes, No, N/A): 1. Is the program purpose clear? 2. Does the program address a specific interest, problem or need? 3. Is the program designed to have a significant impact in addressing the interest, problem or need? 4. Is the program designed to make a unique contribution in addressing the interest, problem or need (i.e., not needlessly redundant of any other Federal, state, local or private efforts)? 5. Is the program optimally designed to address the interest, problem or need? Specific Program Purpose & Design Questions by Program Type: Research and Development Programs: 6. (RD. 1) Does the program effectively articulate potential public benefits? 7. (RD. 2) If an industry-related problem, can the program explain how the market fails to motivate private investment? Section II: Strategic Planning (Yes, No, N/A): 1. Does the program have a limited number of specific, ambitious long- term performance goals that focus on outcomes and meaningfully reflect the purpose of the program? 2. Does the program have a limited number of annual performance goals that demonstrate progress toward achieving the long-term goals? 3. Do all partners (grantees, subgrantees, contractors, etc.) support program-planning efforts by committing to the annual and/or long-term goals of the program? 4. Does the program collaborate and coordinate effectively with related programs that share similar goals and objectives? 5. Are independent and quality evaluations of sufficient scope conducted on a regular basis or as needed to fill gaps in performance information to support program improvements and evaluate effectiveness? 6. Is the program budget aligned with the program goals in such a way that the impact of funding, policy, and legislative changes on performance is readily known? 7. Has the program taken meaningful steps to address its strategic planning deficiencies? Specific Strategic Planning Questions by Program Type: Regulatory-Based Programs: 8. (RD. 1) Are all regulations issued by the program/agency necessary to meet the stated goals of the program, and do all regulations clearly indicate how the rules contribute to achievement of the goals? Capital Assets and Service Acquisition Programs: 8. (Cap. 1) Are acquisition program plans adjusted in response to performance data and changing conditions? 9. (Cap. 2) Has the agency/program conducted a recent, meaningful, credible analysis of alternatives that includes trade-offs between cost, schedule and performance goals? Research and Development Programs: 8. (RD. 1) Is evaluation of the program's continuing relevance to mission, fields of science, and other "customer" needs conducted on a regular basis? 9. (RD. 2) Has the program identified clear priorities? Section III: Program Management (Yes, No, N/A): 1. Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance? 2. Are Federal managers and program partners (grantees, subgrantees, contractors, etc.) held accountable for cost, schedule and performance results? 3. Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose? 4. Does the program have incentives and procedures (e.g., competitive sourcing/cost comparisons, IT improvements) to measure and achieve efficiencies and cost effectiveness in program execution? 5. Does the agency estimate and budget for the full annual costs of operating the program (including all administrative costs and allocated overhead) so that program performance changes are identified with changes in funding levels? 6. Does the program use strong financial management practices? 7. Has the program taken meaningful steps to address its management deficiencies? Specific Program Management Questions by Program Type: Competitive Grant Programs: 8. (Co. 1) Are grant applications independently reviewed based on clear criteria (rather than earmarked) and are awards made based on results of the peer review process? 9. (Co. 2) Does the grant competition encourage the participation of new/first-time grantees through a fair and open application process? 10. (Co. 3) Does the program have oversight practices that provide sufficient knowledge of grantee activities? 11. (Co. 4) Does the program collect performance data on an annual basis and make it available to the public in a transparent and meaningful manner? Block/Formula Grant Programs: 8. (B. 1) Does the program have oversight practices that provide sufficient knowledge of grantee activities? 9. (B. 2) Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner? Regulatory-Based Programs: 8. (Reg. 1) Did the program seek and take into account the views of affected parties including state, local and tribal governments and small businesses, in drafting significant regulations? 9. (Reg. 2) Did the program prepare, where appropriate, a Regulatory Impact Analysis that comports with OMB's economic analysis guidelines and have these RIA analyses and supporting science and economic data been subjected to external peer review by qualified specialists? 10. (Reg. 3) Does the program systematically review its current regulations to ensure consistency among all regulations in accomplishing program goals? 11. (Reg. 4) In developing new regulations, are incremental societal costs and benefits compared? 12. (Reg. 5) Did the regulatory changes to the program maximize net benefits? 13. (Reg. 6) Does the program impose the least burden, to the extent practicable, on regulated entities, taking into account the costs of cumulative final regulations? Capital Assets and Service Acquisition Programs: 8. (Cap. 1) Does the program define the required quality, capability, and performance objectives of deliverables? 9. (Cap. 2) Has the program established appropriate, credible, cost and schedule goals? 10. (Cap. 3) Has the program conducted a recent, credible, cost-benefit analysis that shows a net benefit? 11. (Cap. 4) Does the program have a comprehensive strategy for risk management that appropriately shares risk between the government and contractor? Credit Programs: 8. (Cr. 1) Is the program managed on an ongoing basis to assure credit quality remains sound, collections and disbursements are timely and reporting requirements are fulfilled? 9. (Cr. 2) Does the program consistently meet the requirements of the Federal Credit Reform Act of 1990, the Debt Collection Improvement Act and applicable guidance under OMB Circulars A-1, A-34, and A-129? 10. (Cr. 3) Is the risk of the program to the U.S. Government measured effectively? Research and Development Programs: 8. (RD. 1) Does the program allocate funds through a competitive, merit-based process, or, if not, does it justify funding methods and document how quality is maintained? 9. (RD. 2) Does competition encourage the participation of new/first- time performers through a fair and open application process? 10. (RD. 3) Does the program adequately define appropriate termination points and other decision points? 11. (RD. 4) If the program includes technology development or construction or operation of a facility, does the program clearly define deliverables and required capability/performance characteristics and appropriate, credible cost and schedule goals? Section IV: Program Results (Yes, Large Extent, Small Extent, No): 1. Has the program demonstrated adequate progress in achieving its long-term outcome goal(s)? * Long-Term Goal I: Target: Actual Progress achieved toward goal:: * Long-Term Goal II: Target: Actual Progress achieved toward goal:: * Long-Term Goal III: Target: Actual Progress achieved toward goal:: 2. Does the program (including program partners) achieve its annual performance goals? * Key Goal I: Performance Target: Actual Performance: * Key Goal II: Performance Target: Actual Performance:: * Key Goal III: Performance Target: Actual Performance: Note: Performance targets should reference the performance baseline and years, e.g. achieve a 5% increase over base of X in 2000. 3. Does the program demonstrate improved efficiencies and cost effectiveness in achieving program goals each year? 4. Does the performance of this program compare favorably to other programs with similar purpose and goals? 5. Do independent and quality evaluations of this program indicate that the program is effective and achieving results? Specific Results Questions by Program Type: Regulatory-Based Programs: 6. (Reg. 1) Were programmatic goals (and benefits) achieved at the least incremental societal cost and did the program maximize net benefits? Capital Assets and Service Acquisition Programs: 6. (Cap. 1) Were program goals achieved within budgeted costs and established schedules? Research and Development Programs: 6. (RD. 1) If the program includes construction of a facility, were program goals achieved within budgeted costs and established schedules? Table 9: Side-by-Side of the Fiscal Year 2005 PART and the Fiscal Year 2004 PART Questions: I. Program purpose & design: 1.1; This year‘s question (fiscal year 2005 PART): Is the program purpose clear? (1); Last year‘s question (fiscal year 2004 PART): Same; Comment: [Empty]. 1.2; This year‘s question (fiscal year 2005 PART): Does the program address a specific and existing problem, interest, or need? (2); Last year‘s question (fiscal year 2004 PART): Does the program address a specific interest, problem or need? Comment: Wording clarified. 1.2: (3); Last year‘s question (fiscal year 2004 PART): Is the program designed to have a significant impact in addressing the interest, problem or need? Comment: Dropped; "significant" worked against small programs and was not clear. 1.3; This year‘s question (fiscal year 2005 PART): Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort? (4); Last year‘s question (fiscal year 2004 PART): Is the program designed to make a unique contribution in addressing the interest, problem or need (i.e., is not needlessly redundant of any other Federal, state or, local or private effort)? Comment: Wording clarified. 1.4; This year‘s question (fiscal year 2005 PART): Is the program design free of major flaws that would limit the program's effectiveness or efficiency? (5); Last year‘s question (fiscal year 2004 PART): Is the program optimally designed to address the national, interest, problem or need? Comment: Minor change to clarify focus; "optimally" was too broad. 1.5; This year‘s question (fiscal year 2005 PART): Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly? Last year‘s question (fiscal year 2004 PART): [Empty]; Comment: New question to address distributional design. Specific Program Purpose and Design Questions by Program Type: Research and Development Programs: (RD.1); Last year‘s question (fiscal year 2004 PART): Does the program effectively articulate potential public benefits? Comment: Dropped; covered by 1.2. (RD. 2); Last year‘s question (fiscal year 2004 PART): If an industry-related problem, can the program explain how the market fails to motivate private investment? Comment: Dropped; covered by I.2 and I.5. II. Strategic planning: 2.1; This year‘s question (fiscal year 2005 PART): Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program? (1); Last year‘s question (fiscal year 2004 PART): Does the program have a limited number of specific, ambitious long-term performance goals that focus on outcomes and meaningfully reflect the purpose of the program? Comment: Splits old II.1 into separate questions on existence of (1) long-term performance measures and (2) targets for these measures. Together, the measures and targets comprise the long-term performance goals addressed in last year's question. 2.2; This year‘s question (fiscal year 2005 PART): Does the program have ambitious targets and timeframes for its long-term measures? Last year‘s question (fiscal year 2004 PART): [Empty]; Comment: Splits old II.1; see above. 2.3; This year‘s question (fiscal year 2005 PART): Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals? (2); Last year‘s question (fiscal year 2004 PART): Does the program have a limited number of annual performance goals that demonstrate progress toward achieving the long-term goals? Comment: Splits old II.2 into separate questions on existence of (1) annual performance measures and (2) targets for these measures. Together, the measures and targets comprise the annual performance goals addressed in last year's question. 2.4; This year‘s question (fiscal year 2005 PART): Does the program have baselines and ambitious targets for its annual measures? Last year‘s question (fiscal year 2004 PART): [Empty]; Comment: Splits old II.2; see above. 2.5; This year‘s question (fiscal year 2005 PART): Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program? (3); Last year‘s question (fiscal year 2004 PART): Do all partners (grantees, sub- grantees, contractors, etc.) support program planning efforts by committing to the annual and/or long-term goals of the program? Comment: Wording clarified. 2.5; (4); Last year‘s question (fiscal year 2004 PART): Does the program collaborate and coordinate effectively with related programs that share similar goals and objectives? Comment: Moved to question 3.5. 2.6; This year‘s question (fiscal year 2005 PART): Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need? (5); Last year‘s question (fiscal year 2004 PART): Are independent and quality evaluations of sufficient scope conducted on a regular basis or as needed to fill gaps in performance information to support program improvements and evaluate effectiveness? Comment: Wording clarified. 2.7; This year‘s question (fiscal year 2005 PART): Are budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget? (6); Last year‘s question (fiscal year 2004 PART): Is the program budget aligned with the program goals in such a way that the impact of funding, policy, and legislative changes on performance is readily known? Comment: Modified. 2.8; This year‘s question (fiscal year 2005 PART): Has the program taken meaningful steps to correct its strategic planning deficiencies? (7); Last year‘s question (fiscal year 2004 PART): Same; Comment: [Empty]. Specific Strategic Planning Questions by Program Type: Regulatory Based Programs: 2.RG1; This year‘s question (fiscal year 2005 PART): Are all regulations issued by the program/agency necessary to meet the stated goals of the program, and do all regulations clearly indicate how the rules contribute to achievement of the goals? (Reg. 1); Last year‘s question (fiscal year 2004 PART): Same; Comment: [Empty]. Capital Assets & Service Acquisition Programs: (Cap. 1); Last year‘s question (fiscal year 2004 PART): Are acquisition program plans adjusted in response to performance data and changing conditions? Comment: Dropped; covered in 2.CA1 and 3.CA1. 2.CA1; This year‘s question (fiscal year 2005 PART): Has the agency/program conducted a recent, meaningful, credible analysis of alternatives that includes trade-offs between cost, schedule, risk, and performance goals and used the results to guide the resulting activity? (Cap. 2); Last year‘s question (fiscal year 2004 PART): Has the agency/program conducted a recent, meaningful, credible analysis of alternatives that includes trade-offs between cost, schedule and performance goals? Comment: Minor change. R&D Programs: This year‘s question (fiscal year 2005 PART): R&D programs addressing technology development or the construction or operation of a facility should answer 2.CA1; Last year‘s question (fiscal year 2004 PART): [Empty]; Comment: [Empty]. 2.RD1; This year‘s question (fiscal year 2005 PART): If applicable, does the program assess and compare the potential benefits of efforts within the program to other efforts that have similar goals? (RD. 1); Last year‘s question (fiscal year 2004 PART): Is evaluation of the program's continuing relevance to mission, fields of science, and other "customer" needs conducted on a regular basis? Comment: Modified. 2.RD2; This year‘s question (fiscal year 2005 PART): Does the program use a prioritization process to guide budget requests and funding decisions? (RD. 2); Last year‘s question (fiscal year 2004 PART): Has the program identified clear priorities? Comment: Modified. III. Program management: 3.1; This year‘s question (fiscal year 2005 PART): Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance? (1); Last year‘s question (fiscal year 2004 PART): Same; Comment: [Empty]. 3.2; This year‘s question (fiscal year 2005 PART): Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results? (2); Last year‘s question (fiscal year 2004 PART): Same; Comment: [Empty]. 3.3; This year‘s question (fiscal year 2005 PART): Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose? (3); Last year‘s question (fiscal year 2004 PART): Same; Comment: [Empty]. 3.4; This year‘s question (fiscal year 2005 PART): Does the program have procedures (e.g. competitive sourcing/ cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution? (4); Last year‘s question (fiscal year 2004 PART): Same; Comment: [Empty]. 3.5; This year‘s question (fiscal year 2005 PART): Does the program collaborate and coordinate effectively with related programs? Last year‘s question (fiscal year 2004 PART): [Empty]; Comment: Same as old question 2.4. 3.5; (5); Last year‘s question (fiscal year 2004 PART): Does the agency estimate and budget for the full annual costs of operating the program (including all administrative costs and allocated overhead) so that program performance changes are identified with changes in funding levels? Comment: Now covered by guidance for question 2.7. 3.6; This year‘s question (fiscal year 2005 PART): Does the program use strong financial management practices? (6); Last year‘s question (fiscal year 2004 PART): Same; Comment: [Empty]. 3.7; This year‘s question (fiscal year 2005 PART): Has the program taken meaningful steps to address its management deficiencies? (7); Last year‘s question (fiscal year 2004 PART): Same; Comment: [Empty]. Specific Program Management Questions by Program Type: Competitive Grant Programs: 3.CO1; This year‘s question (fiscal year 2005 PART): Are grants awarded based on a clear competitive process that includes a qualified assessment of merit? (Co. 1); Last year‘s question (fiscal year 2004 PART): Are grant applications independently reviewed based on clear criteria (rather than earmarked) and are awards made based on results of the peer review process? Comment: Modified. Guidance also captures former question Co. 2. 3.C01; (Co.2); Last year‘s question (fiscal year 2004 PART): Does the grant competition encourage the participation of new/first-time grantees through a fair and open application process? Comment: Now considered in guidance for answering 3.CO1, above. 3.CO2; This year‘s question (fiscal year 2005 PART): Does the program have oversight practices that provide sufficient knowledge of grantee activities? (Co. 3); Last year‘s question (fiscal year 2004 PART): Does the agency have sufficient knowledge about grantee activities? Comment: Wording clarified. 3.CO3; This year‘s question (fiscal year 2005 PART): Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner? (Co. 4); Last year‘s question (fiscal year 2004 PART): Same; Comment: [Empty]. Block/Formula Grant Programs: 3.BF1; This year‘s question (fiscal year 2005 PART): Does the program have oversight practices that provide sufficient knowledge of grantee activities? (B.1); Last year‘s question (fiscal year 2004 PART): Same; Comment: [Empty]. 3.BF2; This year‘s question (fiscal year 2005 PART): Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner? (B. 2); Last year‘s question (fiscal year 2004 PART): Same; Comment: [Empty]. Regulatory Based Programs: 3.RG1; This year‘s question (fiscal year 2005 PART): Did the program seek and take into account the views of all affected parties (e.g., consumers; large and small businesses; State, local and tribal governments; beneficiaries; and the general public) when developing significant regulations? (Reg. 1); Last year‘s question (fiscal year 2004 PART): Did the program seek and take into account the views of affected parties including state, local and tribal governments and small businesses in drafting significant regulations? Comment: Wording clarified. 3.RG2; This year‘s question (fiscal year 2005 PART): Did the program prepare adequate regulatory impact analyses if required by Executive Order 12866, regulatory flexibility analyses if required by the Regulatory Flexibility Act and SBREFA, and cost- benefit analyses if required under the Unfunded Mandates Reform Act; and did those analyses comply with OMB guidelines? (Reg. 2); Last year‘s question (fiscal year 2004 PART): Did the program prepare, where appropriate, a Regulatory Impact Analysis (RIA) that comports with OMB's economic analysis guidelines and have these RIA analyses and supporting science and economic data been subjected to external peer review, as appropriate, by qualified specialists? Comment: Minor change. 3.RG3; This year‘s question (fiscal year 2005 PART): Does the program systematically review its current regulations to ensure consistency among all regulations in accomplishing program goals? (Reg. 3); Last year‘s question (fiscal year 2004 PART): Same; Comment: [Empty]. 3.RG3; (Reg. 4); Last year‘s question (fiscal year 2004 PART): In developing new regulations, are incremental societal costs and benefits compared? Comment: Merged into new 3.RG4. 3.RG4; This year‘s question (fiscal year 2005 PART): Are the regulations designed to achieve program goals, to the extent practicable, by maximizing the net benefits of its regulatory activity? (Reg. 5); Last year‘s question (fiscal year 2004 PART): Did the regulatory changes to the program maximize net benefits? Comment: Combines former questions Reg. 4, 5, & 6. 3.RG4; (Reg. 6); Last year‘s question (fiscal year 2004 PART): Does the program impose the least burden, to the extent practicable, on regulated entities, taking into account the costs of cumulative final regulations? Comment: Merged in to new 3.RG4. Capital Assets and Service Acquisition Programs: 3.CA1; This year‘s question (fiscal year 2005 PART): Is the program managed by maintaining clearly defined deliverables, capability/performance characteristics, and appropriate, credible cost and schedule goals? Last year‘s question (fiscal year 2004 PART): [Empty]; Comment: New question, covers old Cap. 1, 2, 3, and 4. (Cap. 1); Last year‘s question (fiscal year 2004 PART): Does the program clearly define the required quality, capability, and performance objectives for deliverables and required capabilities/ performance characteristics? Comment: Merged into new 2.CA1 and 3.CA1. (Cap 2); Last year‘s question (fiscal year 2004 PART): Has the program established appropriate, credible, cost and schedule goals? Comment: Merged into new 2.CA1 and 3.CA1. (Cap 3); Last year‘s question (fiscal year 2004 PART): Has the program conducted a recent, credible, cost-benefit analysis that shows a net benefit? Comment: Merged into new 2.CA1 and 3.CA1. (Cap 4); Last year‘s question (fiscal year 2004 PART): Does the program have a comprehensive strategy for risk management that appropriately shares risk between the government and contractor? Comment: Merged into new 2.CA1 and 3.CA1. Credit Programs: 3.CR1; This year‘s question (fiscal year 2005 PART): Is the program managed on an ongoing basis to assure credit quality remains sound, collections and disbursements are timely, and reporting requirements are fulfilled? (Cr. 1); Last year‘s question (fiscal year 2004 PART): Same; Comment: [Empty]. 3.CR1; (Cr. 2); Last year‘s question (fiscal year 2004 PART): Does the program consistently meet the requirements of the Federal Credit Reform Act of 1990, the Debt Collection Improvement Act and applicable guidance under OMB Circulars A-1, A-11, and A-129? Comment: Merged into new 3.CR2. 3.CR2; This year‘s question (fiscal year 2005 PART): Do the program's credit models adequately provide reliable, consistent, accurate and transparent estimates of costs and the risk to the Government? (Cr. 3); Last year‘s question (fiscal year 2004 PART): Is the risk of the program to the U.S. Government measured effectively? Comment: Combines former Cr. 2 and 3. Research and Development Programs: R&D programs addressing technology development or the construction or operation of a facility should answer 3.CA1. R&D programs that use competitive grants should answer 3.CO1, CO2 and CO3: 3.RD1; This year‘s question (fiscal year 2005 PART): For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality? (RD. 1); Last year‘s question (fiscal year 2004 PART): Does the program allocate funds through a competitive, merit-based process, or, if not, does it justify funding methods and document how quality is maintained? Comment: Modified. (RD. 2); Last year‘s question (fiscal year 2004 PART): Does competition encourage the participation of new/first-time performers through a fair and open application process? Comment: Covered by 3.CO1. (RD. 3); Last year‘s question (fiscal year 2004 PART): Does the program adequately define appropriate termination points and other decision points? Comment: Covered by 2.CA1 and 3.CA1. (RD. 4); Last year‘s question (fiscal year 2004 PART): If the program includes technology development or construction or operation of a facility, does the program clearly define deliverables, capability/performance characteristics, and appropriate, credible cost and schedule goals? Comment: Covered by 2.CA1 and 3.CA1. IV. Program results: 4.1; This year‘s question (fiscal year 2005 PART): Has the program demonstrated adequate progress in achieving its long-term performance goals? (1); Last year‘s question (fiscal year 2004 PART): Has the program demonstrated adequate progress in achieving its long- term outcome goal(s)? Comment: Minor change. 4.2; This year‘s question (fiscal year 2005 PART): Does the program (including program partners) achieve its annual performance goals? (2); Last year‘s question (fiscal year 2004 PART): Same; Comment: [Empty]. 4.3; This year‘s question (fiscal year 2005 PART): Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year? (3); Last year‘s question (fiscal year 2004 PART): Same; Comment: [Empty]. 4.4; This year‘s question (fiscal year 2005 PART): Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals? (4); Last year‘s question (fiscal year 2004 PART): Does the performance of this program compare favorably to other programs with similar purpose and goals? Comment: Minor change. 4.5; This year‘s question (fiscal year 2005 PART): Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results? 5); Last year‘s question (fiscal year 2004 PART): Same; Comment: [Empty]. Specific Results Questions by Program Type: Regulatory Based Programs: 4.RG1; This year‘s question (fiscal year 2005 PART): Were programmatic goals (and benefits) achieved at the least incremental societal cost and did the program maximize net benefits? Last year‘s question (fiscal year 2004 PART): Same; Comment: [Empty]. Capital Assets and Service Acquisition Programs: 4.CA1; This year‘s question (fiscal year 2005 PART): Were program goals achieved within budgeted costs and established schedules? (Cap. 1); Last year‘s question (fiscal year 2004 PART): Same; Comment: [Empty]. Research and Development Programs: This year‘s question (fiscal year 2005 PART): R&D programs addressing technology development or the construction or operation of a facility should answer 4.CA1; (RD. 1); Last year‘s question (fiscal year 2004 PART): If the program includes construction of a facility, were program goals achieved within budgeted costs and established schedules? Comment: Simplified. Source: OMB Web site, [Hyperlink, http://www.whitehouse.gov/omb/part/ bpm861.pdf] http://www.whitehouse.gov/omb/part/bpm861.pdf (downloaded Apr. 7, 2003), 6-12. [End of table] [End of section] Appendix III: Development of PART: Fiscal Year 2003: This administration's efforts to link budget and performance began with the fiscal year 2003 budget, in which the administration announced the "Executive Branch Management Scorecard," a traffic-light grading system to report the work of federal agencies in implementing the President's Management Agenda's five governmentwide initiatives. Each quarter, OMB assessed agencies achievement toward the "standards of success"-- specific goals articulated for each of the five initiatives. Since some of the five initiatives require continual efforts, OMB also assessed agencies' progress toward achieving the standards. The fiscal year 2003 President's Budget also included OMB's assessments of the effectiveness of 130 programs and a brief explanation of the assessments. According to OMB, the assessments were based on OMB staff's knowledge of the programs and professional judgments; specific criteria were not publicly available with which to support OMB's judgments. Fiscal Year 2004: During the spring of 2002, an internal OMB task force--PET--consisting of staff from various OMB divisions, created PART to make the process of rating programs robust and consistent across government programs. During the development of PART, OMB solicited input from interested parties both inside and outside the federal government, including GAO and congressional staff. PART was tested on 67 programs during a series of Spring Review meetings with the OMB Director. Based on these results and other stakeholder feedback, PET recommended a series of refinements to PART, such as using a four-point scale in the Results section as opposed to the "yes/no" format. Another key change was revising the Program Purpose and Design section (Section I) to remove the question "Is the federal role critical?" because it was seen as subjective-- based on an individual's political views. In July 2002, OMB issued PART in final and accompanying instructions for completing the assessments for the President's fiscal year 2004 budget submission. Later that month, OMB provided a series of training sessions on PART for staff from OMB and agencies. Agencies received completed PART assessments during early September 2002 and submitted written appeals to OMB by mid-September. OMB formed the IRP, comprising OMB and agency officials, to conduct consistency reviews[Footnote 35] and provide recommendations on selected PART appeals. The IRP also provided OMB with a broad set of recommendations aimed at improving the PART based on IRP's experience with the consistency audit and appeals. OMB was to finalize all PART assessments by the end of September 2002, although both agency and OMB officials told us that changes and appeals continued through the end of the budget season. RMOs within OMB provided draft summaries of PART results to the Director of OMB during the Director's review of agencies' budget requests. The President's fiscal year 2004 budget (issued February 3, 2003) included a separate volume containing one-page summaries of the PART results for each of the 234 programs that were assessed.[Footnote 36] The relationship between PART and the administration's proposals was presented in agencies' budget justification materials sent to Congress. In an unprecedented move, OMB also posted PART, one-page rating results, and detailed supporting worksheets on its Web site. OMB also included its Web address in the Performance and Management Assessments volume of the budget and, in the budget itself, also described PART and its process and asked for comments on how to improve PART. Figure 3 depicts a time line of the events related to the formulation of the President's budget request, including the key stages of PART development. Figure 3: The PART Process and Budget Formulation Timelines: [See PDF for image] [End of figure] Fiscal Year 2005: For the fiscal year 2005 PART, OMB moved the entire assessment process from the fall to spring. OMB told us that the change was meant to help alleviate the burden of having the PART process overlap the end of the budget season, when workload is already so heavy. Another difference between the 2 years was that agency officials reported that OMB was more collaborative with the agencies in selecting the programs for the fiscal year 2005 PART. Training on the PART assessments to be included in the President's fiscal year 2005 budget began in early May 2003. Agencies submitted PART appeals in early July, and OMB aimed to resolve the appeals and finalize the PART scores by the end of July. In December of 2003, RMOs were to finalize the summaries of PART results, which will be published in February along with the fiscal year 2005 President's Budget. [End of section] Appendix IV: Comments from the Office of Management and Budget: EXECUTIVE OFFICE OF THE PRESIDENT OFFICE OF MANAGEMENT AND BUDGET WASHINGTON, D.C. 20503: DEPUTY DIRECTOR FOR MANAGEMENT: January 16, 2004: Mr. Paul Posner Managing Director Federal Budget and Intergovernmental Relations General Accounting Office: 441 G Street, N W Washington, DC 20548: Dear Mr. Posner: Thank you for the opportunity to comment on the draft GAO report on the PART (Performance Budgeting: Observations on the use of OMB's Program Assessment Rating Tool for the Fiscal Year 2004 Budget, GAO-04-174). We appreciate GAO's extensive review of the PART process. We are particularly pleased that your report recognizes the unprecedented transparency of the PART process and materials that we have posted on our website[NOTE 1] and the extensive efforts OMB has taken to make the PART process consistent across the government. We will continually strive to make the PART as credible, objective, and useful as it can be and believe that your recommendations will help us do that. As you know, OMB is already taking actions to address many of them. For instance: * With respect to centrally monitoring PART recommendations, we have provided a simple format for agencies to follow when reporting the status of recommendation implementation to OMB and I receive these reports semi annually. We will continue to refine this process so that sufficient attention is given to recommendation follow-up. * As the PART relies on separate evaluations of evidence of a program's success, we agree that the judgment about what constitutes a sufficient evaluation should be based on the quality of the evaluation. * Except for programs of insignificant size or impact, we are committed to assessing 100 percent of programs using the PART. We are sensitive to the impact this will have on OMB staff workload and will manage it accordingly. * One of the greatest opportunities for the PART is to compare the performance of, and share best practices among, like programs across government. We will continue to use the PART for that purpose. * We are working diligently to generate the meaningful dialogue with Congress you describe in your recommendations. * We will continue to improve agency and Executive Branch implementation of GPRA by insisting GPRA plans and reports meet the requirements of this important law and the high standards set by the PART. Your report makes valuable conclusions and recommendations about the PART and our overall effort to create a more results-oriented government. I want to note that the PART was designed for and is used in many ways other than just budget formulation. Performance information gleaned from the PART process has not only informed budget decisions, but has also helped direct management, identified opportunities to improve program design, and promoted accountability. We believe that the PART will also greatly improve the goals and measures adopted through the GPRA strategic and performance planning processes. Thank you for the opportunity to review and comment on your draft report. I appreciate your willingness to take our oral and written comments into consideration in the final draft. I look forward to working with you to improve the ways in which we are creating a results-oriented government. Sincerely, Signed by: Clay Johnson, III: NOTES: [I] See, draft report Appendix III, p. 2. [End of section] Appendix V: GAO Contacts and Staff Acknowledgments: GAO Contacts: Paul Posner, (202) 512-9573 Denise Fantone, (202) 512-4997: Acknowledgments: In addition to the above contacts, Kristeen McLain, Jackie Nowicki, and Stephanie Shipman made significant contributions to this report. Thomas Beall, Joseph Byrns, Hashem Dezhbakhsh, Evan Gilman, Patrick Mullen, David Nicholson, and Mark Ramage also made key contributions to this report. (450190): FOOTNOTES [1] Pub. L. No. 103-62 (1993). [2] U.S. General Accounting Office, Performance Budgeting: Past Initiatives Offer Insights for GPRA Implementation, GAO/AIMD-97-46 (Washington, D.C.: Mar. 27, 1997). [3] See Pub. L. No. 103-62, § 2 (1993), 5 U.S.C. § 306 (2003), and 31 U.S.C. §§ 1115-1116 (2003). [4] In addition to budget and performance integration, the other four priorities under the PMA are strategic management of human capital, expanded electronic government, improved financial performance, and competitive sourcing. [5] There is no standard definition for the term "program." For purposes of PART, OMB described the unit of analysis (program) as (1) an activity or set of activities clearly recognized as a program by the public, OMB, and/or Congress; (2) having a discrete level of funding clearly associated with it; and (3) corresponding to the level at which budget decisions are made. [6] The seven major categories are competitive grants, block/formula grants, capital assets and service acquisition programs, credit programs, regulatory-based programs, direct federal programs, and research and development programs. Tax programs were not addressed for the fiscal year 2004 PART process. [7] OMB Circular A-11, Preparation, Submission, and Execution of the Budget, Section 220. [8] OMB defined 20 percent of the budget as either 20 percent of programs or their funding levels so long as all programs are assessed over the 5-year cycle for fiscal years 2004 through 2008 budget requests. [9] The 234 programs assessed for fiscal year 2004 contained a total of 612 recommendations. [10] Pub. L. No. 106-501 (2000). [11] Unobligated balances are defined as portions of available budget authority that the agency has not set aside to cover current legal liabilities. [12] See GAO-03-595T and U.S. General Accounting Office, Managing for Results: Efforts to Strengthen the Link Between Resources and Results at the Administration for Children and Families, GAO-03-9 (Washington, D.C.: Dec. 10, 2002). [13] In the third question pair, a question in the planning section asks about whether the program has long-term goals, and a question in the results section asks whether the agency has made progress in achieving the program's long-term goals. Yet, in 6 of the 115 programs (5 percent) judged not to have adequate long-term goals, credit was given for making progress on their long-term goals even though the guidance again clearly states that a program must receive a "no" if the program received a "no" on the existence of long-term outcome goals. [14] U.S. General Accounting Office, Program Evaluation: Agencies Challenged by New Demand for Information on Program Results, GAO/GGD- 98-53 (Washington, D.C.: Apr. 24, 1998). [15] GAO/GGD-98-53. [16] However, we found 8 cases (out of 118) programs that were rated as "results not demonstrated" despite having both annual and long-term performance goals and evidence that these goals were being met. [17] Because our audit focused on the fiscal year 2004 PART process, our engagement was not limited by OMB's decision to not share its reasoning for shifting the consistency review from IRP to PET or our lack of access to the NAPA review. [18] Memorandum to the President's Management Council, "Where We'd Be Proud To Be," May 21, 2003. [19] OMB Circular A-11, Preparation, Submission, and Execution of the Budget. [20] 31 U.S.C. § 1115(a) (2003). [21] U.S. General Accounting Office, Managing for Results: Agency Progress in Linking Performance Plans With Budgets and Financial Statements, GAO-02-236 (Washington, D.C.: Jan. 4, 2002). [22] 5 U.S.C. § 306(d) (2003). [23] U.S. General Accounting Office, Agencies' Strategic Plans Under GPRA: Key Questions to Facilitate Congressional Review (Version 1), GAO/GGD-10.1.16 (Washington, D.C.: May 1997). [24] H.R. Rep. No. 108-195, p. 8 (2003). [25] H.R. Rep. No. 108-243, pp. 168-69 (2003). [26] The PART program summary sheets are included in the Budget of the United States Government, Fiscal Year 2004, Performance and Management Assessments (Washington, D.C.: February 2003). The summary sheets and worksheets for the 234 programs are on OMB's Web site: http:// www.whitehouse.gov/omb/budget/fy2004/pma.html. [27] These 11 programs are animal welfare, food aid, multifamily housing direct loans and rental assistance, rural electric utility loans and guarantees, and rural water and wastewater grants and loans programs in the Department of Agriculture; the nursing education loan repayment and scholarship program in HHS; the methane hydrates program in DOE; the reclamation hydropower program in DOI; the long-term guarantees program in the U.S. Export-Import Bank; and the climate change and development assistance/population programs in the Agency for International Development. [28] We tested the regression on mandatory programs and as expected the results showed no relationship between the PART scores and the level of funding proposed in the President's Budget. [29] Budget of the United States Government, Fiscal Year 2004, Performance and Management Assessments, 10. [30] For a discussion of this method, see W.H. Greene, Econometric Analysis, Section 10.3 (Upper Saddle River, N.J.: Prentice Hall, 2003). [31] Budget of the United States Government, Fiscal Year 2001, A Citizen's Guide to the Federal Budget (Washington, D.C.: February 2000), http://w3.access.gpo.gov/usbudget/fy2001/guide03.html, (downloaded April 2003), 2. [32] The outlier is the Community Oriented Policing Services program with an estimated 77 percent reduction in funding (see OMB, Budget of the U.S. Government, Fiscal Year 2004, Performance and Management Assessments, (Washington, D.C.: February 2003), 178). The outlier in this case is identified using scatter plot and estimating with and without the outlier. The reported results for small and medium programs are not outlier driven. [33] Budget of the United States Government, Fiscal Year 2004, Performance and Management Assessments, 10. [34] See Greene, 33. [35] According to OMB, IRP performed consistency reviews on a stratified random sample of programs that completed the PART in preparation for the fiscal year 2004 budget. While IRP made recommendations regarding its findings, it did not have the authority to enforce them. [36] Fiscal Year 2004 Budget of the United States Government, Performance and Management Assessments, (Washington, D.C.: February 2003). GAO's Mission: The General Accounting Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO's Web site ( www.gao.gov ) contains abstracts and full-text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics. Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as "Today's Reports," on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select "Subscribe to e-mail alerts" under the "Order GAO Products" heading. Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov Automated answering system: (800) 424-5454 or (202) 512-7470: Public Affairs: Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C. 20548:

The Justia Government Accountability Office site republishes public reports retrieved from the U.S. GAO These reports should not be considered official, and do not necessarily reflect the views of Justia.