Performance Budgeting
Observations on the Use of OMB's Program Assessment Rating Tool for the Fiscal Year 2004 Budget Gao ID: GAO-04-174 January 30, 2004The Office of Management and Budget's (OMB) Program Assessment Rating Tool (PART) is meant to provide a consistent approach to evaluating federal programs during budget formulation. To better understand its potential, congressional requesters asked GAO to examine (1) how PART changed OMB's fiscal year 2004 budget decisionmaking process, (2) PART's relationship to the Government Performance and Results Act of 1993 (GPRA), and (3) PART's strengths and weaknesses as an evaluation tool.
PART helped structure OMB's use of performance information for its internal program and budget analysis, made the use of this information more transparent, and stimulated agency interest in budget and performance integration. OMB and agency staff said this helped OMB staff with varying levels of experience focus on similar issues. Our analysis confirmed that one of PART's major impacts was its ability to highlight OMB's recommended changes in program management and design. Much of PART's potential value lies in the related program recommendations, but realizing these benefits requires sustained attention to implementation and oversight to determine if desired results are achieved. OMB needs to be cognizant of this as it considers capacity and workload issues in PART. There are inherent challenges in assigning a single rating to programs having multiple purposes and goals. OMB devoted considerable effort to promoting consistent ratings, but challenges remain in addressing inconsistencies among OMB staff, such as interpreting PART guidance and defining acceptable measures. Limited credible evidence on results also constrained OMB's ability to rate program effectiveness, as evidenced by the almost 50 percent of programs rated "results not demonstrated." PART is not well integrated with GPRA--the current statutory framework for strategic planning and reporting. By using the PART process to review and sometimes replace GPRA goals and measures, OMB is substituting its judgment for a wide range of stakeholder interests. The PART/GPRA tension was further highlighted by challenges in defining a unit of analysis useful for both program-level budget analysis and agency planning purposes. Although PART can stimulate discussion on program-specific measurement issues, it cannot substitute for GPRA's focus on thematic goals and department- and governmentwide crosscutting comparisons. Moreover, PART does not currently evaluate similar programs together to facilitate trade-offs or make relative comparisons. PART clearly must serve the President's interests. However, the many actors whose input is critical to decisions will not likely use performance information unless they feel it is credible and reflects a consensus on goals. It will be important for OMB to discuss timely with Congress the focus of PART assessments and clarify the results and limitations of PART and the underlying performance information. A more systematic congressional approach to providing its perspective on performance issues and goals could facilitate OMB's understanding of congressional priorities and thus increase PART's usefulness in budget deliberations.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-04-174, Performance Budgeting: Observations on the Use of OMB's Program Assessment Rating Tool for the Fiscal Year 2004 Budget
This is the accessible text file for GAO report number GAO-04-174
entitled 'Performance Budgeting: Observations on the Use of OMB's
Program Assessment Rating Tool for the Fiscal Year 2004 Budget' which
was released on January 30, 2004.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
January 2004:
PERFORMANCE BUDGETING:
Observations on the Use of OMB's Program Assessment Rating Tool for the
Fiscal Year 2004 Budget:
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-174] GAO-04-174:
GAO Highlights:
Highlights of GAO-04-174, a report to congressional requesters
Why GAO Did This Study:
The Office of Management and Budget‘s (OMB) Program Assessment Rating
Tool (PART) is meant to provide a consistent approach to evaluating
federal programs during budget formulation. To better understand its
potential, congressional requesters asked GAO to examine (1) how PART
changed OMB‘s fiscal year 2004 budget decision-making process, (2)
PART‘s relationship to the Government Performance and Results Act of
1993 (GPRA), and (3) PART‘s strengths and weaknesses as an evaluation
tool.
What GAO Found:
PART helped structure OMB‘s use of performance information for its
internal program and budget analysis, made the use of this information
more transparent, and stimulated agency interest in budget and
performance integration. OMB and agency staff said this helped OMB
staff with varying levels of experience focus on similar issues.
Our analysis confirmed that one of PART‘s major impacts was its
ability to highlight OMB‘s recommended changes in program management
and design. Much of PART‘s potential value lies in the related program
recommendations, but realizing these benefits requires sustained
attention to implementation and oversight to determine if desired
results are achieved. OMB needs to be cognizant of this as it
considers capacity and workload issues in PART.
There are inherent challenges in assigning a single rating to programs
having multiple purposes and goals. OMB devoted considerable effort to
promoting consistent ratings, but challenges remain in addressing
inconsistencies among OMB staff, such as interpreting PART guidance
and defining acceptable measures. Limited credible evidence on results
also constrained OMB‘s ability to rate program effectiveness, as
evidenced by the almost 50 percent of programs rated ’results not
demonstrated.“
PART is not well integrated with GPRA”the current statutory framework
for strategic planning and reporting. By using the PART process to
review and sometimes replace GPRA goals and measures, OMB is
substituting its judgment for a wide range of stakeholder interests.
The PART/GPRA tension was further highlighted by challenges in
defining a unit of analysis useful for both program-level budget
analysis and agency planning purposes. Although PART can stimulate
discussion on program-specific measurement issues, it cannot
substitute for GPRA‘s focus on thematic goals and department- and
governmentwide crosscutting comparisons. Moreover, PART does not
currently evaluate similar programs together to facilitate trade-offs
or make relative comparisons.
PART clearly must serve the President‘s interests. However, the many
actors whose input is critical to decisions will not likely use
performance information unless they feel it is credible and reflects a
consensus on goals. It will be important for OMB to discuss timely
with Congress the focus of PART assessments and clarify the results
and limitations of PART and the underlying performance information. A
more systematic congressional approach to providing its perspective on
performance issues and goals could facilitate OMB‘s understanding of
congressional priorities and thus increase PART‘s usefulness in budget
deliberations.
What GAO Recommends:
GAO recommends that OMB
(1) address the capacity demands of PART, (2) strengthen PART
guidance, (3) address evaluation information availability and scope
issues, (4) focus program selection on crosscutting comparisons and
critical operations, (5) broaden the dialogue with congressional
stakeholders, and (6) articulate and implement a complementary
relationship between PART and GPRA.
OMB generally agreed with our findings, conclusions, and
recommendations and stated that it is already taking actions to
address many of our recommendations.
GAO also suggests that Congress consider the need for a structured
approach to articulating its perspective and oversight agenda on
performance goals and priorities for key programs.
www.gao.gov/cgi-bin/getrpt?GAO-04-174.
To view the full product, including the scope and methodology, click
on the link above. For more information, contact Paul Posner at
(202) 512-9573 or posnerp@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
OMB Used the PART to Systematically Assess Program Performance and Make
Results Known, but Follow-up on PART Recommendations Is Uncertain:
Despite OMB's Considerable Efforts to Create a Credible Evaluation
Tool, PART Assessments Require Judgment and Were Constrained by Data
Limitations:
Subjective Terms and a Restrictive Format Contributed to Subjective and
Inconsistent Responses:
There Were Inconsistencies in Defining Acceptable Measures and in
Logically Responding to Question "Pairs":
Disagreements on Performance Information Led to Creation of a "Results
Not Demonstrated" Category:
The Fiscal Year 2004 PART Process Was a Parallel, Competing Approach to
GPRA's Performance Management Framework:
Defining a "Unit of Analysis" That Is Useful for Program-Level Budget
Analysis and Agency Planning Purposes Presents Challenges:
Conclusions and General Observations:
Matter for Congressional Consideration:
Recommendations for Executive Action:
Agency Comments:
Appendixes:
Appendix I: Scope and Methodology:
Appendix II: The Fiscal Year 2004 PART and Differences Between the
Fiscal Year 2004 and 2005 PARTs:
Section I: Program Purpose & Design (Yes, No, N/A):
Section II: Strategic Planning (Yes, No, N/A):
Section III: Program Management (Yes, No, N/A):
Section IV: Program Results (Yes, Large Extent, Small Extent, No):
Appendix III: Development of PART:
Fiscal Year 2003:
Fiscal Year 2004:
Fiscal Year 2005:
Appendix IV: Comments from the Office of Management and Budget:
Appendix V: GAO Contacts and Staff Acknowledgments:
GAO Contacts:
Acknowledgments:
Tables:
Table 1: Overview of Sections of PART Questions:
Table 2: Overview of PART Program Types:
Table 3: The Effect of Overall PART Score on Proposed Funding Changes
(Discretionary Programs):
Table 4: The Effect of Overall PART Score on Proposed Funding Changes
(Small Discretionary Programs):
Table 5: The Effect of Overall PART Score on Proposed Funding Changes
(Medium-Size Discretionary Programs):
Table 6: The Effect of Overall PART Score on Proposed Funding Changes
(Large Discretionary Programs):
Table 7: The Effect of PART Component Scores on Proposed Funding
Changes (All Discretionary Programs):
Table 8: The Effect of PART Component Scores on Proposed Funding
Changes (Small Discretionary Programs):
Table 9: Side-by-Side of the Fiscal Year 2005 PART and the Fiscal Year
2004 PART Questions:
Figures:
Figure 1: Fiscal Year 2004 PART Recommendations:
Figure 2: Number of Discretionary PART Programs by Rating and Funding
Result, Fiscal Years 2003-2004:
Figure 3: The PART Process and Budget Formulation Timelines:
Letter January 30, 2004:
The Honorable George V. Voinovich:
Chairman:
Subcommittee on Oversight of Government Management, the Federal
Workforce and the District of Columbia:
Committee on Governmental Affairs:
United States Senate:
The Honorable Todd R. Platts:
Chairman:
Subcommittee on Government Efficiency and Financial Management:
Committee on Government Reform:
House of Representatives:
The Honorable Sam Brownback:
United States Senate:
The Honorable Todd Tiahrt:
House of Representatives:
Since the 1950s, the federal government has attempted several
governmentwide initiatives designed to better align spending decisions
with expected performance--what is often commonly referred to as
"performance budgeting." Consensus exists that prior efforts--
including the Hoover Commission, the Planning-Programming-Budgeting-
System (PPBS), Management by Objectives, and Zero-Based Budgeting
(ZBB)--failed to significantly shift the focus of the federal budget
process from its long-standing concentration on the items of government
spending to the results of its programs.
In the 1990s, Congress and the executive branch laid out a statutory
and management framework that provides the foundation for strengthening
government performance and accountability, with the Government
Performance and Results Act of 1993[Footnote 1] (GPRA) as its
centerpiece. GPRA is designed to inform congressional and executive
decision making by providing objective information on the relative
effectiveness and efficiency of federal programs and spending. A key
purpose of the act is to create closer and clearer links between the
process of allocating scarce resources:
and the expected results to be achieved with those resources. This type
of integration is critical, as we have learned from prior initiatives
that failed in part because they did not prove to be relevant to budget
decision makers in the executive branch or Congress.[Footnote 2] GPRA
requires not only a connection to the structures used in congressional
budget presentations but also consultation between the executive and
legislative branches on agency strategic plans, which gives Congress an
oversight stake in GPRA's success.[Footnote 3]
In its overall structure, focus, and approach GPRA incorporates two
critical lessons learned from previous reforms. First, any approach
designed to link plans and budgets--that is, to link the responsibility
of the executive to define strategies and approaches with the
legislative "power of the purse"--must explicitly involve both branches
of our government. PPBS and ZBB failed in part because performance
plans and measures were developed in isolation from congressional
oversight and resource allocation processes.
Second, the concept of performance budgeting has and likely will
continue to evolve. Thus, no single definition of performance budgeting
encompasses the range of past and present needs and interests of
federal decision makers. The need for multiple definitions reflects the
differences in the roles various participants play in the budget
process. And, given the complexity and breadth of the federal budget
process, performance budgeting must encompass a variety of perspectives
in its efforts to link resources with results.
This administration has made the integration of performance and budget
information one of five governmentwide management priorities under its
President's Management Agenda (PMA).[Footnote 4] A central element in
this initiative is the Office of Management and Budget's (OMB) Program
Assessment Rating Tool (PART) that OMB describes as a diagnostic tool
meant to provide a consistent approach to evaluating federal programs
as part of the:
executive budget formulation process. The PART is the latest iteration
of 50 years of federal performance budgeting initiatives. It applies 25
questions to all "programs"[Footnote 5] under four broad topics: (1)
program purpose and design, (2) strategic planning, (3) program
management, and (4) program results (i.e., whether a program is meeting
its long-term and annual goals) as well as additional questions that
are specific to one of seven mechanisms or approaches used to deliver
the program.[Footnote 6]
To better understand the PART's potential as a mechanism for assessing
program goals and results, you asked us to examine (1) how the PART
changed OMB's decision-making process in developing the President's
fiscal year 2004 budget request; (2) the PART's relationship to the
GPRA planning process and reporting requirements; and (3) the PART's
strengths and weaknesses as an evaluation tool, including how OMB
ensured that the PART was applied consistently.
To respond to your request, we reviewed OMB materials on the
development and implementation of the PART as well as the results
produced by the PART assessments. To assess consistency of the PART's
application, we performed analyses of OMB data from the PART program
summary and assessment worksheets for each of the 234 programs OMB
reviewed for fiscal year 2004, including a statistical analysis of the
relationship between the PART scores and funding levels in the
President's Budget. We also identified several sets of similar programs
that we examined more closely to determine if comparable or disparate
criteria were applied in producing the PART results for these clusters
of programs. We reviewed 28 programs in nine clusters covering food
safety, water supply, military equipment procurement, provision of
health care, statistical agencies, block grants to assist vulnerable
populations, energy research programs, wildland fire management, and
disability compensation. We also interviewed OMB officials regarding
their experiences with the PART in the fiscal year 2004 budget process.
As part of our examination of the usefulness of the PART as an
evaluation tool and also to obtain agency perspectives on the
relationship between PART and GPRA, we interviewed department and
agency officials, including senior managers, and program, planning, and
budget staffs at (1) the Department of Health and Human Services (HHS),
(2) the Department of Energy (DOE), and (3) the Department of the
Interior (DOI). We selected these three departments because they had a
variety of program types (e.g., block/formula grants, competitive
grants, direct federal, and research and development) that were subject
to the PART and could provide a broad-based perspective on how the PART
was applied to different programs. With the exception of our summary
analyses of all 234 programs, the information obtained from OMB and
agency officials and our review of selected programs is not
generalizable to the PART process for all 234 programs. However, the
consistency and frequency with which similar issues were raised by OMB
and agency officials suggest that our review reliably captures several
significant and salient aspects of the PART as a budget and evaluation
tool.
Our review focused on the fiscal year 2004 PART process. We conducted
our work from May 2003 through October 2003 in accordance with
generally accepted government auditing standards. Detailed information
on our scope and methodology appears in appendix I. OMB provided
written comments on a draft of this report that are reprinted in
appendix IV.
Results in Brief:
The PART has helped to structure and discipline OMB's use of
performance information for its internal program analysis and budget
review, made the use of this information more transparent, and
stimulated agency interest in budget and performance integration. Both
OMB and agency staff noted that this helped ensure that OMB staff with
varying levels of experience focused on the same issues, fostering a
more disciplined approach to discussing program performance with
agencies. Several agency officials also told us that the PART was a
catalyst for bringing agency budget, planning, and program staff
together since none could fully respond to the PART questionnaire
alone.
Our analysis confirmed that one of the PART's major impacts was its
ability to highlight OMB's recommended changes in program management
and design. Over 80 percent of the recommendations made for the 234
programs assessed for the fiscal year 2004 budget process were for
improvements in program design, assessment, and program management;
less than 20 percent were related to funding issues. As OMB and others
recognize, performance is not the only factor in funding decisions.
Determining priorities--including funding priorities--is a function of
competing values and interests. Although OMB generally proposed to
increase funding for programs that received ratings of "effective" or
"moderately effective" and proposed to cut funding for those programs
that were rated "ineffective," our review confirmed OMB's statements
that funding decisions were not applied mechanistically. That is, for
some programs rated "effective" or "moderately effective" OMB
recommended funding decreases, while for several programs judged to be
"ineffective" OMB recommended additional funding in the President's
budget request with which to implement changes.
Much of the potential value of the PART lies in the related program
recommendations and associated improvements, but realization of these
benefits will require sustained attention to implementation and
oversight in order to determine if the desired results are being
achieved. Such attention and oversight takes time, and OMB needs to be
cognizant of this as it considers the capacity and workload issues in
the PART. Currently OMB plans to assess an additional 20 percent of all
federal programs annually. Each year, the number of recommendations
from previous years' evaluations will grow--and a system for monitoring
their implementation will become more critical. OMB encouraged its
Resource Management Offices (RMO) to consider many factors in selecting
programs for the fiscal year 2004 PART assessments, such as continuing
presidential initiatives and programs up for reauthorization. While all
programs would eventually be reviewed over the 5-year period, selecting
related programs for review in a given year would enable decision
makers to analyze the relative efficacy of similar programs in meeting
common or similar outcomes. We recommend that OMB centrally monitor and
report on agency implementation and progress on PART recommendations to
provide a governmentwide picture of progress and a consolidated view of
OMB's workload in this area. In addition, to target scarce analytic
resources and to focus decision makers' attention on the most pressing
policy issues, we recommend that OMB reconsider plans for 100 percent
coverage of federal programs by targeting PART assessments based on
such factors as the relative priorities, costs, and risks associated
with related clusters of programs and activities. We further recommend
that OMB select for review in the same year related or similar programs
or activities to facilitate such comparisons and trade-offs.
Developing a credible evidence-based rating tool to provide bottom-line
ratings for programs was a major impetus in developing the PART.
However, inherent challenges exist in assigning a single "rating" to
programs that often have multiple purposes and goals. Despite the
considerable time and effort OMB has devoted to promoting consistent
application of the PART, the tool is a work in progress. Additional
guidance and considerable revisions are needed to meet OMB's goal of an
objective, evidence-based assessment tool. In addition to difficulties
with the tool itself--such as subjective terminology and a restrictive
yes/no format--providing flexibility to assess multidimensional
programs with multiple purposes and goals often implemented through
multiple actors has led to a reliance on OMB staff judgments to apply
general principles to specific cases. OMB staff were not fully
consistent in interpreting the guidance for complex PART questions and
in defining acceptable measures. In addition, the limited availability
of credible evidence on program results also constrained OMB staff's
ability to use the PART to rate programs' effectiveness. Almost 50
percent of the 234 programs assessed for fiscal year 2004 received a
rating of "results not demonstrated" because OMB decided that program
performance information, performance measures, or both were
insufficient or inadequate. OMB, recognizing many of the limitations
with the PART, modified the PART for fiscal year 2005 based on lessons
learned during the fiscal year 2004 process, but issues remain. We
therefore recommend that OMB continue to improve the PART guidance by
(1) clarifying when output versus outcome measures are acceptable and
(2) better defining an "independent, quality evaluation." We further
recommend that OMB both clarify its expectations regarding the nature,
timing, and amount of evaluation information it wants from agencies for
the purposes of the PART and consider using internal agency evaluations
as evidence on a case-by-case basis.
The PART is not well integrated with GPRA--the current statutory
framework for strategic planning and reporting. According to OMB
officials, GPRA plans were organized at too high a level to be
meaningful for program-level budget decision making. To provide
decision makers with program-specific, outcome-based performance data
useful for executive budget formulation, OMB has stated its intention
to modify GPRA goals and measures with those developed under the PART.
As a result, OMB's judgment about appropriate goals and measures is
substituted for GPRA judgments based on a community of stakeholder
interests. Agency officials
we spoke with expressed confusion about the relationship between GPRA
requirements and the PART process. Many view PART's program-by-program
focus and the substitution of program measures as detrimental to their
GPRA planning and reporting processes. OMB's effort to influence
program goals is further evident in recent OMB Circular A-11
guidance[Footnote 7] that clearly requires each agency to submit a
performance budget for fiscal year 2005, which will replace the annual
GPRA performance plan.
The tension between PART and GPRA was further highlighted by the
challenges in defining a unit of analysis that is useful both for
program-level budget analysis and agency planning purposes. Although
the PART reviews indicated to OMB that GPRA measures are often not
sufficient to help it make judgments about programs, the different
units of analysis used in these two performance initiatives contributed
to this outcome. For the PART, OMB created units of analysis that tied
to discrete funding levels by both disaggregating and aggregating
certain programs. In some cases, disaggregating programs for the PART
reviews ignored the interdependency of programs by artificially
isolating them from the larger contexts in which they operate.
Conversely, in other cases in which OMB aggregated programs with
diverse missions and outcomes for the PART reviews, it became difficult
to settle on a single measure (or set of measures) that accurately
captured the multiple missions of these diverse components. Both of
these "unit of analysis" issues contributed to the lack of available
planning and performance information.
Although the PART can stimulate discussion on program-specific
performance measurement issues, it is not a substitute for GPRA's
strategic, longer-term focus on thematic goals and department-and
governmentwide crosscutting comparisons. GPRA is a broad legislative
framework that was designed to be consultative with Congress and other
stakeholders and allows for varying uses of performance information,
while the PART applies evaluation information to support decisions and
program reviews during the executive budget formulation process.
Moreover, GPRA can anchor the review of programs by providing an
overall strategic context for programs' contributions toward agency
goals. We therefore recommend that OMB seek to achieve the greatest
benefit from both GPRA and PART by articulating and implementing an
integrated, complementary relationship between the two. We further
recommend that OMB continue to improve the PART guidance by expanding
the discussion of how programs--also known as "units of analysis"--are
determined, including recognizing the trade-offs, implications, or both
of such determinations.
As part of the President's budget preparation, the PART clearly must
serve the President's interests. However, experience suggests that
efforts to integrate budget and performance are promoted when Congress
and other key stakeholders have confidence in the credibility of the
analysis and the process used. It is unlikely that the broad range of
players whose input is critical to decisions will use performance
information unless they believe it is relevant, credible, reliable, and
reflective of a consensus about performance goals among a community of
interested parties. Similarly, the measures used to demonstrate
progress toward a goal, no matter how worthwhile, cannot appear to
serve a single set of interests without potentially discouraging use of
this information by others. We therefore recommend that OMB attempt to
build on the strengths of GPRA and PART by seeking to communicate early
in the PART process with congressional appropriators and authorizers
about what performance issues and information are most important to
them in evaluating programs. Furthermore, while Congress has a number
of opportunities to provide its perspective on performance issues and
goals through its authorization, oversight, and appropriations
processes, we suggest that Congress consider the need for a more
structured approach for sharing with the executive branch its
perspective on governmentwide performance matters, including its views
on performance goals and outcomes for key programs and the oversight
agenda.
In commenting on a draft of this report, OMB generally agreed with our
findings, conclusions, and recommendations. OMB outlined actions it is
taking to address many of our recommendations, including refining the
process for monitoring agencies' progress in implementing the PART
recommendations, seeking opportunities for dialogue with Congress on
agencies' performance, and continuing to improve executive branch
implementation of GPRA plans and reports. OMB also suggested some
technical changes throughout the report that we have incorporated as
appropriate. OMB's comments appear in appendix IV. We also received
technical comments on excerpts of the draft provided to the Departments
of the Interior, Energy, and Health and Human Services, which are
incorporated as appropriate.
Background:
The current administration has taken several steps to strengthen and
further performance-resource linkages for which GPRA laid the
groundwork. Central to the budget and performance integration
initiative, the PART is meant to strengthen the process for assessing
the effectiveness of programs by making that process more robust,
transparent, and systematic. As noted above, the PART is a series of
diagnostic questions designed to provide a consistent approach to
rating federal programs. (See app. II for a reproduction of the PART.)
Drawing on available performance and evaluation information, the
questionnaire attempts to determine the strengths and weaknesses of
federal programs with a particular focus on individual program results.
The PART asks, for example, whether a program's long-term goals are
specific, ambitious, and focused on outcomes, and whether annual goals
demonstrate progress toward achieving long-term goals. It is designed
to be evidence based, drawing on a wide array of information, including
authorizing legislation, GPRA strategic plans and performance plans and
reports, financial statements, inspector general and GAO reports, and
independent program evaluations. PART questions are divided into four
sections; each section is given a specific weight in determining the
final numerical rating for a program. Table 1 shows an overview of the
four PART sections and the weights OMB assigned.
Table 1: Overview of Sections of PART Questions:
Section: I. Program Purpose and Design; Description: To assess whether;
* the purpose is clear, and;
* the program design makes sense;
Weight: 20%.
Section: II. Strategic Planning;
Description: To assess whether the
agency sets valid programmatic;
* annual goals, and;
* long-term goals;
Weight: 10%.
Section: III. Program Management;
Description: To rate agency management of the program, including;
* financial oversight, and;
* program improvement efforts;
Weight: 20%.
Section: IV. Program Results/Accountability;
Description: To rate program performance on goals reviewed in;
* the strategic planning section, and;
* through other evaluations;
Weight: 50%.
Source: GAO analysis of the Budget of the United States Government,
Fiscal Year 2004, Performance and Management Assessments (Washington,
D.C.: February 2003).
[End of table]
In addition, each PART program is assessed according to one of seven
approaches to service delivery. Table 2 provides an overview of these
program types and the number and percentage of programs covered by each
type in the fiscal year 2004 President's Budget performance
assessments.
Table 2: Overview of PART Program Types:
Program type: 1. Direct federal; Description: Programs in which support
and services are provided primarily by federal employees; Number/
percentage of programs[A]: 67; 29%.
Program type: 2. Block/formula grant; Description: Programs that
distribute funds to state, local, and tribal governments and other
entities by formula or block grant; Number/percentage of programs[A]:
41; 18%.
Program type: 3. Competitive grant; Description: Programs that
distribute funds to state, local, and tribal governments,
organizations, individuals, and other entities through a competitive
process; Number/percentage of programs[A]: 37; 16%.
Program type: 4. Capital assets and service acquisition; Description:
Programs in which the primary means to achieve goals is the development
and acquisition of capital assets (such as land, structures, equipment,
and intellectual property) or the purchase of services (such as
maintenance and information technology) from a commercial source;
Number/percentage of programs[A]: 34; 15%.
Program type: 5. Research and development; Description: Programs that
focus on creating knowledge or applying it toward the creation of
systems, devices, methods, materials, or technologies; Number/
percentage of programs[A]: 32; 14%.
Program type: 6. Regulatory-based; Description: Programs that employ
regulatory action to achieve program and agency goals through rule
making that implements, interprets, or prescribes law or policy, or
describes procedure or practice requirements. These programs issue
significant regulations, which are subject to OMB review; Number/
percentage of programs[A]: 15; 6%.
Program type: 7. Credit; Description: Programs that provide support
through loans, loan guarantees, and direct credit; Number/percentage
of programs[A]: 4; 2%.
Program type: 8. Mixed[B]; Description: Programs that contain elements
of different program types; Number/percentage of programs[A]: 4; 2%.
Source: GAO summary and analysis of the Budget of the United States
Government, Fiscal Year 2004, Performance and Management Assessments
(Washington, D.C.: February 2003).
[A] Percentages do not add to 100 percent due to rounding.
[B] OMB noted that in rare cases, drawing questions from two of the
seven PART program types--that is, creation of a "mixed" program type-
-yields a more informative assessment.
[End of table]
During the fiscal year 2004 budget cycle, OMB applied the PART to 234
programs (about 20 percent of the fiscal year 2004 President's Budget
request to Congress[Footnote 8]), and gave each program one of four
overall ratings: (1) "effective," (2) "moderately effective," (3)
"adequate," or (4) "ineffective" based on program design, strategic
planning, management, and results. A fifth rating, "results not
demonstrated," was given--independent of a program's numerical score--
if OMB decided that a program's performance information, performance
measures, or both were insufficient or inadequate. The administration
plans to assess an additional 20 percent of the budget each year until
the entire executive branch has been reviewed. For more information on
the development of the PART, see appendix III.
OMB Used the PART to Systematically Assess Program Performance and Make
Results Known, but Follow-up on PART Recommendations Is Uncertain:
The PART clarified OMB's use of performance information in its budget
decision-making process and stimulated new interest in budget and
performance integration. OMB generally proposed budget increases for
programs that received ratings of "effective" or "moderately effective"
and decreased funding requests for those programs that were rated
"ineffective," but there were clear exceptions. Moreover, the more
important role of the PART was not in making resource decisions but in
its support for recommendations to improve program design, assessment,
and management. OMB's ability to use the PART to identify and address
future program improvements and measure progress--a major purpose of
the PART--is predicated on its ability to oversee the implementation of
PART recommendations. However, it is not clear that OMB has a
centralized system to oversee the implementation of such
recommendations or evaluate their effectiveness.
The PART Made Budget and Performance Integration at OMB More
Transparent:
The PART helped structure and discipline the use of performance
information in the budget process and made the use of such information
more transparent throughout the executive branch. According to OMB
senior officials and many of the examiners and branch chiefs, the PART
lent structure to a process that had previously been informal and gave
OMB staff a systematic way of asking performance-related questions.
Both agency and OMB staff noted that this helped ensure that OMB staff
with varying levels of experience focused on the same issues, fostering
a more disciplined approach to discussing performance within OMB and
with agencies. Agency officials told us that by encouraging more
communication between departments and OMB, the PART helps illuminate
both how OMB makes budget decisions and how OMB staff think about
program management. The PART also provided a framework for raising
performance issues during the OMB Director's Reviews. OMB managers and
staff reported that it led to richer discussions on what a program
should be achieving, whether the program was performing effectively,
and how program performance could be improved.
Agencies also reported that the PART process expanded the dialogue
between program, planning, and budget staffs, and stimulated interest
in budget and performance integration. Several agency officials stated
that the PART worksheets were a catalyst for bringing staffs together
since none could fully respond to the questionnaire alone. OMB and
agency officials agreed that the PART led to more interactions between
OMB and agency program and planning staff and, in turn, increased
program managers' awareness of and involvement in the budget process.
According to OMB and several agency officials, the PART process--that
is, responding to the PART questionnaire--involved staff outside of the
performance management area. Additionally, both agency and OMB
officials said that the attention given to programs that were not
routinely reviewed was a positive benefit of the PART process.
Use of Performance Information Was Evident in OMB's Recommendations:
OMB senior officials told us that one of the PART's most notable
impacts was its ability to highlight OMB's recommended changes in
program management and design. As shown in figure 1, we found that 82
percent of PART recommendations addressed program assessment, design,
and management issues; only 18 percent of the recommendations had a
direct link to funding matters.[Footnote 9]
Figure 1: Fiscal Year 2004 PART Recommendations:
[See PDF for image]
[End of figure]
The majority of recommendations relate to changes that go well beyond
funding consideration for one budget cycle. For example, OMB and HHS
officials agree that the Foster Care program as it is currently
designed does not provide appropriate incentives for the permanent
placement of children; the program financially rewards states for
keeping children in foster care instead of the original intent of
providing temporary, safe, and appropriate homes for abused or
neglected children until children can be returned to their families or
other permanent arrangements can be made. The PART assessment provided
support for OMB's recommendation that legislation be introduced that
would create an option for states to participate in an alternate
financing program that would "better meet the needs of each state's
foster care population.":
Performance information included in the PART for the Department of
Labor's (DOL) Community Service Employment for Older Americans program
helped to shape OMB's recommendation to increase competition for the
grants. OMB concluded that although the Older Americans Act of 2000
amendments[Footnote 10] authorize competition for grants in cases in
which grantees repeatedly fail to perform, the programs' 10 national
grantees have historically been the sole recipients of grant funds
regardless of performance. OMB recommended that DOL award national
grants competitively to strengthen service delivery and open the door
to new grantees.
As OMB and others recognize, performance is not the only factor in
funding decisions. Determining priorities--including funding
priorities--is a function of competing values and interests. As seen in
figure 2, we found that PART scores were generally positively related
to proposed funding changes in discretionary programs but not in a
mechanistic way. In other words, PART scores did not automatically
determine funding changes. OMB proposed funding increases for most of
the programs rated "effective" or "moderately effective" and proposed
funding decreases for most of the programs rated "ineffective," but
there were clear exceptions. Programs rated as "results not
demonstrated"--which reflected a range of PART scores--had mixed
results.
Figure 2: Number of Discretionary PART Programs by Rating and Funding
Result, Fiscal Years 2003-2004:
[See PDF for image]
Note: Discretionary programs refer to those programs with budgetary
resources provided in appropriation acts. Because Congress controls
spending for mandatory programs--generally entitlement programs such as
food stamps, Medicare, and veterans' pensions--indirectly rather than
directly through the appropriations process, we excluded them from our
analysis. Of the 234 programs, we could not classify 11 as being either
predominantly mandatory or discretionary; these programs are excluded
from our analysis as well, and are listed in appendix I.
[End of figure]
A large portion of the variability in proposed budget changes could not
be explained by the quantitative measures reported by the PART.
Regressions of PART scores never explained more than about 15 percent
of the proposed budget changes. For only the one-third of discretionary
programs with the smallest budgets, we found that the composite PART
scores had a modest but statistically significant effect on proposed
budget changes (measured in percentage change) between fiscal years
2003 and 2004. For a fuller discussion of the statistical methods used,
see appendix I.
The relationship between performance levels and budget decisions was
not one-dimensional. For example, OMB rated the Department of Defense's
Basic Research program as "effective," but recommended a reduction in
congressionally earmarked projects that it stated did not meet the
program's merit review process. OMB also recommended reducing funding
for DOE's International Nuclear Materials Protection and Cooperation
program (rated "effective") because difficulties in obtaining
international agreements had resulted in the availability of sufficient
unobligated balances[Footnote 11] to make new funding unnecessary.
However, OMB sometimes proposed funding increases for programs that
were rated "ineffective" to implement improvement plans that had been
developed, such as the Internal Revenue Service's new Earned Income Tax
Credit compliance initiatives and DOE's revised environmental cleanup
plans for its Environmental Management (Cleanup) program.
Capacity Issues Could Affect OMB's Ability to Use the PART to Drive
Program Improvements:
OMB has said that a major purpose of the PART is to focus on program
improvements and measure progress. Effectively implementing PART
recommendations aimed at program improvements will require sustained
attention and sufficient oversight of agencies to ensure that the
recommendations are producing desirable results. However, each year,
the number of recommendations from previous years' evaluations will
grow. Currently, OMB plans to assess an additional 20 percent of all
federal programs annually such that all programs would eventually be
reviewed over a 5-year period. OMB encouraged its RMOs to consider a
variety of factors in selecting programs for the fiscal year 2004 PART
assessments, including continuing presidential initiatives and
programs up for reauthorization. Strengthening the focus on selecting
related programs for review in a given year would enable decision
makers to analyze the relative efficacy of similar programs in meeting
common or similar outcomes. As our work has shown, unfocused and
uncoordinated programs waste scarce funds, confuse and frustrate
program customers, and limit overall program effectiveness. Therefore
it is prudent to highlight crosscutting program efforts and clearly
relate and address the contributions of alternative federal strategies
toward meeting similar goals.
Although OMB has created a template for agencies to report on the
status of their recommendations and has reported that agencies are
implementing their PART recommendations, OMB has no central system for
monitoring agency progress or evaluating the effectiveness of changes.
While RMOs are responsible for overseeing agency progress, OMB senior
managers will not have a comprehensive governmentwide picture of
progress on the implementation of PART recommendations, nor will they
have a complete picture of OMB's workload in this area. As OMB has
recognized, following through on the recommendations is essential for
improving program performance and ensuring accountability.
Senior OMB managers readily recognized the increased workload the PART
placed on examiners--in one public forum we attended, a senior OMB
official described many examiners as being very concerned about the
additional workload. However, OMB expects the workload to decline as
OMB and agency staff become more familiar with the PART tool and
process, and as issues with the timing of the PART reviews are
resolved. Agency officials told us that originally, there was no formal
guidance for reassessing PART programs--it varied by RMO. When issued,
OMB's formal PART guidance limited reassessments to (1) updating the
status/implementation of recommendations from the fiscal year 2004 PART
and (2) revisiting specific questions for which new evidence exists.
OMB expected that in most reassessments, only those questions in which
change could be demonstrated would be "reopened." OMB officials
acknowledged that this formal guidance is at least partly due to
resource constraints.
OMB staff were divided on whether the PART assessments made an
appreciable difference in time spent on its budget review process. Many
of those we spoke with told us that their workloads during the
traditional budget season have always been heavy and that PART did not
add significantly to their work, especially since the PART generally
formalized a process already taking place. Those who did acknowledge
workload concerns said that they were surprised at the amount of time
it was taking to reassess programs. In fact, more than one OMB official
told us that reassessing programs was taking almost as long as brand-
new assessments, despite the fact that OMB scaled back the scope of
these reassessments.
Despite OMB's Considerable Efforts to Create a Credible Evaluation
Tool, PART Assessments Require Judgment and Were Constrained by Data
Limitations:
OMB went to great lengths to encourage consistent application of the
PART in the evaluation of government programs, including pilot testing
the instrument, issuing detailed guidance, and conducting consistency
reviews. However, while the instrument can undoubtedly be improved, any
tool that is sophisticated enough to take into account the complexity
of the U.S. government will always require OMB staff to exercise
interpretation and judgment. Providing flexibility to assess
multidimensional programs with multiple purposes and impacts has led to
a reliance on OMB staff judgments to apply general principles to
specific cases. Accordingly, OMB staff were not fully consistent in
interpreting complex questions about agency goals and results. In
addition, the limited availability of credible evidence on program
results also constrained OMB's ability to use the PART to rate
programs' effectiveness.
Inherent Performance Measurement Challenges Make It Difficult to
Meaningfully Interpret a Bottom-Line Rating:
OMB published a single, bottom-line rating for the PART results as well
as individual section scores, which are potentially more useful for
identifying information gaps and program weaknesses. For example, one
program that was rated "adequate" overall got high scores for purpose
(80 percent) and planning (100 percent), but did poorly in being able
to show results (39 percent) and in program management (46 percent).
Thus, the individual section ratings provided a better understanding of
areas needing improvement than the overall rating alone. Bottom-line
ratings inevitably force choices on what best exemplifies a program's
mission--even when a program has multiple goals--and encourages a
determination of the effectiveness of the program even when performance
data are unavailable, the quality of those data is uneven, or they
convey a mixed message on performance.
Many of the outcomes for which federal programs are responsible are
part of a broader effort involving federal, state, local, nonprofit,
and private partners. We have previously reported that it is often
difficult to isolate a particular program's contribution to an outcome
and especially so when it involves third parties.[Footnote 12] This was
reinforced by the results of the fiscal year 2004 PART reviews. One of
the patterns that OMB identified in its ratings was that grant programs
received lower than average ratings. To OMB this suggested the need for
greater effort by agencies to make grantees accountable for achieving
overall program results. However, grant structure and design play a
role in how federal agencies are able to hold third parties responsible
and complicate the process of identifying the individual contributions
of a federal program with multiple partners. In particular, block
grants present implementation challenges, especially in those instances
in which national goals are not compatible with state and local
priorities.
OMB Employed Numerous Tools and Techniques to Promote and Improve
Consistent Application of the PART:
OMB went to great lengths to encourage consistent application of the
PART in the evaluation of government programs. These efforts included
(1) testing the PART in selected agencies before use in the fiscal year
2004 assessment, (2) issuing detailed guidance and worksheets for use
by PART teams, (3) making the Performance Evaluation Team (PET)
available to answer PART implementation questions, (4) establishing an
Interagency Review Panel (IRP) to review consistency of PART
evaluations, and (5) making improvements to the fiscal year 2005
process and guidance based upon the fiscal year 2004 experience.
OMB conducted a pilot test of the PART and released a draft of the PART
questionnaire for public comment prior to its use for the fiscal year
2004 budget cycle. During Spring Review in 2002, OMB and agency staff
piloted the draft PART on 67 programs. The PART was also shared with
and commented on by the Performance Measurement Advisory Council and
other external groups. According to OMB, the results of the Spring
Review and feedback from external groups were used to revise the draft
version of the PART to lessen subjectivity and increase the consistency
of reviews.
OMB issued detailed guidance to help OMB and agency staff consistently
apply the PART and created electronic "templates" or worksheets to aid
in completing PART assessments. This guidance explains the purpose of
each question and describes the evidence required to support a "yes" or
"no" answer. In order to account for different types of programs,
several questions tailored to the seven program types were added to the
PART (primarily in Section III--Program Management). While the PART
guidance cannot be expected to cover every situation, the instructions
established general standards for PART evaluations.
PET addressed in "real time" questions and issues that OMB staff that
were completing the PART evaluations repeatedly raised. PET consisted
of examiners drawn from across the OMB organization representing a
variety of programmatic knowledge and experiences. It served as a
sounding board for OMB staff and a source for sharing experiences,
issues, and useful approaches and also provided training to OMB and
agency staff on the process. For example, in one OMB branch, staff were
grappling with how to apply the PART to a set of block grants. They
went through the instrument with the PET member from their RMO and
continued to consult with that individual throughout the process.
OMB also formed IRP, which consisted of both OMB and agency officials,
to conduct a consistency check of the PART reviews and to review formal
appeals of the process or results for particular questions. During the
fiscal year 2004 budget process, IRP conducted a consistency review of
10 percent of the PART evaluations using a subset of the PART questions
that OMB staff identified as being the most subjective or difficult to
interpret. IRP also reviewed formal agency appeals to determine whether
there was consistent treatment of similar situations.
As an Evaluation Tool, the PART Has Weaknesses in Its Design and, as a
Result, Its Implementation:
Despite the considerable time and effort OMB has devoted to promoting
consistent application of the PART, difficulties both with the tool
itself (such as subjective terminology and a restrictive yes/no format)
and with implementing the tool (including inconsistencies in defining
acceptable measures and contradictory answers to "pairs" of related
questions) aggravated the general performance measurement challenges
described earlier.
Subjective Terms and a Restrictive Format Contributed to Subjective and
Inconsistent Responses:
Many PART questions contain subjective terms that are open to
interpretation. Examples include terminology such as "ambitious" in
describing sought-after performance measures. Because the
appropriateness of a performance measure depends on the program's
purpose, and because program purposes can vary immensely, an ambitious
goal for one program might be unrealistic for a similar but more
narrowly defined program. Some agency officials claimed that having
multiple statutory goals disadvantaged their programs. Without further
guidance, subjective terminology can influence program ratings by
permitting OMB staff's views about a program's purpose to affect
assessments of the program's design and achievements.
Although OMB employed a yes/no format for the PART because OMB believes
it aided standardization, the format resulted in oversimplified answers
to some questions. OMB received comments on the yes/no format in
conducting the PART pilot. Some parties liked the certainty and forced
choice of yes/no. Others felt the format did not adequately distinguish
between the performance of various programs, especially in the results
section (originally in the yes/no format). In response to these
concerns, OMB revised the PART in the spring of 2002 to include four
response choices in the results section (adding "small extent" and
"large extent" to the original two choices "yes" and "no"), while
retaining the dichotomous yes/no format in the other three sections.
OMB acknowledged that a "yes" response should be definite and reflect a
very high standard of performance, and that it would more likely be
difficult to justify a "yes" answer than a "no" answer. Nonetheless,
agency officials have commented that the yes/no format is a crude
reflection of reality, in which progress in planning, management, or
results is more likely to resemble a continuum than an on/off switch.
Moreover, the yes/no format was particularly troublesome for questions
containing multiple criteria for a "yes" answer. As discussed
previously, we conducted an in-depth analysis of PART assessments for
28 related programs in nine clusters and compared the responses to
related questions. That analysis showed six instances in which some OMB
staff gave a "yes" answer for successfully achieving some but not all
of the multiple criteria, while others gave a "no" answer when
presented with a similar situation. For example, Section II, Question
1, asks, "Does the program have a limited number of specific,
ambitious, long-term performance goals that focus on outcomes and
meaningfully reflect the purpose of the program?" The PART defines
successful long-term goals by multiple, distinct characteristics
(program has long-term goals, time frames by which the goals are to be
achieved, etc.), but does not clarify whether a program can receive a
"yes" if each of the characteristics is met, or if most of the
characteristics are met. This contributed to a number of
inconsistencies across program reviews. For example, OMB judged DOI's
Water Reuse and Recycling program "no" on this question, noting that
although DOI set a long-term goal of 500,000 acre-feet per year of
reclaimed water, it failed to establish a time frame for when it would
reach the target. However, OMB judged the Department of Agriculture's
and DOI's Wildland Fire programs "yes" on this question even though the
programs' long-term goals of improved conditions in high-priority
forest acres are not accompanied by specific time frames. In another
example, OMB accepted DOD's recently established long-term strategic
goals for medical training and provision of health care even though it
did not yet have measures or targets for those goals. By breaking out
targets and ambitious time frames separately from the question of
annual goals, agencies have an opportunity to get credit for progress
made.
There Were Inconsistencies in Defining Acceptable Measures and in
Logically Responding to Question "Pairs":
In particular, our analysis of the nine program clusters revealed three
instances in which OMB staff inconsistently defined appropriate
measures--outcome versus output--for programs. Officials also told us
that OMB staff used different standards to define measures as outcome
oriented. This may reflect, in part, the complexity of and relationship
between expected program benefits. Outcomes are generally defined as
the results of outputs--products and services--delivered by a program.
But in some programs, long-term outcomes are expected to occur over
time through multiple steps. In these cases, short-term outcomes--
immediate changes in knowledge and awareness--might be expected to lead
to intermediate outcomes--behavioral changes in the future--and
eventually result in long-term outcomes--benefits to the public.
In the employment and training area, OMB accepted short-term outcomes,
such as obtaining high school diplomas or employment, as a proxy for
long-term goals for the HHS Refugee Assistance program, which aims to
help refugees attain economic self-sufficiency as soon as possible
after they arrive. However, OMB did not accept the same employment rate
measure as a proxy for long-term goals for the Department of
Education's Vocational Rehabilitation program because it had not set
long-term targets beyond a couple of years. In other words, although
neither program contained long-term outcomes, such as participants
gaining economic self-sufficiency, OMB accepted short-term outcomes in
one instance but not the other. Similarly, OMB gave credit for output
measures of claims processing (time, accuracy, and productivity) as a
proxy for long-term goals for the Social Security Administration's
Disability Insurance program, but did not accept the same output
measures for the Veterans Disability Compensation program. OMB took
steps to address this issue for fiscal year 2005.
We also found that three "question pairs" on the PART worksheets are
linked, yet in two of the three "pairs," a disconnect appeared in how
OMB staff responded to these questions for a given program.[Footnote
13] For example, 29 of the 90 programs (32 percent) judged as lacking
"independent and quality evaluations of sufficient scope conducted on a
regular basis" (Section II, Question 5) were also judged as having
"independent and quality evaluations that indicated the program is
effective and achieving results" (Section IV, Question 5). There is a
logical inconsistency in these two responses. In another instance,
there was no linkage between the questions that examine whether a
program has annual goals that demonstrate progress toward achieving
long-term goals and whether the program actually achieves its annual
goals. For example, 15 of the 75 programs (20 percent) judged not to
have adequate annual performance goals (Section II, Question 2) were
nevertheless credited for having made progress on their annual
performance goals (Section IV, Question 2). However, the guidance for
the latter question clearly indicates that a program must receive a
"no" if it received a "no" on the existence of annual goals (Section
II, Question 2). It seems that some raters held programs to a higher
standard for the quality of goals than for progress on them.
The Lack of Performance Information Creates Challenges in Effectively
Measuring Program Performance:
According to OMB, 115 out of 234 programs (49 percent) lacked
"specific, ambitious, long-term performance goals that focus on
outcomes" (Section II, Question 1). In addition, OMB found that 90 out
of 234 programs (38 percent) lacked sufficient "independent, quality
evaluations" (Section II, Question 5). While the validity of these
assessments may be subject to interpretation and debate, our previous
work[Footnote 14] has raised concerns about the capacity of federal
agencies to produce evaluations of program effectiveness.
The lack of evaluations may in part be driven by how OMB defined an
"independent and quality evaluation." To be independent, nonbiased
parties with no conflict of interest would conduct the evaluation, but
agency officials felt that OMB staff started from the default position
that agency-sponsored evaluations are, by definition, biased. However,
our detailed review of 28 PART worksheets found only 7 instances in
which OMB explicitly noted its rejection of evaluations: 1 for being
too old, 3 for not being independent (of the 3, 1 was an internal
agency review and 2 were conducted by industry groups), and the
remaining 3 for not assessing program results. OMB officials have
acknowledged that this issue was a point of friction with agencies and
that beyond GAO, inspectors general, and other government reports that
were automatically presumed to be independent, the independence
standard was considered on a case-by-case basis. In these case-by-case
situations, OMB staff told us that they looked for some degree of
detachment and objectivity in the evaluations. For example, in the case
of one DOE-sponsored evaluation, the OMB examiner attended the meetings
of the review group that conducted the evaluation in order to see
firsthand what sorts of questions the committee posed to the department
officials. In OMB's estimation, there was clear independence. While OMB
changed the fiscal year 2005 guidance to recognize evaluations
contracted out to third parties and agency program evaluation offices
as possibly being sufficiently independent, the new guidance generally
prohibits evaluations conducted by the program itself from being
considered "independent.":
Other reasons evaluation data may be limited include (1) constraints on
federal agencies' ability to influence program outcomes and reliance on
states and others for data for programs for which responsibility has
devolved to the states and (2) the lack of a statutory mandate or
dedicated funds for evaluation, which agency officials told us can
hamper efforts to conduct studies or to improve administrative data
collection.
As we have previously noted, program evaluations can take many forms
and agencies may obtain evaluations in a variety of ways.[Footnote 15]
Some evaluations simply analyze routinely collected program
administrative data; others involve special surveys. The type of
evaluation can greatly affect evaluation cost. Net impact evaluations
compare outcomes for program participants to those of a randomly
assigned control group and are designed for situations in which
external factors are also known to influence those outcomes. However,
the adequacy of an evaluation design can only be determined relative to
the circumstances of the program being evaluated. In addition, agencies
can obtain evaluations by having program or other agency staff collect
and analyze the data, by conducting the work jointly with program
partners (such as state agencies), or by hiring contract firms to do
so. Our survey of 81 federal agency offices conducting evaluations in
1995 of program results found they were most commonly located in
administrative offices at a major subdivision level or in program
offices (43 and 30 percent, respectively). Overall, they reported
conducting 51 percent of their studies in-house, while 34 percent were
contracted out. Depending on the sensitivity of the study questions,
agencies can conduct credible internal evaluations by adopting
procedures to ensure the reliability and validity of data collection
and analysis.
Disagreements on Performance Information Led to Creation of a "Results
Not Demonstrated" Category:
During the PART process OMB created an additional rating category,
"results not demonstrated," which was applied to programs regardless of
their score if OMB decided that one or both of two conditions
pertained: (1) OMB and the agency could not reach agreement on long-
term and annual performance measures and (2) there was inadequate
performance information. Almost 50 percent of the 234 programs assessed
for fiscal year 2004 received this rating of "results not
demonstrated," ranging from high-scoring programs such as the Consumer
Product Safety Commission (83) to low-scoring programs such as the
Department of Veterans Affairs Disability Compensation program (15).
OMB officials said that this rating was given to programs when
agreement could not be reached on long-term and annual performance
measures and was applied regardless of the program's PART score. Our
own review found that OMB generally assigned the "results not
demonstrated" rating as described above.[Footnote 16]
It is important for users of the PART information to interpret the
"results not demonstrated" designation as "unknown effectiveness"
rather than as meaning the program is "ineffective." Having evidence of
poor results is not the same as lacking evidence of effectiveness.
Because the PART guidance sets very high standards for obtaining a
"yes," a "no" answer can mean either that a program did not meet the
standards, or that there is no evidence on whether it met the
standards. In some readily measured areas, lack of evidence of an
action may indicate that the standard probably was not met. However,
because effectiveness is often not readily observed, lack of evidence
on program effectiveness cannot be automatically interpreted as meaning
that a program is ineffective. Furthermore, an agency might have
results for goals established under GPRA, but if OMB and the agency
could not reach agreement on new or revised goals or measures, then OMB
gave a program the rating "results not demonstrated.":
Changes to the PART and Related Guidance for Fiscal Year 2005 Are Meant
to Address Previously Identified Problems:
OMB, recognizing many of the issues we have just discussed, made
modifications to the PART instrument and guidance in time for the
fiscal year 2005 process. OMB said these changes were based upon
lessons learned during the fiscal year 2004 process and input from a
variety of sources, such as PET, IRP, and agency officials, although we
were unable to determine which changes resulted from which
recommendations. Although the PART as used for fiscal year 2005 is very
similar to that for fiscal year 2004, several questions were added,
dropped, merged with other questions, or divided into two questions.
For example, a research and development question used in the fiscal
year 2004 PART that received "not applicable" answers in 13 out of the
32 cases in which it was applied was dropped from the fiscal year 2005
PART. According to OMB officials, several of the multicriteria
questions were split into separate questions in order to reduce
inconsistency, as described earlier in this report. Appendix II
provides more complete information on the guidance changes between
fiscal years 2004 and 2005. To complement the fiscal year 2005 PART
guidance and offer strategies for addressing common performance
measurement challenges, many of which were encountered during the
fiscal year 2004 process, OMB released a separate document, titled
Performance Measurement Challenges and Strategies, which was the result
of a workshop in which agencies participated and identified measurement
challenges and shared best practices and possible work-arounds.
Instead of reestablishing IRP (which included both agency and OMB
representatives) for the fiscal year 2005 process, OMB officials told
us that PET (which included only OMB representatives) would conduct a
consistency review of 25 percent of all PART evaluations, with at least
one consistency check per OMB branch. OMB also told us that it has
asked the National Academy of Public Administration (NAPA) to review
PET's consistency review for the fiscal year 2005 process; the scope
and results of that review were not available to us during our audit
work.[Footnote 17] OMB senior officials cited resources, timing, and
the differing needs of the fiscal year 2004 and 2005 PART processes as
reasons for dropping the IRP review. The absence of agency
participation in this important phase of the PART could hamper ensuring
crucial transparency and credibility.
The Fiscal Year 2004 PART Process Was a Parallel, Competing Approach to
GPRA's Performance Management Framework:
The PART was designed for and is used in the executive branch budget
preparation and review process; as such, the goals and measures used in
the PART must meet OMB's needs. However, GPRA--the current statutory
framework for strategic planning and reporting--is a broader process
involving the development of strategic and performance goals and
objectives to be reported in strategic and annual plans. OMB's desire
to collect performance data that better align with budget decision
units means that the fiscal year 2004 PART process was a parallel
competing structure to the GPRA framework. Although OMB acknowledges
that GPRA was the starting point for the PART, as we explain below, the
emphasis is shifting such that over time the performance measures
developed for the PART and used in the budget process may come to drive
agencies' strategic planning processes.
Agencies told us that in some cases, OMB is replacing PART goals and
measures for those of GPRA. Effective for fiscal year 2005, OMB's
Circular A-11 guidance states that performance budgets are to replace
GPRA's annual performance plans. Agencies see the change as detrimental
to planning and reporting under GPRA and as a resource drain since they
have to respond to both GPRA and PART requirements. Some agency
officials told us that although the PART can stimulate discussion on
program-specific performance measurement issues, it is not a substitute
for GPRA's outcome-oriented, strategic look at thematic goals and
departmentwide program comparisons. Moreover, while the PART does not
eliminate the departmental strategic plans created under GPRA, many OMB
and agency officials told us that the PART is being used to shape the
strategic plans.
OMB's Efforts to Link Performance Information with the Budget Often
Conflict with Agencies' GPRA Planning Efforts:
OMB guidance and officials made clear that GPRA goals, measures, and
reports needed to be modified to provide decision makers with program-
specific, outcome-based performance data that better aligned with the
budget presentation in the President's Budget. According to OMB, such
changes were needed because performance reporting under GPRA had
evolved into a process separate from budget decision making, with GPRA
plans organized at too high a level to be meaningful for program-level
budget analysis and management review. Furthermore, according to OMB
officials, GPRA plans had too many performance measures, which made it
difficult to determine an agency's priorities. However, as some
officials pointed out, the cumulative effect of adding new PART
measures to GPRA plans may actually increase the number of measures
overall; both agency and OMB officials recognize that this is contrary
to goals issued by an OMB official previously responsible for the PART,
indicating his desire to reduce the number of GPRA measures by at least
25 percent in at least 70 percent of federal departments.[Footnote 18]
As a result of these sometimes-conflicting perspectives, agency
officials said that responding to both PART and GPRA requirements
increased their workloads and was a drain on staff resources.
OMB's most recent Circular A-11 guidance clearly requires that each
agency submit a performance budget for fiscal year 2005 and that this
should replace the annual GPRA performance plan.[Footnote 19] These
performance budgets are to include information from the PART
assessments, where available, including all performance goals used in
the assessment of program performance done under the PART process.
Until all programs have been assessed using the PART, the performance
budget will also include performance goals for agency programs that
have not yet been assessed using the PART. OMB's movement from GPRA to
PART is further evident in the fiscal year 2005 PART guidance stating
that while existing GPRA performance goals may be a starting point
during the development of PART performance goals, the GPRA goals in
agency GPRA documents are to be revised significantly, as needed, to
reflect OMB's instructions for developing the PART performance goals.
Lastly, this same guidance states that GPRA plans should be revised to
include any new performance measures used in the PART and unnecessary
measures should be deleted from GPRA plans.
OMB's interest in developing more useful program goals is further
evident in its PART recommendations. Almost half of the fiscal year
2004 PART recommendations related to performance assessment--
developing outcome goals and measures; cost or efficiency measures; and
increasing the tracking/monitoring of data, improving the tracking/
monitoring of data, or both. GPRA was generally the starting point for
PART discussions about goals and measures, and many agency officials
told us that OMB used the PART to modify agencies' existing GPRA goals
and measures. Agency officials reported that the discussions about
goals and measures were one of the main areas of contention during the
PART process. At the same time, agency officials acknowledged that (1)
sometimes OMB staff accepted current GPRA measures and (2) sometimes
the new PART measures and goals were improvements over the old GPRA
measures--the PART measures were more aggressive, more outcome-
oriented, more targeted, or all of the above.
Defining a "Unit of Analysis" That Is Useful for Program-Level Budget
Analysis and Agency Planning Purposes Presents Challenges:
The appropriate unit of analysis or "program" is not always obvious.
What OMB determined was useful for a PART assessment did not
necessarily match agency organization or planning elements. Although
the units of analysis varied across the PART assessments, OMB's
guidance stated that they should be linked to a recognized funding
level in the budget. In some cases, OMB aggregated separate programs
for the purposes of the PART, while in other cases it disaggregated
programs. Aggregating programs to tie them to discrete funding levels
sometimes made it difficult to create a limited, but comprehensive, set
of measures for programs with multiple missions. Disaggregating
programs sometimes ignored the interdependence of programs by
artificially isolating programs from the larger contexts in which they
operate. Both contributed to the lack of available planning and
performance information. For example, aggregating rural water supply
projects as a single unit of analysis may have been a logical choice
for reviewing related activities, but it created problems in
identifying planning and performance information useful for the PART
since these projects are separately administered. In another case, HHS
officials told us that the PART program Substance Abuse Treatment
Programs of Regional and National Significance is an amalgamation of
activities funded in a single budget line, not an actual program. They
said it was a challenge to make these activities look as if they
functioned as a single program.
Disaggregating a program too narrowly can create problems by distorting
its relationship to other programs involved in achieving a common goal.
For example, agency officials described a homeless program in which
outreach workers help homeless persons with emergency needs and refer
them to other agencies for housing and needed services. They said that
their OMB counterparts suggested that the program adopt long-term
outcome measures indicating number of persons housed. Agency officials
argued that chronically homeless people require many services and that
this federal program often supports only some of the services needed at
the initial stages of intervention. The federal program, therefore,
could contribute to, but not be primarily responsible for, affecting
late stages of the intervention process, like housing status.
These issues reveal some of the unresolved tensions between the
President's budget and performance initiative--a detailed budget
perspective--and GPRA--a more strategic planning view. In particular,
agency officials are concerned with problems in trying to respond to
both and overwhelmingly agreed that the PART required a large amount of
agency resources to complete. Moreover, some agency officials said that
the PART (a program-specific review) is not well suited to one of the
key purposes of strategic plans--to convey agencywide, long-term goals
and objectives for all major functions and operations. In addition, the
time horizons are different for the two initiatives--PART assessments
focus on program accomplishments to date while GPRA strategic planning
is long-term and prospective in nature.
Changes Made to GPRA in the PART Process Create Uncertainty About
Opportunities for Substantive Input by Interested Parties and
Congressional Stakeholders:
As noted above, PART goals and measures must meet OMB's needs, while
GPRA is a broader process involving the development of strategic and
performance goals and objectives to be reported in strategic and annual
plans. As a phased reform, GPRA required development of the planning
framework first, but also explicitly encouraged links to the
budget.[Footnote 20] Our work has shown that under GPRA agencies have
made significant progress.[Footnote 21] Additionally, GPRA requires
agencies to consult with Congress and solicit the views of other
stakeholders as they develop their strategic plans.[Footnote 22] We
have previously reported[Footnote 23] that stakeholder involvement
appears critical for getting consensus on goals and measures.
Stakeholder involvement can be particularly important for federal
agencies because they operate in a complex political environment in
which legislative mandates are often broadly stated and some
stakeholders may strongly disagree about the agency's mission and
goals.
The relationship between the PART and its process and the broader GPRA
strategic planning process is still evolving. Some tension between the
level of stakeholder involvement in the development of performance
measures in the GPRA strategic planning process and the process of
developing performance measures for the PART is inevitable. Compared to
the relatively open-ended GPRA process any budget formulation process
is likely to seem closed. An agency's communication with stakeholders,
including Congress, about goals and measures created or modified during
the formulation of the President's budget is likely to be less than
during the development of the agency's own strategic or performance
plan. Since different stakeholders have different needs and no one set
of goals and measures can serve all purposes, the PART can complement
GPRA but should not replace it.
Although these tensions between the need for internal deliberations and
broader consultations are inevitable, if the PART is to be accepted as
a credible element in the development of the President's budget
proposal, congressional understanding and acceptance of the tool and
its analysis will be important. In order for performance information to
more fully inform resource allocations, decision makers must also feel
comfortable with the appropriateness and accuracy of the performance
information and measures associated with these goals. It is unlikely
that decision makers will use performance information unless they
believe it is credible and reliable and reflects a consensus about
performance goals among a community of interested parties. Similarly,
the measures used to demonstrate progress toward a goal, no matter how
worthwhile, cannot serve the interests of a single stakeholder or
purpose without potentially discouraging use of this information by
others.
While it is still too soon to know whether OMB-directed measures will
satisfy the needs of other stakeholders and GPRA's broader planning
purposes, several appropriations subcommittees have stated, in their
appropriations hearings, the need to link the PART with congressional
oversight. For example, the House Committee on Appropriations,
Subcommittee on the Department of the Interior and Related Agencies
notes that while it supports the PMA, the costs of initiatives
associated with it have generally not been requested in annual budget
justifications or through reprogramming procedures.[Footnote 24] The
Subcommittee, therefore, has been unable to evaluate the costs,
benefits, and effectiveness of these initiatives or to weigh the
priority that these initiatives should receive as compared with ongoing
programs funded in the Interior Appropriations bill. Similarly, the
House Report on Treasury and Transportation Appropriations included a
statement in support of the PART, but noted that the administration's
efforts must be linked with the oversight of Congress to maximize the
utility of the PART process, and that if the administration treats as
privileged or confidential the details of its rating process, it is
less likely that Congress will use those results in deciding which
programs to fund. Moreover, the Subcommittee said it expects OMB to
involve the House and Senate Committees on Appropriations in the
development of the PART ratings at all stages in the process.[Footnote
25]
While Congress has a number of opportunities to provide its perspective
on performance issues and performance goals, such as when it
establishes or reauthorizes a new program, during the annual
appropriations process, and in its oversight of federal operations,
opportunities exist for Congress to more systematically articulate
performance goals and outcomes for key programs of major concern and to
allow for timely congressional input in the selection of the PART
programs to be assessed.
Conclusions and General Observations:
OMB, through its development and use of the PART, has more explicitly
infused performance information into the budget formulation process;
increased the attention paid to evaluation and performance information;
and ultimately, we hope, increased the value of this information to
decision makers and other stakeholders. By linking performance
information to the budget process, OMB has provided agencies with a
powerful incentive for improving data quality and availability. The
level of effort and involvement by senior OMB officials and staff
clearly signals the importance of this strategy in meeting the
priorities outlined in the PMA. OMB should be credited with opening up
for scrutiny--and potential criticism--its review of key areas of
federal program performance and then making its assessments available
to a potentially wider audience through its Web site.
While the PART clearly serves the needs of OMB in budget formulation,
questions remain about whether it serves the needs of other key
stakeholders. The PART could be strengthened to enhance its credibility
and prospects for sustainability by such actions as (1) improving
agencies' and OMB's capacity to cope with the demands of the PART, (2)
strengthening the PART guidance, (3) expanding the base of credible
performance information by strategically focusing evaluation
resources, (4) selecting programs for assessment to facilitate
crosscutting comparisons and trade-offs, (5) broadening the dialogue
with congressional stakeholders, and (6) articulating and implementing
a complementary relationship between PART and GPRA.
OMB's ambitious schedule for assessing all federal programs by the
fiscal year 2008 President's Budget will require a tremendous
commitment of OMB's and agencies' resources. Implementation of the PART
recommendations will be a longer-term and potentially more significant
result of the PART process than the scores and ratings. No less
important will be OMB's involvement both in encouraging agency progress
and in signaling its continuing commitment to improving program
management and results through the PART. OMB has created a template by
which agencies report on the status of the recommendations and left
follow-up on the recommendations to each RMO. However, there is no
single focal point for evaluating progress and the results of agency
efforts governmentwide; without this it will be difficult for OMB to
judge the efficacy of the PART and to know whether the increased
workload and trade-offs made with other activities is a good investment
of OMB and agency resources.
The goal of the PART is to evaluate programs systematically,
consistently, and transparently, but in practice, the tool requires OMB
staff to use independent judgment in interpreting the guidance and in
making yes or no decisions for what are often complex federal programs.
These difficulties are compounded by poor or partial program
performance data. Therefore, it is not surprising that we found
inconsistencies in our analysis of the fiscal year 2004 PART
assessments. Recognizing the inherent limitations of any tool to
provide a single performance answer or judgment on complex federal
programs with multiple goals, continued improvements in the PART
guidance, with examples throughout, can nonetheless help encourage a
higher level of consistency as well as transparency.
The PART requires more performance and evaluation information than
agencies currently have, as demonstrated by the fact that OMB rated
over 50 percent of the programs for fiscal year 2004 as "results not
demonstrated" because they "did not have adequate performance goals" or
"had not yet collected data to provide evidence of results." In the
past, we too have noted limitations in the quality of agency
performance and evaluation information and in agency capacity to
produce rigorous evaluations of program effectiveness. Furthermore, our
work has shown that few agencies deployed the rigorous research methods
required to attribute changes in underlying outcomes to program
activities. However, program evaluation information often requires
large amounts of agency resources to produce, and the agency and OMB
may not agree on what is important to measure, particularly when a set
of measures cannot serve multiple purposes. Agreement on what are a
department or agency's critical, high-risk programs and how best to
evaluate them could help leverage limited resources and help determine
what are the most important program evaluation data to collect.
Federal programs are designed and implemented in dynamic environments
where competing program priorities and stakeholders' needs must be
balanced continually and new needs must be addressed. GPRA is a broad
legislative framework that was designed to be consultative with
Congress and other stakeholders and allows for varying uses of
performance information, while the PART applies evaluation information
to support decisions and program reviews during the executive budget
formulation process. While the PART reflects the administration's
management principles and the priority given to using performance
information in OMB's decision-making process, its focus on program-
level assessments cannot substitute for the inclusive, crosscutting
strategic planning required by GPRA. Moreover, GPRA can anchor the
review of programs by providing an overall strategic context for
programs' contributions toward agency goals. Although PART and GPRA
serve different needs, a strategy for integrating the two could help
strengthen both.
Opportunities exist to develop a more strategic approach to the
selection and prioritization of areas to be assessed under the PART
process. Targeting PART assessments based on such factors as the
relative priorities, costs, and risks associated with related clusters
of programs and activities could not only help ration scarce analytic
resources but could also focus decision makers' attention on the most
pressing policy and program issues. Moreover, such an approach could
facilitate the use of PART assessments to review the relative
contributions of similar programs to common or crosscutting goals and
outcomes.
As part of the President's budget preparation, the PART clearly must
serve the President's interests. However, it is unlikely that the broad
range of actors whose input is critical to decisions will use
performance information unless they believe it is credible and reliable
and reflects a consensus about performance goals among a community of
interested parties. Similarly, the measures used to demonstrate
progress toward a goal, no matter how worthwhile, cannot appear to
serve a single set of interests without potentially discouraging use of
this information by others. If the President or OMB wants the PART and
its results to be considered in the congressional debate, it will be
important for OMB to (1) involve congressional stakeholders early in
providing input on the focus of the assessments; (2) clarify any
significant limitations in the assessments as well as the underlying
performance information; and (3) initiate discussions with key
congressional committees about how they can best take advantage of and
leverage PART information in authorizations, appropriations, and
oversight processes.
As we have previously reported, effective congressional oversight can
help improve federal performance by examining the program structures
agencies use to deliver products and services to ensure that the best,
most cost-effective mix of strategies is in place to meet agency and
national goals. While Congress has a number of opportunities to provide
its perspective on performance issues and performance goals, such as
when it establishes or reauthorizes a new program, during the annual
appropriations process, and in its oversight of federal operations, a
more systematic approach could allow Congress to better articulate
performance goals and outcomes for key programs of major concern. Such
an approach could also facilitate OMB's understanding of congressional
priorities and concerns and, as a result, increase the usefulness of
the PART in budget deliberations.
Matter for Congressional Consideration:
In order to facilitate an understanding of congressional priorities and
concerns, we suggest that Congress consider the need for a strategy
that could include (1) establishing a vehicle for communicating
performance goals and measures for key congressional priorities and
concerns; (2) developing a more structured oversight agenda to permit a
more coordinated congressional perspective on crosscutting programs and
policies; and (3) using such an agenda to inform its authorization,
oversight, and appropriations processes.
Recommendations for Executive Action:
We have seven recommendations to OMB for building on and improving the
first year's experience with the PART and its process. We recommend
that the Director of OMB take the following actions:
* Centrally monitor agency implementation and progress on PART
recommendations and report such progress in OMB's budget submission to
Congress. Governmentwide councils may be effective vehicles for
assisting OMB in these efforts.
* Continue to improve the PART guidance by (1) expanding the discussion
of how the unit of analysis is to be determined to include trade-offs
made when defining a unit of analysis, implications of how the unit of
analysis is defined, or both; (2) clarifying when output versus outcome
measures are acceptable; and (3) better defining an "independent,
quality evaluation.":
* Clarify OMB's expectations to agencies regarding the allocation of
scarce evaluation resources among programs, the timing of such
evaluations, as well as the evaluation strategies it wants for the
purposes of the PART, and consider using internal agency evaluations as
evidence on a case-by-case basis--whether conducted by agencies,
contractors, or other parties.
* Reconsider plans for 100 percent coverage of federal programs and,
instead, target for review a significant percentage of major and
meaningful government programs based on such factors as the relative
priorities, costs, and risks associated with related clusters of
programs and activities.
* Maximize the opportunity to review similar programs or activities in
the same year to facilitate comparisons and trade-offs.
* Attempt to generate, early in the PART process, an ongoing,
meaningful dialogue with congressional appropriations, authorization,
and oversight committees about what they consider to be the most
important performance issues and program areas warranting review.
* Seek to achieve the greatest benefit from both GPRA and PART by
articulating and implementing an integrated, complementary
relationship between the two.
Agency Comments:
We provided a draft of this report to OMB for its review and comment.
OMB generally agreed with our findings, conclusions, and
recommendations. In addition, OMB outlined actions it is taking to
address many of our recommendations, including refining the process for
monitoring agencies' progress in implementing the PART recommendations,
seeking opportunities for dialogue with Congress on agencies'
performance, and continuing to improve executive branch implementation
of GPRA plans and reports. OMB officials provided a number of technical
comments and clarifications, which we incorporated as appropriate to
ensure the accuracy of our report. OMB's comments appear in appendix
IV. We also received technical comments on excerpts of the draft
provided to the Departments of the Interior, Energy, and Health and
Human Services. Comments received from the Departments of Energy and
the Interior were incorporated as appropriate. The Department of Health
and Human Services had no comments.
OMB noted that performance information gleaned from the PART process
has not only informed budget decisions but has also helped direct
program management, identified opportunities to improve program design,
and promoted accountability. We agree. As shown in figure 1 in our
report, we found that 82 percent of PART recommendations addressed
program assessment, design, and management issues; only 18 percent of
the recommendations had a direct link to funding matters.
We are sending copies of this report to the Director of OMB,
appropriate congressional committees, and other interested members of
Congress. We will also make copies available to others upon request. In
addition, the report will be available at no charge on the GAO Web site
at [Hyperlink, http://www.gao.gov] http://www.gao.gov.
If you or your staff have questions about this report, please contact
Paul Posner at (202) 512-9573 or [Hyperlink, posnerp@gao.gov]
posnerp@gao.gov. An additional contact and key contributors to this
report are listed in appendix V.
Signed by:
David M. Walker
Comptroller General of the United States:
[End of section]
Appendixes:
Appendix I: Scope and Methodology:
To address the objectives in this report, we reviewed Office of
Management and Budget (OMB) materials and presentations on the
development and implementation of the Program Assessment Rating Tool
(PART) as well as the results of the PART assessments. Our review of
materials included instructions for using PART, OMB's testimony
concerning PART, and public remarks made by OMB officials at relevant
conferences and training. We also reviewed PART-related information on
OMB's Web site, including the OMB worksheets used to support the
assessments, and attended OMB's PART training for the fiscal year 2004
process.
For this report, we focused on the process and final results of the
fiscal year 2004 PART process, but also looked at the initial stages of
the fiscal year 2005 process. We compared the PART guidance for both
years and asked agency and OMB staff to discuss generally the
differences between the 2 fiscal years. We did not review the final
results for the fiscal year 2005 PART, which are embargoed until the
publication of the President's fiscal year 2005 budget request. For the
same reasons, we did not review the results of any reassessments
conducted for fiscal year 2005 on programs originally assessed for
fiscal year 2004. This report presents the experiences of staff from
the three departments and OMB officials who we interviewed. We did not
directly observe the PART process (for either year) in operation nor
did we independently verify the PART assessments as posted on OMB's Web
site or the program or financial information contained in the documents
provided as evidence for the PART assessments. We did, however, take
several steps to ensure that we reliably downloaded and combined the
PART summaries and worksheets with our budget and recommendation
classifications. Our steps included (1) having the computer programs we
used to create and process our consolidated dataset verified by a
second programmer; (2) having transcribed data elements from all
programs checked back to source files; and (3) having selected,
computer-processed data elements checked back to source files for a
random sample of programs and also for specific programs identified in
our analyses.
To better understand the universe of programs OMB assessed for fiscal
year 2004, we developed overall profiles of PART results and examined
relationships between such characteristics as type of program, type of
recommendation, overall rating, total PART score, and answers for each
question on PART. This review enabled us to generally confirm some
information previously reported by OMB, for example, that PART scores
do not automatically determine proposed funding and that grant programs
scored lower overall than other types of programs. It also allowed us
to select a sample of programs for more in-depth review, and this
sample was used to determine which OMB and agency officials we
interviewed.
To gain a better understanding of the PART process at both OMB and
agencies, to inform our examination of the usefulness of PART as an
evaluation tool, and to obtain various perspectives on the relationship
between PART and GPRA, we interviewed officials at OMB and three
selected departments. At OMB, we interviewed a range of staff, such as
associate directors, deputy assistant directors, branch chiefs, and
examiners. Specifically, we interviewed staff in two Resource
Management Offices (RMO). In the Human Resources Programs RMO, we spoke
with staff from the Health Division and the Education and Human
Resources Division. In the Natural Resources, Energy and Science RMO we
interviewed staff from the Energy and Interior Branches. In addition,
we obtained the views of two groups within OMB that were convened
specifically for the PART process: the Performance Evaluation Team
(PET) and the Interagency Review Panel (IRP). The IRP included agency
officials in addition to staff from OMB.
The three departments for which we reviewed the PART process were the
Department of Energy (DOE), the Department of Health and Human Services
(HHS), and the Department of the Interior (DOI). We selected these
three departments based on our data analysis of program types. The
departments selected and their agencies had a variety of program types
(e.g., block/formula grants, competitive grants, direct federal, and
research and development) that were subject to PART and could provide
us with a broad-based perspective on how PART was applied to different
programs employing diverse tools of government. We also chose these
three departments because they had programs under PART review within
the two RMOs at OMB where we did more extensive interviewing, thus
enabling us to develop a more in-depth understanding of how the PART
process operated for a subset of programs. We used this information to
complement our broader profiling of all 234 programs assessed. Within
DOE we studied the experiences of the Office of Science, the Office of
Energy Efficiency and Renewable Energy, and the Office of Fossil
Energy. Within HHS, we studied the experiences of the Administration
for Children and Families, the Health Resources and Services
Administration, and the Substance Abuse and Mental Health Services
Administration. Within DOI, we studied the experiences of the Bureau of
Land Management, the Bureau of Indian Affairs, and the National Park
Service. We interviewed planning, budget, and program staff within each
of the nine agencies as well as those at the department level. We also
reviewed relevant supporting materials provided by these departments in
conjunction with these interviews.
To allow us to describe how PART was used in fiscal year 2004 to
influence changes in future performance, we created a consolidated
dataset in which we classified recommendations OMB made by three areas
in need of improvement: (1) program design, (2) program management, and
(3) program assessment. A fourth category was created for those
recommendations that involved funding issues. We created a consolidated
dataset of information from our analysis of recommendations and
selected information from the PART program summary page and worksheet
for each program.[Footnote 26]
In addition, for approximately 95 percent of the programs, we
identified whether the basis for program funding was mandatory or
discretionary. It was important to separate discretionary and mandatory
programs in our review of PART's potential influence on the President's
budget proposals because funding for mandatory programs is determined
through authorizations, not through the annual appropriations process.
Of the 234 programs that OMB assessed for fiscal year 2004, we
identified 27 mandatory programs and 196 discretionary, but could not
categorize 11 programs as solely mandatory or discretionary because
they were too mixed to classify.[Footnote 27]
For discretionary programs, we explored the relationship between PART
results and proposed budget changes in a series of regression
analyses.[Footnote 28] Using statistical analysis, we found that PART
scores influenced proposed funding changes for discretionary programs;
however, a large amount of variability in these changes remains
unexplained. We examined proposed funding changes between fiscal years
2003 and 2004 (measured by percentage change) and the relationship to
PART scores for the programs assessed in the fiscal year 2004
President's Budget. These scores are the weighted sums of scores for
four PART categories: Program Purpose and Design, Strategic Planning,
Program Management, and Program Results and Accountability. The
corresponding weights assigned by OMB are 0.2, 0.1, 0.2, and 0.5,
respectively.[Footnote 29] Tables in this appendix report regression
results obtained using the method of least squares with
heteroskedasticity-corrected standard errors.[Footnote 30] The same
estimation method is used throughout this analysis.
Overall PART scores have a positive and statistically significant
effect on discretionary program funding. The programs evaluated by OMB
include both mandatory and discretionary programs. Regression results
for mandatory programs showed--as expected--no relationship between
PART scores and the level of funding in the President's Budget
proposal. Assessment ratings, however, can potentially affect the
funding for discretionary programs either in the President's Budget
proposal or in congressional deliberations on spending bills.[Footnote
31] Table 3 reports the regression results for discretionary programs.
Table 3: The Effect of Overall PART Score on Proposed Funding Changes
(Discretionary Programs):
Variable: Overall PART score;
Coefficient estimate: 0.536;
Robust standard error: 0.159;
t-Statistic: 3.38;
P-value: 0.001.
Variable: Constant;
Coefficient estimate: -25.671;
Robust standard error: 8.682;
t-Statistic: -2.96;
P-value: 0.003.
Source: GAO analysis of OMB data.
Notes: R-squared = 0.058, Prob-F = 0.001, N = 196. Originally we
identified 197 discretionary programs. However, no fiscal year 2004
budget estimate is reported for the Disclosed Worker Assistance
program due to grant consolidation at the Department of Labor. (Budget
of the United States Government, Fiscal Year 2004, Performance and
Management Assessments (Washington, D.C.: February 2003), 191.) This
reduced the number of discretionary programs to 196.
[End of table]
The estimated coefficient of the overall score is positive and
significant. These results show that the aggregate PART score has a
positive and statistically significant effect on the proposed change
in discretionary programs' budget, suggesting that programs with
better scores are more likely to receive larger proposed budget
increases.
To examine the effect of program size on our results, we divided all
programs equally into three groups--small, medium, and large--based on
their fiscal year 2003 funding estimate. Regressions similar to those
reported in table 3 were then performed for discretionary programs in
each group. The results, reported in tables 4, 5, and 6 suggest that
the statistically significant effect of overall scores on budget
outcomes exists only for the smaller programs. The estimated
coefficient of the overall score for large programs, which is
significant but only at the 10 percent level, reflects an outlier.
[Footnote 32] Once this outlier is dropped, the estimated coefficient
becomes statistically insignificant.
Table 4: The Effect of Overall PART Score on Proposed Funding Changes
(Small Discretionary Programs):
Variable: Overall PART score;
Coefficient estimate: 1.074;
Robust standard error: 0.404;
t-Statistic: 2.66;
P-value: 0.010.
Variable: Constant;
Coefficient estimate: -50.523;
Robust standard error: 21.155;
t-Statistic: -2.39;
P-value: 0.020.
Source: GAO analysis of OMB data.
Note: R-squared = 0.092, Prob-F = 0.01, N = 71.
[End of table]
Table 5: The Effect of Overall PART Score on Proposed Funding Changes
(Medium-Size Discretionary Programs):
Variable: Overall PART score;
Coefficient estimate: 0.306;
Robust standard error: 0.188;
t-Statistic: 1.62;
P-value: 0.109.
Variable: Constant;
Coefficient estimate: -17.984;
Robust standard error: 12.480;
t-Statistic: -1.44;
P-value: 0.154.
Source: GAO analysis of OMB data.
Note: R-squared = 0.039, Prob-F = 0.109, N = 67.
[End of table]
Table 6: The Effect of Overall PART Score on Proposed Funding Changes
(Large Discretionary Programs):
Variable: Overall PART score;
Coefficient estimate: 0.194;
Robust standard error: 0.109;
t-Statistic: 1.77;
P-value: 0.082.
Variable: Constant;
Coefficient estimate: -8.216;
Robust standard error: 7.778;
t-Statistic: -1.06;
P-value: 0.295.
Source: GAO analysis of OMB data.
Note: R-squared = 0.057, Prob-F = 0.082, N = 58.
[End of table]
The statistical analysis suggests that among the four components of
the PART questionnaire, program purpose, management, and results have
statistically significant effects on proposed funding changes, but the
effects of program purpose and results are more robust across the
estimated models. The overall score is a weighted average of four
components: Program Purpose and Design, Strategic Planning, Program
Management, and Program Results and Accountability.[Footnote 33] To
identify which of the four components contribute to the significant
relationship observed here, we examined the effect of each on proposed
changes in programs' funding levels. Tables 7 and 8 show estimates
from regressions of the proposed funding change on purpose, planning,
management, and results scores for all discretionary programs as well
as small discretionary programs alone.
Table 7: The Effect of PART Component Scores on Proposed Funding
Changes (All Discretionary Programs):
Variable: Purpose;
Coefficient estimate: 0.325;
Robust standard error: 0.127;
t-Statistic: 2.56;
P-value: 0.011.
Variable: Plan;
Coefficient estimate: -0.259;
Robust standard error: 0.199;
t-Statistic: -1.30;
P-value: 0.194.
Variable: Management;
Coefficient estimate: 0.191;
Robust standard error: 0.117;
t-Statistic: 1.63;
P-value: 0.105.
Variable: Results;
Coefficient estimate: 0.363;
Robust standard error: 0.205;
t-Statistic: 1.77;
P-value: 0.078.
Variable: Constant;
Coefficient estimate: -33.096;
Robust standard error: 14.136;
t-Statistic: -2.34;
P-value: 0.020.
Source: GAO analysis of OMB data.
Note: R-squared = 0.087, Prob-F = 0.003, N = 196.
[End of table]
Table 8: The Effect of PART Component Scores on Proposed Funding
Changes (Small Discretionary Programs):
Variable: Purpose;
Coefficient estimate: 0.223;
Robust standard error: 0.274;
t-Statistic: 0.81;
P-value: 0.419.
Variable: Plan;
Coefficient estimate: -0.671;
Robust standard error: 0.543;
t-Statistic: -1.24;
P-value: 0.221.
Variable: Management;
Coefficient estimate: 0.547;
Robust standard error: 0.304;
t-Statistic: 1.80;
P-value: 0.077.
Variable: Results;
Coefficient estimate: 0.956;
Robust standard error: 0.534;
t-Statistic: 1.79;
P-value: 0.078.
Variable: Constant;
Coefficient estimate: -42.455;
Robust standard error: 34.800;
t-Statistic: -1.22;
P-value: 0.227.
Source: GAO analysis of OMB data.
Note: R-squared = 0.149, Prob-F = 0.043, N = 71.
[End of table]
These results suggest that among the four components, program purpose,
management, and results are more likely to affect the proposed budget
changes for discretionary programs. When all discretionary programs are
included, the estimated coefficients are positive and significant for
results (at the 10 percent level) and purpose. When only the small
discretionary programs are included, the estimated coefficients are
positive and significant for both management and results (at the 10
percent level). We also estimated the above regression for medium and
large programs, but coefficient estimates were not statistically
significant, except for the estimated coefficient of purpose for medium
programs.
PART scores explain at most about 15 percent of the proposed funding
changes, leaving a large portion of the variability in proposed funding
changes unexplained. This suggests that most of the variance is due to
institutional factors, program specifics, and other unquantifiable
factors. The coefficient of determination (or R2) is used to measure
the proportion of the total variation in the regression's dependent
variable that is explained by the variation in the regressors
(independent variables).[Footnote 34] The maximum value of this measure
across all estimated regressions is about 15 percent.
Similar analyses were carried out for changes in the proposed budget
for fiscal year 2004 and congressionally appropriated amounts in fiscal
year 2002. Results were qualitatively similar to those reported here.
To assess the strengths and weaknesses of PART as an evaluation tool
and the consistency with which it was applied, we analyzed data from
all 234 programs that OMB reviewed using PART for fiscal year 2004. As
part of our examination of the consistency with which PART was applied
to programs, we also focused on a subset of programs to assess the way
in which certain measurement issues were addressed across those
programs. The issues were selected from those identified in interviews
with officials from the selected agencies described above and our own
review of the PART program summaries and worksheets. Measurement issues
included acceptance of output versus outcome measures of annual and
long-term goals, types of studies accepted as program evaluations,
acknowledgment of related programs, and justifications for judging a
PART question as "not applicable." Programs were selected that formed
clusters, each addressing a similar goal or shared a structural
similarity pertinent to performance measurement, to examine whether
PART assessment issues were handled similarly across programs when
expected. We reviewed the worksheets and compared the treatment of
assessment issues across specific questions within and across programs
in a cluster to identify potential inconsistencies in how the tool was
applied. We reviewed a total of 28 programs in nine clusters. The nine
clusters are food safety, water supply, military equipment procurement,
provision of health care, statistical agencies, block grants to assist
vulnerable populations, energy research programs, wildland fire
management, and disability compensation.
With the exception of our summary analyses of all 234 programs, the
information obtained from OMB and agency interviews, related material,
and review of selected programs is not generalizable to the PART
process for all 234 programs reviewed in fiscal year 2004. We conducted
our review from May through October 2003 in accordance with generally
accepted government auditing standards.
[End of section]
Appendix II: The Fiscal Year 2004 PART and Differences Between the
Fiscal Year 2004 and 2005 PARTs:
Below we have reproduced OMB's fiscal year 2004 PART instrument. We
have also included the comparison of fiscal year 2004 and fiscal year
2005 PART questions that appeared in the fiscal year 2005 PART guidance
(see table 9).
Section I: Program Purpose & Design (Yes, No, N/A):
1. Is the program purpose clear?
2. Does the program address a specific interest, problem or need?
3. Is the program designed to have a significant impact in addressing
the interest, problem or need?
4. Is the program designed to make a unique contribution in addressing
the interest, problem or need (i.e., not needlessly redundant of any
other Federal, state, local or private efforts)?
5. Is the program optimally designed to address the interest, problem
or need?
Specific Program Purpose & Design Questions by Program Type:
Research and Development Programs:
6. (RD. 1) Does the program effectively articulate potential public
benefits?
7. (RD. 2) If an industry-related problem, can the program explain how
the market fails to motivate private investment?
Section II: Strategic Planning (Yes, No, N/A):
1. Does the program have a limited number of specific, ambitious long-
term performance goals that focus on outcomes and meaningfully reflect
the purpose of the program?
2. Does the program have a limited number of annual performance goals
that demonstrate progress toward achieving the long-term goals?
3. Do all partners (grantees, subgrantees, contractors, etc.) support
program-planning efforts by committing to the annual and/or long-term
goals of the program?
4. Does the program collaborate and coordinate effectively with related
programs that share similar goals and objectives?
5. Are independent and quality evaluations of sufficient scope
conducted on a regular basis or as needed to fill gaps in performance
information to support program improvements and evaluate effectiveness?
6. Is the program budget aligned with the program goals in such a way
that the impact of funding, policy, and legislative changes on
performance is readily known?
7. Has the program taken meaningful steps to address its strategic
planning deficiencies?
Specific Strategic Planning Questions by Program Type:
Regulatory-Based Programs:
8. (RD. 1) Are all regulations issued by the program/agency necessary
to meet the stated goals of the program, and do all regulations clearly
indicate how the rules contribute to achievement of the goals?
Capital Assets and Service Acquisition Programs:
8. (Cap. 1) Are acquisition program plans adjusted in response to
performance data and changing conditions?
9. (Cap. 2) Has the agency/program conducted a recent, meaningful,
credible analysis of alternatives that includes trade-offs between
cost, schedule and performance goals?
Research and Development Programs:
8. (RD. 1) Is evaluation of the program's continuing relevance to
mission, fields of science, and other "customer" needs conducted on a
regular basis?
9. (RD. 2) Has the program identified clear priorities?
Section III: Program Management (Yes, No, N/A):
1. Does the agency regularly collect timely and credible performance
information, including information from key program partners, and use
it to manage the program and improve performance?
2. Are Federal managers and program partners (grantees, subgrantees,
contractors, etc.) held accountable for cost, schedule and performance
results?
3. Are all funds (Federal and partners') obligated in a timely manner
and spent for the intended purpose?
4. Does the program have incentives and procedures (e.g., competitive
sourcing/cost comparisons, IT improvements) to measure and achieve
efficiencies and cost effectiveness in program execution?
5. Does the agency estimate and budget for the full annual costs of
operating the program (including all administrative costs and allocated
overhead) so that program performance changes are identified with
changes in funding levels?
6. Does the program use strong financial management practices?
7. Has the program taken meaningful steps to address its management
deficiencies?
Specific Program Management Questions by Program Type:
Competitive Grant Programs:
8. (Co. 1) Are grant applications independently reviewed based on clear
criteria (rather than earmarked) and are awards made based on results
of the peer review process?
9. (Co. 2) Does the grant competition encourage the participation of
new/first-time grantees through a fair and open application process?
10. (Co. 3) Does the program have oversight practices that provide
sufficient knowledge of grantee activities?
11. (Co. 4) Does the program collect performance data on an annual
basis and make it available to the public in a transparent and
meaningful manner?
Block/Formula Grant Programs:
8. (B. 1) Does the program have oversight practices that provide
sufficient knowledge of grantee activities?
9. (B. 2) Does the program collect grantee performance data on an
annual basis and make it available to the public in a transparent and
meaningful manner?
Regulatory-Based Programs:
8. (Reg. 1) Did the program seek and take into account the views of
affected parties including state, local and tribal governments and
small businesses, in drafting significant regulations?
9. (Reg. 2) Did the program prepare, where appropriate, a Regulatory
Impact Analysis that comports with OMB's economic analysis guidelines
and have these RIA analyses and supporting science and economic data
been subjected to external peer review by qualified specialists?
10. (Reg. 3) Does the program systematically review its current
regulations to ensure consistency among all regulations in
accomplishing program goals?
11. (Reg. 4) In developing new regulations, are incremental societal
costs and benefits compared?
12. (Reg. 5) Did the regulatory changes to the program maximize net
benefits?
13. (Reg. 6) Does the program impose the least burden, to the extent
practicable, on regulated entities, taking into account the costs of
cumulative final regulations?
Capital Assets and Service Acquisition Programs:
8. (Cap. 1) Does the program define the required quality, capability,
and performance objectives of deliverables?
9. (Cap. 2) Has the program established appropriate, credible, cost and
schedule goals?
10. (Cap. 3) Has the program conducted a recent, credible, cost-benefit
analysis that shows a net benefit?
11. (Cap. 4) Does the program have a comprehensive strategy for risk
management that appropriately shares risk between the government and
contractor?
Credit Programs:
8. (Cr. 1) Is the program managed on an ongoing basis to assure credit
quality remains sound, collections and disbursements are timely and
reporting requirements are fulfilled?
9. (Cr. 2) Does the program consistently meet the requirements of the
Federal Credit Reform Act of 1990, the Debt Collection Improvement Act
and applicable guidance under OMB Circulars A-1, A-34, and A-129?
10. (Cr. 3) Is the risk of the program to the U.S. Government measured
effectively?
Research and Development Programs:
8. (RD. 1) Does the program allocate funds through a competitive,
merit-based process, or, if not, does it justify funding methods and
document how quality is maintained?
9. (RD. 2) Does competition encourage the participation of new/first-
time performers through a fair and open application process?
10. (RD. 3) Does the program adequately define appropriate termination
points and other decision points?
11. (RD. 4) If the program includes technology development or
construction or operation of a facility, does the program clearly
define deliverables and required capability/performance
characteristics and appropriate, credible cost and schedule goals?
Section IV: Program Results (Yes, Large Extent, Small Extent, No):
1. Has the program demonstrated adequate progress in achieving its
long-term outcome goal(s)?
* Long-Term Goal I: Target: Actual Progress achieved toward goal::
* Long-Term Goal II: Target: Actual Progress achieved toward goal::
* Long-Term Goal III: Target: Actual Progress achieved toward goal::
2. Does the program (including program partners) achieve its annual
performance goals?
* Key Goal I: Performance Target: Actual Performance:
* Key Goal II: Performance Target: Actual Performance::
* Key Goal III: Performance Target: Actual Performance:
Note: Performance targets should reference the performance baseline and
years, e.g. achieve a 5% increase over base of X in 2000.
3. Does the program demonstrate improved efficiencies and cost
effectiveness in achieving program goals each year?
4. Does the performance of this program compare favorably to other
programs with similar purpose and goals?
5. Do independent and quality evaluations of this program indicate that
the program is effective and achieving results?
Specific Results Questions by Program Type:
Regulatory-Based Programs:
6. (Reg. 1) Were programmatic goals (and benefits) achieved at the
least incremental societal cost and did the program maximize net
benefits?
Capital Assets and Service Acquisition Programs:
6. (Cap. 1) Were program goals achieved within budgeted costs and
established schedules?
Research and Development Programs:
6. (RD. 1) If the program includes construction of a facility, were
program goals achieved within budgeted costs and established schedules?
Table 9: Side-by-Side of the Fiscal Year 2005 PART and the Fiscal Year
2004 PART Questions:
I. Program purpose & design:
1.1;
This year‘s question (fiscal year 2005 PART): Is the program purpose
clear? (1);
Last year‘s question (fiscal year 2004 PART): Same;
Comment: [Empty].
1.2;
This year‘s question (fiscal year 2005 PART): Does the program address
a specific and existing problem, interest, or need? (2);
Last year‘s question (fiscal year 2004 PART): Does the program address
a specific interest, problem or need?
Comment: Wording clarified.
1.2: (3);
Last year‘s question (fiscal year 2004 PART): Is the program designed
to have a significant impact in addressing the interest, problem or
need?
Comment: Dropped; "significant" worked against small programs and was
not clear.
1.3;
This year‘s question (fiscal year 2005 PART): Is the program designed
so that it is not redundant or duplicative of any other Federal,
state, local or private effort? (4);
Last year‘s question (fiscal year 2004 PART): Is the program designed
to make a unique contribution in addressing the interest, problem or
need (i.e., is not needlessly redundant of any other Federal, state
or, local or private effort)?
Comment: Wording clarified.
1.4;
This year‘s question (fiscal year 2005 PART): Is the program design
free of major flaws that would limit the program's effectiveness or
efficiency? (5);
Last year‘s question (fiscal year 2004 PART): Is the program optimally
designed to address the national, interest, problem or need?
Comment: Minor change to clarify focus; "optimally" was too broad.
1.5;
This year‘s question (fiscal year 2005 PART): Is the program
effectively targeted, so that resources will reach intended
beneficiaries and/or otherwise address the program's purpose
directly?
Last year‘s question (fiscal year 2004 PART): [Empty];
Comment: New question to address distributional design.
Specific Program Purpose and Design Questions by Program Type:
Research and Development Programs:
(RD.1);
Last year‘s question (fiscal year 2004 PART): Does the program
effectively articulate potential public benefits?
Comment: Dropped; covered by 1.2.
(RD. 2);
Last year‘s question (fiscal year 2004 PART): If an industry-related
problem, can the program explain how the market fails to motivate
private investment?
Comment: Dropped; covered by I.2 and I.5.
II. Strategic planning:
2.1;
This year‘s question (fiscal year 2005 PART): Does the program have a
limited number of specific long-term performance measures that focus
on outcomes and meaningfully reflect the purpose of the program? (1);
Last year‘s question (fiscal year 2004 PART): Does the program have a
limited number of specific, ambitious long-term performance goals that
focus on outcomes and meaningfully reflect the purpose of the program?
Comment: Splits old II.1 into separate questions on existence of (1)
long-term performance measures and (2) targets for these measures.
Together, the measures and targets comprise the long-term performance
goals addressed in last year's question.
2.2;
This year‘s question (fiscal year 2005 PART): Does the program have
ambitious targets and timeframes for its long-term measures?
Last year‘s question (fiscal year 2004 PART): [Empty];
Comment: Splits old II.1; see above.
2.3;
This year‘s question (fiscal year 2005 PART): Does the program have a
limited number of specific annual performance measures that can
demonstrate progress toward achieving the program's long-term goals?
(2);
Last year‘s question (fiscal year 2004 PART): Does the program have a
limited number of annual performance goals that demonstrate progress
toward achieving the long-term goals?
Comment: Splits old II.2 into separate questions on existence of (1)
annual performance measures and (2) targets for these measures.
Together, the measures and targets comprise the annual performance
goals addressed in last year's question.
2.4;
This year‘s question (fiscal year 2005 PART): Does the program have
baselines and ambitious targets for its annual measures?
Last year‘s question (fiscal year 2004 PART): [Empty];
Comment: Splits old II.2; see above.
2.5;
This year‘s question (fiscal year 2005 PART): Do all partners
(including grantees, sub-grantees, contractors, cost-sharing partners,
and other government partners) commit to and work toward the annual
and/or long-term goals of the program? (3);
Last year‘s question (fiscal year 2004 PART): Do all partners
(grantees, sub- grantees, contractors, etc.) support program planning
efforts by committing to the annual and/or long-term goals of the
program?
Comment: Wording clarified.
2.5; (4);
Last year‘s question (fiscal year 2004 PART): Does the program
collaborate and coordinate effectively with related programs that
share similar goals and objectives?
Comment: Moved to question 3.5.
2.6;
This year‘s question (fiscal year 2005 PART): Are independent
evaluations of sufficient scope and quality conducted on a regular
basis or as needed to support program improvements and evaluate
effectiveness and relevance to the problem, interest, or need? (5);
Last year‘s question (fiscal year 2004 PART): Are independent and
quality evaluations of sufficient scope conducted on a regular basis
or as needed to fill gaps in performance information to support
program improvements and evaluate effectiveness?
Comment: Wording clarified.
2.7;
This year‘s question (fiscal year 2005 PART): Are budget requests
explicitly tied to accomplishment of the annual and long-term
performance goals, and are the resource needs presented in a complete
and transparent manner in the program's budget? (6);
Last year‘s question (fiscal year 2004 PART): Is the program budget
aligned with the program goals in such a way that the impact of
funding, policy, and legislative changes on performance is readily
known?
Comment: Modified.
2.8;
This year‘s question (fiscal year 2005 PART): Has the program taken
meaningful steps to correct its strategic planning deficiencies? (7);
Last year‘s question (fiscal year 2004 PART): Same;
Comment: [Empty].
Specific Strategic Planning Questions by Program Type:
Regulatory Based Programs:
2.RG1;
This year‘s question (fiscal year 2005 PART): Are all regulations
issued by the program/agency necessary to meet the stated goals of the
program, and do all regulations clearly indicate how the rules
contribute to achievement of the goals? (Reg. 1);
Last year‘s question (fiscal year 2004 PART): Same;
Comment: [Empty].
Capital Assets & Service Acquisition Programs:
(Cap. 1);
Last year‘s question (fiscal year 2004 PART): Are acquisition program
plans adjusted in response to performance data and changing
conditions?
Comment: Dropped; covered in 2.CA1 and 3.CA1.
2.CA1;
This year‘s question (fiscal year 2005 PART): Has the agency/program
conducted a recent, meaningful, credible analysis of alternatives that
includes trade-offs between cost, schedule, risk, and performance
goals and used the results to guide the resulting activity? (Cap. 2);
Last year‘s question (fiscal year 2004 PART): Has the agency/program
conducted a recent, meaningful, credible analysis of alternatives that
includes trade-offs between cost, schedule and performance goals?
Comment: Minor change.
R&D Programs:
This year‘s question (fiscal year 2005 PART): R&D programs addressing
technology development or the construction or operation of a facility
should answer 2.CA1;
Last year‘s question (fiscal year 2004 PART): [Empty];
Comment: [Empty].
2.RD1;
This year‘s question (fiscal year 2005 PART): If applicable, does the
program assess and compare the potential benefits of efforts within
the program to other efforts that have similar goals? (RD. 1);
Last year‘s question (fiscal year 2004 PART): Is evaluation of the
program's continuing relevance to mission, fields of science, and
other "customer" needs conducted on a regular basis?
Comment: Modified.
2.RD2;
This year‘s question (fiscal year 2005 PART): Does the program use a
prioritization process to guide budget requests and funding decisions?
(RD. 2);
Last year‘s question (fiscal year 2004 PART): Has the program
identified clear priorities?
Comment: Modified.
III. Program management:
3.1;
This year‘s question (fiscal year 2005 PART): Does the agency
regularly collect timely and credible performance information,
including information from key program partners, and use it to manage
the program and improve performance? (1);
Last year‘s question (fiscal year 2004 PART): Same;
Comment: [Empty].
3.2;
This year‘s question (fiscal year 2005 PART): Are Federal managers and
program partners (including grantees, sub-grantees, contractors,
cost-sharing partners, and other government partners) held accountable
for cost, schedule and performance results? (2);
Last year‘s question (fiscal year 2004 PART): Same;
Comment: [Empty].
3.3;
This year‘s question (fiscal year 2005 PART): Are funds (Federal and
partners') obligated in a timely manner and spent for the intended
purpose? (3);
Last year‘s question (fiscal year 2004 PART): Same;
Comment: [Empty].
3.4;
This year‘s question (fiscal year 2005 PART): Does the program have
procedures (e.g. competitive sourcing/ cost comparisons, IT
improvements, appropriate incentives) to measure and achieve
efficiencies and cost effectiveness in program execution? (4);
Last year‘s question (fiscal year 2004 PART): Same;
Comment: [Empty].
3.5;
This year‘s question (fiscal year 2005 PART): Does the program
collaborate and coordinate effectively with related programs?
Last year‘s question (fiscal year 2004 PART): [Empty];
Comment: Same as old question 2.4.
3.5; (5);
Last year‘s question (fiscal year 2004 PART): Does the agency estimate
and budget for the full annual costs of operating the program
(including all administrative costs and allocated overhead) so that
program performance changes are identified with changes in funding
levels?
Comment: Now covered by guidance for question 2.7.
3.6;
This year‘s question (fiscal year 2005 PART): Does the program use
strong financial management practices? (6);
Last year‘s question (fiscal year 2004 PART): Same;
Comment: [Empty].
3.7;
This year‘s question (fiscal year 2005 PART): Has the program taken
meaningful steps to address its management deficiencies? (7);
Last year‘s question (fiscal year 2004 PART): Same;
Comment: [Empty].
Specific Program Management Questions by Program Type:
Competitive Grant Programs:
3.CO1;
This year‘s question (fiscal year 2005 PART): Are grants awarded based
on a clear competitive process that includes a qualified assessment of
merit? (Co. 1);
Last year‘s question (fiscal year 2004 PART): Are grant applications
independently reviewed based on clear criteria (rather than earmarked)
and are awards made based on results of the peer review process?
Comment: Modified. Guidance also captures former question Co. 2.
3.C01; (Co.2);
Last year‘s question (fiscal year 2004 PART): Does the grant
competition encourage the participation of new/first-time grantees
through a fair and open application process?
Comment: Now considered in guidance for answering 3.CO1, above.
3.CO2;
This year‘s question (fiscal year 2005 PART): Does the program have
oversight practices that provide sufficient knowledge of grantee
activities? (Co. 3);
Last year‘s question (fiscal year 2004 PART): Does the agency have
sufficient knowledge about grantee activities?
Comment: Wording clarified.
3.CO3;
This year‘s question (fiscal year 2005 PART): Does the program collect
grantee performance data on an annual basis and make it available to
the public in a transparent and meaningful manner? (Co. 4);
Last year‘s question (fiscal year 2004 PART): Same;
Comment: [Empty].
Block/Formula Grant Programs:
3.BF1;
This year‘s question (fiscal year 2005 PART): Does the program have
oversight practices that provide sufficient knowledge of grantee
activities? (B.1);
Last year‘s question (fiscal year 2004 PART): Same;
Comment: [Empty].
3.BF2;
This year‘s question (fiscal year 2005 PART): Does the program collect
grantee performance data on an annual basis and make it available to
the public in a transparent and meaningful manner? (B. 2);
Last year‘s question (fiscal year 2004 PART): Same;
Comment: [Empty].
Regulatory Based Programs:
3.RG1;
This year‘s question (fiscal year 2005 PART): Did the program seek and
take into account the views of all affected parties (e.g., consumers;
large and small businesses; State, local and tribal governments;
beneficiaries; and the general public) when developing significant
regulations? (Reg. 1);
Last year‘s question (fiscal year 2004 PART): Did the program seek and
take into account the views of affected parties including state, local
and tribal governments and small businesses in drafting significant
regulations?
Comment: Wording clarified.
3.RG2;
This year‘s question (fiscal year 2005 PART): Did the program prepare
adequate regulatory impact analyses if required by Executive Order
12866, regulatory flexibility analyses if required by the Regulatory
Flexibility Act and SBREFA, and cost- benefit analyses if required
under the Unfunded Mandates Reform Act; and did those analyses comply
with OMB guidelines? (Reg. 2);
Last year‘s question (fiscal year 2004 PART): Did the program prepare,
where appropriate, a Regulatory Impact Analysis (RIA) that comports
with OMB's economic analysis guidelines and have these RIA analyses
and supporting science and economic data been subjected to external
peer review, as appropriate, by qualified specialists?
Comment: Minor change.
3.RG3;
This year‘s question (fiscal year 2005 PART): Does the program
systematically review its current regulations to ensure consistency
among all regulations in accomplishing program goals? (Reg. 3);
Last year‘s question (fiscal year 2004 PART): Same;
Comment: [Empty].
3.RG3; (Reg. 4);
Last year‘s question (fiscal year 2004 PART): In developing new
regulations, are incremental societal costs and benefits compared?
Comment: Merged into new 3.RG4.
3.RG4;
This year‘s question (fiscal year 2005 PART): Are the regulations
designed to achieve program goals, to the extent practicable, by
maximizing the net benefits of its regulatory activity? (Reg. 5);
Last year‘s question (fiscal year 2004 PART): Did the regulatory
changes to the program maximize net benefits?
Comment: Combines former questions Reg. 4, 5, & 6.
3.RG4; (Reg. 6);
Last year‘s question (fiscal year 2004 PART): Does the program impose
the least burden, to the extent practicable, on regulated entities,
taking into account the costs of cumulative final regulations?
Comment: Merged in to new 3.RG4.
Capital Assets and Service Acquisition Programs:
3.CA1;
This year‘s question (fiscal year 2005 PART): Is the program managed
by maintaining clearly defined deliverables, capability/performance
characteristics, and appropriate, credible cost and schedule goals?
Last year‘s question (fiscal year 2004 PART): [Empty];
Comment: New question, covers old Cap. 1, 2, 3, and 4.
(Cap. 1);
Last year‘s question (fiscal year 2004 PART): Does the program clearly
define the required quality, capability, and performance objectives
for deliverables and required capabilities/ performance
characteristics?
Comment: Merged into new 2.CA1 and 3.CA1.
(Cap 2);
Last year‘s question (fiscal year 2004 PART): Has the program
established appropriate, credible, cost and schedule goals?
Comment: Merged into new 2.CA1 and 3.CA1.
(Cap 3);
Last year‘s question (fiscal year 2004 PART): Has the program
conducted a recent, credible, cost-benefit analysis that shows a net
benefit?
Comment: Merged into new 2.CA1 and 3.CA1.
(Cap 4);
Last year‘s question (fiscal year 2004 PART): Does the program have a
comprehensive strategy for risk management that appropriately shares
risk between the government and contractor?
Comment: Merged into new 2.CA1 and 3.CA1.
Credit Programs:
3.CR1;
This year‘s question (fiscal year 2005 PART): Is the program managed
on an ongoing basis to assure credit quality remains sound,
collections and disbursements are timely, and reporting requirements
are fulfilled? (Cr. 1);
Last year‘s question (fiscal year 2004 PART): Same;
Comment: [Empty].
3.CR1; (Cr. 2);
Last year‘s question (fiscal year 2004 PART): Does the program
consistently meet the requirements of the Federal Credit Reform Act of
1990, the Debt Collection Improvement Act and applicable guidance
under OMB Circulars A-1, A-11, and A-129?
Comment: Merged into new 3.CR2.
3.CR2;
This year‘s question (fiscal year 2005 PART): Do the program's credit
models adequately provide reliable, consistent, accurate and
transparent estimates of costs and the risk to the Government? (Cr.
3);
Last year‘s question (fiscal year 2004 PART): Is the risk of the
program to the U.S. Government measured effectively?
Comment: Combines former Cr. 2 and 3.
Research and Development Programs:
R&D programs addressing technology development or the construction or
operation of a facility should answer 3.CA1. R&D programs that use
competitive grants should answer 3.CO1, CO2 and CO3:
3.RD1;
This year‘s question (fiscal year 2005 PART): For R&D programs other
than competitive grants programs, does the program allocate funds and
use management processes that maintain program quality? (RD. 1);
Last year‘s question (fiscal year 2004 PART): Does the program
allocate funds through a competitive, merit-based process, or, if not,
does it justify funding methods and document how quality is
maintained?
Comment: Modified.
(RD. 2);
Last year‘s question (fiscal year 2004 PART): Does competition
encourage the participation of new/first-time performers through a
fair and open application process?
Comment: Covered by 3.CO1.
(RD. 3);
Last year‘s question (fiscal year 2004 PART): Does the program
adequately define appropriate termination points and other decision
points?
Comment: Covered by 2.CA1 and 3.CA1.
(RD. 4);
Last year‘s question (fiscal year 2004 PART): If the program includes
technology development or construction or operation of a facility,
does the program clearly define deliverables, capability/performance
characteristics, and appropriate, credible cost and schedule goals?
Comment: Covered by 2.CA1 and 3.CA1.
IV. Program results:
4.1;
This year‘s question (fiscal year 2005 PART): Has the program
demonstrated adequate progress in achieving its long-term performance
goals? (1);
Last year‘s question (fiscal year 2004 PART): Has the program
demonstrated adequate progress in achieving its long- term outcome
goal(s)?
Comment: Minor change.
4.2;
This year‘s question (fiscal year 2005 PART): Does the program
(including program partners) achieve its annual performance goals?
(2);
Last year‘s question (fiscal year 2004 PART): Same;
Comment: [Empty].
4.3;
This year‘s question (fiscal year 2005 PART): Does the program
demonstrate improved efficiencies or cost effectiveness in achieving
program goals each year? (3);
Last year‘s question (fiscal year 2004 PART): Same;
Comment: [Empty].
4.4;
This year‘s question (fiscal year 2005 PART): Does the performance of
this program compare favorably to other programs, including
government, private, etc., with similar purpose and goals? (4);
Last year‘s question (fiscal year 2004 PART): Does the performance of
this program compare favorably to other programs with similar purpose
and goals?
Comment: Minor change.
4.5;
This year‘s question (fiscal year 2005 PART): Do independent
evaluations of sufficient scope and quality indicate that the program
is effective and achieving results? 5);
Last year‘s question (fiscal year 2004 PART): Same;
Comment: [Empty].
Specific Results Questions by Program Type:
Regulatory Based Programs:
4.RG1;
This year‘s question (fiscal year 2005 PART): Were programmatic goals
(and benefits) achieved at the least incremental societal cost and did
the program maximize net benefits?
Last year‘s question (fiscal year 2004 PART): Same;
Comment: [Empty].
Capital Assets and Service Acquisition Programs:
4.CA1;
This year‘s question (fiscal year 2005 PART): Were program goals
achieved within budgeted costs and established schedules? (Cap. 1);
Last year‘s question (fiscal year 2004 PART): Same;
Comment: [Empty].
Research and Development Programs:
This year‘s question (fiscal year 2005 PART): R&D programs addressing
technology development or the construction or operation of a facility
should answer 4.CA1; (RD. 1);
Last year‘s question (fiscal year 2004 PART): If the program includes
construction of a facility, were program goals achieved within
budgeted costs and established schedules?
Comment: Simplified.
Source: OMB Web site, [Hyperlink, http://www.whitehouse.gov/omb/part/
bpm861.pdf] http://www.whitehouse.gov/omb/part/bpm861.pdf (downloaded
Apr. 7, 2003), 6-12.
[End of table]
[End of section]
Appendix III: Development of PART:
Fiscal Year 2003:
This administration's efforts to link budget and performance began with
the fiscal year 2003 budget, in which the administration announced the
"Executive Branch Management Scorecard," a traffic-light grading system
to report the work of federal agencies in implementing the President's
Management Agenda's five governmentwide initiatives. Each quarter, OMB
assessed agencies achievement toward the "standards of success"--
specific goals articulated for each of the five initiatives. Since some
of the five initiatives require continual efforts, OMB also assessed
agencies' progress toward achieving the standards. The fiscal year 2003
President's Budget also included OMB's assessments of the effectiveness
of 130 programs and a brief explanation of the assessments. According
to OMB, the assessments were based on OMB staff's knowledge of the
programs and professional judgments; specific criteria were not
publicly available with which to support OMB's judgments.
Fiscal Year 2004:
During the spring of 2002, an internal OMB task force--PET--consisting
of staff from various OMB divisions, created PART to make the process
of rating programs robust and consistent across government programs.
During the development of PART, OMB solicited input from interested
parties both inside and outside the federal government, including GAO
and congressional staff. PART was tested on 67 programs during a series
of Spring Review meetings with the OMB Director. Based on these results
and other stakeholder feedback, PET recommended a series of refinements
to PART, such as using a four-point scale in the Results section as
opposed to the "yes/no" format. Another key change was revising the
Program Purpose and Design section (Section I) to remove the question
"Is the federal role critical?" because it was seen as subjective--
based on an individual's political views.
In July 2002, OMB issued PART in final and accompanying instructions
for completing the assessments for the President's fiscal year 2004
budget submission. Later that month, OMB provided a series of training
sessions on PART for staff from OMB and agencies. Agencies received
completed PART assessments during early September 2002 and submitted
written appeals to OMB by mid-September. OMB formed the IRP, comprising
OMB and agency officials, to conduct consistency reviews[Footnote 35]
and provide recommendations on selected PART appeals. The IRP also
provided OMB with a broad set of recommendations aimed at improving the
PART based on IRP's experience with the consistency audit and appeals.
OMB was to finalize all PART assessments by the end of September 2002,
although both agency and OMB officials told us that changes and appeals
continued through the end of the budget season. RMOs within OMB
provided draft summaries of PART results to the Director of OMB during
the Director's review of agencies' budget requests. The President's
fiscal year 2004 budget (issued February 3, 2003) included a separate
volume containing one-page summaries of the PART results for each of
the 234 programs that were assessed.[Footnote 36]
The relationship between PART and the administration's proposals was
presented in agencies' budget justification materials sent to Congress.
In an unprecedented move, OMB also posted PART, one-page rating
results, and detailed supporting worksheets on its Web site. OMB also
included its Web address in the Performance and Management Assessments
volume of the budget and, in the budget itself, also described PART and
its process and asked for comments on how to improve PART.
Figure 3 depicts a time line of the events related to the formulation
of the President's budget request, including the key stages of PART
development.
Figure 3: The PART Process and Budget Formulation Timelines:
[See PDF for image]
[End of figure]
Fiscal Year 2005:
For the fiscal year 2005 PART, OMB moved the entire assessment process
from the fall to spring. OMB told us that the change was meant to help
alleviate the burden of having the PART process overlap the end of the
budget season, when workload is already so heavy. Another difference
between the 2 years was that agency officials reported that OMB was
more collaborative with the agencies in selecting the programs for the
fiscal year 2005 PART.
Training on the PART assessments to be included in the President's
fiscal year 2005 budget began in early May 2003. Agencies submitted
PART appeals in early July, and OMB aimed to resolve the appeals and
finalize the PART scores by the end of July. In December of 2003, RMOs
were to finalize the summaries of PART results, which will be published
in February along with the fiscal year 2005 President's Budget.
[End of section]
Appendix IV: Comments from the Office of Management and Budget:
EXECUTIVE OFFICE OF THE PRESIDENT
OFFICE OF MANAGEMENT AND BUDGET
WASHINGTON, D.C. 20503:
DEPUTY DIRECTOR FOR MANAGEMENT:
January 16, 2004:
Mr. Paul Posner
Managing Director
Federal Budget and Intergovernmental Relations
General Accounting Office:
441 G Street, N W Washington, DC 20548:
Dear Mr. Posner:
Thank you for the opportunity to comment on the draft GAO report on the
PART (Performance Budgeting: Observations on the use of OMB's Program
Assessment Rating Tool for the Fiscal Year 2004 Budget, GAO-04-174).
We appreciate GAO's extensive review of the PART process. We are
particularly pleased that your report recognizes the unprecedented
transparency of the PART process and materials that we have posted on
our website[NOTE 1] and the extensive efforts OMB has taken to make
the PART process consistent across the government. We will continually
strive to make the PART as credible, objective, and useful as it can
be and believe that your recommendations will help us do that. As you
know, OMB is already taking actions to address many of them. For
instance:
* With respect to centrally monitoring PART recommendations, we have
provided a simple format for agencies to follow when reporting the
status of recommendation implementation to OMB and I receive these
reports semi annually. We will continue to refine this process so that
sufficient attention is given to recommendation follow-up.
* As the PART relies on separate evaluations of evidence of a program's
success, we agree that the judgment about what constitutes a sufficient
evaluation should be based on the quality of the evaluation.
* Except for programs of insignificant size or impact, we are committed
to assessing 100 percent of programs using the PART. We are sensitive
to the impact this will have on OMB staff workload and will manage it
accordingly. * One of the greatest opportunities for the PART is to
compare the performance of, and share best practices among, like
programs across government. We will continue to use the PART for that
purpose.
* We are working diligently to generate the meaningful dialogue with
Congress you describe in your recommendations.
* We will continue to improve agency and Executive Branch implementation
of GPRA by insisting GPRA plans and reports meet the requirements of
this important law and the high standards set by the PART.
Your report makes valuable conclusions and recommendations about the
PART and our overall effort to create a more results-oriented
government. I want to note that the PART was designed for and is used
in many ways other than just budget formulation. Performance
information gleaned from the PART process has not only informed budget
decisions, but has also helped direct management, identified
opportunities to improve program design, and promoted accountability.
We believe that the PART will also greatly improve the goals and
measures adopted through the GPRA strategic and performance planning
processes.
Thank you for the opportunity to review and comment on your draft
report. I appreciate your willingness to take our oral and written
comments into consideration in the final draft. I look forward to
working with you to improve the ways in which we are creating a
results-oriented government.
Sincerely,
Signed by:
Clay Johnson, III:
NOTES:
[I] See, draft report Appendix III, p. 2.
[End of section]
Appendix V: GAO Contacts and Staff Acknowledgments:
GAO Contacts:
Paul Posner, (202) 512-9573 Denise Fantone, (202) 512-4997:
Acknowledgments:
In addition to the above contacts, Kristeen McLain, Jackie Nowicki, and
Stephanie Shipman made significant contributions to this report. Thomas
Beall, Joseph Byrns, Hashem Dezhbakhsh, Evan Gilman, Patrick Mullen,
David Nicholson, and Mark Ramage also made key contributions to this
report.
(450190):
FOOTNOTES
[1] Pub. L. No. 103-62 (1993).
[2] U.S. General Accounting Office, Performance Budgeting: Past
Initiatives Offer Insights for GPRA Implementation, GAO/AIMD-97-46
(Washington, D.C.: Mar. 27, 1997).
[3] See Pub. L. No. 103-62, § 2 (1993), 5 U.S.C. § 306 (2003), and 31
U.S.C. §§ 1115-1116 (2003).
[4] In addition to budget and performance integration, the other four
priorities under the PMA are strategic management of human capital,
expanded electronic government, improved financial performance, and
competitive sourcing.
[5] There is no standard definition for the term "program." For
purposes of PART, OMB described the unit of analysis (program) as (1)
an activity or set of activities clearly recognized as a program by the
public, OMB, and/or Congress; (2) having a discrete level of funding
clearly associated with it; and (3) corresponding to the level at which
budget decisions are made.
[6] The seven major categories are competitive grants, block/formula
grants, capital assets and service acquisition programs, credit
programs, regulatory-based programs, direct federal programs, and
research and development programs. Tax programs were not addressed for
the fiscal year 2004 PART process.
[7] OMB Circular A-11, Preparation, Submission, and Execution of the
Budget, Section 220.
[8] OMB defined 20 percent of the budget as either 20 percent of
programs or their funding levels so long as all programs are assessed
over the 5-year cycle for fiscal years 2004 through 2008 budget
requests.
[9] The 234 programs assessed for fiscal year 2004 contained a total of
612 recommendations.
[10] Pub. L. No. 106-501 (2000).
[11] Unobligated balances are defined as portions of available budget
authority that the agency has not set aside to cover current legal
liabilities.
[12] See GAO-03-595T and U.S. General Accounting Office, Managing for
Results: Efforts to Strengthen the Link Between Resources and Results
at the Administration for Children and Families, GAO-03-9 (Washington,
D.C.: Dec. 10, 2002).
[13] In the third question pair, a question in the planning section
asks about whether the program has long-term goals, and a question in
the results section asks whether the agency has made progress in
achieving the program's long-term goals. Yet, in 6 of the 115 programs
(5 percent) judged not to have adequate long-term goals, credit was
given for making progress on their long-term goals even though the
guidance again clearly states that a program must receive a "no" if the
program received a "no" on the existence of long-term outcome goals.
[14] U.S. General Accounting Office, Program Evaluation: Agencies
Challenged by New Demand for Information on Program Results, GAO/GGD-
98-53 (Washington, D.C.: Apr. 24, 1998).
[15] GAO/GGD-98-53.
[16] However, we found 8 cases (out of 118) programs that were rated as
"results not demonstrated" despite having both annual and long-term
performance goals and evidence that these goals were being met.
[17] Because our audit focused on the fiscal year 2004 PART process,
our engagement was not limited by OMB's decision to not share its
reasoning for shifting the consistency review from IRP to PET or our
lack of access to the NAPA review.
[18] Memorandum to the President's Management Council, "Where We'd Be
Proud To Be," May 21, 2003.
[19] OMB Circular A-11, Preparation, Submission, and Execution of the
Budget.
[20] 31 U.S.C. § 1115(a) (2003).
[21] U.S. General Accounting Office, Managing for Results: Agency
Progress in Linking Performance Plans With Budgets and Financial
Statements, GAO-02-236 (Washington, D.C.: Jan. 4, 2002).
[22] 5 U.S.C. § 306(d) (2003).
[23] U.S. General Accounting Office, Agencies' Strategic Plans Under
GPRA: Key Questions to Facilitate Congressional Review (Version 1),
GAO/GGD-10.1.16 (Washington, D.C.: May 1997).
[24] H.R. Rep. No. 108-195, p. 8 (2003).
[25] H.R. Rep. No. 108-243, pp. 168-69 (2003).
[26] The PART program summary sheets are included in the Budget of the
United States Government, Fiscal Year 2004, Performance and Management
Assessments (Washington, D.C.: February 2003). The summary sheets and
worksheets for the 234 programs are on OMB's Web site: http://
www.whitehouse.gov/omb/budget/fy2004/pma.html.
[27] These 11 programs are animal welfare, food aid, multifamily
housing direct loans and rental assistance, rural electric utility
loans and guarantees, and rural water and wastewater grants and loans
programs in the Department of Agriculture; the nursing education loan
repayment and scholarship program in HHS; the methane hydrates program
in DOE; the reclamation hydropower program in DOI; the long-term
guarantees program in the U.S. Export-Import Bank; and the climate
change and development assistance/population programs in the Agency for
International Development.
[28] We tested the regression on mandatory programs and as expected the
results showed no relationship between the PART scores and the level of
funding proposed in the President's Budget.
[29] Budget of the United States Government, Fiscal Year 2004,
Performance and Management Assessments, 10.
[30] For a discussion of this method, see W.H. Greene, Econometric
Analysis, Section 10.3 (Upper Saddle River, N.J.: Prentice Hall, 2003).
[31] Budget of the United States Government, Fiscal Year 2001, A
Citizen's Guide to the Federal Budget (Washington, D.C.: February
2000), http://w3.access.gpo.gov/usbudget/fy2001/guide03.html,
(downloaded April 2003), 2.
[32] The outlier is the Community Oriented Policing Services program
with an estimated 77 percent reduction in funding (see OMB, Budget of
the U.S. Government, Fiscal Year 2004, Performance and Management
Assessments, (Washington, D.C.: February 2003), 178). The outlier in
this case is identified using scatter plot and estimating with and
without the outlier. The reported results for small and medium programs
are not outlier driven.
[33] Budget of the United States Government, Fiscal Year 2004,
Performance and Management Assessments, 10.
[34] See Greene, 33.
[35] According to OMB, IRP performed consistency reviews on a
stratified random sample of programs that completed the PART in
preparation for the fiscal year 2004 budget. While IRP made
recommendations regarding its findings, it did not have the authority
to enforce them.
[36] Fiscal Year 2004 Budget of the United States Government,
Performance and Management Assessments, (Washington, D.C.: February
2003).
GAO's Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office
441 G Street NW,
Room LM Washington,
D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.
General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.
20548:
