Grants Management
Enhancing Performance Accountability Provisions Could Lead to Better Results Gao ID: GAO-06-1046 September 29, 2006Maximizing the extent to which grants achieve their long-term performance goals is critical to successfully addressing the challenges of the 21st century. While performance accountability mechanisms are fairly new to federal grants, they have been used in contracts for some time and lessons learned have begun to inform federal grant design. Given this, GAO was asked to examine (1) challenges to performance accountability in federal grants, (2) mechanisms being used to improve grant performance, and (3) strategies the federal government can use to encourage the use of these mechanisms. GAO performed a content analysis of relevant literature and interviewed experts. To illustrate the mechanisms and strategies found in the literature, GAO used examples from the literature and selected additional case illustrations--two federal grant programs (vocational education and child support enforcement) and two nonfederal contracts--for further study.
Accountability provisions in federal grants can vary widely. They can be financial (e.g., bonus payments) or nonfinancial (e.g., altered oversight or flexibility), and can be employed by various actors at different stages in the grant life cycle. Mechanisms need to be tailored to specific situations since there is no "one-size-fits-all" solution. Collectively, five key strategies appear to facilitate the effective design and implementation of performance accountability mechanisms. They are as follows: 1. ensure mechanisms are of sufficient value to motivate desired behaviors, 2. periodically renegotiate and revise mechanisms and measures, 3. ensure appropriate measurement selection, 4. ensure grantor and grantee technical capacity, and 5. allow for phased implementation. In addition to these strategies, collaboration, oversight, and feedback also appear critical to the success of performance accountability mechanisms. Opportunities exist to improve the design and implementation of federal grants. A results-focused design can enable and facilitate the use of accountability provisions. National program evaluation studies and demonstration grants can provide valuable information to support oversight of and knowledge about accountability mechanisms. Finally, the Office of Management and Budget (OMB), agencies, and grantees can benefit from sharing good practices and lessons learned regarding performance accountability provisions. OMB recognized the value in sharing information on performance accountability mechanisms, but has not yet focused on this issue.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-06-1046, Grants Management: Enhancing Performance Accountability Provisions Could Lead to Better Results
This is the accessible text file for GAO report number GAO-06-1046
entitled 'Grants Management: Enhancing Performance Accountability
Provisions Could Lead to Better Results' which was released on
September 29, 2006.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Chairman, Subcommittee on Government Management, Finance
and Accountability, Committee on Government Reform, House of
Representatives:
United States Government Accountability Office:
GAO:
September 2006:
Grants Management:
Enhancing Performance Accountability Provisions Could Lead to Better
Results:
Grants Management:
GAO-06-1046:
GAO Highlights:
Highlights of GAO-06-1046, a report to the Chairman, Subcommittee on
Government Management, Finance and Accountability, Committee on
Government Reform, House of Representatives
Why GAO Did This Study:
Maximizing the extent to which grants achieve their long-term
performance goals is critical to successfully addressing the challenges
of the 21st century. While performance accountability mechanisms are
fairly new to federal grants, they have been used in contracts for some
time and lessons learned have begun to inform federal grant design.
Given this, GAO was asked to examine (1) challenges to performance
accountability in federal grants, (2) mechanisms being used to improve
grant performance, and (3) strategies the federal government can use to
encourage the use of these mechanisms. GAO performed a content analysis
of relevant literature and interviewed experts. To illustrate the
mechanisms and strategies found in the literature, GAO used examples
from the literature and selected additional case illustrations”two
federal grant programs (vocational education and child support
enforcement) and two nonfederal contracts”for further study.
What GAO Found:
Accountability provisions in federal grants can vary widely. They can
be financial (e.g., bonus payments) or nonfinancial (e.g., altered
oversight or flexibility), and can be employed by various actors at
different stages in the grant life cycle (see figure below). Mechanisms
need to be tailored to specific situations since there is no ’one-size-
fits-all“ solution. Collectively, five key strategies appear to
facilitate the effective design and implementation of performance
accountability mechanisms. They are as follows: 1. ensure mechanisms
are of sufficient value to motivate desired behaviors, 2. periodically
renegotiate and revise mechanisms and measures, 3. ensure appropriate
measurement selection, 4. ensure grantor and grantee technical
capacity, and 5. allow for phased implementation.
In addition to these strategies, collaboration, oversight, and feedback
also appear critical to the success of performance accountability
mechanisms.
Opportunities exist to improve the design and implementation of federal
grants. A results-focused design can enable and facilitate the use of
accountability provisions. National program evaluation studies and
demonstration grants can provide valuable information to support
oversight of and knowledge about accountability mechanisms. Finally,
the Office of Management and Budget (OMB), agencies, and grantees can
benefit from sharing good practices and lessons learned regarding
performance accountability provisions. OMB recognized the value in
sharing information on performance accountability mechanisms, but has
not yet focused on this issue.
Figure: Accountability Provisions Can Be Used by Various Actors
throughout the Grant Life Cycle:
[See PDF for Image]
Source: GAO.
[End of Figure]
What GAO Recommends:
GAO recommends that the Director of OMB work with agencies and Congress
to encourage the use of performance accountability mechanisms in grant
design and implementation by promoting the practices in this report and
encouraging knowledge transfer among agencies and grantees. OMB
generally agreed with our findings and recommendation.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-1046].
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Bernice Steinhardt at
(202) 512-6543 or steinhardtb@gao.gov.
[End of Section]
Contents:
Letter1:
Results in Brief:
Background:
Trade-offs and Challenges Exist in Ensuring Performance Accountability
in Federal Grants:
Accountability Mechanisms Can Improve Performance and Performance
Accountability:
Strategies Support Successful Selection, Design, and Implementation of
Performance Accountability Mechanisms:
Various Opportunities Exist at the Federal Level to Enhance Performance
Accountability in Grants:
Conclusions:
Recommendation for Executive Action:
Agency Comments:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: GAO Contact and Staff Acknowledgments:
Bibliography:
Tables:
Table 1: Examples of Accountability Provisions:
Table 2: Sources and Types of Authorization and Guidance for
Performance Accountability Mechanisms:
Table 3: Examples of Ways to Tailor Performance Measures and
Mechanisms:
Figure:
Figure 1: Accountability Provisions Can Be Used at Different Points in
the Grant Life Cycle by Various Users:
United States Government Accountability Office:
Washington, DC 20548:
September 29, 2006:
The Honorable Todd Platts:
Chairman, Subcommittee on Government Management, Finance and
Accountability:
Committee on Government Reform:
House of Representatives:
Dear Mr. Chairman:
The federal government faces an array of challenges and opportunities
to enhance performance, ensure accountability, and position the nation
for the future. A number of overarching trends--including the nation's
long-term fiscal imbalance--drive the need to reexamine what the
federal government does, how it does it, who does it, and how it gets
financed. Because grants to state and local governments constituted
nearly 20 percent of total federal outlays in fiscal year 2005,
maximizing the extent to which grants achieve their long-term
performance goals and objectives is critical to successfully addressing
the challenges of the 21st century.
In recent years, interest in federal grant performance accountability
has grown. For the purposes of this report, performance accountability
is defined as the mechanisms by which individuals or organizations are
held accountable for meeting specified performance-related
expectations. Consistent with the decade-long trend toward an increased
results orientation and expectation for performance accountability as
evidenced by the Government Performance and Results Act of 1993
(GPRA),[Footnote 1] performance accountability mechanisms in federal
grants have become more common. For example, performance assessment
mechanisms are present in grants authorized by both the Job Training
Partnership Act and its successor program, the Workforce Investment Act
of 1988 (WIA) and the No Child Left Behind Act.[Footnote 2] More
recently, the Office of Management and Budget (OMB) developed the
Program Assessment Rating Tool (PART), which, among other things, holds
federal programs and their partners accountable for performance. Simply
monitoring and reporting performance can also encourage performance
accountability and performance improvements. In this report, we have
focused on specific mechanisms that are meant to encourage performance
incentives--such as rewards given or penalties imposed--when
performance exceeds or fails to meet specified levels.
While performance accountability mechanisms are fairly new to some
federal grants, they have been used in contracts and loans for some
time. Moreover, lessons learned from performance-based contracting have
begun to inform federal grant design, for example, in the case of WIA
grants requirements. In addition, some states award their federal pass-
through grants to subgrantees as contracts with performance
accountability requirements.[Footnote 3]
Given this growing body of experience, you asked us to examine ways to
infuse effective performance accountability mechanisms and practices
into the federal grant process. Specifically, our objectives were to
identify (1) What kinds of challenges to performance accountability
exist in federal grants? (2) What kinds of mechanisms are being used to
improve grant performance, and how? and (3) Given the findings of
questions 1 and 2, what strategies can the federal government use to
encourage the use of these mechanisms, as appropriate? For the purposes
of this report, we were interested specifically in the mechanisms by
which individuals or organizations are held accountable for meeting
specified performance-related expectations that are directly tied to a
grant.
To address our objectives, we conducted a literature review that
included our prior reports, and interviewed experts in the area of
federal grant and contract performance accountability to identify the:
(1) challenges to performance accountability that exist in federal
grants, (2) types of performance accountability mechanisms--defined as
rewards and penalties--used, and (3) key strategies that appear to
encourage the successful implementation of performance accountability
mechanisms. Our identification of types of mechanisms and strategies
was developed by conducting a content analysis of selected literature
from our review that met our criteria for addressing the issue of
accountability. Recognizing that grants have increasingly assumed
features traditionally associated with contracts, we drew on
experiences from both performance-based contracting and grants to help
identify valuable lessons learned that could inform efforts to improve
performance accountability in federal grants. To illustrate these
mechanisms and strategies, and to supplement our findings and the many
case examples identified by our content analysis, we use relevant case
examples found in the literature. We also selected four additional
cases for further in-depth illustrations. These four cases were
selected based on our literature review, interviews with experts, and
reviews of prior GAO work because they are good examples of where (1) a
performance mechanism was present and (2) there is reason to believe
that performance improved. We selected two federal grant programs and
two nonfederal contracts: (1) the federal vocational education grants
authorized by the Carl D. Perkins Vocational and Technical Education
Act of 1998 (Perkins III), which are passed through states to secondary
and postsecondary schools for career and technical education; (2) the
federal Child Support Enforcement (CSE) program, authorized by the
Social Security Act, Title IV, part D, which ensures that children are
financially supported by both parents; (3) the real property management
contract between the Ontario Provincial Government's Ontario Realty
Corporation (ORC) and SNC-Lavalin ProFac, Inc. (ProFac), a private
property management company; and (4) the contract between the
Massachusetts Division of Medical Assistance (DMA) and the
Massachusetts Behavioral Health Partnership (MBHP) for the provision of
mental health and substance abuse services for residents covered by the
MassHealth Medicaid program.
To develop the federal grant case illustrations and obtain perspectives
on the strategies we identified, we interviewed federal headquarters
and regional program and finance officials from the federal agencies
that administer the grant programs--the Departments of Education and
Health and Human Services. In addition, we visited selected grantees
and subgrantees from among these programs that federal and state
officials identified as being particularly successful--or as facing
particular challenges--with performance accountability. To develop the
contracting case illustrations, we interviewed, conducted site visits,
or both with both contractors and contracting agencies. For both the
grant and contract cases, we reviewed the authorizing legislation or
contract, guidance, documentation, and prior studies, and interviewed
relevant officials to obtain perspectives on the strategies we
identified.
See appendix I for a more detailed discussion of our scope and
methodology.
We conducted our work from December 2005 through August 2006 in offices
in Washington, D.C; Harrisburg, Lancaster, Norristown, and
Philadelphia, Pennsylvania; Eloy, Glendale, and Phoenix, Arizona; and
Boston, Massachusetts, in accordance with generally accepted government
auditing standards.
Results in Brief:
Although there are various ways to design grants to encourage
performance accountability, in general, there are three factors that
particularly affect the degree of performance accountability that can
be achieved, including whether a grant (1) includes performance-
oriented objectives in addition to fiscally oriented objectives, (2)
operates as a distinct program or as a funding stream, and (3) supports
a limited or diverse array of objectives. Because design features that
encourage performance accountability can limit state and local grantee
flexibility, achieving these twin goals can be a delicate balancing
act, and has implications for the accountability relationship between
levels of government and the information needed to support
accountability. Even in federal grants with designs that favor
performance accountability, grant implementation challenges related to
developing performance goals and measures as well as collecting and
reporting performance data can influence the extent of performance
accountability achieved.
Accountability mechanisms available for use in grants vary widely and
can be financial or nonfinancial in nature. A financial mechanism could
reward performance with increased funding or a onetime bonus payment;
nonfinancial mechanisms include such things as altered oversight or
flexibility. Financial mechanisms also vary the degree of risk sharing
between the grantor and the grantee. Many mechanisms can be employed by
Congress, agencies, or grant recipients at different points throughout
the grant life cycle. Mechanisms are flexible and need to be tailored
to specific situations since not all mechanisms are appropriate to all
situations, and there is no "one-size-fits-all" or "magic bullet"
solution to performance accountability.
Collectively, five key strategies appear to facilitate the effective
design and implementation of performance accountability mechanisms.
They are as follows:
* Ensure mechanisms are of sufficient value. The value of the rewards
and penalties--whether financial or nonfinancial--and the cost of
improved performance are adequate to motivate desired behaviors and
provide a meaningful return to both the grantor and the grantee.
* Periodically renegotiate and revise mechanisms and measures. Provide
for and use the flexibility to reevaluate performance accountability
mechanisms and associated performance measures at regular, scheduled
intervals and allow time to learn from each cycle to improve
performance.
* Ensure appropriate measurement selection. Measures should represent
performance that is within the grantee's sphere of influence, and can
reasonably be achieved and evaluated within the specified time frame,
and should be tested over time to minimize the potential for unintended
consequences and perverse incentives.
* Ensure grantor and grantee technical capacity. Grantors and grantees
should have the necessary knowledge about performance accountability
mechanisms and the ability to effectively implement them.
* Ensure phased implementation. Allow time to design, test, and revise
measurement systems before linking them to accountability mechanisms.
In addition to these strategies, we noted extensive use of partnerships
and collaborations and regular and effective oversight and feedback,
which appeared critical to the success of accountability provisions in
a third-party environment. We have previously reported that these
practices are often associated with both high-performing
organizations[Footnote 4] and organizations that effectively used
performance information to manage.[Footnote 5]
The experiences with and strategies related to federal grant
accountability provisions described in this report suggest a number of
opportunities for Congress and the executive branch to improve the
design and implementation of performance accountability mechanisms.
First, a results-focused design can help encourage performance
accountability in general and specifically provide for--or at least not
prohibit--the use of accountability mechanisms to encourage desired
behavior. In addition, the use of national program evaluation studies
and research and demonstration grants can provide valuable information
to assist in agency and congressional oversight of and knowledge about
accountability mechanisms. Because credible performance information and
performance measures form the basis for well-functioning accountability
provisions, it remains critical for Congress and the executive branch
to continue to encourage the development and use of such measures.
Finally, OMB and agencies--as well as grantees--can benefit from
sharing good practices and lessons learned about experiences with
performance accountability provisions in federal grants, as this is an
efficient and effective way to increase grantor and grantee knowledge,
understanding, and use of these provisions.
OMB, as the focal point for overall management in the executive branch,
plays a key role in improving the performance of federal programs. It
uses a number of vehicles, such as Web sites with information about
performance measures and grants targeted to federal agencies and
informal workshops and seminars, to encourage general performance
improvement in federal programs. OMB staff told us that focusing
specifically on performance accountability provisions in grants is
necessary and useful, but that to date, they have focused their efforts
on encouraging and enhancing agency capacity to develop high-quality,
results-based program performance measures since improving the quality
of measures and data necessarily precedes tying them to accountability
provisions.
We are therefore recommending that the Director of OMB encourage and
assist federal agencies in working with the Congress to expand the
effective use of performance accountability mechanisms, focusing on the
practices in this report, when federal grant programs are being created
or reauthorized. We further recommend that OMB offer opportunities for
knowledge transfer among federal agencies and encourage agencies to
share leading practices and lessons learned in implementing grant
accountability mechanisms. Possible vehicles for the collection and
dissemination of this information include good practices guides and
workshops and Web sites such as results.gov, grants.gov, and
expectmore.gov.
On August 22, 2006, we provided a draft of this report to the Director
of OMB and the Secretaries of Education and Health and Human Services.
We also provided relevant sections of a draft of this report to the
grantees and contractors highlighted in this report. We received
technical comments from all three agencies, which were incorporated as
appropriate. In addition, OMB agreed with our recommendation but
suggested we broaden it to address the role of federal agencies and
Congress in the grant redesign and reauthorization process. We agree,
and have amended our recommendation accordingly.
Background:
Grants, along with contracts and cooperative agreements, are tools used
by the federal government to achieve national priorities via nonfederal
parties, including state and local governments, educational
institutions, and nonprofit organizations. Diverse in structure and
purpose, grants can be generally classified as either categorical or
block, with categorical grants allowing less recipient discretion than
block grants. For example, the Community Services Block Grant provides
funds to states and is sometimes passed to local agencies to support a
variety of efforts that reduce poverty, revitalize low-income
communities, and lead to self-sufficiency among low-income families and
individuals, while giving the agencies broad discretion in how the
funds can be spent. In practice, the "categorical" and "block" grant
labels represent the ends of a continuum and overlap considerably in
its middle range.
Grant funds may also be grouped by their method of allocating funds.
Formula grants allocate funds based on distribution formulas prescribed
by legislation or administrative regulation and often narrowly define
the eligible recipients as state agencies. On the other hand,
categorical grants are generally awarded on a competitive basis to
applicants meeting broader eligibility requirements.
Despite substantial variation among grants, grants generally follow a
similar life cycle and include announcement, application, award,
postaward, and closeout phases. Once established through legislation,
which may specify particular objectives and eligibility and other
requirements, a grant program may be further defined by grantor agency
requirements. For competitive grant programs, the public is notified of
the grant opportunity announcement, and potential grantees must submit
their applications for agency review. In the awards stage, the agency
identifies successful applicants or legislatively defined grant
recipients and awards funding. The postaward stage includes payment
processing, agency monitoring, and grantee reporting, which may include
financial and performance information. The closeout phase includes
preparation of final reports, financial reconciliation, and any
required accounting for property.
Traditionally, grant accountability has referred to legal or financial
compliance. The Single Audit Act,[Footnote 6] for example, requires
grantees to conduct an overall financial compliance audit to promote
accountability. As such, at a minimum all grantees are held accountable
for sound financial management and use of federal funds to support
allowable activities. Beyond that, however, accountability for
performance varies from grant to grant. As discussed earlier, this
historical focus on financial accountability has expanded in response
to increasing expectations of demonstrable performance and performance
accountability for all government programs. For example, the
Comptroller General's Domestic Working Group issued its Guide to
Opportunities for Improving Grant Accountability, highlighting
innovative approaches and promising practices in grants management--
focused both on ensuring grant funds are spent properly as well as
achieving their desired results.[Footnote 7]
While performance accountability in grants is a relatively new pursuit,
it has been used in contracts for a number of years. To illustrate
performance accountability mechanisms and the strategies that
contribute to their successful design and implementation, we examined
four cases: (1) the federal CSE program, (2) the federal Perkins III
Career and Technical Education Program, (3) a performance-based
contract between the Massachusetts DMA and MBHP, and (4) a performance-
based contract between the Canadian Ontario Realty Corporation (ORC)
and ProFac.
Child Support Enforcement:
The CSE program was established in 1975 by Title IV-D of the Social
Security Act (Pub. L. No. 93-647). CSE functions in all states and
territories through state or local social services departments,
attorneys general offices, or departments of revenue in order to ensure
that children are financially supported by both of their parents. State
programs work toward establishing paternity, locating parents,
establishing and enforcing support orders, and collecting and
distributing child support payments. The federal Office of Child
Support Enforcement (OCSE), an office of the Department of Health and
Human Services' Administration for Children and Families, oversees the
development, management, and operation of state CSE programs and
provides financial support (66 percent of total operating costs) to
states. In fiscal year 2005 federal expenditures on CSE were $3.5
billion, with states spending $1.8 billion. Total collections in fiscal
year 2005 were more than $23 billion. The total legally owed support
for fiscal year 2005 was $29 billion, with $17.4 billion of that
collected. Total arrears (past due payments) for all previous years
combined was $107 billion. Over $7 billion of those past due payments
were collected and distributed in fiscal year 2005.
The Child Support Performance and Incentive Act of 1998 (Pub. L. No.
105-200) linked incentive payments to performance, and in fiscal year
2005, OCSE made over $450 million in incentive payments to states. This
act changed the original CSE incentive program from awarding incentives
based solely on cost-effectiveness to awards based on meeting specific
performance targets in five outcome areas: paternity establishment,
order establishment, current collections, past due collections, and
cost-effectiveness. The performance measures and targets are defined in
the text of the act, which also provides a formula for determining the
amount of each incentive payment. Additionally, the act established an
alternative penalty system for those states not yet in compliance with
the statewide automated data processing system required by Title IV-D
Sec. 454(A) of the Social Security Act. The new incentive program was
phased in from 2000 through 2002.
Carl D. Perkins Career and Technical Education:
Effective July 1, 1999, the Carl D. Perkins Vocational and Technical
Education Act of 1998 (Perkins III, Pub. L. No. 105-332) amends earlier
legislation to evaluate and improve vocational and technical
education.[Footnote 8] Each year under Perkins III, Congress has
appropriated more than $1.1 billion in grants to states for career and
technical education. The Office of Vocational and Adult Education
(OVAE), an office of the Department of Education, administers the
grants established in Perkins III, a pass-through grant to states,
which administer the distribution of the funds to local school
districts.
Perkins III defines major roles for OVAE and states in establishing
performance accountability systems for vocational and technical
education. States are given the responsibility for developing
performance measures and data collection systems related to four
required core performance indicators: academic and technical skill
attainment, completion, placement and retention, and nontraditional
participation and completion. OVAE negotiates these performance
measures with states to ensure that they are sufficiently rigorous.
States not meeting their performance levels for 1 year are required to
complete a program improvement plan. States not meeting their
performance levels for 2 years are subject to financial sanctions,
although no state has failed to meet its overall levels for 2
consecutive years. States have also been eligible to receive incentive
funds if they exceeded performance goals for the Perkins III grant as
well as targets established by Title I and Title II of WIA. Title I of
WIA supports workforce investment programs. Title II, also known as the
Adult Education and Family Literacy Act, provides adult education funds
to states. Governors have the authority to allocate the incentive funds
for use in any of the three program areas.
Massachusetts Division of Medical Assistance Contract with the
Massachusetts Behavioral Health Partnership:
Beginning in 1996, the Massachusetts DMA entered into a 5-year
contract, which was renewed in 2001, with MBHP to manage mental health
and substance abuse services for roughly 300,000 people covered by the
MassHealth Primary Care Clinician Plan--part of the Massachusetts
Medicaid program. Of the individuals covered by the plan, more than
half are children 18 or younger, including 20,000 children in the
custody of the commonwealth's Departments of Social Services and Youth
Services.
The structure of the contract between DMA and MBHP involves a base
contract, which governs requirements related to administrative and
medical operations. These requirements continue through the life of the
contract, or until they are modified through amendments. In addition to
the base contract, there are performance incentive projects that focus
on research and development projects. The majority of earnings
available to MBHP come from the organization's successful completion of
these contractually defined incentive projects, which are renegotiated
annually. Earnings are achieved only after the successful completion of
specified goals and objectives, as documented and reviewed by the
state.
Ontario Realty Corporation's Contract with ProFac:
ProFac, a facilities management company, was awarded a 5-year contract
in 1999 (since renewed) by ORC to provide facilities management
services for approximately 30 million square feet of space in 2,100
building sites owned by the Ontario government. About 280 ProFac
managers, engineers, technicians, and support staff provide these
services.
The contract between ORC and ProFac links performance to a 10 percent
quarterly management fee holdback: on a monthly basis, ORC only
reimburses ProFac for 90 percent of its administrative costs, retaining
the other 10 percent, a "holdback," which ORC returns to ProFac on a
quarterly basis only if ProFac obtains a sufficient level of
performance. This contract also links performance to an annual share-
in-savings arrangement: ORC sets a budget each year based on prior
years' actual expenditures and budget projections; if ProFac spends
less than the budget, it is able to share the difference, a share in
savings, only if it reaches a sufficient operational level. ProFac's
performance on 30 key performance indicators (KPI) determines its
operational level. These KPIs are accumulated to determine a score in
four performance objectives: management performance, financial
performance, asset integrity, and customer service. The performance
objectives are scored and then weighted according to ORC's priorities
to determine a total performance rating.
Trade-offs and Challenges Exist in Ensuring Performance Accountability
in Federal Grants:
The trade-offs and challenges associated with performance
accountability in federal grants largely depend on several key aspects
of grant design and implementation. As we have previously reported,
performance accountability tends to be greater (and grantee flexibility
lower) in programs with certain types of design features. Because
design features that encourage performance accountability can limit
state and local grantee flexibility, achieving these twin goals can be
a delicate balancing act and has implications for the accountability
relationship between levels of government and the information needed to
support accountability. Even in federal grants with designs that favor
performance accountability, grant implementation challenges related to
developing performance goals and measures as well as collecting and
reporting performance data can influence the extent of performance
accountability achieved.
Grant Design Features Affect the Balance between Accountability and
Flexibility:
Although there are various ways to design grants to encourage
performance accountability, in general, there are three factors that
particularly affect the degree of performance accountability that can
be achieved, including whether a grant (1) includes performance-
oriented objectives in addition to fiscally oriented objectives, (2)
operates as a distinct program or as a funding stream, and (3) supports
a limited or diverse array of objectives.
As we discussed previously, federal grants have traditionally focused
on fiscal or legal accountability, such as holding states accountable
for using federal grant funds to supplement rather than to supplant
their own spending on a particular activity. However, federal grants
that also include performance-oriented objectives--as well as the
provisions that implement them--provide the basis for performance
measurement and accountability for results, and signal a federal role
in managing performance over the grant. Ideally, both types of
objectives would be present in federal grants. Performance-related
objectives focus on service or production activities and their results.
For example, the central objective of the grants for Special Programs
for the Aging--Nutrition Services, is to provide nutritious meals to
needy older Americans to improve nutrition and reduce social isolation.
In contrast, fiscal or financial assistance objectives focus on
providing dollars to support or expand activities. Typical fiscal
objectives include increasing support for meritorious goods or
underfunded services and targeting grant funding to needy
jurisdictions. For example, the objective of Title VI Innovative
Education grants is to provide funds to support local education reform
efforts. When objectives are purely fiscal, accountability to the
federal agency tends to focus on fiscal matters, such as holding states
accountable for using federal grant funds to supplement rather than to
supplant their own spending on a particular activity.
Even when performance-oriented objectives are present, whether federal
grants operate as distinct programs or as part of a larger funding
stream directly affects who can be held accountable and for
what.[Footnote 9] A grant that operates as a program has performance
requirements and objectives and carries out specific programwide
functions through a distinct delivery system, such that grant-funded
activities, clients, and products are clearly identifiable. This type
of grant gives the federal agency a role in managing performance and
makes it easier to obtain uniform information about performance
attributable to the grant funds. It is possible to identify which
activities were supported; the amount of federal funds allocated to
each; and to various extents, the results grantees achieved with
federal funds.
In contrast, funds from grants that operate as part of a funding stream
are merged with funds from state or local sources (and sometimes from
other federal sources) to support state or local activities allowable
under the flexible grant. These programs are managed at the state or
local level, with the federal role limited accordingly. When grants are
part of a funding stream, it is possible to identify which activities
federal funds supported and the amount allocated to each, but once the
grant funds are combined with the overall budget for a state or local
activity, federal dollars lose their identification and their specific
results cannot be separated out. This is particularly the case when the
federal share is small, with most funding coming from other sources.
The program outcome measures available in such programs are likely to
be for outcomes of the state or local service delivery program, not the
federal program from which the funding originated. Thus, grantees would
generally be held accountable for overall outcomes, regardless of the
funding source. For example, projects such as Oregon Option and the
National Performance Review were designed to promote accountability for
federal and/or national priorities, regardless of the funding source.
They encourage grantors and grantees to work toward collaboratively
developed outcomes. These intergovernmental partnerships can be
particularly useful when funds come from a combination of federal,
state, local, and private sources, or when the federal funding share is
small.
Federal grants vary along a continuum, at one end supporting a single
major activity common to all grantees (such as categorical grants), and
at the other end, allowing unrestricted choice by the recipient among a
wide variety of allowable activities, (such as block grants).
Flexibility is narrowest, but accountability to the federal level
clearest, in programs that focus on a single major activity.
Flexibility is broadest in programs designed to support diverse state
or local activities, but finding a common performance metric can be
extremely challenging since these activities can vary considerably from
state to state. That said, we have previously reported on options for
building accountability provisions into block grants that help balance
states' flexibility to select a mix of activities and services that
will best allow them to achieve a particular national outcome with
accountability for achieving that outcome. These options include (1)
relying on state processes both to manage block grant funds and to
monitor and assess compliance and (2) emphasizing results-based
evaluation rather than examining specific program or administrative
activities.
Implementation Issues Present Further Performance Accountability
Challenges:
In addition to these design features, we have previously reported on a
number of performance accountability challenges encountered in many
grant programs during the grant implementation phase.
Lack of consensus on goals and performance measures: The priorities of
states, tribes, local communities, and the federal government are not
always the same. To ensure that grantees work toward national
priorities, they need to be involved in the development of performance
goals and measures. Lack of agreement on goals and measures--
particularly when the federal funding is a small portion of the funding
stream--could lead to grantees making choices that do not necessarily
support the achievement of national goals.
Reliance on performance data from state and local partners and other
third parties: Even if grantees collect data on similar activities,
outcomes, and services, absent common data definitions Congress and
program managers will lack comparable information, limiting the ability
to compare state efforts or draw meaningful conclusions about the
relative effectiveness of different strategies. We have previously
reported that agencies relying on third parties for performance data
also have difficulty ascertaining the accuracy and quality of the data.
Further, programs often rely on state administrative systems for
performance information. For some programs--such as many of the
Administration for Children and Families' programs---since final
reports are not due until 90 to 120 days after the end of the federal
fiscal year, there is a delay in available data.
Onerous and inconsistent grant administration processes and
requirements[Footnote 10]: Multiple grants maybe available for the same
or similar purposes, meaning that federal grant recipients must
navigate through a myriad of federal grant programs in order to find
the appropriate source of funds to finance projects that meet local
needs and address local issues. Sometimes programs meant to address
common problems have potentially conflicting requirements. Variations
in performance accountability requirements among these grants can limit
the degree of performance accountability achieved. We have recently
reported that while this situation is improving because of OMB's
efforts to streamline the grants application process, problems still
exist.[Footnote 11]
Prohibition of performance information collection[Footnote 12]: Because
states are principally responsible for implementing block grants at the
state level, the block grant statutory prohibitions and requirements,
and federal regulations and guidance are generally kept to a minimum.
Sometimes federal agencies are prohibited from imposing reporting
requirements because they are seen as burdensome. Clearly, this limits
the extent to which federal agencies can oversee grantee performance.
Nevertheless, even with these trade-offs and challenges, agencies have
been able to shift toward increased performance accountability in
federal grants and the use of accountability provisions to ensure that
grantees achieve real results through the programs, activities, and
services financed with federal funds. The accountability provisions
described in this report, along with strategies for their effective
use, can help address the challenges noted above.
Accountability Mechanisms Can Improve Performance and Performance
Accountability:
We found a number of accountability provisions, specific actions that
can be taken--that is, rewards given or penalties imposed when
performance exceeds or fails to meet specified performance levels--that
Congress, granting agencies, and grantees can use at different points
in the grant life cycle to improve both grant performance and
performance accountability. These examples demonstrate that
accountability provisions can result in significant performance
improvement and are flexible enough to accommodate a variety of
situations.
A Variety of Accountability Mechanisms Exist:
We found that a wide variety of accountability provisions are being
used in both grant and contracting situations. A selection of these
provisions is shown in table 1. This list is not intended to be
exhaustive; rather, it is meant to illustrate the variety of mechanisms
available. Some mechanisms may be more appropriate in certain
situations than others, but all of these mechanisms can be used to
either encourage improved performance or discourage poor performance.
For example, public recognition and increased funding are two different
mechanisms that can both be used to encourage and reward good
performance. Similarly, mechanisms such as reduced funding or increased
oversight can be used to discourage or penalize poor performance.
Table 1: Examples of Accountability Provisions:
Rewards;
Accountability mechanism: Praise;
Definition: Public recognition of good performance, for example,
through the press, Web sites, intranets, newsletters, hearings,
testimony, and award ceremonies.
Rewards;
Accountability mechanism: Bonus;
Definition: Onetime cash payment.
Rewards or penalties;
Accountability mechanism: Increase/decrease flexibility;
Definition: Increase or decrease in grantee's flexibility by issuing
administrative, programmatic, or financial waivers from requirements
and restrictions or by adding award conditions.
Rewards or penalties;
Accountability mechanism: Increase/decrease workload;
Definition: Manipulate the workload (e.g., case load).
Rewards or penalties;
Accountability mechanism: Increase/decrease award term;
Definition: Increase or decrease in the length or term of the grant.
Rewards or penalties;
Accountability mechanism: Increase/decrease oversight;
Definition: Increase or decrease in the degree of oversight.
Rewards or penalties;
Accountability mechanism: Increase/decrease funding rate;
Definition: Increase or decrease in the per-unit reimbursement rate
(e.g., case rate). Either partial or full funding can be based on a
unit rate.
Rewards or penalties;
Accountability mechanism: Increase/decrease funding level;
Definition: Increase or decrease in funding. Either the entire award
can be tied to performance or a portion of funding above an established
baseline (i.e., an incentive portion). Examples include share in
savings (grantee keeps a portion of dollars saved) and milestones
(payments linked to a predefined chronological series of performance
levels typically combining process, output, and outcome measures).
Rewards or penalties;
Accountability mechanism: Use of past performance;
Definition: Use past performance of grantee to inform selection of
future recipients.
Penalties;
Accountability mechanism: Reproof;
Definition: Public reprimand for poor performance, for example, through
the press, Web sites, intranets, newsletters, hearings, and testimony.
Penalties;
Accountability mechanism: Reperformance;
Definition: Grantee must reperform the service at its own cost to meet
performance agreements.
Penalties;
Accountability mechanism: Impose financial penalty or sanction;
Definition: Includes a onetime reduction in the value of an award.
Sanctions may also be in one area to influence actions in another area
(crossover sanctions). Also includes suspending or withholding a
payment (temporarily halting grant payments and/or work), suspending or
terminating the award (canceling the current grant or temporarily
excluding grantee from future awards), or finally, debarment
(permanently exclude grantee from future grant awards).
Source: GAO.
[End of table]
In addition, mechanisms can be either financial or nonfinancial in
nature. A financial mechanism would be an increase in funding or a
bonus. For example, the CSE program employs a financial incentive in
the form of a bonus to encourage states to work toward the program's
five performance goals: states are eligible for a bonus every year
based on performance. Nonfinancial mechanisms would include altered
oversight or flexibility. For example, as part of the National
Environmental Performance Partnership System, the Environmental
Protection Agency affords states with high environmental performance
levels greater flexibility in spending their grant funds.
Financial mechanisms also vary by their degree of risk or risk sharing
between the grantor and the grantee. Grantee risk increases as the
amount of money tied to performance increases. For example, bonuses--
money awarded over and above the base grant amount--represent the least
risk, while an outcome-based milestone payment plan where the entire
grant award is based on performance represents much higher financial
risk to the grantee. Nonfinancial actions, such as altering flexibility
or oversight, would be relatively risk neutral.
Accountability Provisions Can Be Employed at Different Phases of the
Grant Life Cycle:
Accountability mechanisms can be used in different phases of the grant
life cycle by different actors, including Congress, granting agencies,
and grantees themselves, and the lessons learned from one grant cycle
can be used to improve a performance accountability mechanism in the
next (see fig. 1). For example, when reauthorizing the CSE program,
Congress revised the original CSE incentive payments, which were solely
based on cost efficiency, to create an incentive program tied to
performance measures that reflect CSE's five key goals: (1) paternity
establishment, (2) order establishment, (3) current collections, (4)
collection of payments in arrears, and (5) cost-effectiveness (design/
redesign phase).
Figure 1: Accountability Provisions Can Be Used at Different Points in
the Grant Life Cycle by Various Users:
[See PDF for image]
Source: GAO.
[End of figure]
In contrast, performance measures and targets for Perkins III are
created during the implementation phase. Specifically, each state is
required by law to create its own performance measures linked to four
core indicators: (1) student attainment of challenging state-
established academic and vocational technical skill proficiencies; (2)
student attainment of secondary diploma or postsecondary degree or
credential; (3) student placement in employment, pursuit of further
education, or both; and (4) student participation in and completion of
vocational technical education programs that lead to nontraditional
training and employment. The Department of Education periodically
negotiates the performance targets for each state measure (postaward
phase).
The use of past performance can inform and improve the recipient
selection process (application phase). Specifically, the Florida
Department of Children and Families has reported considerable success
using past performance in recipient selection--contractors that do not
meet their performance measures and standards are ineligible to be
awarded future contracts.
Other mechanisms, such as altered flexibility or oversight, can be used
by the granting agency--or even the grantee--to encourage improved
performance during the term of the award (postaward phase). For
example, according to the literature we reviewed, Minnesota's
Department of Human Services Refugee Services Section increases its
oversight of local agencies if their performance drops below 80 percent
on their key performance measures, including job placement rates, which
nearly doubled over 5 fiscal years.
Importantly, grantees can also use these provisions to extend
accountability to subgrantees and contractors. This is significant
because many federal grants are ultimately passed through states to
subgrantees. Some accountability provisions, such as public award and
recognition, can even be employed by stakeholders or interested
parties. For example, the National Association for State Directors of
Career and Technical Education Consortium annually recognizes high-
performing career and technical administrators and teachers and
provides opportunities to share lessons learned and best practices.
Even when performance accountability provisions are absent from or
limited by a grant's legislation, agencies and grantees may still be
able to include these types of provisions in the terms and conditions
of the grants or subgrants or in contracts as long as the authorizing
legislation does not specifically prohibit their use. OMB Circular A-
110 provides that for grants awarded to nonprofits, a number of
accountability mechanisms may be used--including withholding payments,
termination of award, and "other remedies that may be legally
available"--if the grantee materially fails to comply with the terms
and conditions of the award. These terms and conditions can be
specified in federal statute, regulation, assurance, applications, or
the notice of award. For grants to state and local governments,
however, OMB Circular A-102 contains no detailed accountability
provisions and defers to the requirements specified in the authorizing
legislation.
Various authorities govern the use of accountability provisions (see
table 2). Provisions set by Congress, such as increased flexibility in
the form of waivers from statutory restrictions, are generally laid out
in authorization or appropriations legislation. As stated earlier,
granting agencies can include accountability provisions in regulations,
grant announcements, the request for proposal, and the notice of award.
Grantees (and subgrantees) can use accountability provisions, such as
formal recognition, to improve their performance internally. For
example, the CSE program in Montgomery County, Pennsylvania, recognizes
performance-improving suggestions from individual employees by publicly
praising and inducting them into the office's "all-star team." These
employees also receive a T-shirt with a picture of a stork--the
program's mascot--with the program's motto Striving Toward Optimizing
our Resources for Kids.
Table 2: Sources and Types of Authorization and Guidance for
Performance Accountability Mechanisms:
Authorizing body: Congress;
Authorizing/implementing vehicles: Authorization, appropriations, other
legislation.
Authorizing body: Grantor;
Authorizing/implementing vehicles: Regulations, announcements, the
request for proposal in the Federal Register, notice of award.
Authorizing body: Grantee;
Authorizing/implementing vehicles: Internal personnel policies and
practices, use with subgrantees and contractors.
Source: GAO.
[End of table]
Accountability Mechanisms Can Be Tailored to Specific Situations:
Selecting appropriate performance measures and linking them to
performance accountability mechanisms is not a one-size-fits-all
process; rather, accountability provisions are tailored to reflect the
program's characteristics. In addition to the range of accountability
mechanisms available, we found a number of ways mechanisms were
tailored and combined to reflect a variety of circumstances. Table 3
describes how measures and mechanisms can be designed and triggered to
either reward or penalize performance.
Table 3: Examples of Ways to Tailor Performance Measures and
Mechanisms:
Tailored measure and mechanism: Stretch goals;
Description: An action is taken based on reaching a secondary, higher
goal. For example, a grantor awards the grantee when performance
exceeds one performance target and reaches a secondary, higher target.
Tailored measure and mechanism: Hurdles/triggers;
Description: Specific conditions or performance levels that must be met
before actions can be taken. For example, once a recipient meets a
minimum performance level or requirement, it becomes eligible for an
award based on performance.
Tailored measure and mechanism: Dead bands;
Description: Ranges of performance are established for which the
mechanism does not provide an award or penalty. For example,
performance achieved within a certain range is not eligible for an
award or penalty, but performance above or below that range is subject
to them.
Tailored measure and mechanism: Step up/step down;
Description: Actions are taken based on a number of preset performance
levels. Step up is similar to a series of stretch goals. For example, a
recipient receives a reward in which the value is determined by which
performance interval it reached. Conversely, each time a recipient's
performance drops, it is subject to increasing penalties or deductions.
Tailored measure and mechanism: Formula;
Description: Payments are made based on a formula--a mathematical
weighting of a number of factors. For example, a recipient's entire
award or an incentive portion may be based on a formula containing a
performance component.
Tailored measure and mechanism: Share in savings/share in revenue;
Description: Payments represent a share of a specific source of funds.
Share in revenue is used to refer to situations where performance is
defined by revenue generation and payments are based on some
formulation of revenue generated, typically a percentage. For example,
a recipient responsible for increasing financial collections is
eligible to receive a portion of the additional funds collected; Share
in savings is used to refer to situations where performance is defined
by cost savings and payments are based on some formulation of cost
savings, typically a percentage. For example, a recipient that
identifies efficiencies that result in administrative cost savings is
eligible to receive a portion of the savings.
Tailored measure and mechanism: Milestones;
Description: Payments are linked to a predefined chronological series
of performance levels representing processes, outputs, and outcomes.
For example, a recipient is paid 33 percent of the performance-based
portion of award as each of three milestones is completed successfully.
Tailored measure and mechanism: Floating measures;
Description: Used to refer to situations where one or sets of
performance measures can be changed during the term of an agreement.
These measures are selected from a larger set of predefined performance
measures. For example, a grantor assesses grantee performance on 25
performance measures, but at any given time provides awards or
penalties based on the performance of a subset of the measures. The
subset the grantor provides the award or penalty for can change during
the course of the agreement, with notification.
Tailored measure and mechanism: Indexes;
Description: Used to refer to a situation where individual performance
measures are weighted to create a single index. For example, a number
of measures are weighted according to priority, and then the combined
weight is used as a single performance measure.
Tailored measure and mechanism: Variable target;
Description: Situation where performance is defined by a variable or
relative measure, such as performance of a third party in the same
issue area. This method is used, for example, in situations where
performance can be directly affected by external factors, such as the
economy. For example, grantors assess the performance of a single grant
recipient by comparing its performance to other recipients of the same
grant.
Source: GAO, based on literature review.
[End of table]
For example, to encourage its contractor, ProFac, to cut costs while
maintaining a high-level of performance, ORC modified a basic financial
incentive to include a share-in-savings feature. ProFac is eligible to
share a portion of the savings if it spends less than its yearly
budget--often referred to as share in savings. To ensure that ProFac
does not cut costs to the detriment of high performance, ORC also
requires that ProFac achieve a performance rating of 80 percent or
higher to share in these cost savings.
The CSE performance measures are an example of a "step up" provision--
for each increasing performance percentage interval there is a
corresponding increase in the incentive percentage paid. Each time a
state moves to the next highest interval, it receives a higher
percentage of the incentive for that measure. Conversely, the
alternative penalty procedure for failure to implement a statewide
child support data processing system acts as a "step down" mechanism.
For each year the state fails to implement such a system, but shows a
good faith effort to attempt to do so, the state will be penalized at
increasing intervals--during the first year of noncompliance, the state
will receive a 4 percent penalty, the second year an 8 percent penalty,
the third year a 16 percent penalty, and so on.
Strategies Support Successful Selection, Design, and Implementation of
Performance Accountability Mechanisms:
Collectively, five key strategies appear to facilitate the effective
selection, design, and implementation of performance accountability
mechanisms.[Footnote 13] These strategies are:
* ensure mechanisms are of sufficient value,
* periodically renegotiate and revise mechanisms,
* ensure appropriate measurement selection and usage,
* ensure grantor and grantee technical capacity, and:
* implement system in stages.
In addition to these strategies, we noted extensive use of partnerships
and collaborations and regular and effective oversight and feedback,
which appeared critical to the success of accountability provisions in
a third-party environment. We have previously reported that these
practices are often associated with high-performing organizations and
organizations that effectively used performance information to
manage.[Footnote 14]
Ensure Mechanisms Are of Sufficient Value:
There are a number of factors to consider when designing accountability
mechanisms that help to ensure the mechanisms are of sufficient value
and motivate performance improvement. Ensuring sufficient value
requires that:
* both the grantor and grantee are able to determine the value of the
rewards and penalties and the cost of improved performance--be they
financial or nonfinancial--and provide a meaningful return to both the
grantor and the grantee and:
* rewards or penalties should be consistently applied to maintain the
value of the mechanisms to both the grantor and grantee.
Understand the Value of Performance:
According to the literature we reviewed, both the grantor and grantee
should understand what a particular level of performance is worth to
them and what it will cost them to achieve that level of performance.
When the value of performance is not properly identified, funds could
be wasted and grantees may not respond to the mechanism. For example,
we found one case where the contracting agency offered and ultimately
paid a $250,000 bonus to a contractor for completing a pipeline 2-1/2
months earlier than scheduled. However, because the contracting agency
did not actually need the pipeline to be completed for several years
after the original contractual deadline, the contractor paid $250,000
for a level of performance it did not need. Although the recipient
responded to the incentive, the contracting agency did not properly
calculate the value of the performance improvement to the agency,
resulting in wasted funds.
For a grantor, considering how accountability provisions support its
strategic priorities can assist in determining the value of
performance. The size of the associated rewards and penalties should be
commensurate with the priority. For example, successful pay-for-
performance programs reserve large rewards for achieving an
organization's most important priorities, or those that lead to large
benefits, and provide smaller incentives for achieving goals that reap
smaller benefits or are of lesser importance. For example, in the
health care field, for certain conditions such as heart attack or
stroke, delays in administering appropriate therapy greatly increase
the risk of mortality and disability. Therefore, the incentives to
treat these conditions quickly and appropriately should be larger than
the incentives for other practices that should be encouraged yet
produce fewer direct effects on mortality and illness, such as avoiding
the use of ineffective antibiotics to treat the common cold.
Based on our literature review, it appears that insufficiently valued
incentives are one of the main reasons that accountability provisions
fail. When an incentive is of sufficient value, the expected return
outweighs the expected risk, and recipients are motivated to pursue the
performance improvement. From 1975 through 1997 the CSE program
included an incentive program that focused on cost-effectiveness.
States were guaranteed an "incentive payment" from 6 to 10 percent of
their total collections. In practice, the 4 percentage point difference
between the minimum and maximum payment was reportedly not large enough
to motivate states to increase collections enough to earn the 10
percent bonus. The new incentive system, established by the Child
Support Performance and Incentive Act of 1998[Footnote 15] only
provides incentive payments to states that meet one or more of the
act's five outcome-based performance goals and associated
targets,[Footnote 16] and penalizes states that fall below threshold
levels in certain areas. A review of the new incentive system in a
sample of nine states found that the median score on each of the five
performance measures increased from fiscal years 2000 to 2002, the time
period that the incentive system was implemented.[Footnote 17]
Motivating grantees to work toward federal outcomes is particularly
challenging in grants where the federal investment is relatively small.
Officials at Arizona's Department of Adult Education, Career and
Technical Education Division, told us that state funds in joint
technological education districts outweighed federal funds for career
and technical education (CTE) programs by more than four to one, and
some districts did not want to accept federal CTE funds because, in
their view, complying with the federal performance requirements was not
worth the amount of funds they would receive. In order to ensure that
the financial value of the Perkins III grants was large enough to
motivate districts to meet the Perkins III reporting and performance
requirements, the Arizona Career and Technical Division requires
districts be in Perkins III compliance in order to receive CTE-related
state funds, thereby creating a large incentive for local school
districts to comply with the Perkins III requirements. Indeed, one
district we spoke with lost Perkins III funding, but it was not until
the state linked Perkins III compliance to state funding, and the
district lost the rest of its CTE funding from the state, that the
district started to make significant improvement toward meeting Perkins
III requirements.
In addition, the grantor and grantee should understand the trade-off
between the financial risk--the possibility performance will not
improve sufficiently despite the resource investment--and the potential
return--what will be gained if performance goals are met or exceeded--
in order to determine whether to pursue any particular performance
improvement. Accountability provisions that contain financial
incentives and sanctions can shift risk between the grantor and
grantee. That is, the more the grant award depends on performance, the
greater the financial risk to the grantees: if they invest but do not
perform sufficiently, they do not get paid. Conversely, in grants with
limited or no performance accountability provisions, the grantor bears
the bulk of the financial risk, since the grantee would receive the
grant funds regardless of the results achieved.
Ensure Effective Distribution:
The ability of a performance accountability mechanism to influence
performance also depends on the effective distribution of
organizational rewards and penalties to individuals within the
organization who are directly responsible for the desired
performance.[Footnote 18] For example, Glendale Union School District
officials provide significant financial incentives to every school
employee with whom a student has contact, including teachers,
administrative staff, and other support staff--including the
maintenance staff and bus drivers. The district's philosophy is that
all employees influence the school's atmosphere and academic
achievement and therefore contribute to any success it enjoys.
Incentive funds are distributed based on a school's performance on 13
academic, involvement, and satisfaction-related measures. According to
a district official, the program, started 5 years ago, has increased
camaraderie and collaboration among school employees, which the
official said has contributed to academic improvement.
In Pennsylvania, the state passes along a portion of the state-earned
federal incentive payments to the counties, according to each county's
proportionate share of the aggregate state CSE expenditures and to
reflect its relative score for each performance measure, following the
performance targets defined in legislation. Pennsylvania codifies the
performance expectations and incentive payment procedures through
cooperative agreements with each county.
Execute Mechanisms Consistently:
The grantor must execute the mechanisms consistently and as designed to
preserve the value of the mechanisms and to avoid introducing
unnecessary risk. For example, if rewards are not paid as promised the
grantee could learn that its additional efforts are not worth the cost-
-or risk--and may not make the additional effort to improve
performance. Similarly, if rewards are paid indiscriminately or if
penalties are not levied as expected, the grantee could learn that no
additional effort or investment is required in order to benefit. In
both cases, the system breaks down and the intended value of the
accountability provision is lost.
We have reported on an agency with the authority to levy penalties for
poor performance that resisted doing so. For example, the Federal
Transit Administration (FTA) has several enforcement tools to deal with
grantees' noncompliance, including warning letters, suspension of
funds, and grant termination. However, traditionally, FTA had been
reluctant to use these tools to enforce compliance, opting instead to
work with grantees in an effort to continually promote transit
development. Reviews also showed that FTA's oversight was superficial
and inconsistent and that FTA seldom used its enforcement authority to
compel grantees to correct weaknesses, even those that were long-
standing. Consequently, federal dollars had been placed at risk.
However, in response to our 1992 report, FTA established a new
enforcement policy, developed detailed guidance on carrying out
enforcement actions, and has since demonstrated a greater willingness
to use these actions against grantees that do not comply with federal
transit requirements. The Department of Defense (DOD), on the other
hand, has paid billions in incentive and award fees for only
"acceptable, average, expected, good, or satisfactory" performance.
Despite paying billions in fees, DOD has little evidence to support its
belief that these fees improve contractor performance and acquisition
outcomes. The department has not compiled data, conducted analyses, or
developed performance measures to evaluate the effectiveness of award
and incentive fees. Using accountability mechanisms in this manner
undermines their effectiveness as a motivational tool and marginalizes
their use in holding grantees and contractors accountable for outcome-
based results.[Footnote 19]
Periodically Renegotiate and Revise Mechanisms and Measures:
Organizations need to allow for and use the flexibility to revise,
update, or improve performance accountability mechanisms in order to
respond to changing needs. In the literature we reviewed, we found a
number of reasons for why accountability provisions may need to be
revised. For example, unintended consequences associated with
performance measures may be discovered only after full implementation.
Organizational priorities may change. Technology may be introduced that
substantially alters performance expectations. In addition,
expectations that were previously considered stretch goals can become
the norm over time--for example, as productivity gains are realized
rewarding such performance may no longer make sense. Finally, efforts
to reevaluate and revise should consider whether established
accountability provisions are still effective at motivating performance
improvements.
For example, the DMA/MBHP contract demonstrates a situation in which
the entire accountability system experienced a revision to adjust to
contract progression. Incentives in this contract were initially
designed to motivate operational performance, such as processing time
for billing, and performance targets were revised upward each year as
performance improved. This upward revision helped ensure that
performance continued to improve. Once MBHP's performance reached the
highest levels of industry performance in these areas, further
improvements were no longer a priority.
As a result, DMA and MBHP used the annual review to revise the
incentive system from motivating operational improvement to completing
projects designed to improve performance in areas that would add value
to the services MBHP provides, such as a project on providing
behavioral health assistance to the homeless.
There are a number of ways to accommodate the need for periodic
revision. For example, congressional amendments to or reauthorizations
of grant programs allow policymakers the opportunity to revisit and
modify existing provisions and to add flexibility for agencies that can
lead to improved effectiveness. Agencies can include renegotiation and
revision policies in regulations, guidance, and the terms and
conditions of a grant award. Providing for periodic revision may be
particularly important where performance measures are specified in
legislation, because agency flexibility to respond to changing needs is
significantly reduced. For example, as we have discussed, the Child
Support Performance and Incentive Act of 1998 specifies the five
performance measures, the performance targets, and the percentage of
incentive payments that states can earn for performance. Initially,
states made changes and saw improvement in these areas. Recently, both
state and federal program officials have expressed concern about the
long-term sustainability of such aggressive targets. For example,
program officials said that even states that have in the past met the
90 percent performance target for the paternity measure are concerned
because more recent annual rates have dropped back down closer to 80
percent. According to officials, states initially conducted an
extensive caseload cleanup to improve performance on the five incentive
measures when the incentive program was enacted in 1998, and much of
the backlog of cases that could be addressed relatively easily has
been. However, since the measures and performance targets are
legislatively defined and the CSE program is permanently authorized,
the agency does not currently have the flexibility to revise the
measures or performance targets.
In contrast, state agencies, in negotiation with the Department of
Education, can periodically revise their Perkins III CTE performance
measures and targets during annual negotiations of their state plans.
At program introduction, program targets are set through the
negotiation process between states and OVAE. From this process,
performance targets negotiated initially reflect a realistic level of
what states can actually produce. Next, through annual application
updates, the legislation allows renegotiation of performance levels
with states. Among other factors, OVAE officials attributed the
program's success to this ongoing ability to renegotiate and revise the
program's measures. Although the Perkins III legislation is similar to
the Child Support Performance and Incentive Act of 1998, in that
Perkins defines the four core indicators tied to the performance
measures used in the incentive program, it provides flexibility that
the Child Support Performance and Incentive Act of 1998 lacks. The
flexibility to revise or update the performance measures is built into
the Perkins III legislation.
Accountability systems by their very nature assume that performance can
be improved. However, performance improvements depend on adequate time
for and ability of participants to learn from prior actions and use
what they have learned to improve performance from one period to the
next.[Footnote 20] Depending upon the complexity of the task, this
process can take many cycles. Therefore, accountability systems should
not be abandoned prematurely; rather, they should be assessed, revised,
and improved.
Ensure Appropriate Measurement Selection and Usage:
Selecting and using appropriate types of performance measures is
important to the effective use of accountability mechanisms. We have
previously reported on general attributes of good performance measures,
noting that measures should be linked to agency goals and missions; be
clearly stated; include measurable targets; and be objective, reliable,
and balanced.[Footnote 21] Specifically, we found four of these
characteristics that highlight key features of performance measures
that can help ensure the successful linking of performance measures and
rewards and penalties:
* the performance being measured should be within the recipient's
sphere of influence,
* the performance measures should be suitable to the mechanism
evaluation cycle, and:
* the performance measures and performance data should be tested.
Performance Should Be within Recipient's Ability and Influence:
Performance measures tied to rewards and penalties should represent
performance that can be sufficiently influenced by the grant
recipient's actions. Absent this linkage, the grantee may have little
motivation to change behavior to improve performance, and the granting
agency risks wasting funds by either rewarding efforts that cannot
reasonably be tied to grantee behavior or penalizing a grantee for
outcomes that even its best efforts may not have prevented. For
example, the Temporary Assistance to Needy Families (TANF) bonus
payments[Footnote 22] rewarded states for reducing out-of-wedlock
births. Several studies report, however, that there does not appear to
be a link between the existence of these programs, or increases in
efforts to deliver program services, and the TANF bonus payment. Many
state officials perceive the outcome measure as inappropriate,
relatively difficult to influence, or both, and discourage attempts to
do so. According to one study, several states reported that they did
not compete or did not continue to compete for the bonus funds because,
among other reasons, their actions would not sufficiently affect the
out-of-wedlock birth rate; therefore they directed their efforts to
activities that were more directly under their influence.
In another example, the Perkins III CTE program has a financial
incentive system that assesses state performance through performance
measures that support its four core indicators--one of which encourages
participation in and completion of programs leading to nontraditional
employment.[Footnote 23] State and local officials in Arizona said
their ability to affect performance for this indicator is very limited.
They told us that although they have tried to address the barriers to
nontraditional employment, they found that cultural and demographic
influences have limited their ability to improve performance every
year. Because performance has not improved as a result of their
efforts, they focus most of their energy on efforts to improve
performance in the other three core indicator areas, which reflect
performance that is more directly under their control.[Footnote 24]
Measures Should Be Suitable to the Mechanism Cycle:
Measures should assess performance that can be observed, achieved, and
reported frequently enough to inform the use of awards and penalties on
a timely basis. For example, an annual reward or penalty should be tied
to a measure that is also assessed annually.
ORC uses performance measures that can be assessed in a relatively
short period of time and that support program outcomes. ORC holds back
10 percent of ProFac's management fee each month. Each quarter, ProFac
has an opportunity to earn the holdback on the basis of its performance
during the prior quarter on 30 KPIs. For example, 1 of the quarterly
indicators tied to its overall customer service objective is the
"overall customer satisfaction rate with project delivery." The
quarterly assessment is based on performance information gathered
through customer satisfaction surveys of local managers and facility
management contacts for all alteration projects, capital repairs, or
both completed in the previous quarter. Both ORC and ProFac officials
credit the frequency of evaluation for motivating ProFac to maintain
high performance throughout the year.
OVAE uses the timing of its grant funding distribution cycle to its
advantage in order to motivate states to meet federal performance
accountability requirements. OVAE disburses grant funds in two pieces:
a small portion in July and the remainder in October. States that did
not provide complete, timely performance data, or missed their
performance targets in the prior year, may have "conditions" put on the
July portion of the funding; if conditions are not met during that
quarter, the October funding is withheld.
Measures and Data Should Be Tested:
Performance measures that trigger accountability mechanisms should be
well functioning and time tested before they are linked to rewards and
penalties to minimize the potential for unintended consequences.
Although our literature review did not specify how long this could
take, one study in our review noted that many leading companies use and
test their measurement systems for years before linking them to
accountability provisions.
Performance data should also be tested to make sure they are credible,
reliable, and valid. Absent these attributes, organizations lack the
basis for sound decisions about rewards and penalties. Data quality is
so critical to performance accountability and oversight of grants that
several organizations use it as the principal performance measure for
performance-based funding. Pinellas County, Florida, alters the case
funding rates paid to its ambulatory service contractor based on data
quality. This "altered funding rate" provision links case reimbursement
rates directly to data quality. For example, data that are incomplete,
illegible, inaccurate, altered, or lacking evidence of medical
necessity--and limit the county's ability to claim for payment or use
its data processing procedures--result in reduced reimbursements to the
ambulatory service contractor for the affected cases. Pinellas County
reports that as a result, ongoing data quality issues are minimal. In
another example, the Child Support Performance and Incentive Act of
1998 prohibits the payment of financial incentives to states for
performance in program areas where state data have failed an annual
data reliability test. This requirement ensures that incentive payments
are based on reliable and complete performance information.
Ensure Grantor and Grantee Technical Capacity:
Grantor and grantee capacity--specifically, the knowledge about
performance accountability mechanisms and the ability to effectively
implement them--is critical to the effectiveness of performance
accountability systems. For example, when the Air Force implemented its
performance-based contracting program, it found that employee training
focusing on how the performance-based aspects of the contracts should
work were most critical. Specifically, practices such as providing a
step-by-step approach to the process that outlined who should be
involved at each step, how much of their time and effort would be
required at each step, and what their specific roles and
responsibilities would be were critical to employees understanding what
was needed to create mechanisms to improve performance.
In addition, federal CSE staff in Region III provide a "Child Support
Enforcement Incentives 101" presentation to state and county CSE staff
throughout the region to explain how the performance measures and
incentive payments work. This training presentation is tailored to the
experience of CSE staff and the demographics of the county, state, or
both (large urban, rural, large interstate caseloads, etc.) but strives
to provide a clear and consistent message: the everyday activities of
CSE staff directly affect the amount of child support available to
children and their families, and drive the amount of incentive payments
the county specifically, and the state in general, earns. The
presentation includes interactive exercises to show how each employee's
casework feeds into outcome-based program results.
Implement System in Stages:
Organizations go through a number of stages designing, testing, and
revising measurement systems before linking them to accountability
mechanisms. This longer, phased implementation allows organizations to
ensure the system is effectively designed before tying it to rewards
and penalties. During these stages, organizations can conduct pilot
tests, create financial models, and conduct behavioral modeling to
understand and modify a system prior to full implementation. For
example, according to one expert, the Tennessee Valley Authority
completes a "readiness test," an assessment of measurement
effectiveness and suitability, before allowing pay for performance or
similar financial incentive systems to be pinned to that measure. This
helps avoid unintended consequences associated with poorly designed
measures. Phased implementation also allows organizations to adjust to
new demands on their time and resources; set up or modify data
collection systems; and ensure the credibility, validity, and
reliability of the data before they are used to measure performance.
For example, the CSE incentive program was implemented in three stages
to allow states to learn about the new incentives and performance
measures. The five performance areas attached to incentives were
developed and legislatively defined in 1998. In 1999, the new data
measures were used by the states and audited for data reliability for
the first time. In year one, one-third of the total incentive funds
were allocated based on the new formula and the remaining funds were
allocated based on the old system. In year two, two-thirds of the
funding was allocated using the new system, and the remaining funds
were allocated based on the old system. In year three, all incentive
funding was allocated according to the new formula.
Collaboration and Oversight Also Key to Success:
In addition to these strategies described above, we saw extensive use
of partnerships and collaborations and regular and effective oversight
and feedback. We have previously reported that these practices are
often associated with high-performing organizations[Footnote 25] and
organizations that effectively used performance information to manage.
Designing and implementing accountability provisions in a collaborative
environment can help develop and encourage buy-in and support and lead
to improvements. For example, Arizona state and local CTE officials
said the state's focus has shifted from a compliance-focused "audit,"
ensuring performance data were properly collected and reported, to a
true partnership in which state and local officials work together to
identify and replicate successes, find solutions to challenges, and
thereby improve performance. State CTE staff spend several days each
year meeting with local CTE officials and providing regular assistance
through on-site technical assistance teams, phone calls, and e-mails.
Oversight and feedback are critical to creating and sustaining
effective performance accountability provisions. We have previously
reported on oversight practices, noting specifically the value of
feedback provided through performance monitoring plans and tools such
as site visits, document reviews, and evaluations. For example, OVAE
employs a number of tools to provide feedback and assistance to states
implementing the Perkins III vocational education program. Among these
tools are:
* establishing state guidance that outlines how to meet the Perkins III
performance requirements,
* developing a peer-to-peer mentorship program among states and with
OVAE to share experiences and good practices,
* conducting monthly conference calls with state directors and data
specialists to discuss challenges and solutions to data collection and
quality,
* offering data quality "institutes" and conferences to share
performance measurement and data quality and collection practices, and:
* providing technical assistance to states.
An OVAE official said providing these types of oversight and feedback
activities generated ideas and discussion to help states improve their
performance; the state CTE officials with whom we spoke agreed.
Various Opportunities Exist at the Federal Level to Enhance Performance
Accountability in Grants:
The experiences with and strategies related to federal grant
accountability provisions described in this report suggest a number of
opportunities for Congress and the executive branch to improve the
design and implementation of performance accountability mechanisms.
First, a results-focused design can help encourage performance
accountability in general and specifically provide for--or at least not
prohibit--the use of accountability mechanisms to encourage desired
behavior. In addition, the use of national program evaluation studies
and research and demonstration grants can provide valuable information
to assist in agency and congressional oversight of and knowledge about
accountability mechanisms. Because credible performance information and
performance measures form the basis for well-functioning accountability
provisions, it remains critical for Congress and the executive branch
to continue to encourage their development and use. Finally, OMB and
agencies can commit to sharing good practices and lessons learned from
experiences with performance accountability provisions in federal
grants--an efficient and effective way to increase grantor and grantee
knowledge, understanding, and use of these provisions.
A Results-Focused Design Encourages Performance Accountability:
Considering grant design features and their implications for grantee
flexibility and accountability can help policymakers provide for
appropriate accountability provisions, whatever type of grant design is
selected. We have previously reported that policy options reflected in
grant design collectively establish (1) the degree of flexibility
afforded to states or localities; (2) the relevance of performance
objectives for grantee accountability; (3) whether accountability for
performance rests at the federal, state, or local level; and (4)
prospects for measuring performance through grantee reporting and
oversight.[Footnote 26] Under a results-oriented approach, federal
policymakers would specify national goals and objectives in statute,
enact a process for establishing them, or adopt some combination of the
two. As a result, when designing or reauthorizing grants, it is
important to consider questions like the following:
* Is there a need for national performance objectives in this policy
area? If so, grantees may be required to use uniform performance
measures--as in the CSE program--to gauge progress. This allows for
comparisons across grantees, and the supporting performance data
collected from grantees have the advantage of being program specific.
However, uniform activities, objectives, and measures may not exist or
may not be desirable, especially under flexible grant program designs.
In these cases, Congress may instead decide to allow grantees to
establish their own program objectives. For example, the Child Care and
Development Block Grant requires states to certify that they have
requirements in effect to protect the health and safety of children
whose child care is subsidized by the block grant. These requirements
must cover the areas of preventing and controlling for infectious
diseases, physical premise safety, and health and safety training.
However, the specificity and stringency of these requirements and the
manner in which they are enforced is left to the states. The Perkins
III legislation outlines several performance areas and requires states
to determine the measures they will use to measures progress in these
statutorily defined areas. Performance targets for these measures are
negotiated with OVAE. In these cases, the federal role in monitoring
the grants is generally limited to collecting information on state and
local program efforts and accomplishments as well as evaluating and
disseminating information on best practices. Another option is to grant
temporary exemptions (waivers) from certain federal program
requirements to grantees that demonstrate that the flexibility granted
can lead to performance improvements. For example, Oregon Option is an
intergovernmental partnership that seeks to improve performance on
benchmarks for a broad variety of initiatives, including childhood
immunization, employment for the disabled, wild salmon recovery,
juvenile justice, welfare reform, and child nutrition, by waiving
administrative rules or seeking statutory change.
* In all cases, what accountability provisions are needed to support
attainment of national performance objectives? These might include
constraints on activities and funds distribution or operational
objectives, standards, and criteria for performance. These can be set
for the program as a whole or delegated to the level of government
responsible for program management. Additional considerations are as
follows: What data are needed for grantee accountability, and is it
feasible to collect these data from providers? Is it possible to
collect data at the project level? Will the contribution of federal
funds be distinguishable from state, local, and private funds? If the
answer to several of these questions is no, is additional information
needed for program oversight? If so, how will such information be
gathered and reported? The answers to questions such as these provide
the basis for setting grantee reporting requirements.
Careful Use of National Program Evaluation Studies and Research and
Demonstration Grants Can Help Assess Mechanism Performance:
Congress has a number of opportunities to conduct oversight, such as
when it establishes or reauthorizes a new program, during the annual
appropriations process, and during hearings focused on program and
agency operations. Providing for--or at a minimum, not prohibiting--
performance accountability mechanisms can provide timely, targeted
performance information and help policymakers ensure that federal
grants focus on their goals, providing another basis for congressional
oversight.
National program evaluations have the potential to answer questions
about both overall program performance as well as the effectiveness of
performance accountability mechanisms, in terms of their
implementation, outcomes, impacts, and cost-effectiveness. However,
national programwide evaluations are expensive in terms of dollars and
time and frequently require capacities and resources beyond those
provided for program management. Also, while evaluations of multiple
sites provide valuable information, programwide evaluation data are
typically periodic and often cover too few sites to support national
estimates of performance. In these cases, research and demonstration
projects often can provide better information on the effectiveness of
various service delivery methods and approaches. Knowledge to support
effective practice is well established in some subject areas and can be
incorporated into program provisions (such as service standards) or in
companion technical assistance or knowledge dissemination programs.
Encourage Development and Use of Credible Performance Information and
Performance Measures:
As we discussed earlier, performance accountability provisions rely on
a supply of credible, reliable, and valid data and high-quality
performance measures. We found organizations that recognizing the
importance of data quality, tied incentives to increasing the supply of
this type of information. Unfortunately, as our work on PART[Footnote
27] and GPRA implementation shows, the credibility of performance data
has been a long-standing weakness. OMB, through its development and use
of PART, has provided agencies with a powerful incentive for improving
data quality and availability. However, improving the supply of
performance information is in and of itself insufficient to sustain
performance management and achieve real improvements in management and
program results. Rather, it needs to be accompanied by a demand for and
use of that information by decision makers and managers alike. Key
stakeholder outreach and involvement is critical to building demand
and, therefore, success. Lack of consensus by a community of interested
parties on goals and measures and the way that they are presented can
detract from the credibility of performance information and,
subsequently, its use. While congressional buy-in is critical to
sustain any major management initiative, it is especially important for
performance accountability given Congress's constitutional role in
setting national priorities and allocating the resources to achieve
them. Recognizing this, policymakers could use incentives to encourage
program partners to agree on performance measures and targets against
which performance will be judged.
Share Good Practices and Lessons Learned:
We and others have frequently reported on the benefits of sharing
promising practices and lessons learned to promote performance
accountability in general in federal programs and program partners. We
believe sharing good practices related to the effective design and
implementation of performance accountability mechanisms carries similar
benefits. As noted earlier, some state and local agencies' programs
have used this type of information sharing among themselves and their
grantees and contractors as a means of performance improvement.
OMB, as the focal point for overall management in the executive branch,
plays a key role in promoting performance improvement in federal
programs and has developed or contributed to a number of tools to share
information and encourage improvements to federal grants and program
performance. For example, www.grants.gov includes information on grant
opportunities, resources to assist in writing grant proposals, and a
newsletter highlighting recent grant success stories, and
www.results.gov has information on best practices related to the
President's Management Agenda initiatives--one of which is Budget and
Performance Integration (BPI). Successful implementation of BPI depends
significantly on federal agencies' ability to ensure federal program
partners work toward program goals and are held accountable for
results. Expectmore.gov provides information on PART assessments and
improvement plans; these assessments consider, among other things,
whether the agency regularly collects timely and credible performance
information to manage its programs, and whether the performance
measurements are used to increase accountability. OMB's own Web site
also contains information on and examples of what it considers to be
high-quality PART performance measures; discussion papers on
measurement topics, such as how to effectively measure what you are
trying to prevent; and strategies to address some of the challenges of
measuring research and development programs.
OMB hosts a number of standing work groups and committees--comprising
agency and OMB staff--to address important grant-related issues, all of
which could accommodate a more specific focus on grants accountability
provisions. For example, OMB's Chief Financial Officer's Council has a
standing grants policy committee that focuses on grant application and
reporting streamlining. Agency BPI leads meet monthly and recently
developed a subgroup to share lessons learned related to efficiency
measures that balance effectiveness, quality, and cost. They also
discuss strategies to address the challenges of efficiency measures in
the grant context and to develop additional guidance for agencies in
this area.
In addition, OMB hosted a Block and Formula Grant workshop in October
2005 for federal officials aimed at identifying and sharing best
practices in grants management and performance measurement. OMB staff
agreed that the workshop was a valuable, efficient, and effective way
to share information and lessons learned and that collectively the
participants increased their knowledge and understanding of ways to
enhance grant performance. They also noted that the real difficulty
comes in "what to do next," in other words, implementing the strategies
gleaned from these sessions.
OMB staff told us that focusing specifically on performance
accountability provisions in grants is necessary and useful, but that
to date, they have focused their governmentwide efforts primarily on
encouraging and enhancing agency capacity to develop high-quality,
results-based program performance measures since improving the quality
of measures and data necessarily precedes tying them to accountability
provisions. The Block and Formula Grant workshop addressed issues of
measurement and accountability, and several block grant programs have
been working to strengthen grantee accountability.
Conclusions:
As the challenges of the 21st century grow, it will become increasingly
important for Congress, OMB, and executive agencies to consider how the
federal government can maximize performance and results. This will be
particularly important for federal grant program managers, given the
significant amount of federal resources invested in these tools.
Because many national objectives can only be achieved through state,
local, and nongovernmental organizations, enhancing performance
accountability below the federal level is equally important. In this
report, we identify a variety of accountability mechanisms as well as
key strategies to enhance their use. Collectively, these can help
enhance and sustain performance accountability in grants at all levels
of government.
As the cases we described illustrate, rewards and penalties are
fundamental tools to help drive and motivate desired behaviors, but
performance accountability mechanisms are not one size fits all; there
is no universal transferable mechanism applicable to all programs. The
specific mechanisms used by agencies and programs and highlighted
throughout this report may not be universally adopted by other federal
agencies and programs seeking to improve their own programs.
Nevertheless, many can be tailored to specific grant programs, and the
key strategies can be adapted to address the specific accountability
challenges each agency faces.
Like all successful change initiatives, the progress currently under
way to move from traditional fiscal accountability in grants to greater
accountability for performance will take time; accountability
provisions--and the performance measures associated with them--can take
many years to mature. Although some federal programs are well on their
way to collecting and reporting on reliable, credible, and valid data
that support high-quality outcome goals agreed to by all program
partners, many others are still struggling with how to define
appropriate outcome measures. It will be critical to proceed
thoughtfully and implement performance accountability in phases,
building in enough opportunities to learn from mistakes and revise
measures and mechanisms to reap the benefits of performance management
while minimizing perverse incentives and unintended consequences.
As with all challenges, starting with small steps is often the best way
forward. Accountability provisions can be used to bring program
partners together to identify common ground. For example, programs that
struggle with defining appropriate outcome goals, measures, and targets
may wish to tie incentives to reaching agreement on them. Those that
struggle with poor data quality and data definitions could reward
grantees for progress in this area. Performance accountability--
especially in the early stages--must be constructive, not punitive.
Even if penalties are employed to promote performance accountability,
there should be a constructive, collaborative approach to performance
improvement that precedes them. Tying performance to lower risk,
nonfinancial mechanisms may at first be more acceptable until
performance measures have been time tested and revised as needed and
grantees have had time to collect the necessary data to support the
measures. Above all, a collaborative process that includes Congress,
the executive branch, and grantees will be critical to developing
successful performance accountability systems.
Accountability provisions assume that performance can be improved--but
this requires information sharing and feedback. OMB has a central role
in overseeing the performance and accountability in the federal
government, and has used its role to promote general results-oriented
performance measurement and management practices in federal grants
through Web sites, guidance, work groups, and workshops. Each of these
tools and strategies could be expanded on to specifically promote and
encourage performance accountability in federal grants, both among
related federal grant programs--programs that have a common purpose--
and federal grant types--such as categorical grants, block grants, and
funding streams. Sharing good practices and lessons learned and
providing feedback on performance are valuable practices that can
leverage resources to enhance knowledge and further performance
accountability. Leading practices can be shared within and among
agencies, grant programs, grantees, and even grant types. OMB
recognized the value in sharing information on performance
accountability mechanisms, but has not yet focused on this issue.
Recommendation for Executive Action:
We are therefore recommending that the Director of OMB encourage and
assist federal agencies in working with the Congress to expand the
effective use of performance accountability mechanisms, focusing on the
practices in this report, when federal grant programs are being created
or reauthorized. We further recommend that OMB offer opportunities for
knowledge transfer among federal agencies and encourage agencies to
share leading practices and lessons learned in implementing grant
accountability mechanisms. Possible vehicles for the collection and
dissemination of this information include good practices guides and
workshops and Web sites such as results.gov, grants.gov, and
expectmore.gov.
Agency Comments:
On August 22, 2006, we provided a draft of this report to the Director
of OMB and the Secretaries of Education and Health and Human Services.
We also provided relevant sections of a draft of this report to the
grantees and contractors highlighted in this report. We received
technical comments from all three agencies, which were incorporated as
appropriate. In addition, OMB agreed with our recommendation but
suggested we broaden it to address the role of federal agencies and
Congress in the grant design and reauthorization process. We agree, and
have amended our recommendation accordingly.
We are sending copies of this report to the Director of the Office of
Management and Budget, the Secretaries of Education and Health and
Human Services, and other interested parties. We will also make copies
available to others upon request. In addition, the report is available
at no charge on GAO's Web site at [Hyperlink, http://www.gao.gov].
Please contact me on (202) 512-6543 or steinhardtb@gao.gov if you or
your staff have any questions about this report. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. Key contributors to this report are
acknowledged in appendix II.
Sincerely yours,
Signed by:
Bernice Steinhardt:
Director, Strategic Issues:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
The objectives of this report were to identify (1) the challenges to
performance accountability in grants; (2) the kinds of mechanisms that
are being used to improve grant performance and how; and (3) given the
findings of questions 1 and 2, what strategies the federal government
can use to encourage the use of these mechanisms, as appropriate.
To meet the first and second objectives, we interviewed experts in
grants and performance management, including individuals from the
following organizations: the School of Public Policy at the University
of Maryland, the University of Central Florida, the John F. Kennedy
School of Government at Harvard University, the John C. Stennis
Institute of Government at Mississippi State University, the Public and
International Affairs Department at George Mason University, the
Political Science Department at the University of New Hampshire,
Measurement International, and the American Productivity and Quality
Center.
Based on our literature review, we developed a coding scheme for
identifying (1) types of performance accountability mechanisms and (2)
strategies used to successfully design and implement these mechanisms.
We used these codes in a content analysis we conducted on a subset of
the documents we reviewed. We chose the documents for content analysis
based on the following criteria:
* discussed accountability systems, mechanisms, or both, discussed
general practices that facilitated to the effective use of
accountability mechanisms, or provided case examples;
* published in 1993 or later;
* found in major electronic databases; and:
* published in the United States.
The content analysis was conducted by two analysts, with the second
analyst conducting a dependent review. Discrepancies in coding were
discussed and agreement reached between the two analysts. Our analysis
produced an inventory of performance accountability mechanism types and
five strategies used to facilitate the effective design and
implementation of performance accountability mechanisms. See the
bibliography for documents included in our review.
To illustrate the mechanisms and strategies identified through our
content analysis, we used relevant case examples found in the
literature. To further illustrate the mechanisms and design and
implementation strategies, we also selected four additional case
illustrations--two federal grant programs and two nonfederal contract
cases. These four cases were selected based on our literature review,
interviews with experts, and reviews of prior GAO work because they are
good examples of where (1) a performance mechanism was present and (2)
there is reason to believe that performance improved at least in part
because of the mechanism.
To screen and develop the grant case illustrations, we interviewed
regional and headquarters federal agency officials and officials at
county/local offices. We also reviewed grant legislation, program
guidance, and prior studies. To develop contract case illustrations, we
interviewed officials both from the contracting agencies and the
contractors and reviewed the contract.
To address our third objective, we synthesized prior GAO work, and we
interviewed officials at the Office of Management and Budget.
We conducted our work from December 2005 through August 2006 in
accordance with generally accepted government auditing standards.
[End of section]
Appendix II: GAO Contact and Staff Acknowledgments:
GAO Contact:
Bernice Steinhardt (202) 512-6543 or steinhardtb@gao.gov:
Acknowledgments:
Jackie Nowicki (Assistant Director) and Chelsa Gurkin (Senior Analyst-
in-Charge) managed this assignment. David Bobruff, Katie Hamer, and
Anne Marie Morillon made significant contributions to all aspects of
the work. Kate France significantly contributed to the initial research
and design of the assignment. In addition, Tom Beall and Jay Smale
provided methodological assistance, Amy Rosewarne provided key
assistance with message development, and Donna Miller developed the
report's graphics.
[End of section]
Bibliography:
American Productivity & Quality Center. Achieving Organizational
Excellence Through the Performance Measurement System. Houston: 1999.
American Productivity & Quality Center. Measure What Matters: Aligning
Performance Measures with Business Strategy. Houston: 1999.
Ashworth, Karl, and others. "When Welfare-to-Work Programs Seem to Work
Well: Explaining Why Riverside and Portland Shine So Brightly."
Industrial & Labor Relations Review, vol. 59, iss. 1 (2005).
Ausink, John, and others. Implementing Performance-Based Services
Acquisition (PBSA): Perspectives from an Air Logistics Center and a
Product Center. A Documented Briefing prepared by the RAND Corporation
for the U.S. Air Force. 2002.
Ausink, John, Frank Camm, and Charles Cannon. Performance-Based
Contracting in the Air Force: A Report on Experiences in the Field. A
Documented Briefing prepared by the RAND Corporation for the U.S. Air
Force. 2001.
Baker, George. "Distortion and Risk in Optimal Incentive Contracts."
The Journal of Human Resources, vol. 37, no. 4 (2002).
Banta, Trudy W., and others. "Performance Finding Comes of Age in
Tennessee." The Journal of Higher Education, vol. 67, no. 1 (1996).
Conlon, Timothy. "Grants Management in the 21st Century: Three
Innovative Policy Responses." IBM Center for the Business of
Government: Financial Management Series. Washington, D.C.: October
2005.
Courty, Pascal, and Gerald Marschke. "Measuring Government Performance:
Lessons from a Federal Job-Training Program." The American Economic
Review, vol. 82, no. 2 (1997).
Dodds, Dan R. "Performance Incentives Can Spark Greater Productivity."
School Administrator, vol. 55, iss. 3 (1998).
Domestic Working Group: Grant Accountability Project. Guide to
Opportunities for Improving Grant Accountability. Washington, D.C.:
October 2005.
Dye, Jane Lawler, and Harriet B. Presser. "The State Bonus to Reward a
Decrease in 'Illegitimacy': Flawed Methods and Questionable Effects."
Family Planning Perspectives, vol. 31, no. 3 (1999).
Eberts, Randall, Kevin Hollenbeck, and Joe Stone. "Teacher Performance
Incentives and Student Outcomes." The Journal of Human Resources, vol.
37, no. 4 (2002).
Garber, Alan M. "Evidence-Based Guidelines as a Foundation for
Performance Incentives." Health Affairs, vol. 24, no. 1 (2005).
Gordon, Stephen B. "Performance Incentive Contracting: Using the
Purchasing Process to Find Money Rather Than Spend It." Government
Finance Review, August (1998).
Hatry, Harry P., and others. How Federal Programs Use Outcome
Information: Opportunities for Federal Managers. Washington, D.C.: IBM
Center for the Business of Government, May 2003.
Hildebrandt, Gregory G. "The Use of Performance Incentives in DOD
Contracting." Acquisition Review Quarterly, Spring (1998).
Johnston, Jocelyn M., and Barbara S. Romzek. "State Social Services
Contracting: Exploring the Determinants of Effective Contract
Accountability." Public Administration Review, vol. 65, iss. 4 (2005).
Kathuria, Sandeep. "An Overview of Share-in Savings Contracting."
Contract Management, vol. 45, no. 11 (2005).
King, Donald C., and Paul D. Tolchinsky. "Do Goals Mediate the Effects
of Incentives on Performance?" Academy of Management Review, vol. 5,
no. 3 (1980).
Lane, Nancy E. "Performance Incentives in The Massachusetts Behavioral
Health Program." Administration and Policy in Mental Health, vol. 32,
no. 4 (2005).
Marcus, Alfred A., Barry M. Mitnick, and Kiran Verma. "Making Incentive
Systems Work: Incentive Regulation in the Nuclear Power Industry."
Journal of Public Administration Research and Theory: J-PART, vol. 9,
no. 3 (1999).
Martin, Lawrence L. "Making Performance-Based Contracting Perform: What
the Federal Government Can Learn from State and Local Governments." IBM
Endowment for the Business of Government: New Ways to Manage Series.
Washington, D.C.: 2002.
Martin, Lawrence L. Performance-Based Contracting (PBC) for Human
Services: A Review of the Literature. Orlando: Center for Community
Partnerships, University of Central Florida, 2003.
Metzenbaum, Shelley H. "Performance Accountability: The Five Building
Blocks and Six Essential Practices." IBM Center for the Business of
Government: Managing for Performance and Results Series. Washington,
D.C.: 2006.
Rand Corporation. Organizational Improvement and Accountability:
Lessons for Education from Other Sectors. 1st ed. Santa Monica, Calif.:
2004.
Sammer, Joanne. "Making Incentive$ Pay." Industry Week: Leadership in
Manufacturing. (Cleveland: Penton Media Inc., August 2002).
http://www.industryweek.com/ReadArticle.aspx?ArticleID=1121 (downloaded
Mar. 9, 2006).
Tsimbinos, John M. "Are You Willing to Pay for Success?" Bottomline,
vol. 7, no. 6 (1990).
U.S. Department of Defense, Deputy Under Secretary of Defense for
Acquisition Reform. Incentive Strategies for Defense Acquisitions.
Washington, D.C.: 2001.
U.S. Department of Energy, Office of Inspector General. Inspection of
Selected Aspects of the Office of River Protection Performance-Based
Incentive Program. Washington, D.C.: 2001.
FOOTNOTES
[1] Pub. L. No. 103-62, 107 Stat. 285 (1993).
[2] Pub. L. No. 105-220, Pub. L. No. 107-110.
[3] Pass-through grants are federal grants given to state governments
that are subsequently distributed to county, municipal, or township
governments.
[4] The other attributes of high-performing organizations are a clear,
well-articulated, and compelling mission; a focus on the needs of the
clients and customers; and the strategic management of people.
[5] GAO, Comptroller General's Forum: High-Performing Organizations:
Metrics, Means, and Mechanisms for Achieving High Performance in the
21st Century Public Management Environment, GAO-04-343SP (Washington,
D.C.: Feb. 13, 2004), and Managing for Results: Enhancing Agency Use of
Performance Information for Management Decision Making, GAO-05-927
(Washington, D.C.: Sept. 9, 2005).
[6] Pub. L. No. 98-502.
[7] Domestic Working Group, Grant Accountability Project, Guide to
Opportunities for Improving Grant Accountability. October 2005. This
guide states that it is designed to provide government executives at
the federal, state and local levels with ideas for better managing
grants. The guide focuses on specific steps taken by various agencies.
The intent is to share useful and innovative approaches taken, so that
others can consider using them.
[8] On August 12, 2006, the Carl D. Perkins Career and Technical
Education Improvement Act of 2006 (Perkins IV) became Pub. L. No. 109-
270. Perkins IV includes some revisions to the performance and
accountability provisions.
[9] GAO, Grant Programs: Design Features Shape Flexibility,
Accountability, and Performance Information, GAO/GGD-98-137
(Washington, D.C.: June 22, 1998).
[10] GAO, Federal Assistance: Grant System Continues to Be Highly
Fragmented, GAO-03-718T (Washington, D.C.: Apr. 29, 2003).
[11] Pub. L. No. 106-107, the Federal Financial Assistance Management
Improvement Act of 1999 requires OMB to coordinate agency efforts to
streamline the administrative requirements of federal grants and engage
and involve grantees in developing and implementing their reform goals
and implementation plans. The act also requires GAO to evaluate the
reform efforts. See GAO, Grants Management: Grantees' Concerns with
Efforts to Streamline and simplify Processes. GAO-06-566 (Washington,
D.C.: July 28, 2006).
[12] GAO, Block Grants: Issues in Designing Accountability Provisions.
GAO/AIMD-95-226 (Washington, D.C.: Sept. 1, 1995).
[13] We identified these strategies through our literature review, and
illustrate them with examples from the literature and from our
additional four case illustrations.
[14] GAO-04-343SP, GAO-05-927.
[15] Pub. L. No. 105-200.
[16] The five performance goals are (1) paternity establishment, (2)
child support order establishment, (3) collections on current support
due, (4) collections on arrears, and (5) cost-effectiveness.
[17] The Lewin Group, Study of the Implementation of the Performance-
Based Incentive System, October 2003.
[18] GAO, Results-Oriented Cultures: Creating a Clear Linkage between
Individual Performance and Organizational Success, GAO-03-488
(Washington, D.C.: Mar. 14, 2003).
[19] GAO, Defense Acquisitions: DOD Has Paid Billions in Award and
Incentive Fees Regardless of Acquisition Outcomes, GAO-06-66
(Washington, D.C.: Dec. 19, 2005).
[20] GAO, Tax Administration: IRS Needs to Further Refine Its Tax
Filing Season Performance Measures, GAO-03-143 (Washington, D.C.: Nov.
22, 2002).
[21] GAO-03-143.
[22] The Personal Responsibility and Work Opportunity Reconciliation
Act of 1996 authorized the Bonus to Reward Decrease in Illegitimacy
Ratio, a provision intended to motivate states to pursue nonmarital
birth prevention programs. This provision awarded up to $25 million in
each of fiscal years 1999 through 2002 to as many as five states
showing the largest reduction in nonmarital births.
[23] Nontraditional employment relates to the participation of students
in fields in which their gender constitutes less than 25 percent of the
individuals employed in that field (e.g., female students participating
in automotive repair programs).
[24] However, as discussed earlier, states can periodically revise
their Perkins III CTE performance measures and targets during annual
negotiations of their state plan.
[25] The other attributes of high-performing organizations are a clear,
well-articulated, and compelling mission; a focus on the needs of the
clients and customers; and the strategic management of people.
[26] GAO/GGD-98-137.
[27] OMB developed PART as a diagnostic tool meant to provide a
consistent approach to assessing federal programs during the executive
budget formulation process. PART covers four broad topics for all
programs selected for review: (1) program purpose and design, (2)
strategic planning, (3) program management, and (4) program results. We
have previously reported on PART in GAO, Performance Budgeting: PART
Focuses Attention on Program Performance, but More Can Be Done to
Engage Congress, GAO-06-28 (Washington, D.C.: Oct. 28, 2005).
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
