Workforce Investment Act
Improvements Needed in Performance Measures to Provide a More Accurate Picture of WIA's Effectiveness Gao ID: GAO-02-275 February 1, 2002Congress passed the Workforce Investment Act (WIA) in 1998 to bring most federally funded employment and training services into a single, one one-stop center system. GAO assessed three programs that provide service through this system. States and localities have begun to implement the new performance measurement system for the three WIA-funded programs but report several challenges. States had to change the way they collected and reported performance data. They also faced challenges in implementing these measures due to their complexity and the resource demands created by new measures. Some developed new procedures to obtain access to sensitive records. The performance levels are of particular concern to state and local officials because failure to meet them can result in financial sanctions. As a result, states may be choosing to serve only those job seekers who are most likely to be successful. Even when fully implemented, performance measures may not provide a true picture of WIA-funded program performance because data are neither comparable across states nor timely. The measures include many of the indicators relevant to an employment and training program, such as getting and keeping jobs and increasing wages and skills. Although measures exist to gauge the performance of the three WIA-funded programs, there are no measures to gauge the performance of the one-stop system as a whole. At least 17 programs provide services through the one-stop system, and most have their own performance measures. Although these performance measures may be used for assessing outcomes for individual programs, they cannot be used to measure the success of the overall system.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-02-275, Workforce Investment Act: Improvements Needed in Performance Measures to Provide a More Accurate Picture of WIA's Effectiveness
This is the accessible text file for GAO report number GAO-02-275
entitled 'Workforce Investment Act: Improvements Needed in Performance
Measures to Provide a More Accurate Picture of WIA's Effectiveness'
which was released on February 1, 2002.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
United States General Accounting Office:
GAO:
Report to Congressional Requesters:
February 2002:
Workforce Investment Act:
Improvements Needed in Performance Measures to Provide a More Accurate
Picture of WIA's Effectiveness:
GAO-02-275:
Contents:
Letter:
Results in Brief:
Background:
States and Localities Are Taking Action to Implement WIA Performance
Measures but Face Challenges in Doing So:
Performance Measures May Not Accurately Assess Performance of the Three
WIA-Funded Programs:
Existing Performance Measures Fail to Gauge Overall One-Stop
Performance:
Conclusions:
Recommendations for Executive Action:
Agency Comments:
Appendix I: WIA Performance Measures for Adults, Dislocated Workers,
and Older Youth:
Appendix II: Comments from the Department of Labor:
Appendix III: GAO Contacts and Staff Acknowledgments:
GAO Contacts:
Staff Acknowledgments:
Related GAO Products:
Tables:
Table 1: Performance Measures for the Three WIA-Funded Programs:
Table 2: WIA‘s Mandatory Programs and Related Federal Agencies:
Table 3: Comparing Wage Replacement Rates for Two Job Seekers:
Figures:
Figure 1: Timing of Data Collection by Quarter for Selected Performance
Measures:
Figure 2: Oregon‘s Bean Counter:
Figure 3: Number of Months it Takes to Receive UI Data for Job
Placement:
Figure 4: Comparison of Performance Measures for 17 One-Stop Programs:
Abbreviations:
GAO: General Accounting Office:
GPRA: Government Performance and Results Act:
JTPA: Job Training Partnership Act:
MDA: Mississippi Development Authority:
MIS: management information system:
OMB: Office of Management and Budget:
TEGL: Training and Employment Guidance Letter:
UI: Unemployment Insurance:
WIA: Workforce Investment Act:
WRIS: Wage Record Interchange System:
[End of section]
United States General Accounting Office:
Washington, DC 20548:
February 1, 2002:
The Honorable Edward M. Kennedy:
Chairman:
Committee on Health, Education, Labor and Pensions:
United States Senate:
The Honorable James M. Jeffords:
United States Senate:
The Congress passed the Workforce Investment Act (WIA) in 1998 to begin
unifying a fragmented employment and training system and to better
serve job seekers and employers. To create a more comprehensive
workforce investment system, WIA requires states and localities to
bring together most federally funded employment and training services
into a single system, called the one-stop center system. Seventeen
programs across four federal agencies”programs such as the Employment
Service (Wagner-Peyser) and Adult Education and Literacy”must provide
services through this one-stop system.[Footnote 1] Three of these
programs, whose funding is authorized by WIA under Title I to provide
services to adults, dislocated workers, and youth, replace those
previously funded under the Job Training Partnership Act (JTPA).
[Footnote 2] These three new WIA programs, authorized at about $3.7
billion in fiscal year 2001, have performance measures established
under WIA that states and localities must track in order to demonstrate
the effectiveness of the programs. These performance measures gauge
program results in the areas of job placement, employment retention,
and earnings change, as well as skill attainment and customer
satisfaction. States are held accountable by the U.S. Department of
Labor for their performance in these areas. If they fail to meet their
expected performance levels, they may suffer financial sanctions; if
they meet or exceed their levels, they may be eligible to receive
additional funds, called incentive grants.[Footnote 3] In order to make
objective decisions about which states receive incentives and
sanctions, WIA seeks to ensure that states collect and report
comparable performance data on all participants.
In an effort to assess the effectiveness of WIA‘s performance measures
and whether they will yield useful information that can clearly
demonstrate performance under WIA, we assessed (1) the progress states
and localities have made and the issues they have faced in implementing
performance measures for the three WIA-funded programs; (2) how useful
the WIA performance measures are in accurately gauging the performance
of the three WIA-funded programs; and (3) beyond gauging the
performance of the three WIA-funded programs, how well the performance
of the one-stop system is being measured.
Our review is based on a survey of WIA program administrators in all 50
states; visits to Florida, Indiana, Missouri, Oregon, and Pennsylvania,
where we interviewed state and local workforce investment officials;
interviews with Labor officials and with national associations
representing state and local workforce development officials; and a
review of relevant documents. We selected the five states for site
visits based on a variety of factors, including their experience using
unemployment insurance (UI) wage record data to measure outcomes, the
status of their data collection systems, and their progress in
developing additional measures for state and local use. Our review
focused on measures for Title I Adult, Dislocated Worker, and Older
Youth programs. We did not review the performance measures for younger
youth (age 14-18).[Footnote 4] We conducted our work between December
2000 and August 2001 in accordance with generally accepted government
auditing standards.
Results in Brief:
States and localities have taken action to implement the new
performance measurement system for the three WIA-funded programs but
reported confronting a number of challenges in doing so. As part of
their implementation efforts, states had to change the way they
collected and reported performance data from the way they were
collected for previous programs. Most states decided to develop new
automated data systems, but 15 states do not have their systems
completely in place. The timing of the guidance from Labor on how to
report the data--not issued until well into the first year of
implementation--delayed the development of these systems. States and
localities also faced challenges in implementing these measures due to
their complexity and the resource demands created by new measures; some
had to develop new procedures to obtain access to sensitive records.
For example, a new measure to assess customer satisfaction for job
seekers and employers requires states to implement complicated
procedures to conduct a customer satisfaction survey. In addition,
state officials expressed concern that the levels of performance that
they are expected to achieve are too high. Because the program was in
its first year, there were little or no data that could be used as a
point of comparison, or baseline, for establishing performance levels.
Furthermore, state and local officials believed that the established
performance levels did not take into account local economic conditions.
The performance levels are of particular concern to state and local
officials because failure to meet them can result in financial
sanctions. As a result, states may be choosing to serve only those job
seekers who are most likely to be successful. For example, to meet
their performance levels to replace earnings for workers who have been
dislocated from their jobs, local staff might provide WIA services only
to job seekers who have the best prospect of finding work that can
match or surpass their previous earnings.
Even when fully implemented, WIA performance measures may still not
provide a true picture of WIA-funded program performance largely
because data are not comparable across states or timely. The measures
include many of the indicators relevant to an employment and training
program, such as getting and keeping jobs and increasing wages and
skills. However, data may not accurately gauge performance and are not
comparable across states for a variety of reasons. For example, Labor‘s
guidance on enrolling job seekers”a process called registration”does
not provide clear direction on when to start collecting performance
data on participants, and we found that states and localities differed
on whom they tracked and when. In addition, UI wage records, required
in order to track performance outcomes, are not easily used to identify
employment across state lines, and some states are more successful than
others in obtaining this information. Only seven states are currently
using a clearinghouse”called the Wage Record Interchange System
(WRIS)”that allows states to exchange UI data. Many of the remaining
states are reluctant to participate, often citing cost concerns. In
addition, the time lag associated with UI data becoming available (as
much as 9 months or longer) results in states not being able to use the
data to manage their programs in the short term.
While there are measures designed to gauge the performance of the three
WIA-funded programs, there are no measures to gauge the performance of
the one-stop system as a whole. At least 17 programs provide services
through the one-stop system and most have their own performance
measures. Although these performance measures may be used for assessing
outcomes for individual programs, they cannot be used to measure the
success of the overall system. For example, no program has a measure to
track job seekers who use only self-service or informational activities
offered through the one-stop, which may constitute a large proportion
of job seekers. Not knowing how many job seekers use the one-stop‘s
services limits the one-stop‘s ability to assess its impact.
Furthermore, state and local officials told us that having multiple
performance measures has impeded coordination among programs. There has
been limited progress in developing overall performance measures for
the one-stop system. Labor convened a working group to develop
additional indicators of the one-stop system‘s performance, but they
have not yet issued them.
To provide the Congress and the public with a more accurate picture of
WIA performance, we are making recommendations to Labor in this report
to postpone the implementation of financial sanctions, to expedite the
development of criteria for renegotiating performance levels, to more
clearly define policies and measures, and to develop ways to address
the challenges in using UI data. In its written comments, Labor
concurred with our findings and recommendations.
Background:
Labor required states to implement major provisions of WIA Title I by
July 1, 2000, although some states began implementing provisions of WIA
as early as July 1999. Services provided under WIA represent a marked
change from those provided under the previous program, allowing for a
greater array of services to the general public. WIA is designed to
provide for greater accountability over what existed previously: it
established new performance measures and a new requirement to use UI
data to track and report on the achievements of the three WIA-funded
programs. WIA also requires that many federal programs work together to
provide employment and training services through the one-stop system.
WIA-Funded Services Represent a Change from those Funded under JTPA:
Program services provided under WIA represent a marked change from
those provided under JTPA. When WIA was enacted in 1998, it replaced
the JTPA programs for economically disadvantaged adults and youth and
for dislocated workers with three new programs”WIA Adult, Dislocated
Worker, and Youth”that provide a broader range of services to the
general public, no longer using income to determine eligibility for all
program services.[Footnote 5] The newly authorized WIA programs
[Footnote 6] no longer focus exclusively on training but provide for
three tiers, or levels, of service for adults and dislocated workers:
core, intensive, and training. Core services include basic services
such as job searches and labor market information. These activities may
be self-service or require some staff assistance. Intensive services
include such activities as comprehensive assessment and case
management”activities that require greater staff involvement. Training
services include such activities as occupational skills or on-the-job
training. These tiers of WIA-funded services are provided sequentially.
That is, in order to receive intensive services, job seekers must first
receive at least one core service; to receive training services, a job
seeker must first receive at least one core service and then at least
one intensive service. Key to moving from core to a higher level of
services is that the services are needed to help job seekers become
self-sufficient. Labor‘s guidance provides for monitoring and tracking
to begin when job seekers receive core services that require
significant staff assistance. Job seekers who receive core services
that are self-service in nature are not included in the performance
measures.
WIA Performance Measures Are Designed to Increase Accountability for
Three WIA-Funded Programs:
WIA is designed to provide for greater accountability than the
accountability provided for under JTPA. It does so by establishing new
performance measures and a new requirement to use UI data to track and
report on the achievements of the three WIA-funded programs. According
to Labor, performance data collected from the states in support of the
measures are intended to be comparable across states in order to
maintain objectivity in determining incentives and sanctions. They are
also intended to provide information to support Labor‘s performance
goals under the Government Performance and Results Act (GPRA)[Footnote
7] and for program evaluation. Some of the measures that relate to
adults, dislocated workers, and older youth are similar to those used
under JTPA, including job placement, job retention, and wage gains or
replacement. Attainment of a credential[Footnote 8]”a degree or
certification of skills or training completed”and customer satisfaction
for both job seekers and employers are new under WIA. (See table 1 for
a complete list of the WIA performance measures and appendix I for a
more complete explanation of the performance measures discussed in this
report.)
Table 1: Performance Measures for the Three WIA-Funded Programs:
WIA funding stream: Adult;
Performance measure:
1. Entered employment rate;
2. Employment retention rate at 6 months;
3. Average earnings change in 6 months;
4. Entered employment and credential rate.
WIA funding stream: Dislocated worker;
Performance measure:
5. Entered employment rate;
6. Employment retention rate at 6 months;
7. Earnings replacement rate in 6 months;
8. Entered employment and credential rate.
WIA funding stream: Older youth (age 19-21);
Performance measure:
9. Entered employment rate;
10. Employment retention rate at 6 months;
11. Average earnings change in 6 months;
12. Entered employment/education/training and credential rate.
WIA funding stream: Younger youth (age 14-18);
Performance measure:
13. Skill attainment rate;
14. Diploma or equivalent attainment;
15. Placement and retention rate.
WIA funding stream: Customer satisfaction;
Performance measure:
16. Customer satisfaction for participants;
17. Customer satisfaction for employers.
Source: U.S. Department of Labor Employment and Training
Administration, Training and Employment Guidance Letter No. 7-99 (Mar.
3, 2000).
[End of table]
In contrast to JTPA, for which data on outcomes were obtained through
follow-ups with job seekers, WIA requires states to use UI wage records
to track outcomes. According to Labor‘s guidance, if a program
participant does not appear in the UI wage records, states may use
supplemental data sources, such as follow-ups with participants and
employers, to track entered employment, retention, and credential
attainment. However, only UI wage records may be used to calculate
earnings change and replacement.
Unlike JTPA, which established expected performance levels using a
computer model, WIA requires states to negotiate with Labor to
establish expected performance levels for each measure. States, in
turn, must negotiate performance levels with each local area. The law
requires that these negotiations take into account differences in
economic conditions, participant characteristics, and services
provided. To derive equitable performance levels, Labor and the states
use historical data to develop their estimates of expected performance
levels. These estimates provide the basis for negotiations.
WIA holds states accountable for achieving their performance levels by
tying those levels to financial sanctions and incentive funding. States
that meet their performance levels under WIA are eligible to receive
incentive grants that may generally range from $750,000 to $3 million.
[Footnote 9] To be eligible for an incentive grant, states must also
meet levels established under the Department of Education‘s Vocational
Education (Perkins Act) and Adult Education and Literacy programs.
States that do not meet their performance levels under WIA are subject
to sanctions.[Footnote 10] If a state fails to meet its performance
levels for 1 year, Labor provides technical assistance, if requested.
If a state fails to meet its performance levels for 2 consecutive
years, it may be subject to up to a 5-percent reduction in its annual
WIA formula grant. Under JTPA, the most stringent sanction was the
possible reorganization of the local service delivery organization.
WIA Requires that Many Federal Programs Work Together to Provide
Services through the One-Stop System:
In addition to establishing the three new programs, WIA requires that
states use the one-stop center system to provide services for these and
many other employment and training programs. This system was developed
by states prior to WIA through One-Stop Planning and Implementation
Grants from Labor. About 17 programs funded through four federal
agencies are now required to provide services through the one-stop
center under WIA.[Footnote 11] Table 2 shows the programs that WIA
requires to provide services through the one-stop centers (termed
mandatory programs) and the related federal agency.
Table 2. WIA‘s Mandatory Programs and Related Federal Agencies:
Federal agency: Department of Labor;
Mandatory programs:
WIA adult;
WIA dislocated worker;
WIA youth;
Employment Service (Wagner-Peyser);
Trade adjustment assistance programs;
Veterans‘ employment and training programs;
Unemployment Insurance;
Job Corps;
Welfare-to-Work grant-funded programs;
Senior Community Service Employment Program;
Employment and training for migrant and seasonal farm workers;
Employment and training for Native Americans.
Federal agency: Department of Education;
Mandatory programs:
Vocational Rehabilitation Program;
Adult Education and Literacy;
Vocational Education (Perkins Act).
Federal agency: Department of Health and Human Services;
Mandatory programs:
Community Block Grant Services.
Federal agency: Department of Housing and Urban Development (HUD)
Mandatory programs:
HUD-administered employment and training.
[End of table]
Under WIA, employers are expected to play a key role in establishing
regional workforce development policies, deciding how services should
be provided in the one-stop, and overseeing one-stop operations.
Employers, who are encouraged to use the one-stop system to fill their
job vacancies, are also seen as key one-stop customers under WIA.
States and Localities Are Taking Action to Implement WIA Performance
Measures but Face Challenges in Doing So:
States and localities are taking action to implement performance
measures for the three WIA-funded programs, but they have confronted
several challenges in doing so. To implement these measures, states and
localities had to change the way they collected and reported
performance data. Most states we surveyed had to create new automated
data systems to collect and report WIA data. Many state systems,
however, are still not completely in place. The lack of final guidance
from Labor on how to report the data slowed the development of these
systems. States and localities also faced challenges in implementing
the measures due to their complexity and the resource demands they
created, and some had to develop new procedures to obtain UI wage
records. In addition, states faced a new negotiation process with Labor
to set performance levels for each measure. Many states believe these
levels are too high because little or no baseline data were used, and
the negotiations did not sufficiently account for differences in
economic conditions and populations served. Under WIA, performance
levels are now tied to incentives and sanctions so that states can be
financially rewarded if they meet them or penalized if they do not.
States reported that the need to meet these performance levels may lead
local staff to focus WIA-funded services on job seekers who are most
likely to succeed in their job search or who are most able to make wage
gains.
Management Information Systems Are Still Being Developed to Collect and
Report WIA Data:
As part of implementing WIA performance measures, states had to develop
automated data systems to track the activities of individual WIA
participants and report on performance.[Footnote 12] Based on our
survey, most states developed a new automated data system, or
management information system (MIS), to collect and report WIA
performance data at the state level. The remaining states adapted their
previous data collection systems used under JTPA. However, 15 states,
regardless of whether they were developing a new system or adapting
their existing system, reported that, as of August 2001, they did not
have their system completely in place. All states expect to have
completed their systems by July 2002. In some states, local areas do
not use the state MIS to collect local WIA performance information. In
these states, local areas must develop their own systems, taking time
and resources to do so.
The lack of timely reporting guidance slowed the development of the
data systems. Final guidance on how states must report their
performance to Labor was issued to the states in March 2001”8 months
after the states were required to implement major provisions of WIA and
begin collecting data.[Footnote 13] Lack of final guidance resulted in
delays and costly program changes as states and local areas developed
and adjusted their final systems. For example, one local area we
visited decided to continue using its old system and delayed the
development of a new system pending final guidance because it would be
too costly and time-consuming to develop a system that might need to be
changed. All states, regardless of whether or not they had implemented
a new system, had to make changes in their automated MIS systems to
accommodate the final guidance.
Complex Performance Measures Stymied State and Local Efforts to
Implement Them:
States and localities reported that the complexity of WIA‘s new
performance measures made them difficult and time-consuming to
implement. Many of the states that we surveyed commented that the
measures were hard to follow because the calculations for the measures
are complex and sometimes confusing, specifically who to include in the
measures, when to collect the data for the measures, and how to
calculate the measures.[Footnote 14]
* Knowing who to include in the measures. It is difficult to know
whether or not a job seeker should be counted in the measure for a
program, even if he or she is served by the program. For example, a
participant in the adult program who is already employed must be
included in the retention and wage gain measures but cannot be counted
in the entered employment measure. Yet, for the dislocated worker
program, the entered employment measure can include those who may be
employed when they enter the program.[Footnote 15]
* Knowing when to measure performance for participants. The data for
different performance measures can be collected in different quarters
of the program year. For example, for customer satisfaction, data can
be collected at two points in time, depending on how the participant
exited the program (see figure 1). Data on entered employment for
participants in the adult program are collected in the third quarter
after exit; retention data are collected in the fifth quarter after
exit.[Footnote 16] Data on earnings change and replacement are
collected at two points in time: pre-program earnings data can be
collected at registration, and post-program earnings data are collected
in the fifth quarter after exit.[Footnote 17] If a program participant
does not appear in the UI wage records, local staff can collect
supplemental data to establish employment for the participant, but this
must be recorded within 30 days after the WIA participant is found
missing from the wage records. For entered employment, staff can
collect supplemental data in the fourth quarter after a participant
leaves (or exits) the program, but for retention, staff can collect it
in the sixth quarter after exit. Supplemental data cannot be used to
measure earnings change and replacement. Because the timing of data
collection is complex and can be confusing, one local area in Oregon
developed a tool it calls the ’bean counter“ to help local staff
determine when to follow-up with participants so their performance
counts in the calculations (see figure 2).
Figure 1: Timing of Data Collection by Quarter for Selected Performance
Measures:
[Refer to PDF for image]
This figure illustrates the following timeline:
Program Year 1999 Quarter 3, Jan.-March:
Participant registers in WIA;
Measure: Earnings change/replacement: Collect data.
Program Year 1999 Quarter 4, April-June:
Program Year 2000 Quarter 1, July-September:
Participant exits WIA;
Program Year 2000 Quarter 2, October-December:
Measure: Customer satisfaction: Collect data.
Program Year 2000 Quarter 3, Jan.-March:
Measure: Customer satisfaction: Collect data.
Measure: Entered employment: Collect data.
Program Year 2000 Quarter 4, April-June:
Measure: Entered employment: 30 days to get supplemental data.
Program Year 2001 Quarter 1, July-September:
Measure: Retention: Collect data.
Measure: Earnings change/replacement: Collect data.
Program Year 2001 Quarter 2, October-December:
Measure: Retention: 30 days to get supplemental data.
Source: U.S. Department of Labor TEGL 7-99.
[End of figure]
Figure 2: Oregon‘s Bean Counter:
[Refer to PDF for image]
This figure is an illustration of the Oregon Bean Counter. By sliding
the paper card in the sleeve, staff can determine the appropriate
quarter in which to follow up with participants.
Source: Worksystems Inc.
[End of figure]
* Knowing how to calculate the measures. In order to calculate the
measures, states must account for a variety of factors. The type and
combination of these factors determine the calculations that will be
used. For example, in calculating some of the measures for the adult
program, states must consider (1) whether the job seeker is employed at
registration, (2) whether he or she is employed at both the first and
third quarters after exit, and (3) whether the data source used to
confirm employment was UI records or supplemental data. This
information in various combinations results in 14 different ways that
adult participants can be grouped in order to calculate the measures.
In addition to noting the complexity of the measures, state and local
officials said that the new measures taxed their resources. States had
to develop procedures to collect data for the new customer satisfaction
measure in compliance with detailed guidance from Labor. The guidance
calls for states to conduct a telephone survey from a random sample
large enough to obtain 500 completed surveys from both participants and
employers. Because this guidance changed over time, states had to
revise their procedures accordingly. For example, a revision to the
guidance issued in October 2001 required states to maintain an up-to-
date list of participants‘ names and addresses from which to sample”a
requirement that was originally voluntary.
One indication of states‘ progress in implementing these measures may
be reflected in their ability to submit complete quarterly reports.
[Footnote 18] Quarterly reports require data on all 17 performance
measures. For the quarterly report that was due in May 2001, all states
submitted their reports, but, according to Labor, only 16 states were
able to provide data for all 17 performance measures. For the quarterly
report that was due in August 2001”more than 1 year after WIA
implementation”all states submitted their reports, but only 23 could
provide data on employer customer satisfaction.[Footnote 19] According
to Labor, states could not fully report on customer satisfaction
because they have not yet fully implemented procedures to measure it.
One state had to compile the data manually because its MIS was not
fully operational.
States Adopt New Procedures to Access Sensitive Information:
WIA‘s new requirement that states use UI wage records to measure
outcomes has led states to adopt new procedures to access these and
other sensitive records. Unlike JTPA, which relied on surveys of
participants to collect information on employment and earnings, WIA
requires UI wage records to be used as the primary data source of
employment and wage information”and the only data source for some
measures, according to Labor‘s guidance. To obtain employment and
earnings information, states match information collected on individual
WIA participants against state UI wage records. To access UI data from
the state agency that oversees the UI database, some states had to
establish data-sharing agreements. In Mississippi, for example, the
agency responsible for overseeing WIA”the Mississippi Development
Authority (MDA)”had to make arrangements with the agency that oversees
the UI data”the Mississippi Employment Security Commission”to have them
match the wage records and provide the results to MDA. In addition,
some states may be more rigorous in protecting the confidentiality of
UI records through privacy laws, which may add obstacles to collecting
performance data. For example, Oregon law prohibits the release of WIA
participants‘ records without informed consent. Consequently, program
providers had to enter into an agreement that established a protocol
for collecting and sharing the data”one that developed safeguards to
protect confidentiality. In addition, the state had to develop a
process to ensure that WIA participants consented to the use of their
protected records in this way.
States Believe Performance Levels Are Set Too High:
All the states we visited believed that some of the established
performance levels for their measures were set too high for them to
meet”either because they were set in absence of historical or baseline
data or because negotiations did not sufficiently account for
variations in economic conditions or population served. States reported
that limitations in available baseline data made it difficult to set
fair, realistic performance levels. The new measures on credentials and
customer satisfaction, for instance, had no prior data available on
which to set performance levels. Where baseline data were available,
such as for the wage-related measures, the data were collected under
JTPA, a program whose goals were different from those of WIA.[Footnote
20] In addition, some states believe that the performance levels did
not account for variations in economic conditions, such as the slow
growth in new or existing businesses that some areas have experienced.
Performance levels also did not account for the many economically
disadvantaged or hard-to-serve individuals seeking services in some
local areas.
States Report that Performance Levels May Determine Who Receives WIA-
Funded Services:
Many states reported that the need to meet performance levels may be
the driving factor in deciding who receives WIA-funded services at the
local level. All the states we visited told us that local areas are not
registering many WIA participants, largely attributing the low number
of WIA participants to concerns by local staff about meeting
performance levels.
Local staff are reluctant to provide WIA-funded services to job seekers
who may be less likely to get and keep a job. One state official
described how local areas were carefully screening potential
participants and holding meetings to decide whether to register them.
As a result, individuals who are eligible for and may benefit from WIA-
funded services may not be receiving services that are tracked under
WIA.
Performance levels for the measures that track earnings change for
adults and earnings replacement for dislocated workers may be
especially problematic. Several state officials reported that local
staff were reluctant to register already employed adults or dislocated
workers. Officials in one state reported that some local areas had not
yet registered any dislocated workers. State and local officials
explained that it would be hard to increase the earnings of adults who
are already employed or replace the wages of dislocated workers, who
are often laid off from high-paying, low-skilled jobs or from jobs that
required skills that are now obsolete. In addition, for dislocated
workers, employers may provide severance pay or workers might work
overtime prior to a plant closure, increasing these workers‘ earnings
before they are dislocated. As a result, many dislocated workers who
come to the one-stop center have earned high wages just prior to being
dislocated, making it hard to replace”let alone increase” their
earnings. If high wages are earned before dislocation and lower wages
are earned after job placement through WIA, the wage change will be
negative, depressing the wage replacement level. As a result, a local
area may not meet its performance level for this measure, discouraging
service to those who may need it.
A hypothetical example involving two workers dislocated at the same
time illustrates this point (see table 3). One worker is a sales clerk
with limited skills earning $25,000, the other a long-time factory
worker with obsolete skills earning $60,000. Both are laid off from
work and go to their local one-stop center seeking job placement
assistance. The clerk is placed in a new job as a receptionist paying
$25,000. By calculating his wage replacement from his salary as a
clerk, the one-stop can claim a wage replacement rate of 100 percent.
The factory worker eventually gets a job as a security guard earning
$30,000, netting a wage replacement rate of 50 percent. As this example
shows, a one-stop center can meet its performance levels more easily by
serving the clerk than by serving the factory worker even though both
job seekers may need the one-stop system‘s resources to find a job or
enhance their skills.
Table 3: Comparing Wage Replacement Rates for Two Job Seekers:
Scenario 1: Sales Clerk;
Sales clerk earning $25,000;
* Gets laid off;
* Cannot find a job;
* Comes to one-stop center for job placement assistance placement
assistance;
* The center gets him placed as a receptionist making $25,000;
Wage replacement rate: $25,000/$25,000 = 100%.
Scenario 2: Factory Worker;
Factory worker earning $60,000 after 20 years of employment;
* Gets laid off;
* Cannot find a job;
* Comes to one-stop center for job placement assistance placement
assistance;
• The center gets him placed as a security guard earning $30,000;
Wage replacement rate: $30,000/$60,000 = 50%.
Note: Example uses annual full-time earnings. When replacement rates
are actually calculated using UI wage records, they will be based on
earnings reported on a quarterly basis.
[End of table]
Some states and Labor are making efforts to address this disincentive
to serve certain job seekers. Indiana instituted a policy allowing
local areas to adjust their dislocated worker wage replacement rate in
light of the significant dislocations they are facing.[Footnote 21]
Texas uses a regression model to establish local performance levels
that adjust for differences in factors, such as economic conditions and
the characteristics of individuals served. Without this policy, said a
Texas official, WIA programs would have registered fewer workers.
Similarly, Michigan substantially reduced the penalties to local areas
for failing to meet performance levels and found that the number of
registered participants increased as a result of instituting less
threatening sanctions. WIA requires that states be allowed to
renegotiate their performance levels based on unanticipated
circumstances. Labor is currently developing criteria that states can
use to renegotiate their performance levels based on unanticipated
circumstances, such as changes in economic conditions due to plant
closings or shifts in unemployment for the current and future years.
The guidance is expected to be released soon.
Performance Measures May Not Accurately Assess Performance of the Three
WIA-Funded Programs:
Even when fully implemented, WIA performance measures may still not
provide an accurate picture of performance for the three WIA-funded
programs largely because data are neither comparable across states nor
timely. State and local officials generally supported many of the
performance measures as relevant indicators of the success of an
employment and training program. However, the performance data
collected and reported by states and localities are not comparable
largely because of the lack of clear guidance on when to collect and
report performance data and what constitutes a credential”the
attainment of a certified skill or degree. In addition, while UI wage
records are one of the best available sources of employment and
earnings data, limitations in the data may hinder the ability of states
and local areas to meet their performance levels and use the measures
for short-term program management.
Performance Measures Include Relevant Indicators, but Lack of Clear
Guidance Affects Accuracy and Comparability of Performance Data:
State and local officials in the states we visited generally support
many of the performance measures as relevant indicators of the success
of an employment and training program. For example, several officials
cited the wage-related measures, such as job placement, retention, and
earnings change, as important indicators of a successful employment and
training program. The measures are also generally consistent with the
goal of WIA to help individuals get and keep jobs and increase their
wages and skills. In addition, the states noted that the measures
provide a good basis for long-term evaluation.
However, the performance data collected and reported by states and
localities are not comparable”a critical component in creating a level
playing field from which states‘ relative performance can be evaluated.
While there are various reasons that performance data are not
comparable, one of the chief reasons is the lack of clear guidance for
collecting and reporting performance data on participants. Labor has
provided detailed written guidance to states on who should be
registered under WIA and when this registration should occur, but the
guidance is open to interpretation in some areas.[Footnote 22] The lack
of a uniform understanding of when registration occurs and thus who
should be counted toward the measures raises questions about both the
accuracy and comparability of states‘ performance data. For example,
the guidance tells states to register adults and dislocated workers who
need significant staff assistance designed to help with job seeking or
acquiring occupational skills, but the state can decide what
constitutes significant staff assistance.[Footnote 23] The guidance
provides examples of when to register job seekers, but it sometimes
requires staff to make subtle and subjective distinctions. For example,
those who receive initial assessment of skill levels and the need for
supportive services are not to be registered; those requiring
comprehensive assessment or staff-assisted job search and placement
assistance, including career counseling, are to be registered. In
another example involving the classification of workshops, job seekers
who participate are to be registered in some cases, but not in others.
Labor has allowed states and local areas flexibility in implementing
the registration policy, and we found that local areas differed on when
they registered WIA job seekers. In one local area we visited, the one-
stop center registers most job seekers who come into the center, even
if staff assistance is minimal. At this center, a general orientation
is sufficient for the job seeker to be registered under WIA. In
contrast, another center in the same state registers only those job
seekers who require significant staff assistance and are likely to
benefit from intensive services. Similar disparities occurred in other
states we visited. Labor has said there is little consistency across
states in registering participants and has convened a work group to
develop additional guidance on registration, but as yet, the issues
remain unresolved.
The lack of a definition for the credential measure is also leading to
performance data that are not accurate or comparable across states.
Labor allows the states and local areas to determine what constitutes a
credential and to develop a statewide list of approved credentials with
input from employers. Because states and, in many cases, local areas
must define what constitutes a credential, what is currently counted as
a credential differs within and across states. Some states may strictly
define credentials to include only diplomas from accredited
institutions or use only formal training completion criteria as defined
by education partners. Other states may expand their criteria to count
a broad variety of credentials, such as job readiness, on-the-job
experience, and completion of workshops. Labor officials note that
states‘ performance levels for the credential measure are negotiated to
take state and local definitions into account, and the measure is
intended to help local employers gauge the readiness and skill level of
job seekers. Nevertheless, given the broad range of definitions states
and localities employ, the outcomes on the credential measure may be of
limited value, even within a single state.
Challenges in Using UI Wage Record Data Also May Affect the Accuracy
and Comparability of Performance Data:
UI wage records are one of the best available data sources for tracking
the employment and earnings of individuals”a significant improvement
over the less objective self-reporting methods of JTPA”but the
limitations of the database pose challenges that need to be addressed.
These challenges, if unresolved, may hinder states‘ ability to meet
their performance levels. As we have reported in prior work,[Footnote
24] one such limitation is that UI wage records, while covering about
94 percent of workers, exclude certain employment categories, such as
self-employed persons, most independent contractors, military
personnel, federal government workers, and postal workers. States,
therefore, must develop alternative methods to track WIA participants
who are employed in these uncovered occupations. Pennsylvania, for
example, developed a partnership with other states in its region to
share the cost of purchasing the rights to federal civil service and
military personnel data. And Florida has developed agreements with the
Department of Defense, the Office of Personnel Management, and the U.S.
Postal Service to access employment and wage information on an annual
basis. Our survey data indicate that 33 states are using additional or
supplemental data to compensate for uncovered occupations, with only 27
of those using the supplemental data to count towards their performance
levels. Thus, at least 23 states have not used additional data to help
them meet their performance levels.
Another limitation is that state UI databases include only wage record
information on job seekers who get jobs within their state; they do not
track job seekers who find jobs in other states. States cannot readily
access UI wage records from other states to track outcomes under WIA,
making it difficult to track individuals who receive services in one
state but get a job in another. Over one-third of all of the states we
surveyed reported that an estimated 16 to 30 percent of cases are not
being picked up by their state‘s UI wage record system. To fill in
these gaps, seven states have agreements with other states”often those
that share a common border”to exchange UI information. Indiana, for
example, established an agreement with Illinois to trade data. If data
are missing on particular participants, Illinois sends the cases to
Indiana to see if the Indiana UI wage records have information on the
job seeker. The value of these agreements, however, may be limited
because job seekers may find work in a state that does not have an
agreement with the one in which they received services.
Another way to obtain UI data on workers who are employed out of state
is through WRIS, a clearinghouse that makes UI wage records available
to states seeking employment and wage information on their WIA
participants. This information can provide outcome data on WIA
participants to help states meet their performance levels. While WRIS
was available for states to use by July 2001, only 7 out of the 50
states are currently able and ready to participate, with 8 others in
various stages of completing the requirements for participation.
[Footnote 25] Although many states have shown an interest in a system
such as WRIS, many are reluctant to participate because Labor, while
agreeing to cover all the costs of operating WRIS for its first year,
has not yet agreed to pay for subsequent years. The estimated total
cost of operating WRIS is $2 million annually, but states have not been
given a definitive answer about how much it would cost them to
participate after this first year if Labor does not continue funding.
[Footnote 26] Because of this uncertainty regarding future costs,
states are hesitant to commit to participation in WRIS. If not all
states participate, the value of WRIS will be diminished”even for
participating states”because no data will be available from
nonparticipating states‘ UI wage records.
Lack of Timely Data Limits Use of Performance Measures for Short-Term
Program Management:
The lack of timely data, due to the time lag in obtaining UI wage
records, makes it difficult for state and local officials to use the
performance measures for short-term program management”because, for the
wage-related measures, current available data on the measures will
reflect performance from the previous program year. While UI wage
records are the best available data source for documenting employment,
the data collection and reporting process is slow and time-consuming.
Data are generally collected from employers only once every quarter,
and employers have 30 days after the quarter ends to report the data.
In many states, employers”especially small businesses”are allowed to
submit data in paper format, which then must be converted to electronic
media. After data entry, information must be checked for errors and
corrected. All of these steps take time. As a result, WIA program
administrators are unable to get a timely picture of program
performance. For example, we asked states in our survey how quickly job
placement outcome data would be available to them from UI wage records.
On the basis of our survey, we found that for 30 states, the earliest
time period that job placement data would be available is 6 months or
more after an individual entered employment, with 15 states reporting
that it may take 9 months or longer. Similarly, for the employment
retention measure, over half of states report that obtaining this
information could take a year or longer. (See figure 3.)
Figure 3: Number of Months it Takes to Receive UI Data for Job
Placement:
[Refer to PDF for image]
This figure is a vertical bar graph depicting the following data:
Shortest time period to receive UI job placement data:
3 months: 4 states;
4 months: 10 states;
5 months: 6 states;
6 months: 19 states;
7 months: 6 states;
8 months: 3 states;
9 months: 1 state;
11 months: 1 state.
Longest time period to receive UI job placement data:
4 months: 1 state;
5 months: 7 states;
6 months: 3 states;
7 months: 9 states;
8 months: 15 states;
9 months: 6 states;
10 months: 2 states;
11 months: 2 states;
12 months: 1 state;
13 months: 2 states;
14+ months: 2 states.
Note: Our survey asked two questions regarding UI time frames. One
question asked for the earliest date states would expect to receive UI
data; the other asked for the latest date.
Source: GAO survey.
[End of figure]
The time delay in receiving UI wage record data makes it difficult for
state and local officials to use the performance measures to gauge the
effectiveness of their services. States report that not being able to
get performance results in the same program year is a problem: it makes
it difficult to manage programs and improve one-stop services. Labor
reports that the performance measures are not intended to be a
management tool. State and local officials, therefore, must develop
alternative methods if they want to assess the quality of their
services so they can identify problems and improve programs in a timely
way. Labor has encouraged these efforts, but, while some local areas
are finding ways to collect data to help them manage their programs,
there is no cohesive effort at the federal level to share strategies
and promising approaches for the Adult and Dislocated Worker Programs.
[Footnote 27]
Existing Performance Measures Fail to Gauge Overall One-Stop
Performance:
Although there are performance measures for the three WIA-funded
programs and most of the programs required to provide services in the
one-stop, no measures exist to assess how well the overall one-stop
systems are working. The success of the one-stop system as a whole is
not captured by the program measures of individual one-stop partners.
Furthermore, combining the performance measures from mandatory programs
does not provide a comprehensive picture of one-stop performance. Even
when measures appear the same, comparing them is difficult because of
differences in definitions and calculations. Beyond failing to provide
a complete evaluation of one-stop performance, state officials reported
that the separate reporting requirements of the partner programs have
hampered coordination within their one-stop systems. While WIA did not
establish any comprehensive measures to assess the overall one-stop
system, it required that Labor take the lead in developing optional
measures to help states assess progress toward their workforce
investment goals. Labor has made limited progress on such performance
measures, and only a few states have developed their own overall
measures.
Existing Performance Measures Fail to Capture Important One-Stop
Features:
The existing performance measures for participating one-stop programs
fail to capture important one-stop features. First, it is difficult to
get an unduplicated count of job seekers using the one-stop. While an
individual may have multiple outcome measures for the services received
through each of the programs at a one-stop, there is no single outcome
measure for multiple services. In addition, separate reporting systems
for each of the programs make it difficult to disaggregate data and
track an individual‘s outcome for those receiving multiple services.
Second, other important aspects of one-stop performance are not
included within the existing measures. Customer satisfaction measures
used in support of WIA-funded programs and the Employment Service fail
to measure how job seekers and employers believe they are being served
by the one-stop system as a whole. Instead, these measures show
satisfaction with the individual programs. Employer satisfaction is
important under WIA because WIA created a more private-sector driven
system. Capturing customer satisfaction of the system as a whole would
reflect whether job seekers are successful at attaining the services
they need to get jobs and would assess whether employers are satisfied
with job applicants sent to them from different one-stop programs.
Finally, state and local officials expressed concern that a large
portion of one-stop participants are not included in performance
measures. Many job seekers use self-service and informational
activities but they are not tracked and counted in any program
measures. While staff time and resources are used to establish and
maintain self-service resource rooms and web sites, job seekers who use
only these services will not be included in any of the performance
measures. Without any information on individuals who use self-service,
it will be difficult for Labor to show how effectively one-stops are
being used.
Multiple Measures Cannot Be Used to Measure the One-Stop System‘s
Performance:
Performance measures for different programs often track similar
outcomes, as figure 4 shows. However, the measures cannot be combined
to obtain an overall view of one-stop performance. Although the same
terms are used in various performance measures, their definitions are
not identical. For example, while WIA older and younger youth programs
define youth as being between the ages of 14 and 21, the laws governing
Job Corps and HUD‘s Youthbuild define youth as being between the ages
of 16 and 24. Similarly, the definition of veterans is different for
the Employment Service and Veterans‘ Employment and Training Service
program. The differences in definitions mean that assessing the
outcomes for youth or veterans by combining the performance measures of
individual programs within the one-stop setting would be difficult.
Figure 4: Comparison of Performance Measures for 17 One-Stop Programs:
[Refer to PDF for image]
This figure illustrates the following information:
Department of Labor: WIA Adult[C];
Placement in employment: [Check];
Retention in employment[A]: [Check];
Earnings: [Check];
Skill improvement[B]: [Check];
Customer satisfaction: [Check];
Other: [Empty].
Department of Labor: WIA Dislocated Worker[C];
Placement in employment: [Check];
Retention in employment[A]: [Check];
Earnings: [Check];
Skill improvement[B]: [Check];
Customer satisfaction: [Check];
Other: [Empty].
Department of Labor: WIA Youth (age 14-21)[C];
Placement in employment: [Check];
Retention in employment[A]: [Check];
Earnings: [Check];
Skill improvement[B]: [Check];
Customer satisfaction: [Check];
Other: [Empty].
Department of Labor: Wagner-Peyser Employment Service;
Placement in employment: [Check];
Retention in employment[A]: [Check];
Earnings: [Empty];
Skill improvement[B]: [Empty];
Customer satisfaction: [Check];
Other: [Empty].
Department of Labor: Trade Adjustment Assistance Training Program[D];
Placement in employment: [Check];
Retention in employment[A]: [Check];
Earnings: [Check];
Skill improvement[B]: [Empty];
Customer satisfaction: [Empty];
Other: [Empty].
Department of Labor: Employment and training services to Veterans[E];
Placement in employment: [Check];
Retention in employment[A]: [Check];
Earnings: [Empty];
Skill improvement[B]: [Empty];
Customer satisfaction: [Empty];
Other: [Check].
Department of Labor: Unemployment Insurance[F];
Placement in employment: [Empty];
Retention in employment[A]: [Empty];
Earnings: [Empty];
Skill improvement[B]: [Empty];
Customer satisfaction: [Empty];
Other: [Empty].
Department of Labor: Job Corps;
Placement in employment: [Check];
Retention in employment[A]: [Check];
Earnings: [Check];
Skill improvement[B]: [Check];
Customer satisfaction: [Empty];
Other: [Empty].
Department of Labor: Welfare-to-Work Program;
Placement in employment: [Check];
Retention in employment[A]: [Check];
Earnings: [Check];
Skill improvement[B]: [Empty];
Customer satisfaction: [Empty];
Other: [Empty].
Department of Labor: Senior Community Service Employment Program
(SCSEP)[G];
Placement in employment: [Check];
Retention in employment[A]: [Check];
Earnings: [Empty];
Skill improvement[B]: [Empty];
Customer satisfaction: [Check];
Other: [Check].
Department of Labor: Migrant and Seasonal Farmworker Employment and
Training Program[H];
Placement in employment: [Check];
Retention in employment[A]: [Check];
Earnings: [Check];
Skill improvement[B]: [Check];
Customer satisfaction: [Check];
Other: [Empty].
Department of Labor: Native American Employment and Training
Programs[I];
Placement in employment: [Empty];
Retention in employment[A]: [Empty];
Earnings: [Empty];
Skill improvement[B]: [Empty];
Customer satisfaction: [Empty];
Other: [Check].
Department of Education: Vocational Rehabilitation Services Program;
Placement in employment: [Check];
Retention in employment[A]: [Empty];
Earnings: [Check];
Skill improvement[B]: [Empty];
Customer satisfaction: [Empty];
Other: [Check].
Department of Education: Adult Education and Literacy;
Placement in employment: [Check];
Retention in employment[A]: [Check];
Earnings: [Empty];
Skill improvement[B]: [Check];
Customer satisfaction: [Empty];
Other: [Check].
Department of Education: Carl D. Perkins Vocational Education
Program[J];
Placement in employment: [Check];
Retention in employment[A]: [Check];
Earnings: [Empty];
Skill improvement[B]: [Check];
Customer satisfaction: [Empty];
Other: [Empty].
HHS: Community Services Block Grant[K];
Placement in employment: [Check];
Retention in employment[A]: [Check];
Earnings: [Check];
Skill improvement[B]: [Empty];
Customer satisfaction: [Empty];
Other: [Check].
HUD: HUD-administered employment and training (Youthbuild);
Placement in employment: [Empty];
Retention in employment[A]: [Empty];
Earnings: [Empty];
Skill improvement[B]: [Check];
Customer satisfaction: [Empty];
Other: [Check].
[A] May include retention in education and/or training.
[B] May include the attainment of a credential or placement in
education or training.
[C] Two measures for customer satisfaction”one for employers, one for
job seekers”are used for all WIA-funded programs.
[D] This program reports by submitting individual records on program
participants.
[E] Based on proposed regulations.
[F] Unemployment Insurance does not have participant performance
measures.
[G] SCSEP is required to develop performance measures, which are
expected to be in place for program year 2003, according to a Labor
official.
[H] Grantees for this program are required to use the core indicators
of performance common to WIA adult and youth programs.
[I] Individual grantees propose performance in their individual grantee
plans, which Labor must approve.
[J] Measures must include either placement or retention and may include
both.
[K] For this program, states must report on at least one measure from
each of six national goals. The six national goals are (1) number
employed, (2) number maintaining 90-day retention, (3) number making
progress toward literacy/GED, (4) number gaining health care coverage
through employment, (5) increase in the availability and affordability
of essential services, and (6) increase in access to community services
and resources by low-income people. States may select from among
national measures or develop their own.
Source: Based on agency regulations and documents.
[End of figure]
Besides variations in definitions, there are also variations in how
measures are calculated in different programs. For example, while the
entered employment rate for WIA‘s adult program is defined as the
percentage of workers who get a job by the end of the first quarter
after exit, the entered employment rate for the Employment Service is
defined as the percentage of workers who get a job or changed employers
in the first or second quarter after registration. As a result,
performance data from these separate programs cannot be combined to
yield a single overall score to assess various performance outcomes of
the one-stop system. The Office of Management and Budget (OMB) convened
a work group representing federal WIA partners to look at common
definitions and measures across programs, which may issue guidance to
states and localities on WIA performance measures and federal
requirements.
Separate Measures May Impede Coordination:
Beyond failing to provide an overall picture of one-stop performance,
state officials reported that separate performance measures impede the
cooperation of one-stop partners.[Footnote 28] As a result, even though
WIA was meant to establish a more coordinated workforce development
system through the use of one-stops, over one-third of states surveyed
expressed concern that individual program performance measures may
impede this process. Some states even believed that separate measures
caused competition among programs. They said that if staff did not
understand that a participant could be counted in more than one
program, they might not direct them to other one-stop programs. For
example, one state reported in its written comments to our survey that
competition for participants and duplication of services due to lack of
coordination with other programs would continue as long as each program
is required to meet its own performance and participation levels.
Fourteen states volunteered in their written comments to our survey
that the federal government should work to coordinate performance
measures across programs or develop systemwide measures. In addition,
while states agree that systemwide measures are needed, they caution
against making any additional measures mandatory, since states are
still adapting to the existing measures.
Limited Progress on Developing Performance Measures for One-Stop
Systems:
Although WIA did not establish one-stop measures, it does require that
Labor develop additional optional measures to assist states in
assessing progress toward their workforce investment goals,[Footnote
29] which Labor has interpreted to include one-stop measures. Labor
began developing workforce development performance measures to capture
overall one-stop use after one-stop systems were piloted. Since the
passage of WIA, Labor has continued its efforts to develop systemwide
measures, but it has made limited progress. Labor convened a working
group in September 2001 to develop additional indicators of one-stop
performance. Partner representatives at this group included national
workforce-related organizations including the National Association of
State Workforce Agencies, National Governors‘ Association, National
Association of Counties, U.S. Conference of Mayors, and representatives
of states and regional boards. This group is working to develop a menu
of indicators that will help provide a comprehensive picture of WIA
system activity. Such measures may include capturing information on
self-service customers and the cost of services at the one-stop.
[Footnote 30] These measures, while optional, would help provide
information on overall one-stop use across the country if all states
report on at least some of the measures. Labor plans to have guidelines
for these optional indicators in place for use in program year 2002,
which begins on July 1, 2002. In order for states to be able to
implement them for the coming program year, Labor will need to provide
final guidance well before July 2002.
In addition to these national efforts, some states, on their own
initiative, have attempted to develop additional measures for one-stop
systems, but these efforts are not coordinated and do not allow for
nationwide assessment of the one-stop system. According to our survey,
eight states have created or are developing additional systemwide
measures, but of those, only three are reporting them to Labor.
Pennsylvania, for example, developed five measures specific to its one-
stop system‘s performance. These indicators, intended to measure the
overall effectiveness of the one-stop system, include median cycle time
to fill a job, and the percentage of employers and individuals using
services through the one-stop. Florida, on the other hand, has
developed ’tiers“ of measures that focus on the outcome of their
workforce development programs. In the first tier, state-generated
systemwide indicators measure many employment and training programs
together. The second tier clusters similar types of programs and
captures measures relevant to particular groups (e.g., continued
education status of youth in youth programs). The third tier captures
all the federally mandated measurements, as well as measures for the
other tiers, such as caseloads for specific programs. In this way,
Florida has attempted to measure the system overall as well as outcomes
for individual programs. Measures developed by other states include the
number of people using the resource rooms at one-stops and the increase
over time in the number of unemployed people getting a job. Despite
these states‘ efforts, the absence of nationally established systemwide
measures means that Labor cannot ensure nationwide comparability.
Conclusions:
WIA represents a fundamental shift in the way federally funded
employment and training services are provided to job seekers and in the
way WIA programs measure and monitor success. Despite obstacles, in
just over a year states have made good progress in implementing the new
requirements under WIA”developing new processes and designing new
systems. Labor, for its part, has been working to find ways to allow
states and localities greater flexibility to design their programs to
meet local needs and has been actively seeking opportunities for states
to have input into the process, particularly in the area of performance
measurement. But given the challenges states have faced in implementing
the new performance measurement system, more time is needed before the
measures can meaningfully gauge the success of the programs.
This new performance measurement system under WIA is a high-stakes
game”a state‘s future funding and, therefore, its ability to serve its
citizens may depend upon how well it performs compared to how well it
is expected to perform. It should be no surprise that states and
localities are designing their systems and processes in ways that will
enhance their ability to meet their performance levels. Because states
see the current performance levels as too high for the current economy,
states and localities may choose not to serve those job seekers who may
be helped by their services, but who may not help in achieving their
negotiated performance levels. Unless the performance levels can be
adjusted to truly reflect differences in economic conditions and the
population served, local areas will continue to have a disincentive to
serve some job seekers that could be helped.
WIA‘s requirement to use UI data to track outcomes is a step in the
right direction”it provides federal, state, and local government
entities with an objective means to evaluate program success. But it
brings challenges that need to be addressed, and states will need help
to do so. Establishing the means to routinely share data across state
lines through WRIS and developing ways to share promising approaches in
the use of supplemental data sources and in managing the assessment of
short-term program needs would go far in moderating these challenges.
Without this help and the cooperative efforts of states and localities
toward this end, developing a useful performance measurement system
will take longer and cost more.
In general, WIA‘s performance measurement system captures some useful
information, but it may not capture all the right information. The
measure to track credentials has limited value because it lacks a
standard definition for what‘s being measured. For other measures, the
lack of clear definitions for whom to track limits their usefulness in
drawing conclusions about program success at both the state and
national levels. Without clear definitions and processes, the measures
will not provide the Congress with a true picture of how well the
programs are performing. Furthermore, WIA performance measures gauge
only WIA-funded services; yet there is widespread agreement that
measures are needed to gauge the effectiveness of the entire one-stop
system. The system‘s narrow focus on program outcomes for a limited
number of participants misses a key requirement of WIA to support the
movement toward a coordinated system. In fact, the measures may foster
the opposite”a siloed approach that encourages competition among
programs and limits their cooperation. Without global one-stop
measures, the Congress will not be able to assess how well states and
localities are doing in meeting WIA‘s requirement to coordinate
services. The lack of such measures may, instead, send a signal to
states that service coordination is a minor goal.
Recommendations for Executive Action:
To give states and local areas more time to implement WIA performance
measures and establish baseline data needed to determine performance
levels, we recommend that the Department of Labor:
* delay the application of financial sanctions for at least 1 year or
until it is judged that states have their data systems sufficiently in
place to successfully track WIA outcomes.
To eliminate possible disincentives to serve some job seekers and
ensure that states and local areas will not be unduly penalized for
economic downturns, we recommend that the Department of Labor:
* expedite the release of guidance on revising negotiated performance
levels and allow states to immediately begin the process of re-
negotiation.
To ensure uniformity in data collection and reporting so that
performance results are more accurate and comparable across states, we
recommend that the Department of Labor:
* provide clearer guidance using objective criteria on who should or
should not be registered as a WIA participant for tracking purposes
and, once the guidance is released, work proactively with states to
implement it, and:
* issue guidance delineating a clear definition for what constitutes a
credential, and, once the guidance is released, ensure that states use
it to report on this indicator.
To help states address the challenges of using UI data to measure
outcomes, we recommend that the Department of Labor:
* continue to fully fund the Wage Record Interchange System in order to
facilitate the sharing of UI data across state lines;
* develop ways for states to share promising approaches in the use of
supplemental data sources in closing the data gaps for covered and
uncovered employment in UI; and;
* develop ways for states to share promising approaches that help
states address the UI timeliness issue, providing methods to help
states monitor and improve their programs in a timely manner.
To help states measure one-stop performance, we recommend that the
Department of Labor:
* ensure that the development of optional one-stop system measures is
completed in enough time for states to implement them at the beginning
of program year 2002.
Agency Comments:
We provided a draft of this report to Labor for its review and comment.
Labor‘s comments are in appendix II. We incorporated comments and
clarifications where appropriate.
Labor generally agreed with our findings and recommendations, noting
that they are consistent with information they have gathered from state
and local partners. In its comments, Labor expressed concern that
negotiated performance levels may be determining who receives WIA-
funded services, indicating that it will work with states and local
areas to address this issue. Labor also commented on our finding
regarding the lack of clear guidance on certain policies, stressing the
importance of state and local flexibility in determining specific
policies and practices to fit local needs. While state and local
flexibility is important, we continue to be concerned that the lack of
a uniform understanding of when registration occurs and what
constitutes a credential raises questions about both the accuracy and
the comparability of states‘ performance data. We are pleased to note
that Labor is in the process of reviewing this issue. Finally, Labor
cites its efforts to collaborate with states and local areas in
developing a performance accountability system and increasing
partnerships. We commend Labor for obtaining states‘ input and
participation in developing such a system.
We are sending copies of this report to the Secretary of Labor,
relevant congressional committees, and others who are interested.
Copies will also be made available to others upon request. The report
is also available on GAO‘s home page at [hyperlink,
http://www.gao.gov].
Please contact me on (202) 512-7215 if you or your staff have any
questions about this report. Other major contributors to this report
are listed in appendix III.
Signed by:
Sigurd R. Nilsen:
Director, Education, Workforce, and Income Security Issues:
[End of section]
Appendix I: WIA Performance Measures for Adults, Dislocated Workers,
and Older Youth:
The following table includes descriptions of the performance measures
reviewed in this report. It does not include the calculations required
to attain the final value of the performance measures.
Adults: Entered employment rate;
Definition: Of those who did not have a job when they registered for
WIA, the percentage of adults who got a job by the end of the 1st
quarter after exit. This measure excludes participants who are employed
at the time of registration.
Adults: Employment retention rate at 6 months;
Definition: Of those who had a job in the 1st quarter after exit, the
percentage of adults who have a job in the 3rd quarter after exit.
Adults: Average earnings change in 6 months;
Definition: Of those who had a job in the 1st quarter after exit, the
post-program earnings increases as compared with pre-program earnings.
Adults: Employment and credential rate;
Definition: Of those adults who received WIA training services, the
percentage who were employed in the 1st quarter after exit and received
a credential by the end of the 3rd quarter after exit.
Dislocated workers: Entered employment rate;
Definition: The percentage of dislocated workers who got a job by the
end of the 1st quarter after exit. This measure includes dislocated
workers who are employed at the time of registration.
Dislocated workers: Employment retention rate at 6 months;
Definition: Of those who had a job in the 1st quarter after exit, the
percentage of dislocated workers who have a job in the 3rd quarter
after exit.
Dislocated workers: Earnings replacement rate in 6 months;
Definition: Of those who had a job in the 1st quarter after exit, the
percentage of pre-program earnings being earned post-program. Since it
may be difficult to find dislocated workers jobs with equivalent or
better wages, this measure captures the percentage of earnings of the
new job in relation to the old.
Dislocated workers: Employment and credential rate;
Definition: Of those dislocated workers who received WIA training
services, the percentage who were employed in the 1st quarter after
exit and received a credential by the end of the 3rd quarter after
exit.
Older youth (age 19-21): Entered employment rate;
Definition: Of those who are not employed at registration and who are
not enrolled in post-secondary education or advanced training in the
1st quarter after exit, the percentage of older youth who have gotten a
job by the end of the 1st quarter after exit. This measure also
excludes youth that move on to post-secondary education or advanced
training and not employment.
Older youth (age 19-21): Employment retention rate at 6 months;
Definition: Of those who are employed in the 1st quarter after exit and
who are not enrolled in post-secondary education or advanced training
in the 3rd quarter after exit, the percentage of older youth that are
employed in the 3rd quarter after exit.
Older youth (age 19-21): Average earnings change in 6 months;
Definition: Of those who had a job in the 1st quarter after exit and
who are not enrolled in post-secondary education or advanced training,
the post-program earnings increases as compared with pre-program
earnings.
Older youth (age 19-21): Employment/education/training and credential
rate;
Definition: The percentage of older youth who are in employment, post-
secondary education, or advanced training in the 1st quarter after exit
and received a credential by the end of the 3rd quarter after exit.
Customer satisfaction[A]: Employer customer satisfaction;
Definition: The average of three statewide survey questions rated 1-10:
(1 being ’very dissatisfied“ and 10 being ’very satisfied“):
* was the employer satisfied with services;
* did the service meet the expectations of the customer;
* how well did the service compare to the ideal set of services.
Customer satisfaction[A]: Participant customer satisfaction;
Definition: The average of three statewide survey questions rated 1-10:
* was the participant satisfied with services;
* did the service meet the expectations of the customer;
* how well did the service compare to the ideal set of services.
[A] A statewide telephone survey of a sample of 500 is conducted for
all the WIA-funded programs.
Source: U.S. Department of Labor TEGL 7-99.
[End of table]
[End of section]
Appendix II: Comments From the Department of Labor:
U.S. Department of Labor:
"A Proud Member of America's Workforce Network"
Assistant Secretary for Employment and Training:
Washington, D.C. 20210:
January 16, 2002:
Mr. Sigurd R. Nilsen:
Director:
Education, Workforce and Income Security Issues:
U.S. General Accounting Office:
Washington, DC 20548:
Dear Mr. Nilsen:
Thank you for the opportunity to provide comments on the draft GAO
report, Workforce Investment Act: Improvements Needed in Performance
Measures to Provide More Accurate Picture of WIA's Effectiveness (GAO-
02-275).
In general, GAO's findings and recommendations are consistent with what
we have heard in discussions of the performance accountability system
both within the Department and with our state and local partners. The
Department of Labor's approach to performance accountability under
title I of the Workforce Investment Act (WIA) has been based on: (1)
the system change created by WIA; (2) state and local flexibility; and
(3) increased partnerships.
The System Change Created by WIA: The enactment of WIA brought dramatic
changes to many aspects of the nation's workforce system. Many of these
changes were in the area of service delivery. WIA streamlined
eligibility and service strategy procedures in order to allow
individuals to be served in an expedited and customer-focused manner.
The delivery of services to customers is the first priority, and
administrative processes such as reporting and paperwork become less
important than program performance. The discussion on pages 14 and 15
of the draft related to states reporting that performance levels may
determine who receives WIA-funded services causes us concern that not
all areas are taking advantage of the opportunities that WIA provides.
In 2002, ETA staff will work closely with states and local areas to
clarify any misconceptions in this crucial area and to stress that the
priority of the workforce investment system is to provide quality
services to customers.
With the focus on service delivery, performance accountability becomes
even more important. For example, WIA contains sets of core indicators
designed to measure the youth, adult, and dislocated worker programs,
as well as customer satisfaction indicators. While we agree that
measures which assess the performance of the One-Stop system as a whole
are needed, WIA does not contain comprehensive measures for the One-
Stop delivery system. As noted in your report, last September we began
to gather information and recommendations on system measures to support
state use of additional performance indicators in preparation for WIA
reauthorization discussions.
State and Local Flexibility: In implementing WIA, the Department has
developed policies and guidance designed to support the key WIA reform
principle of increased state and local flexibility. Accordingly, when
developing the regulations and performance accountability guidelines,
in many instances the Department set broad parameters or definitions,
leaving the process of establishing specific policies or definitions up
to the states and local areas. Two examples of this flexibility are
touched upon in the draft report: (1) the point of registration for
adults and dislocated workers, and (2) the definition of a credential
for performance accountability purposes. As you noted in your report,
we have provided guidance on establishing a point of registration and
on the states' authority to define credentials, but in both cases the
needs of the job seeker and employer customers at the state and local
level were given priority over a one-size-fits-all national
requirement. Based on state and local input, the Department is in the
process of reviewing these policy decisions. No decision regarding
revisions to these existing policies has been made at this time.
Increased Partnership: WIA also calls for increased partnerships
between all levels of the workforce system. Accordingly, the WIA title
I performance accountability system has been developed through an
unprecedented collaborative process. The Department has made great
efforts to ensure that the performance accountability system for title
I was designed in partnership with states and local areas. A series of
consultation papers were published for public comment in the Federal
Register beginning in March 1999. The March 1999 consultation paper
specifically asked for comment on several of the issues discussed in
the draft report, such as (1) the point at which adult and youth
registrants are counted for different performance measures; (2) the use
of wage records for performance measurement considering availability,
completeness, and accuracy; and (3) identifying possible unintended
effects resulting from definitions/policies around performance
measurements.
The public was also provided an opportunity to comment on the interim
final regulations, published in April 1999, which covered the point of
registration issue and the performance accountability system. The
Department hosted two rounds of town hall meetings to solicit comments
from the public on all issues related to the implementation of WIA,
including the area of performance. Proposed WIA title I reporting
requirements were published in the Federal Register for comment, and
consultation with workforce partners continued to ensure that all
comments and points of view had been considered.
We are impressed by the progress that our state and local partners have
made in implementing the performance accountability provisions of WIA
title I. However, due to the complexity of building a comprehensive and
seamless workforce system, we anticipate that the performance
infrastructure will continue to develop over the next several years,
and that lessons learned from states and local areas will be used to
improve the system as it evolves.
We have enclosed some additional specific comments on the GAO's
findings. We will provide a formal response to GAO's recommendations
for executive action when the report is final. In addition, we expect
that the issue of performance accountability will be a key issue
analyzed as part of the upcoming WIA reauthorization discussions.
Sincerely,
Signed by:
Emily Stover DeRocco:
Enclosure:
[End of section]
Appendix III: GAO Contacts and Staff Acknowledgments:
GAO Contacts:
Dianne Blank (202) 512-5654:
Ronni Schwartz (202) 512-7033:
Staff Acknowledgments:
Abbey Frank, Mikki Holmes, and Amanda Ahlstrand made significant
contributions to this report. In addition, James Wright assisted in the
study design and the national survey; Jessica Botsford and Richard
Burkard provided legal support; and Patrick DiBattista and Barbara
Alsip assisted in the message and report development.
[End of section]
Related GAO Products:
U.S. General Accounting Office. Workforce Investment Act: Better
Guidance Needed to Address Concerns Over New Requirements. [hyperlink,
http://www.gao.gov/products/GAO-02-72]. Washington, D.C.: 2001.
U.S. General Accounting Office. Veterans‘ Employment and Training
Service: Proposed Performance Measurement System Improved, But Further
Changes Needed. [hyperlink, http://www.gao.gov/products/GAO-01-580].
Washington, D.C.: 2001.
U.S. General Accounting Office. Multiple Employment and Training
Programs: Overlapping Programs Indicate Need for Closer Examination of
Structure. [hyperlink, http://www.gao.gov/products/GAO-01-71].
Washington, D.C.: 2000.
U.S. General Accounting Office. Workforce Investment Act:
Implementation Status and the Integration of TANF Services. [hyperlink,
http://www.gao.gov/products/GAO/T-HEHS-00-145]. Washington, D.C.: 2000.
[End of section]
Footnotes:
[1] In addition to the Department of Labor, which oversees the
implementation of WIA, programs funded through the Departments of
Education, Health and Human Services, and Housing and Urban Development
are required to provide services for certain of their programs through
this one-stop center system.
[2] While WIA reauthorizes or amends other existing programs, including
Adult Education and Literacy, Employment Service (Wagner-Peyser), and
Vocational Rehabilitation, the performance measures apply only to the
adult, dislocated worker, and youth programs under Title I, which, for
this report, we will refer to as ’WIA-funded programs.“
[3] States must also meet performance levels for programs authorized
under the Adult Education and Family Literacy Act and Carl D. Perkins
Vocational and Applied Technology Education Act (Perkins) to be
eligible for incentive grants.
[4] We also did not review measures for eligible training providers,
which were reviewed in our study, U.S. General Accounting Office,
Workforce Investment Act: Better Guidance Needed to Address Concerns
Over New Requirements, [hyperlink,
http://www.gao.gov/products/GAO-02-72] (Washington, D.C.: 2001).
[5] WIA‘s Youth program uses low income as an eligibility requirement.
[6] Authorized through fiscal year 2003, WIA‘s three new programs had a
budget authority of about $3.7 billion in fiscal year 2001.
[7] GPRA was intended to focus government decision making, management,
and accountability on the results and outcomes achieved by federal
programs.
[8] Guidance from Labor defines a credential as a nationally recognized
degree or certificate or a recognized state/locally defined credential.
[9] According to Labor‘s guidance, states must achieve a cumulative
score of 100 percent or more in each program area”adult, dislocated
worker, and youth”and for customer satisfaction and at least 80 percent
of the negotiated performance level for all 17 measures. Failing to
achieve the 80-percent level for any one performance measure
disqualifies a state from being considered for an incentive grant.
[10] According to Labor, while the agency has sole responsibility for
administering sanctions under WIA, awarding incentives is a joint
effort involving Labor and the Department of Education.
[11] In addition, three other categories of programs are required to
provide services through the one-stop center: Youth Opportunity Grants;
demonstration, pilot, multi-service, research, and multi-state
projects; and national emergency grants. Because they are of limited
scope, we did not include them in our total.
[12] WIA requires states to establish and operate management
information systems based on guidelines established by Labor, which are
designed to promote the efficient collection of information.
[13] U.S. Department of Labor Employment and Training Administration,
Training and Employment Guidance Letter No. 14-00 (Mar. 5, 2001).
[14] Calculations for the measures are found in U.S. Department of
Labor Employment and Training Administration, Training and Employment
Guidance Letter (TEGL) No. 7-99 (Mar. 3, 2000).
[15] Dislocated workers may include those who have obtained interim
employment after being displaced.
[16] States must wait two full quarters after exit before collecting
data because of delays in available wage records.
[17] Data for pre-program earnings may be collected at registration or
at any time until the fifth quarter after exit.
[18] As authorized by WIA, Labor requires states to submit quarterly
summary reports reflecting the state‘s performance levels and other
activities.
[19] The 23 states are a subset of the 53 states and territories that
report WIA data to Labor.
[20] While JTPA focused on providing training to the economically
disadvantaged, including the hardest-to-serve, WIA provides a broader
range of services to all individuals, regardless of their eligibility
for other services.
[21] The adjusted levels would apply only to the dislocated workers
from the company that has been affected, not to the entire local area.
[22] TEGL No. 7-99.
[23] All youth who receive WIA-funded services are required to be
registered.
[24] See U.S. General Accounting Office, Veterans‘ Employment and
Training Service: Proposed Performance Measurement System Improved, But
Further Changes Needed, GAO-01-580 (Washington, D.C.: 2001).
[25] To participate, states are required to be technically and
administratively ready. For example, states must be able to respond to
queries for UI wage data and establish internal security measures to
ensure the confidentiality of data.
[26] A national association is currently working a contractor to study
the costs associated with WRIS.
[27] Labor‘s Office of Youth Services has taken action to help youth
programs in this area, such as creating a set of short-term performance
measures.
[28] This issue was also raised in a study that found that the
continued emphasis on measuring performance in categorical programs
undermines the integration expected in WIA. Evelyn Ganzglass, Martin
Jensen, Neil Ridley, Martin Simon, and Chris Thompson. Transforming
State Workforce Development Systems: Case Studies of Five Leading
States (National Governors‘ Association, 2001).
[29] Section 136 (i)(1) requires Labor to work with workforce
development system partners to develop terms for a menu of additional
indicators of performance to help states assess their progress toward
their workforce investment goals. The purpose of developing these
additional measures is to ensure nationwide comparability. 30Labor is
currently seeking OMB approval to collect data on the costs and usage
of WIA and Wagner-Peyser funded services that do not require
registration. Labor issued a Federal Register Notice on January 16,
2002 seeking public comment on this proposal.
[End of section]
GAO‘s Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO‘s commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO‘s Web site [hyperlink,
http://www.gao.gov] contains abstracts and full text files of current
reports and testimony and an expanding archive of older products. The
Web site features a search engine to help you locate documents using
key words and phrases. You can print these documents in their entirety,
including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as ’Today‘s Reports,“ on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
[hyperlink, http://www.gao.gov] and select ’Subscribe to daily E-mail
alert for newly released products“ under the GAO Reports heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov:
(202) 512-4800:
U.S. General Accounting Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
