Employment and Training Administration
More Actions Needed to Improve Transparency and Accountability of Its Research Program Gao ID: GAO-11-285 March 15, 2011To help guide the nation's workforce development system, the Department of Labor's (Labor) Employment and Training Administration (ETA) conducts research in areas related to job training and employment. Building upon our earlier work, GAO examined the following: (1) To what extent do ETA's research priorities reflect key national employment and training issues and how useful were the studies funded under them? (2) What steps has ETA taken to improve its research program? (3) How has ETA improved the availability of its research since our last review in January 2010? To answer these questions, GAO reviewed ETA's research reports disseminated between January 2008 and March 2010 costing $1 million or more, as well as ongoing studies costing $2 million or more. GAO also convened a virtual expert panel, interviewed Labor and ETA officials, and reviewed relevant documents.
ETA's 2007 to 2012 research plan generally addressed key employment and training issues, but some studies were limited in their usefulness. Most experts on our panel reported that the areas in ETA's plan reflected key national employment and training issues at least to a moderate extent. ETA invested most of its research and evaluation resources in the areas of Unemployment Insurance and increasing labor market participation of underutilized groups. Of the $96 million that ETA invested in the 58 research reports we reviewed, more than half--or about $56 million--funded studies in these two areas. The methodological approaches and statistical procedures researchers used in the studies we reviewed were generally consistent with the questions posed, but the studies were not always useful for informing policy and practice. For example, in one study, shortcomings in the data collection phase limited the strength of the findings. Experts suggested that ETA include more varied and rigorous methodologies in its future research projects. They also reported that future research should address additional areas, including a focus on employment and training approaches that work and for whom. Labor and ETA have taken steps to improve the way research is conducted, but additional actions are needed. In acknowledging the need for more rigorous evaluations to inform its policies, Labor recently established the Chief Evaluation Office to oversee departmentwide research and evaluation efforts. In addition, ETA made changes to some of its research practices. For example, ETA has begun involving outside experts in developing its research plan. Despite these improvements, ETA's process lacks critical elements needed to ensure that current improvements become part of its routine practices. For example, ETA's process lacks a formal provision to consult with the newly established Chief Evaluation Officer at important points in the research process. Moreover, ETA's current research practices fall short of ensuring research transparency and accountability--essential elements of a sound research program. For example, its research and evaluation center lacks safeguards to protect it from undue outside influence. ETA has recently begun efforts to increase the rigor of its research designs, but has faced design and implementation challenges. For example, some of ETA's ongoing research studies face challenges in recruiting large enough sample sizes to meet the studies' objectives. ETA has improved the availability of its research findings, but it lacks a plan for assessing the usability of its Web-based search page--the primary tool for making ETA's research publicly available. ETA recently improved the timeliness with which it disseminates its research reports, decreasing the average number of days to release its reports to the public from 804 days in 2008 to 76 days in 2009. ETA has taken steps to update its online, Web-based search page. However, the agency has not developed a formal plan for assessing the overall effectiveness of its Web-based search page, including user satisfaction. In addition to its research database, ETA uses a variety of other methods to disseminate its research, including providing its research reports at conferences and internal briefings. Experts suggested that ETA consider other effective dissemination methods, such as publishing a one-page summary of research findings. GAO recommends that ETA formally include the Chief Evaluation Officer in its research process, create a mechanism to enhance the transparency and accountability of its research program, and develop a plan to ensure that research reports are accessible through its Web-based search page. Labor agreed with our recommendations and noted its ongoing efforts to improve its research program. While these efforts are important, GAO stresses the need for additional actions to fully address the recommendations.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: George A. Scott Team: Government Accountability Office: Education, Workforce, and Income Security Phone: (202) 512-5932GAO-11-285, Employment and Training Administration: More Actions Needed to Improve Transparency and Accountability of Its Research Program
This is the accessible text file for GAO report number GAO-11-285
entitled 'Employment And Training Administration: More Actions Needed
to Improve Transparency and Accountability of Its Research Program'
which was released on April 14, 2011.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
United States Government Accountability Office:
GAO:
Report to Congressional Committees:
March 2011:
Employment And Training Administration:
More Actions Needed to Improve Transparency and Accountability of Its
Research Program:
GAO-11-285:
GAO Highlights:
Highlights of GAO-11-285, a report to congressional committees.
Why GAO Did This Study:
To help guide the nation‘s workforce development system, the
Department of Labor‘s (Labor) Employment and Training Administration
(ETA) conducts research in areas related to job training and
employment. Building upon our earlier work, GAO examined the
following: (1) To what extent do ETA‘s research priorities reflect key
national employment and training issues and how useful were the
studies funded under them? (2) What steps has ETA taken to improve its
research program? (3) How has ETA improved the availability of its
research since our last review in January 2010? To answer these
questions, GAO reviewed ETA‘s research reports disseminated between
January 2008 and March 2010 costing $1 million or more, as well as
ongoing studies costing $2 million or more. GAO also convened a
virtual expert panel, interviewed Labor and ETA officials, and
reviewed relevant documents.
What GAO Found:
ETA‘s 2007 to 2012 research plan generally addressed key employment
and training issues, but some studies were limited in their
usefulness. Most experts on our panel reported that the areas in ETA‘s
plan reflected key national employment and training issues at least to
a moderate extent. ETA invested most of its research and evaluation
resources in the areas of Unemployment Insurance and increasing labor
market participation of underutilized groups. Of the $96 million that
ETA invested in the 58 research reports we reviewed, more than half”or
about $56 million”funded studies in these two areas. The
methodological approaches and statistical procedures researchers used
in the studies we reviewed were generally consistent with the
questions posed, but the studies were not always useful for informing
policy and practice. For example, in one study, shortcomings in the
data collection phase limited the strength of the findings. Experts
suggested that ETA include more varied and rigorous methodologies in
its future research projects. They also reported that future research
should address additional areas, including a focus on employment and
training approaches that work and for whom.
Labor and ETA have taken steps to improve the way research is
conducted, but additional actions are needed. In acknowledging the
need for more rigorous evaluations to inform its policies, Labor
recently established the Chief Evaluation Office to oversee
departmentwide research and evaluation efforts. In addition, ETA made
changes to some of its research practices. For example, ETA has begun
involving outside experts in developing its research plan. Despite
these improvements, ETA‘s process lacks critical elements needed to
ensure that current improvements become part of its routine practices.
For example, ETA‘s process lacks a formal provision to consult with
the newly established Chief Evaluation Officer at important points in
the research process. Moreover, ETA‘s current research practices fall
short of ensuring research transparency and accountability”essential
elements of a sound research program. For example, its research and
evaluation center lacks safeguards to protect it from undue outside
influence. ETA has recently begun efforts to increase the rigor of its
research designs, but has faced design and implementation challenges.
For example, some of ETA‘s ongoing research studies face challenges in
recruiting large enough sample sizes to meet the studies‘ objectives.
ETA has improved the availability of its research findings, but it
lacks a plan for assessing the usability of its Web-based search page”
the primary tool for making ETA‘s research publicly available. ETA
recently improved the timeliness with which it disseminates its
research reports, decreasing the average number of days to release its
reports to the public from 804 days in 2008 to 76 days in 2009. ETA
has taken steps to update its online, Web-based search page. However,
the agency has not developed a formal plan for assessing the overall
effectiveness of its Web-based search page, including user
satisfaction. In addition to its research database, ETA uses a variety
of other methods to disseminate its research, including providing its
research reports at conferences and internal briefings. Experts
suggested that ETA consider other effective dissemination methods,
such as publishing a one-page summary of research findings.
What GAO Recommends:
GAO recommends that ETA formally include the Chief Evaluation Officer
in its research process, create a mechanism to enhance the
transparency and accountability of its research program, and develop a
plan to ensure that research reports are accessible through its Web-
based search page. Labor agreed with our recommendations and noted its
ongoing efforts to improve its research program. While these efforts
are important, GAO stresses the need for additional actions to fully
address the recommendations.
View [hyperlink, http://www.gao.gov/products/GAO-11-285] or key
components. For more information, contact George Scott at (202) 512-
7215 or scottg@gao.gov.
[End of section]
Contents:
Letter:
Background:
ETA's Research Areas Generally Reflect Key Issues, but Some Studies
Are of Limited Usefulness:
ETA Has Taken Steps to Improve Its Research Program, but Additional
Actions Are Needed:
ETA Has Improved the Availability of Its Research but There Are
Opportunities to Improve Its Search Page and Dissemination Methods:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Status of Prior GAO Recommendations to the Department of
Labor, as of January 2011:
Appendix II: Scope and Methodology:
Appendix III: The Panel's Ratings of Key Employment and Training
Issues, Populations, and Programs That ETA Should Address in Its
Future Research:
Appendix IV: Characteristics of Research Studies Disseminated between
January 2009 and March 2010 That Cost $1 Million or More:
Appendix V: Delphi Phase I and Phase II Questionnaires:
Appendix VI: Experts Who Agreed to Participate in GAO's Delphi Panel:
Appendix VII: Comments from the Department of Labor:
Appendix VIII: GAO Contact and Staff Acknowledgments:
GAO Related Products:
Tables:
Table 1: Research and Evaluation Terms and Definitions:
Table 2: ETA's Research Areas, with Examples of Topics Covered in the
2007 to 2012 Research Plan:
Table 3: Estimated Funding by Research Area for Research Studies
Disseminated between January 2008 and March 2010:
Table 4: Characteristics of Three Ongoing Studies:
Table 5: Descriptive Statistics on Key Employment and Training Issues
ETA Should Address in Its Future Research:
Table 6: Descriptive Statistics on Key Populations ETA Should Address
in Its Future Research:
Table 7: Descriptive Statistics on Key Employment and Training
Programs ETA Should Address in Its Future Research:
Figures:
Figure 1: ETA's Combined Evaluations and Pilots, Demonstrations, and
Research Budgets from Fiscal Years 2007 to 2010:
Figure 2: Key Phases of the Research Process:
Figure 3: Issue Areas Ranked among the Top Three by Experts for ETA to
Address in Future Research:
Figure 4: Unemployment Issues Frequently Cited by Experts as Important
for ETA to Address in Future Research:
Figure 5: Expert Panel Views on the Extent to Which ETA's Research Has
Informed Policy and Practice:
Figure 6: Methodological Approaches Frequently Cited by Experts as
Very Important for ETA to Address in Future Research:
Figure 7: ETA's 8-Step Research Process:
Figure 8: Experts' Opinion on the Effectiveness of Various Research
Dissemination Methods:
Abbreviations:
ETA: Employment and Training Administration:
Project GATE II: Project Growing America Through Entrepreneurship II:
Labor: Department of Labor:
OMB: Office of Management and Budget:
OPDR: Office of Policy Development and Research:
TEN: Training and Employment Notice:
UI: Unemployment Insurance:
WIA:Workforce Investment Act of 1998:
WIRED:Workforce Innovation in Regional Economic Development:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
March 15, 2011:
The Honorable Tom Harkin:
Chairman:
Subcommittee on Labor, Health and Human Services, Education, and
Related Agencies:
Committee on Appropriations:
United States Senate:
The Honorable Michael B. Enzi:
Ranking Member:
Committee on Health, Education, Labor, and Pensions:
United States Senate:
The Honorable Patty Murray:
Chairman:
The Honorable Johnny Isakson:
Ranking Member:
Subcommittee on Employment and Workplace Safety:
Committee on Health, Education, Labor, and Pensions:
United States Senate:
Across the country, the workforce development systems' one-stop
centers serve as the key access point for services that are crucial in
today's economy--services that include Unemployment Insurance (UI)
benefits, job training, and employment assistance. The Department of
Labor's (Labor) Employment and Training Administration (ETA) is
responsible for guiding the nearly $13 billion public workforce
development system. Its mission is to help make the U.S. labor market
function more efficiently by developing policies that lead to high-
quality job training, employment, labor market information, and income
maintenance services. To help shape its policies, ETA conducts
evaluations and research studies on a range of employment-related
topics. Congress appropriated about $103 million to ETA's research and
evaluation budget line items for 2010.[Footnote 1]
Over the last decade, however, ETA's research and evaluation program
has fallen short in its efforts to conduct research that can help
answer urgent workforce policy questions. In 2008 and again in 2009,
we faulted ETA for failing to conduct research and evaluations that
would lead to an understanding of what works and what
doesn't.[Footnote 2] For example, in January 2010, we reported on
shortcomings in ETA's research structure and processes.[Footnote 3] We
found that ETA's research and evaluation center, the Office of Policy
Development and Research (OPDR),
* lacked independent authority to make key decisions about its
research;
* maintained processes that were unclear and that lacked transparency
and accountability;
* lacked a standard process for ensuring stakeholder involvement or
other strategies to ensure that research addressed key national
priorities; and:
* had been slow to distribute its research findings and slow to
respond to its statutory mandate to evaluate the Workforce Investment
Act of 1998 (WIA).
Based on these findings, we made several recommendations to Labor to
improve ETA's research program. (For information on the status of
those recommendations, see appendix I.)
As ETA's leadership moves forward to help the nation meet its current
employment challenges, questions remain about how well ETA's research
has prepared the workforce development system for the challenges of
today. Against this backdrop, you asked us to build upon our January
2010 review and further examine ETA's research program. Specifically,
we answered the following questions:
1. To what extent do ETA's research priorities reflect key national
employment and training issues and how useful were the studies funded
under them?
2. What steps has ETA taken to improve its research program?
3. How has ETA improved, if at all, the availability of its research
since our last review in January 2010 and what other steps could ETA
take to further ensure its research findings are readily available?
To address our objectives, we reviewed the 58 research and evaluation
reports that ETA disseminated between January 2008 and March 2010 and
assessed the methodological soundness of the 11 completed studies that
cost $1 million or more. In addition, we reviewed the 10 ongoing
studies costing $2 million or more to determine if research practices
or the soundness of research designs had changed over time. In
addition, we convened a virtual (Delphi) expert panel of academics,
researchers, and consultants to obtain their opinions on ETA's
research priorities and dissemination methods. To learn how ETA
determines what research to conduct, we interviewed Labor officials
and reviewed relevant agency and budget documents. In addition, we
conducted site visits to two local workforce agencies in Pennsylvania
and Virginia that are implementing ETA's ongoing research studies to
obtain information about challenges and issues associated with
participating in studies. To evaluate the availability of ETA's
research, we analyzed dissemination time frames for all publications
released between January 2008 and March 2010 and we tested the ability
of ETA's research database to support searches generally available to
users of research databases. (See appendix II for more details on our
objectives, scope, and methodology.)
We conducted this performance audit from March 2009 through March 2011
in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives.
Background:
WIA sets forth various requirements for the Secretary of Labor
relating to research and evaluation of federally funded employment-
related programs and activities. The law calls upon the Secretary of
Labor to publish in the Federal Register every 2 years a plan that
describes its pilot, demonstration, and research priorities for the
next 5 years regarding employment and training. Specifically, WIA
requires the Secretary to:
* develop the research plan after consulting with states, localities,
and other interested parties;
* send the plan to the appropriate committees of Congress; and:
* take into account such factors as the likelihood that the results of
the projects will be useful to policymakers and stakeholders in
addressing employment and training problems.[Footnote 4]
Within ETA, OPDR's Division of Research and Evaluation plans,
conducts, and disseminates employment and training-related research
and evaluations. Nearly all of the agency's research and evaluation
studies are conducted under contract; these contractors represent a
range of research organizations and academic institutions.
Furthermore, OPDR plans and conducts its research and evaluation
activities in consultation with ETA's program offices, such as the
Office of Workforce Investment and the Office of Trade Adjustment
Assistance.
ETA's research and evaluation funding is divided into two separate
budget line items:
Pilots, demonstrations, and research. Efforts in this category are
focused on developing and testing new ways to approach problems and to
deliver services. Under WIA, pilots and demonstrations shall be
carried out "for the purpose of developing and implementing techniques
and approaches, and demonstrating the effectiveness of specialized
methods, in addressing employment and training needs." WIA also states
that the Secretary shall "carry out research projects that will
contribute to the solution of employment and training problems in the
United States."
Evaluations. Efforts in this category are focused on continuing
evaluations of certain programs and activities carried out under WIA.
These evaluations must address:
* the effectiveness of these programs and activities carried out under
WIA in relation to their cost;
* the effectiveness of the performance measures relating to these
programs and activities;
* the effectiveness of the structure and mechanisms for delivery of
services through these programs and activities;
* the impact of the programs and activities on the community and
participants involved, and on related programs and activities;
* the extent to which such programs and activities meet the needs of
various demographic groups; and:
* such other factors as may be appropriate.
In program year 2010, ETA's combined budget appropriation for
conducting evaluations and pilots, demonstrations, and research was
about $103 million--or nearly $34 million above what the agency
requested.[Footnote 5] (See figure 1.) About $84 million of the 2010
funds were designated by the Congress for specific projects, including
$30 million for Transitional Jobs activities for ex-offenders, and
another $5.5 million for competitive grants addressing the employment
and training needs of young parents. According to agency documents, in
2008 and 2009, the Congress similarly increased ETA's requested budget
for pilots, demonstrations, and research, at the same time
specifically designating how the majority of those funds would be
used, including $4.9 million in 2008 and $5 million in 2009 for the
young parents' demonstration.
Figure 1: ETA's Combined Evaluations and Pilots, Demonstrations, and
Research Budgets from 2007 to 2010:
[Refer to PDF for image: stacked vertical bar graph]
Year: 2007;
Funds requested: $22.6 million;
Funds designated for congressionally mandated projects: $0;
Funds not designated for congressionally mandated projects: $19.6
million;
ETA's combined budgets: $19.6 million.
Year: 2008;
Funds requested: $20.0 million;
Funds designated for congressionally mandated projects: $48.5 million;
Funds not designated for congressionally mandated projects: $4.8
million;
ETA's combined budgets: $53.3 million.
Year: 2009;
Funds requested: $25.0 million;
Funds designated for congressionally mandated projects: $46.3 million;
Funds not designated for congressionally mandated projects: $9.4
million;
ETA's combined budgets: $55.7 million.
Year: 2010;
Funds requested: $69.1 million;
Funds designated for congressionally mandated projects: $84.4 million;
Funds not designated for congressionally mandated projects: $18.7
million;
ETA's combined budgets: $103.1 million.
Source: GAO analysis of ETA budget documents.
[End of figure]
Key Elements of Sound Research and Evaluation Programs:
While there is no single or ideal way for government agencies to
conduct research, several leading national organizations have
developed guidelines that identify key elements that promote a sound
research program.[Footnote 6] These guidelines identify five elements
as key: agency resources, professional competence, independence,
evaluation policies and procedures, and evaluation plans.
Resources. Research should be supported through stable, continuous
funding sources and through special one-time funds for evaluation
projects of interest to executive branch and congressional
policymakers.
Professional competence. Research should be performed by professionals
with appropriate training and experience for the evaluation activity
(such as performing a study, planning an evaluation agenda, reviewing
evaluation results, or performing a statistical analysis).
Independence. Although the heads of federal agencies and their
component organizations should participate in establishing evaluation
agendas, budgets, schedules, and priorities, the independence of
evaluators must be maintained with respect to the design, conduct, and
results of their evaluation studies.
Evaluation policy and procedures. Each federal agency and its
evaluation centers should publish policies and procedures and adopt
quality standards to guide evaluations within its purview. Such
policies and procedures should identify the kinds of evaluations to be
performed and the criteria and administrative steps for developing
evaluation plans and setting priorities, including selecting
evaluation approaches to use, consulting experts, ensuring evaluation
product quality, and publishing reports.
Evaluation plans. Each federal agency should require its major program
components to prepare annual and multiyear evaluation plans and to
update these plans annually. The planning should take into account the
need for evaluation results to inform program budgeting,
reauthorization, agency strategic plans, program management, and
responses to critical issues concerning program effectiveness. These
plans should include an appropriate mix of short-and long-term studies
to produce results for short-or long-term policy or management
decisions. To the extent practical, the plans should be developed in
consultation with program stakeholders.
Furthermore, leading organizations, including the American Evaluation
Association and the National Academy of Sciences, emphasize the need
for research programs to establish specific policies and procedures to
guide research activities. Based on several key elements identified by
these organizations, we developed a framework comprised of five
phases--agenda setting, selecting research, designing research,
conducting research, and disseminating research results. (See figure
2.)
Figure 2: Key Phases of the Research Process:
[Refer to PDF for image: illustration]
Agenda setting:
Identify general research priority areas where the agency will focus
its attention.
Selecting research:
Identify the criteria agencies use to select the research studies that
they will fund.
Designing research:
Develop research questions and appropriate methodologies for research
projects.
Conducting research:
Perform and monitor the implementation of research projects.
Disseminating research:
Make available sound research to policymakers, researchers, government
officials, and the general public.
Source: GAO analysis of key elements of a sound research process
identified by leading national organizations,including the American
Evaluation Association and the National Academy of Sciences.
[End of figure]
Agenda setting. Agencies should establish a structured process for
developing their research priorities. The process should identify how
agencies set research priority areas and provide for updating the
areas on a regular basis. The process should also allow for the
consideration of critical issues and state how internal and external
stakeholders will be included in developing the plan.
Selecting research. At this phase, the process should identify how the
research program's staff identifies and selects studies to fund,
including the criteria it uses to make those decisions. Steps might
describe how the staff assembles a list of potential studies, works
with internal program offices, and makes final decisions.
Designing research. During the design phase, the process should
identify steps taken to select appropriate research approaches and
methods and the safeguards in place to ensure appropriate tradeoffs
are made between what is desirable and what is practical and between
the relative strengths and weaknesses of different methods.
Conducting research. At this stage, the process should include
policies and procedures to guide the conduct of research. The process
should ensure that key events, activities, and time frames are
specified and that knowledgeable staff in the sponsoring agency
monitor the implementation of the research.
Disseminating research. This process should describe how research
findings are made available to the public and disseminated to all
potential users. These dissemination methods should include safeguards
to ensure research findings are disseminated in a timely manner and
are accessible through the Internet with user-friendly search and
retrieval technologies.
Research Terminology in This Report:
In this report, we use several technical terms in describing ETA's
research designs and study characteristics. (See table 1.)
Table 1: Research and Evaluation Terms and Definitions:
Term: Bias;
Definition: The extent to which a measurement or a sampling or
analytic method systematically underestimates or overestimates a value.
Term: Cross-sectional data;
Definition: Observations collected on subjects or events at a single
point in time.
Term: Descriptive study;
Definition: Provides descriptive information about specific conditions
or events, such as the number of individuals who received a particular
type of job training.
Term: Experimental design;
Definition: Units of study are randomly assigned to a treatment group
or to one or more comparison groups. The program's effects are
estimated by comparing outcomes for the treatment group with outcomes
for each comparison group.
Term: External validity/generalizability (used interchangeably with
generalizability);
Definition: The extent to which a finding applies (or can be
generalized) to persons, objects, settings, or times other than those
that were the subject of study.
Term: Longitudinal data;
Definition: Sometimes called "time series data," observations
collected over a period of time; the sample (instances or cases) may
or may not be the same each time but the population remains constant.
Term: Meta-analysis;
Definition: A systematic approach to summarizing or analyzing
evaluation findings across studies, usually quantitative, allowing
analysis of variation in program effect sizes by study methods or
program characteristics.
Term: Random assignment;
Definition: A method for assigning subjects to two or more groups by
chance, as in the flip of a coin.
Term: Secondary analysis;
Definition: The reanalysis of data collected in previous studies or
surveys to address new questions or use methods not previously
employed.
Term: Quasi-experimental design;
Definition: Units of study are assigned to a treatment and a
comparison group without random assignment.
Term: Representative sample;
Definition: A sample that has approximately the same distribution of
characteristics as the population from which it was drawn.
Term: Treatment group;
Definition: The subjects of the intervention being studied.
Source: GAO analysis of program evaluation and methodology documents.
[End of table]
ETA's Research Areas Generally Reflect Key Issues, but Some Studies
Are of Limited Usefulness:
Experts Thought ETA's 2007 to 2012 Research Plan Reflected Key Areas,
but They Also Suggested New Ones for Future Research:
Our expert panel generally considered ETA's research areas to be the
right ones for the period the research plan covered. About three-
fourths of the panel members reported that ETA's 2007 to 2012 research
agenda reflected key employment and training issues to at least a
moderate extent. However, a few experts commented that some of ETA's
research areas may be too broad and lack specificity.
The areas in ETA's 2007 to 2012 research plan covered a range of
issues, from job training to postsecondary education. Table 2
illustrates the scope of ETA's research areas.
Table 2: ETA's Research Areas, with Examples of Topics Covered in the
2007 to 2012 Research Plan:
Research area: Integration of workforce and regional economic
development;
Example topic areas[A]:
* Job training initiatives to produce high-skill, high-wage jobs;
* Strategic partnerships between private business sector and public
entities;
* Regional and economic development.
Research area: Methods of expanding U.S. workforce skills;
Example topic areas[A]:
* Rapid response services for dislocated workers;
* Costs and benefits of apprenticeship;
* Trade adjustment assistance.
Research area: Increasing the Labor market participation of
underutilized populations;
Example topic areas[A]:
* Enhanced services for the hard-to-employ;
* Employment-centered programs for ex-offenders;
* Current strategies to employ and retain older workers.
Research area: Using state-level administrative data to measure
progress and outcomes;
Example topic areas[A]:
* Examination of local workforce investment areas;
* WIA and employment outcomes.
Research area: Postsecondary education and job training;
Example topic areas[A]:
* The role of community colleges in workforce development and training;
* Career advancement accounts;
* Growing America through entrepreneurship.
Research area: Unemployment Insurance;
Example topic areas[A]:
* UI benefits;
* Characteristics of unemployed workers.
Source: GAO analysis of ETA's research agenda.
[A] This is not a complete or comprehensive list of examples of the
types of research conducted by ETA under each research area.
[End of table]
With regard to the specific studies within these research areas, ETA
invested most of its research and evaluation resources in work that
focused on increasing the labor market participation of underutilized
workers and on UI. Of the estimated $96 million that supported the 58
research reports published between January 2008 and March 2010, more
than half--about $56 million--funded research that addressed these two
research areas. Other areas received far less funding. For example,
funding for studies addressing the methods of expanding U.S. workforce
skills and using state-level administrative data to measure progress
and outcomes accounted for about $6.5 million, or about 6.7 percent of
the cost of studies published during the period we examined. (See
table 3.) Overall, the individual studies that ETA funded addressed a
wide variety of issues and ranged in cost from about $15,000 to a high
of about $22 million.
Table 3: Estimated Funding by Research Area for Research Studies
Disseminated between January 2008 and March 2010:
Research area: Increasing the Labor market participation of
underutilized populations;
Total amount: $28.9 million.
Research area: Unemployment Insurance;
Total amount: $26.8 million.
Research area: Postsecondary education and job training;
Total amount: $12.4 million.
Research area: Integration of workforce and regional economic
development;
Total amount: $9.9 million.
Research area: Methods of expanding U.S. workforce skills;
Total amount: $4.0 million.
Research area: Using state-level administrative data to measure
progress and outcomes;
Total amount: $2.5 million.
Research area: Other[A];
Total amount: $11.6 million.
Research area: Total;
Total amount: $96.1 million.
Source: GAO analysis of study cost information provided by ETA.
[A] "Other" includes studies that were not part of ETA's 2007 to 2012
priorities.
[End of table]
In addition to the research areas covered in ETA's 2007 to 2012
research plan, experts from our virtual panel suggested that ETA
incorporate additional research areas in its future research agenda.
Of the research areas identified, over half of our experts (28 of 39)
ranked the identification of employment and training approaches that
work, and for whom, as one of the top areas that ETA's future research
should address. (See figure 3.) Without such focus, experts commented
that it will be difficult to know how to improve the nation's
workforce system. Other issues ranked at the top by experts included
research on job creation strategies and the impact of long-term and
short-term training. (See appendix III for more information on issue-
area rankings.)
Figure 3: Issue Areas Ranked among the Top Three by Experts for ETA to
Address in Future Research:
[Refer to PDF for image: horizontal bar graph]
Which areas would you rank as the top 3 areas that ETA should address
in future research?
Employment and training approaches that work and for whom:
Number of respondents answering: 28.
Job creation strategies:
Number of respondents answering: 18.
Impact of long-term and/or short-term training:
Number of respondents answering: 18.
Long-term outcomes of employment and training programs:
Number of respondents answering: 15.
Effective performance measurement systems:
Number of respondents answering: 9.
Issues related to Unemployment Insurance:
Number of respondents answering: 8.
Linkages between the public workforce system and economic development
entities:
Number of respondents answering: 6.
Short-term outcomes of employment and training programs:
Number of respondents answering: 4.
Needs of different labor markets or industries:
Number of respondents answering: 4.
One-stop center management and operations:
Number of respondents answering: 3.
Value of various credentials:
Number of respondents answering: 3.
Employment and training strategies for various economic conditions:
Number of respondents answering: 1.
Source: GAO‘s survey of ETA‘s research priorities and dissemination
methods.
[End of figure]
In addition to identifying overall employment and training areas,
including issues related to UI, experts also identified specific
aspects of the UI system that could be examined in ETA's future
research. In particular, most experts (34 of 39 respondents) reported
that it would be at least moderately important, in the future, for ETA
to research the linkage between UI and employment and safety net
programs, such as Temporary Assistance for Needy Families or the
Supplemental Nutrition Assistance Program. (See figure 4.) This area
of research may be particularly important given the role that these
programs play in supporting individuals during economic downturns. In
addition, many experts (24 of 39 respondents) mentioned that ETA
should make the examination of the incentives and disincentives in the
UI system a research priority, given the challenge of supporting
unemployed workers during difficult economic times, while promoting
self-sufficiency through employment.
Figure 4: Unemployment Issues Frequently Cited by Experts as Important
for ETA to Address in Future Research:
[Refer to PDF for image: stacked horizontal bar graph]
Linkage between UI programs and safety net programs:
Number of respondents answering:
Not at all important: 0;
Somewhat important: 8;
Moderately important: 8;
Very important: 11;
Extremely important: 15.
Examination of incentives and disincentives of UI system:
Number of respondents answering:
Not at all important: 0;
Somewhat important: 5;
Moderately important: 9;
Very important: 17;
Extremely important: 7.
Modernization of UI system:
Number of respondents answering:
Not at all important: 0;
Somewhat important: 3;
Moderately important: 13;
Very important: 15;
Extremely important: 6.
Costs and benefits of UI system in different business cycles:
Number of respondents answering:
Not at all important: 1;
Somewhat important: 6;
Moderately important: 8;
Very important: 18;
Extremely important: 3.
Accountability mechanisms in UI system:
Number of respondents answering:
Not at all important: 1;
Somewhat important: 7;
Moderately important: 12;
Very important: 15;
Extremely important: 2.
Source: GAO‘s survey of ETA‘s research priorities and dissemination
methods.
[End of figure]
Experts also reported that it is important to fund research on what
works for selected population groups. Of the population groups
identified, the experts on our virtual panel most often ranked the
long-term unemployed, economically disadvantaged workers, and adults
with low basic skills as the top populations on which to focus future
research. Specifically, several experts commented that research could
help to identify the challenges some of these groups face, as well as
identify effective strategies that may help these population groups
obtain employment. (See appendix III for a complete list of responses
to these items.)
In addition to population groups, experts also identified several
employment and training programs that they believe warrant research
attention. In particular, experts most often ranked three components
of the WIA program--WIA Adult, WIA Dislocated Workers, and WIA Youth
[Footnote 7]--as key to evaluate in ETA's future research. Among those
three, WIA Adult was ranked the highest. (See appendix III for a
complete list of experts' responses on employment and training
programs to evaluate.)
ETA's Research Studies Generally Answered the Questions Posed, but
Their Usefulness Was Limited:
Research organizations and academic institutions with responsibility
for implementing ETA-funded research generally used methodologies
appropriate for the questions posed, but the studies were not always
useful for informing policy and practice. From January 2008 through
March 2010, ETA published 17 large research and evaluation reports--14
evaluations and 3 research reports--that each cost $1 million or more.
Four of these reports were designed to demonstrate what works and for
whom. Each of these four reports compared the employment-related
outcomes of individuals or regions who participated in training or
employment programs with the employment outcomes of similar
individuals who did not participate in the programs. The remaining 13
reports were descriptive and were not designed to assess program
outcomes.
In several studies we examined that cost $1 million or more, we found
that, for a number of reasons, ETA's research studies were limited in
their usefulness and in their ability to inform policy and practices.
For example, in a study of the Prisoner Re-entry Initiative,
shortcomings in the data collection phase limited the strength of the
findings and, as a result, limited the study's opportunity to
influence policy directions. Among other things, while the study
provided information on employment-centered opportunities for ex-
offenders, the study relied on self-reported baseline data, did not
account for differences across sites where services were received,
lacked the capacity to record differences in the intensity of those
services, and researchers failed to ensure that data collectors were
properly trained.
In another study, researchers did not control for bias in selecting
participants, compromising their ability to draw conclusions about the
cause and effect of program outcomes. Authors of this study on the
Workforce Innovations in Regional Economic Development (WIRED)
initiative acknowledged that the study would be unable to attribute
outcomes to program services because it did not use random assignment
in selecting participating regions. We have previously criticized ETA
for failing to adequately provide for evaluating the effectiveness of
its WIRED initiative.[Footnote 8]
Moreover, some studies were limited due to observation periods that
did not match the needs of the studies' objectives. For example, an
evaluation of an entrepreneurship training project was unable to
assess the effectiveness of the project in meeting its long-term goals
of increasing business ownership and self-sufficiency because the time
frames for the study were too short. In this study, data collection
was limited to 18 months after participants were randomly assigned, a
period far shorter than the 60-month period recommended by experts.
[Footnote 9] (See appendix IV for additional information on the
methodological characteristics of these studies.)
Experts generally agreed that ETA's research had limited usefulness in
informing policy and practice. Over one-third of the 39 experts
reported that over the past 5 years, ETA's research informed
employment and training policy and state and local practices to a
little extent or not at all. (See figure 5.) Some experts commented
that the design of these studies and the length of time to complete
them and disseminate results reduced their usefulness. For example,
many of the reports that we reviewed costing $1 million or more were
multiyear projects that took, in most cases, about 3 to 5 years to
complete. Some experts commented that the inclusion of shorter-length
studies may be useful in times of rapidly changing economic
conditions. At least one expert noted that some mixed-methods studies
would be useful--studies that would allow for short-term interim
findings that could facilitate changes in practice during the course
of the research study.
Figure 5: Expert Panel Views on the Extent to Which ETA's Research Has
Informed Policy and Practice:
[Refer to PDF for image: stacked horizontal bar graph]
In the past 5 years, to what extent has ETA‘s research informed
employment and training policy:
Number of respondents answering:
Don‘t know: 1;
Not at all: 1;
Little extent: 12;
Some extent: 17;
Moderate extent: 8;
Great extent: 0.
In the past 5 years, to what extent has ETA‘s research informed state
and local employment and training practices:
Number of respondents answering:
Don‘t know: 4;
Not at all: 1;
Little extent: 15;
Some extent: 13;
Moderate extent: 5;
Great extent: 1.
Source: GAO‘s survey of ETA‘s research priorities and dissemination
methods.
[End of figure]
Members of our expert panel stressed the importance of ETA
incorporating varied methodological approaches into its future
research proposals to best position the agency to address key
employment and training issues. Twenty-seven of the 39 experts
reported it was very important that ETA evaluate its pilots and
demonstrations. Twenty-three reported that it was very important that
more randomized experimental research designs be integrated into ETA's
future research.[Footnote 10] (See figure 6.) While several experts
noted that these randomized experiments will allow ETA to identify the
effectiveness of particular interventions or strategies, at least one
expert suggested that ETA should be strategic in choosing the
interventions it tests more rigorously, basing those decisions on what
appears most promising in preliminary studies.
Figure 6: Methodological Approaches Frequently Cited by Experts as
Very Important for ETA to Address in Future Research:
[Refer to PDF for image: stacked horizontal bar graph]
Evaluation of pilots and demonstrations:
Number of respondents answering:
Very unimportant: 0;
Somewhat unimportant: 2;
Neither important nor unimportant: 0;
Somewhat important: 9;
Very important: 27.
Randomized experimental design:
Number of respondents answering:
Very unimportant: 0;
Somewhat unimportant: 0;
Neither important nor unimportant: 3;
Somewhat important: 12;
Very important: 23.
Quasi-experimental design:
Number of respondents answering:
Very unimportant: 1;
Somewhat unimportant: 1;
Neither important nor unimportant: 3;
Somewhat important: 17;
Very important: 16.
Analysis of administrative data for research purposes:
Number of respondents answering:
Very unimportant: 0;
Somewhat unimportant: 1;
Neither important nor unimportant: 5;
Somewhat important: 19;
Very important: 14.
Descriptive studies:
Number of respondents answering:
Very unimportant: 0;
Somewhat unimportant: 3;
Neither important nor unimportant: 6;
Somewhat important: 18;
Very important: 12.
Meta-analyses (i.e., review or analysis of prior research):
Number of respondents answering:
Very unimportant: 2;
Somewhat unimportant: 2;
Neither important nor unimportant: 11;
Somewhat important: 18;
Very important: 6.
Secondary analysis of existing research data:
Number of respondents answering:
Very unimportant: 1;
Somewhat unimportant: 4;
Neither important nor unimportant: 13;
Somewhat important: 17;
Very important: 4.
Source: GAO‘s survey of ETA‘s research priorities and dissemination
methods.
[End of figure]
Furthermore, 16 of the 39 experts also reported that it is very
important for ETA to consider including more quasi-experimental
studies in the future. As previously discussed, such studies would
include designs that compare outcomes between groups with similar
characteristics, but do not use random assignment. By including more
quasi-experimental designs, ETA may be able to better understand the
link between services and outcomes in those settings where random
assignment is not possible, ethical, or practical.[Footnote 11]
ETA Has Taken Steps to Improve Its Research Program, but Additional
Actions Are Needed:
Labor Has Taken Steps to Reform Its Research Program:
Labor has taken several steps designed to improve the way it conducts
research, both at the department level and within ETA.
Department-level efforts. Labor has changed the organizational
structure of research within the department. In 2010, acknowledging
the need for better and more rigorous evaluations to inform its
policy, Labor established the Chief Evaluation Office to oversee the
department's research and evaluation efforts. The office, which
resides within the Office of the Assistant Secretary for Policy, has
no authority to direct research within Labor's agencies, according to
officials. It does, however, manage evaluations supported by funds
from a departmentwide account, oversee departmentwide evaluations, and
provide consultation to Labor agencies, including ETA. Specifically,
the office is responsible for creating and maintaining a comprehensive
inventory of past, ongoing, and planned evaluation activities within
Labor and for ensuring that Labor's evaluation program and findings
are transparent, credible, and accessible to the public. In fiscal
year 2010, the Chief Evaluation Office had an estimated budget of $8.5
million, and two of its four staff were on board by the beginning of
fiscal year 2011.
ETA efforts. ETA has recently made changes to some of its research
practices--chief among them is the involvement of stakeholders and
outside experts in the research process. We previously criticized ETA
for failing to consistently involve a broad range of stakeholders,
outside experts, or the general public in deciding what areas of
research it should undertake. We recommended that ETA take steps to
routinely involve outside experts in the research agenda-setting
process.[Footnote 12] For the upcoming 2010 to 2015 research plan, ETA
has awarded a grant to the Heldrich Center at Rutgers University to
convene an expert panel to help inform the research plan. The center
is expected to issue a report in May 2011 that outlines the panel's
recommendations for research areas to include in the plan. In
addition, ETA will work with other Labor agencies, as well as the
Departments of Education and Health and Human Services, before
finalizing its research agenda. Officials told us that they will also
solicit public comments before the research plan is finalized.
In addition to engaging stakeholders, ETA has also established a
formal research process. As we previously reported, ETA developed and
documented its research process in 2007.[Footnote 13] The agency's
actions were in response to a request by the Office of Management and
Budget (OMB) to establish more formal policies and procedures to guide
its research--a request that came out of OMB's concerns about the
manner in which ETA's research was being carried out. Prior to 2007,
ETA lacked a documented research process, and its research was often
conducted in an ad hoc manner. ETA's current research process
identifies the steps, activities, and time frames it uses to carry out
its research. Figure 7 illustrates critical components of ETA's 8-step
research process.
Figure 7: ETA's 8-Step Research Process:
[Refer to PDF for image: illustrated table]
Research stage: Agenda setting;
Steps in ETA‘s process: No formalized process;
Critical components of each step: (none).
Research stage: Selecting research;
Steps in ETA‘s process: ETA Step 1: Select projects for evaluation and
pilots, demonstrations, and research (PD&R) funding;
Critical components of each step:
* Review ETA/Labor stated priorities and recommendations in the 5-year
plan and solicit project ideas from within;
* Review congressional mandates and requests to co-fund projects from
other agencies;
* Review existing project commitments;
* Submit agenda to Assistant Secretary of ETA and finalize plan;
* Prepare approval memos for approved projects.
Research stage: Designing research;
Steps in ETA‘s process: ETA Step 2: Design Statements of Work (SOW),
Requests for Proposal (RFP) and Solicitations for Grant Applications
(SGA);
Critical components of each step:
* Review time frames, available funding, and methodology for each
project idea and determine appropriate funding method;
* Prepare draft reflecting research questions and methodology and
obtain feedback from program offices;
* Revise SOWs, RFPs, or SGAs, as appropriate, and obtain final
approval from Assistant Secretary;
* Submit to OMB for review any SGA, SOW, or RFP that costs $250,000 or
more.
Research stage: Designing research;
Steps in ETA‘s process: ETA Step 3: Contractor/grantee selection
process;
Critical components of each step:
* Work with the Office of Grants and Contracts Management to select
contractors or grantee;
* Ensure contracts and/or grants are awarded in accordance with
Federal Acquisition Regulations and departmental procurement rules.
Research stage: Conducting research;
Steps in ETA‘s process: ETA Step 4: Initial project design (Post-award
implementation);
Critical components of each step:
* Hold initial meetings to discuss study objectives and gather
information;
* Review contractor‘s draft design report and approve? Review
grantee‘s implementation plans and approve;
* Submit contractor‘s final design report, and if applicable, submit
report for peer review.
Research stage: Conducting research;
Steps in ETA‘s process: ETA Step 5: Paperwork Reduction Act clearance
process;
Critical components of each step:
* Prepare and submit an information collection request for internal
review to reduce burdens associated with federal information
collection and reporting requirements;
* Inform the public about the information collection request and allow
60 days for comment;
* Finalize information collection request and submit to OMB for review
and approval.
Research stage: Conducting research;
Steps in ETA‘s process: ETA Step 6: Project monitoring;
Critical components of each step:
* Monitor project implementation and raise any issues to senior
management;
* Develop corrective action plans (as needed) and review/approve
project progress and finance reports.
Research stage: Disseminating research;
Steps in ETA‘s process: ETA Steps 7 and 8: Review and dissemination of
reports.
Critical components of each step:
* Review and finalize report deliverables, working with contractors
and internal offices to incorporate revisions;
* Finalize research reports and obtain approval from the Assistant
Secretary;
* Produce Training and Employment Notice (TEN) and abstract to
announce the availability of approved research reports;
* Disseminate reports through ETA‘s research Web site and in hard copy
(as available);
* Alert OMB of reports submitted for dissemination and not approved
within 9 months.
Source: GAO analysis of ETA documentation.
[End of figure]
ETA's process contains several of the key elements identified by
leading organizations as important for guiding research activities.
For example, the process includes specific steps the agency should
take to identify the types of evaluations it will perform, as well as
the administrative steps it should take to develop evaluation plans
and select the research projects to fund.[Footnote 14] In addition,
the process also specifies key events and time frames, and provides
for monitoring the implementation of the research. For example, the
process stipulates that ETA should alert OMB of research reports that
have not been approved for dissemination within 9 months of being
submitted and allows contractors to publicly release their research
reports within those same time frames.[Footnote 15]
Some Areas of ETA's Research Program Merit Further Attention:
Despite ETA's efforts, more action is needed to improve its research
program. While ETA has taken steps to document its research process,
its process lacks specific details in some areas, creating ambiguities
that could undermine efforts to adhere to a formal process. For
example, as we previously reported, its process lacks clear criteria,
such as a dollar threshold or a particular methodological design
feature, for determining which projects require peer review. And while
the process specifies the actions project officers should take if
reports are not released in a timely manner, it does not specify the
consequences for failing to do so. We previously recommended that ETA
establish more specific processes, including time frames for
disseminating research reports. ETA has taken some action, such as
revising the performance standards for project officers to hold them
accountable for meeting time frames, but these steps do not fully
satisfy the recommendation because the changes are not yet reflected
in the formal research process.
Moreover, ETA's process is missing some critical elements that are
needed to ensure that the current improvements become routine
practices.
* Consulting with the Chief Evaluation Officer. ETA's process lacks a
formal provision requiring consultation with the newly established
Chief Evaluation Officer at important points in the research process.
For example, it contains no provision for consulting with the Chief
Evaluation Officer when developing its annual list of research
projects or when determining how ETA will invest its research and
evaluation resources. Such consultation could help Labor better
coordinate its research and evaluation efforts and better leverage its
research funding. Moreover, the process contains no provision for
involving the Chief Evaluation Officer in the early stages of
developing its research projects. In the recent past, Labor officials
told us that ETA has had difficulty developing requests for research
and evaluation proposals that can pass OMB technical reviews. In
particular, OMB has been critical of ETA's research designs because
they failed to provide for adequate sample size and appropriate
methodologies that are needed to obtain useful results. In addition,
OMB has also expressed concerns with ETA's reliance on process
evaluations rather than focusing on outcomes. These difficulties have
resulted in delays in the research process. ETA has begun to consult
with the Chief Evaluation Officer; however, these consultations are
not a routine component in the formal process.
* Setting the research agenda. ETA's current process, as documented,
begins with phase two--selecting specific research studies--and misses
the important first step of setting the overall research agenda. This
first phase of the process should include the steps that ETA will take
to establish its research priorities and to update them on a regular
basis. It should also include provisions for ensuring critical issues
are considered and internal and external stakeholders are included in
developing the plan. Officials noted that they plan to incorporate the
agenda-setting phase into its formal process, but have not yet done
so. Setting the research agenda is key to ensuring that an appropriate
mix of studies is included in future research. Failing to make this
phase part of the formal process, including the specific steps to
involve outside stakeholders that are currently under way, may leave
ETA with little assurance that these efforts will continue in the
future.
Beyond ETA's process for conducting research, current research
practices fall short of ensuring research transparency and
accountability--essential elements of a sound research and evaluation
program. The research program has few, if any, safeguards to protect
it from undue influence. According to officials, at times in the past
decade, many key research decisions have been made outside of the
office that is responsible for research. For example, decisions about
which research studies would and would not be publicly released were
made at the highest levels within ETA, and the criteria used to make
those decisions were unclear. Of the 34 reports that ETA released to
the public in 2008, 20 had waited between 2 and 5 years to be approved
for public release.[Footnote 16] Several reports that had experienced
long delays had relatively positive and potentially useful findings
for the workforce system, according to our analysis. Among the studies
delayed by almost 5 years was an evaluation of labor exchange services
in the one-stop system that found certain employment services to be
highly cost-effective in some situations. Another study, delayed for
about 3.5 years, was a compendium of past and ongoing experimental
studies of the workforce system, including early findings and
recommendations for future research.
In our previous report, we noted that ETA's research and evaluation
center lacked a specific mechanism to insulate it from undue
influence. We reported that other federal agencies, such as the
Department of Education's Institute of Education Sciences and the
National Science Foundation, engage advisory bodies in the research
process. While not without tradeoffs in terms of additional time and
effort, such an approach may serve to protect the research program
from undue influence and improve accountability. ETA is currently
involving outside experts in setting the research agenda for 2010 to
2015, but is not involving experts more broadly on research policy and
practices.
ETA Has Recently Included More Random Assignment Studies in Its
Research Program:
ETA has recently begun to include more rigorous studies in its ongoing
research. Of the 10 large, ongoing studies costing $2 million or more
that began during the period of our review, three--the WIA Gold
Standard Evaluation of the Adult and Dislocated Worker Programs, the
Impact Evaluation of the Young Parents Demonstration, and the
Evaluation of Project Growing America Through Entrepreneurship II
(Project GATE II)--use experimental design with random assignment, as
recommended by our experts.[Footnote 17] These ongoing studies--which
range in cost from $2 million to nearly $23 million--have the
potential to determine the effectiveness of some of the program
services. Table 4 outlines some key characteristics of these three
studies.
Table 4: Characteristics of Three Ongoing Studies:
Study: Purpose;
WIA Gold Standard Evaluation: This experimental evaluation will
provide rigorous, nationally representative estimates of the WIA's net
impacts. The study will also produce a detailed description of the
program's implementation and estimates of its benefits and costs. Key
questions include:
* Does access to WIA-intensive training services lead adult and
dislocated workers to achieve better educational and employment
outcomes than they would achieve in the absence of those services?
* Does the effectiveness of WIA vary by population subgroup?
* Is the effectiveness of WIA--and its components--commensurate with
its costs?
Impact Evaluation of the Young Parents Demonstration: This
experimental evaluation will determine the impacts of supplemental
short-term mentoring services to complement core workforce development
programs, including the following:
* Measure differences in services that the participants in the
treatment and control groups receive;
* Assess how services received differ across sites and among
individuals in the treatment and control groups;
* Identify the resource requirements involved in providing services
and identify implementation issues that might have affected outcomes;
Evaluation of Project Growing America Through Entrepreneurship II
(Project GATE II): This experimental evaluation will compare the
outcomes of randomly assigned Project GATE II participants to the
outcomes of individuals who did not receive Project GATE II services;
* Measure the impacts of Project GATE on participants' labor market
and self-employment outcomes;
* Test the viability of providing Project GATE services to dislocated
workers in rural areas over the age of 45;
* Assess the linkages between the local one-stop centers' programs and
other key groups providing services.
Study: Scope;
WIA Gold Standard Evaluation: Thirty local workforce investment areas,
selected at random, stratified to ensure that they are representative
in terms of the size, geography, and customer training rates.
Following this selection, a subset of participants at each of the 30
selected sites will be randomly assigned to one of three groups;
Impact Evaluation of the Young Parents Demonstration: Five to 10 sites
serving approximately 2,500 participants;
Evaluation of Project Growing America Through Entrepreneurship II
(Project GATE II): Examination of four state grants.
Study: Performance period;
WIA Gold Standard Evaluation: 7 years. A subset of participants will
be tracked and administered surveys at 15 and 36 months after random
assignment;
Impact Evaluation of the Young Parents Demonstration: 7 years. A
sample of participants will be tracked and administered a follow-up
survey approximately 12 months after random assignment. An additional
survey will be conducted 30 months after random assignment;
Evaluation of Project Growing America Through Entrepreneurship II
(Project GATE II): 3 years. A sample of project participants will be
tracked and administered a follow-up survey approximately 12 months
after random assignment.
Study: Peer review;
WIA Gold Standard Evaluation: The contractor will coordinate three
meetings of the peer review board made up of 10 experts. ETA will
select seven and the contractor three. The purpose of the board will
be to review and comment on the project methodology and key project
deliverables;
Impact Evaluation of the Young Parents Demonstration: The contractor
will convene up to four peer review panels composed of four to five
researchers selected by ETA to review the evaluation methodology,
analytic approaches and key deliverables;
Evaluation of Project Growing America Through Entrepreneurship II
(Project GATE II): No requirements under contract. However, ETA
officials told us that they have plans for an independent peer review.
Study: Cost;
WIA Gold Standard Evaluation: Total: $22,951,040 (Evaluation contract);
Impact Evaluation of the Young Parents Demonstration: Total:
$6,154,570; (Impact evaluation contract: $4.8 million; process
evaluation: $717,000; and technical assistance contracts: $199,998 and
$437,572);
Evaluation of Project Growing America Through Entrepreneurship II
(Project GATE II): Total: $2,014,996; (Impact evaluation: $1.5 million;
technical assistance contracts: $485,000 and $29,996).
Source: GAO analysis of ETA's documents.
[End of table]
Experimental designs with random assignment are an important means to
understand whether various program components or services are
effective, but they are also often difficult to design and implement
in real-world settings.[Footnote 18] For example, in doing evaluations
of employment and training programs, researchers often have difficulty
in recruiting sample sizes large enough to detect meaningful outcomes.
Because employment and training services may vary by location, and
participants and their socio-economic environments are diverse,
researchers must find ways to standardize procedures and treatment or
service options. This often means recruiting relatively large samples.
However, studies can be intrusive, often requiring program sites to
change how they operate or to increase the resources available to
participants. As a result, recruiting sites and sufficient numbers of
participants may be difficult.
Some of ETA's ongoing research studies face challenges in recruiting
sample sizes large enough to meet the studies' objectives. For
example, based on an OMB review, it was determined that the sample
size for the Impact Evaluation of the Young Parents Demonstration had
to be much larger in order to be able to assess the effectiveness of
the program. At that time, ETA had already awarded two phases of
grants. After consulting with the new Chief Evaluation Officer, ETA
changed the number of participants required for the third phase from
100 to 400 to obtain a sample large enough to address OMB's concerns
and provide reliable estimates. However, grantees found it difficult
to recruit even the 100 participants in the smaller sample, and it
remains unclear whether they will be able to recruit all of the needed
participants for the expanded design.
The WIA Gold Standard Evaluation of the Adult and Dislocated Worker
Programs:
The WIA Gold Standard Evaluation illustrates ETA's difficulties in
planning and executing large-scale, rigorous random assignment
studies. WIA required that the Secretary of Labor conduct at least one
multi-site control-group evaluation of the services and programs under
WIA by the end of fiscal year 2005.[Footnote 19] ETA, however, delayed
executing such a study, finally soliciting proposals in November 2007
and awarding the contract in June 2008. The contractor submitted the
initial design report in January 2009 and provided ETA with design
revisions in May 2010. Officials tell us researchers will soon begin
randomly assigning participants. ETA expects to receive the first
report (on implementation) during the winter of 2012-2013 and the
final report in 2015--10 years later than the WIA-mandated time frame.
An OMB-selected panel of government experts--a technical working group
composed of experts chosen by ETA, the evaluation contractor, and OMB
staff--reviewed the original design for this study.[Footnote 20]
Reviewers agreed the design contained many strengths, including:
* the selection of an experimental design and a net impact approach;
* the addition of a process or implementation study to evaluate
differences among sites and other implementation and data collection
issues;
* the use of administrative and survey data;
* the collection of information on services received by participants
in the control group; and:
* the collection of a wide range of outcome data for participants.
However, reviewers raised several concerns regarding the design. For
example, they were skeptical that the researchers would be able to
obtain a sufficiently large and representative sample to draw
meaningful conclusions about the effectiveness of the national
workforce system. In order to maximize participation, officials told
us that the Assistant Secretary of ETA made personal phone calls to
all selected sites to emphasize the importance of the study, offered
an open door policy to site officials to discuss issues, and followed
up with an appreciation letter. Furthermore, ETA required the
evaluation contractor to provide reimbursement payments to each site
to offset implementation costs.
Reviewers also had several other concerns regarding which groups would
be included in the study and which groups would not. For example, some
experts raised concerns about getting accurate information on the
youth program because of the large, one-time infusion of funds the
program received from the American Recovery and Reinvestment Act of
2009. Reviewers were further concerned about the appropriateness of
the evaluation objectives, the adequacy of steps taken to account for
the effect of variation in services across sites on evaluation
outcomes, and the external validity or generalizability of the study.
In order to address these concerns, ETA made substantial adjustments
to the original design. Specifically, ETA officials told us that based
on an agreement with OMB, they instructed the contractor to drop the
youth component from the evaluation and to focus only on the Adult and
Dislocated Worker programs. While we received information on the new
design and time frames for the WIA Gold Standard Evaluation, a
finalized design plan is not yet available. According to officials, a
finalized design is being prepared and will be available in June 2011.
ETA Has Improved the Availability of Its Research but There Are
Opportunities to Improve Its Search Page and Dissemination Methods:
ETA Has Improved the Timeliness of Its Disseminated Research:
ETA has recently improved the timeliness with which it disseminates
its research reports. In our last review in January 2010, we found
that 20 of the 34 reports that ETA disseminated in 2008 had been
waiting 2 to 5 years to be publicly released.[Footnote 21] The 34
research reports published by ETA in 2008 took, on average, 804 days
from the time the report was submitted to ETA until the time it was
posted to ETA's research database. By contrast, from 2009 through the
first quarter of 2010, the average time between submission and public
release was 76 days, which represents a more than 90 percent
improvement in dissemination time compared with 2008. Additionally,
there were no research reports in 2009 that were delayed for more than
6 months. Further, the average time to dissemination improved
significantly even when we excluded such outliers as the 20 research
reports that were delayed for 2 years or more. Without these outliers,
average time to dissemination for reports in 2008 was 100 days,
indicating that time to dissemination in 2009 through the first
quarter of 2010 still improved by 24 percent.
ETA Has Improved Its Research Database but Lacks Plans for Assessing
the Usability of Its Search Page:
In 2010, ETA updated its online, Web-based search page in order to
improve the usability of its research database--the primary tool for
making ETA research available to policymakers and the general public.
Officials told us that ETA's old Web-based search page was so error-
prone and difficult to use that they opted to substitute it with one
that had not yet completed internal testing. Our review of the old Web-
based search page confirmed that it had serious limitations and did
not consistently return the same results. For example, when we
searched the database by title for a known ETA research report titled
Registered Apprenticeship, we successfully retrieved that report once.
One month later, when we entered the exact same search terms, we were
unable to retrieve the report. (For a more complete description of our
analysis of ETA's search capability, see appendix II.)
In our review of the updated Web-based search page, we found that the
updates make the research database more usable. Labor officials told
us they have taken other steps, as well, in efforts to improve its Web-
based search page. For example, they have developed a project plan
that articulates the steps Labor will take to update ETA's Web-based
search page. In addition, they have assigned a database administrator
whose responsibilities include performing daily quality control spot
checks in order to monitor performance and address technical problems.
Although these changes have the potential to improve the usability of
ETA's database, Labor has not developed a formal plan for assessing
the overall effectiveness of its Web-based search page, including user
satisfaction. Labor has made a number of changes to the way the page
operates, but it has not provided users with tips on how to use the
search functions, even though it is an industry standard to do
so.[Footnote 22] Even skilled users who were familiar with the old Web-
based search page may need guidance on the exact meaning of new terms
and functions now available on the new page. For example, the old Web-
based search page gave users the option of searching by "key word,"
which is no longer an option in the new page. Instead, "key word"
searches have been replaced with a variety of other options, including
the ability to search the full text or abstract of a research report.
However, there is no guidance on the Web site on how to use these new
search options. Industry best practices suggest that a Web site
evaluation plan that incorporates data from routine reviews of Web
site performance and that assesses user satisfaction can help agencies
ensure the usability of their Web sites. ETA currently has no plans to
do such assessments.
ETA Uses Various Methods to Disseminate Research, but Experts Suggest
Additional Methods:
At present, ETA's research database is the primary method that ETA
uses to make its research reports publicly available, according to
officials. In order to call attention to new reports available in that
database, ETA sends a Training and Employment Notice, also commonly
known as a TEN, to an e-mail list of the more than 40,000 subscribers
who have signed up to receive them. ETA's research process specifies
that for each new research report that is approved for dissemination,
ETA must draft a TEN and an abstract before it is posted to ETA's Web
site. Beyond posting reports to its database, ETA also distributes
hard copies of some of its research reports.
In addition to electronic distribution, ETA also organizes various
presentations to disseminate its research findings. These
presentations, however, are done on an ad hoc basis. As mentioned in
our prior report, ETA hosted a research conference in 2009 to present
some of its research findings, renewing a practice that had been
discontinued in 2003. As ETA looks to the future, officials tell us
they will plan and organize similar research conferences as resources
permit. In addition to these research conferences, ETA's regional
offices occasionally hold smaller, regional conferences as well.
Beyond these formal conferences, ETA also hosts an internal briefing
series at Labor headquarters where research contractors present their
findings to various officials. For each of these briefings, ETA has a
list of stakeholders that it invites, including various Labor
officials, outside agency officials, congressional staff, and other
outside stakeholders.
Experts who participated in our virtual panel provided their views on
the effectiveness of different methods for disseminating research
reports, and several of those rated more highly are methods currently
employed by ETA. (See figure 8.)
Figure 8: Experts' Opinion on the Effectiveness of Various Research
Dissemination Methods:
[Refer to PDF for image: stacked horizontal bar graph]
E-mail notifications announcing newly released research:
Number of respondents answering:
No response or left blank: 0;
Not at all effective: 0;
Somewhat effective: 3;
Moderately effective: 6;
Very effective: 19;
Extremely effective: 11.
Searchable database of papers on ETA‘s Web site:
Number of respondents answering:
No response or left blank: 0;
Not at all effective: 0;
Somewhat effective: 3;
Moderately effective: 10;
Very effective: 10;
Extremely effective: 16.
Distributing a one-page summary of the research findings from each
report:
Number of respondents answering:
No response or left blank: 0;
Not at all effective: 0;
Somewhat effective: 4;
Moderately effective: 9;
Very effective: 17;
Extremely effective: 9.
Briefings at ETA for external audiences (including stakeholders and
policymakers):
Number of respondents answering:
No response or left blank: 1;
Not at all effective: 0;
Somewhat effective: 7;
Moderately effective: 9;
Very effective: 14;
Extremely effective: 8.
Publicizing research findings through press articles, industry
publications, etc.:
Number of respondents answering:
No response or left blank: 0;
Not at all effective: 0;
Somewhat effective: 7;
Moderately effective: 13;
Very effective: 14;
Extremely effective: 5.
Presenting at outside conferences:
Number of respondents answering:
No response or left blank: 0;
Not at all effective: 0;
Somewhat effective: 6;
Moderately effective: 14;
Very effective: 15;
Extremely effective: 4.
National ETA-sponsored conferences:
Number of respondents answering:
No response or left blank: 2;
Not at all effective: 0;
Somewhat effective: 5;
Moderately effective: 16;
Very effective: 13;
Extremely effective: 3.
Regional ETA-sponsored conferences:
Number of respondents answering:
No response or left blank: 1;
Not at all effective: 0;
Somewhat effective: 4;
Moderately effective: 19;
Very effective: 13;
Extremely effective: 2.
Publishing articles in journals:
Number of respondents answering:
No response or left blank: 0;
Not at all effective: 0;
Somewhat effective: 14;
Moderately effective: 11;
Very effective: 8;
Extremely effective: 6.
Contracting with relevant organizations or associations to disseminate
research:
Number of respondents answering:
No response or left blank: 1;
Not at all effective: 0;
Somewhat effective: 7;
Moderately effective: 17;
Very effective: 10;
Extremely effective: 4.
Webinars:
Number of respondents answering:
No response or left blank: 0;
Not at all effective: 1;
Somewhat effective: 9;
Moderately effective: 17;
Very effective: 9;
Extremely effective: 3.
Distributing a compendium of ETA‘s research:
Number of respondents answering:
No response or left blank: 0;
Not at all effective: 2;
Somewhat effective: 7;
Moderately effective: 20;
Very effective: 5;
Extremely effective: 5.
Videos or podcasts:
Number of respondents answering:
No response or left blank: 2;
Not at all effective: 4;
Somewhat effective: 13;
Moderately effective: 12;
Very effective: 8;
Extremely effective: 0.
Social media (such as Twitter):
Number of respondents answering:
No response or left blank: 3;
Not at all effective: 11;
Somewhat effective: 12;
Moderately effective: 10;
Very effective: 3;
Extremely effective: 0.
Source: GAO‘s survey of ETA‘s research priorities and dissemination
methods.
[End of figure]
Most of the experts (30 of the 39 respondents) in our panel reported
that using e-mail notifications, a searchable database of ETA papers,
and briefings at ETA for external audiences (including stakeholders
and policymakers) would be very effective or extremely effective
approaches for disseminating research. In addition, a majority of the
experts (26 of the 39 respondents) in our panel reported that
publishing one-page summaries of research findings, not currently done
by ETA, would be very or extremely effective.
Conclusions:
ETA plays an important role in developing workforce policies and
helping to identify the most effective and efficient ways to train and
employ workers for jobs in the twenty-first century. With the current
economic crisis and high unemployment rates, ETA's role has become
even more critical. The agency has made some improvements in its
research program, even since our last review a year ago. But officials
can do more to ensure that the progress continues in the years to come.
ETA has taken a major step forward in establishing a formal research
process--one that documents most actions that must be taken in the
life cycle of a research or evaluation project. But, it is missing
some key elements that could help ensure the continuation of current
practices. While ETA is currently using outside advisory bodies to
help it establish its research agenda, the formal process does not
include the agenda-setting phase. Officials tell us they have plans to
incorporate this phase in the future, and we urge them to do so.
Without a formalized agenda-setting phase, ETA may miss opportunities
to ensure that its research agenda addresses the most critical
employment and training issues and that outside stakeholders are
routinely involved. Moreover, ETA's process has not formalized the now
ad hoc advisory role of the Chief Evaluation Officer. Absent the
routine involvement of the Chief Evaluation Officer at key steps in
the process, ETA may find it difficult to ensure that research
proposals are asking the right questions, are methodologically sound,
and that they can quickly pass critical OMB reviews.
ETA's research findings are now available to the public on its Web
site in far less time than it took in 2008. Despite this clear
improvement, ETA has not taken the necessary steps to ensure that
research products remain readily available to the public. The decision
regarding what and when to make research publicly available is left in
the hands of too few, and the process lacks needed safeguards to
ensure transparency and accountability. Absent safeguards, key
research decisions may again be made in ways that harm the credibility
of the program and prevent important research findings from being used
to inform policy and practice.
ETA's Web-based search page is the primary means ETA uses to make the
research studies it funds readily available to the public. And, while
ETA has improved the functionality of its Web site, no effort has been
made to ensure that the problems that plagued the system in the past
do not recur. Absent such efforts, ETA will have little assurance that
its research findings are actually available to users.
Recommendations for Executive Action:
To improve ETA's research program, we recommend that the Secretary of
Labor require ETA to take the following three actions:
* Formally incorporate into its research process the routine
involvement of the Chief Evaluation Officer at key milestones,
including at the development of ETA's annual research agenda and
spending priorities, as well as at the early stages of developing
specific research projects.
* Develop a mechanism to enhance the transparency and accountability
of ETA's research program. For example, such a mechanism might include
involving advisory bodies or other entities outside ETA, in efforts to
develop ETA's research policies and processes.
* Develop a formal plan for ensuring that ETA's research products are
easily accessible to stakeholders and to the general public through
its searchable database. Such a plan could involve requiring Labor to
assess the overall effectiveness of its Web-based search page,
including user satisfaction with search features.
Agency Comments and Our Evaluation:
We provided a draft of this report to the Department of Labor for
review and comment. Labor provided written comments, which are
reproduced in appendix VII. In addition, ETA provided technical
comments, which we incorporated where appropriate.
In its response, Labor generally agreed with our findings and all of
our recommendations, noting its ongoing efforts in support of the
recommendations.
* Regarding our recommendation to formally incorporate into its
research process the routine involvement of the Chief Evaluation
Officer at key research milestones, Labor noted that it is currently
taking steps to do so. Officials reported that they have worked
closely with this office in various aspects of its research, including
discussing research, demonstration projects, and evaluations in the
early stages of development and plans to continue this collaboration
in the future. However, ETA's comments did not discuss plans to update
its documentation on the formal research process. We found in our
review that involving the Chief Evaluation Officer was not an official
component of ETA's documented research process, and it occurred on an
ad hoc basis. As ETA moves forward, we urge the agency to modify its
current research process and document the involvement of the Chief
Evaluation Officer at critical research milestones.
* Regarding our recommendation for ETA to develop a mechanism to
enhance the transparency and accountability of its research program,
officials cited several steps they are taking to improve the program,
including involving outside experts in the development of their 5-year
research plan and establishing advisory and peer review groups to
review major evaluations. While officials note they plan to engage
outside experts in broader research policies and processes, we
encourage ETA to formalize this involvement. Moreover, we encourage
ETA to continue to move forward in its efforts to further clarify
components of its research process that are not well defined,
including, for example, the criteria to be used when deciding when a
peer review should be performed.
* Regarding our recommendation to develop a formal plan to ensure that
disseminated research is easily accessible to stakeholders and the
general public, officials cited specific steps the agency has taken to
improve its Web-based research database. While these actions are a
step in the right direction, we believe that it is still important for
Labor to develop a formal and comprehensive plan to ensure that
disseminated research continues to be accessible to the public.
Furthermore, Labor expressed concerns about how we characterized the
agency's budget for pilots, demonstrations, and research. Recognizing
these concerns, we made changes to the report to better capture the
amount of funding ETA has available for research.
As agreed with your offices, unless you publicly announce the contents
of this report earlier, we plan no further distribution until 30 days
from the report date. At that time, we will send copies to the
appropriate congressional committees, the Secretary of Labor, and
other interested parties. The report will also be available at no
charge on GAO's Web site at [hyperlink, http://www.gao.gov].
If you or your staffs have any questions about this report, please
contact me at (202) 512-7215 or scottg@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. GAO staff who made key contributions to
this report are listed in appendix VIII.
Signed by:
George A. Scott:
Director, Education, Workforce and Income Security Issues:
[End of section]
Appendix I: Status of Prior GAO Recommendations to the Department of
Labor, as of January 2011:
GAO's recommendations: The Secretary of Labor should
Department of Labor's response: [Empty];
Status: [Empty].
GAO's recommendations: The Secretary of Labor should take steps to
clarify the Employment and Training Administration's (ETA) revised
organizational structure and ensure that the Office of Policy
Development and Research (OPDR) reports directly to ETA's Assistant
Secretary;
Department of Labor's response: The Department of Labor (Labor) does
not agree with this recommendation as written. According to Labor
officials, the Administrator of OPDR currently reports to the Deputy
Assistant Secretary, not directly to ETA's Assistant Secretary.
However, Labor officials acknowledge that important functions such as
research and evaluation should not have too many intermediary
reporting layers. To facilitate communication, officials further noted
that the OPDR Administrator, the Deputy Assistant Secretary, and the
Chief Evaluation Officer meet on a monthly basis with the Assistant
Secretary to discuss evaluations;
Status: Labor has taken no action.
GAO's recommendations: The Secretary of Labor should provide
sufficient authority to ETA's research and evaluation center to plan,
conduct, and disseminate research;
Department of Labor's response: Labor agrees with this recommendation,
but authority to make key decisions still resides with the Office of
the Assistant Secretary for ETA. OPDR currently provides
recommendations to this office regarding plans for conducting and
disseminating research. In an effort to improve evaluations
departmentwide, the Secretary of Labor recently established the Chief
Evaluation Office to monitor evaluation efforts across the department.
OPDR has begun to work informally with the Chief Evaluation Officer
and the Chief Economist to design and implement research and
evaluation projects;
Status: Labor's actions do not completely satisfy recommendation.
GAO's recommendations: The Secretary of Labor should direct ETA's
research and evaluation center to establish more specific processes,
including time frames for dissemination of research, to promote
transparency and accountability;
Department of Labor's response: Labor agrees with this recommendation.
ETA reports that it has taken some steps to establish more specific
processes regarding dissemination of research, citing changes in
performance standards for project officers. However, our
recommendation would make broader changes to their research process
and no such changes are reflected in the documents the agency provided;
Status: Labor's actions do not completely satisfy recommendation.
GAO's recommendations: The Secretary of Labor should create an
information system to track research projects at all phases to ensure
timely completion and dissemination;
Department of Labor's response: Labor agrees with this recommendation.
Officials report that they have begun working on a centralized,
electronic tracking system for its research projects. However, the
work is still under way and no time frames have been provided for its
completion. Currently, OPDR uses an Excel document to keep inventory
of all research, demonstration, and evaluation projects;
Status: Labor's actions do not completely satisfy recommendation.
GAO's recommendations: The Secretary of Labor should instruct ETA's
research and evaluation center to develop processes to routinely
involve outside experts in setting its research agenda and to the
extent required, do so consistent with the Federal Advisory Committee
Act;
Department of Labor's response: Labor agrees with this recommendation.
OPDR has taken steps to engage outside experts in setting its 5-year
research plan for 2011 and collaborate with the research and
evaluation centers of other federal agencies, such as the Departments
of Education and Health and Human Services. OPDR also plans to convene
an expert panel, solicit public comments, and incorporate feedback
from its 2009 Reemployment Research Conference and its 2010 ETA
Reemployment Summit. However, despite these current efforts, OPDR has
not formally incorporated them in its standard research process;
Status: Labor's actions do not completely satisfy recommendation.
Source: GAO analysis of GAO-10-243.
[End of table]
[End of section]
Appendix II: Scope and Methodology:
We were asked to review the Employment and Training Administration's
(ETA) research program to better understand its approach to conducting
and disseminating research. Specifically, we answered the following
research questions: (1) To what extent do ETA's research priorities
reflect key national employment and training issues and how useful
were the studies funded under them? (2) What steps has ETA taken to
improve its research program? (3) How has ETA improved, if at all, the
availability of its research since our last review in January 2010 and
what other steps could ETA take to further ensure its research
findings are readily available?
To answer our research questions, we convened a virtual panel using a
modified Delphi technique to obtain selected employment and training
experts' opinions on ETA's research priorities and dissemination
methods. We also visited two workforce agencies in Pennsylvania and
Virginia that are implementing two of ETA's ongoing research studies
to learn about implementation issues and how research is being
conducted. In addition, we reviewed 58 ETA-funded research and
evaluation reports disseminated between January 2008 and March 2010
and assessed the methodological soundness of completed studies that
cost $1 million or more. We also reviewed ETA's ongoing studies that
cost $2 million or more. To determine the availability of ETA's
research, we measured the time between when the final version of a
research report was submitted to ETA's Office of Policy Development
and Research (ODPR) and when it was posted on ETA's Web site. We also
conducted a series of systematic searches to test the reliability of
ETA's research database. Furthermore, we interviewed Department of
Labor (Labor) and ETA officials to better understand ETA's research
capacity, processes, and the use of research findings to inform policy
and practice. Lastly, we reviewed relevant agency documents and
policies, as well as relevant federal laws.
Web-Based Expert Panel:
We convened a nongeneralizable Web-based virtual panel of 41
employment and training experts to obtain their opinions on ETA's
research priorities and dissemination methods. We employed a modified
version of the Delphi method to organize and gather these experts'
opinions.[Footnote 23] To encourage participation by our experts, we
promised that responses would not be individually identifiable and
that results would generally be provided in summary form. To select
the panel, we asked several employment and training experts, on the
basis of their experience and expertise, to identify other experts who
were knowledgeable of ETA and the research it conducts and
disseminates. After receiving nominations from experts, we reviewed
the list to ensure that it reflected a range of perspectives and
backgrounds, including academics, researchers, and consultants.
Our Delphi process entailed two survey phases. (See appendix V for a
copy of our phase I and phase II questionnaires.) In phase I, which
ran from June 22, 2010, to August 9, 2010, we asked the panel to
respond to five open-ended questions about ETA's research priorities
and dissemination methods. We developed these questions based on our
study objectives and pretested them with four experts by phone to
ensure the questionnaire was clear, unbiased, and did not place an
undue burden on respondents. All relevant changes were made before we
deployed the first Web-based questionnaire to experts.
After the experts completed the open-ended questions in the first
questionnaire, we performed a content analysis of the responses in
order to identify the most important issues raised by our experts. Two
members of our team categorized experts' responses to each of the
questions. Any disagreements were discussed until consensus was
reached. Thirty-six of the 41 panelists selected completed phase I of
the survey (about an 88 percent response rate). Those that did not
complete phase I were allowed to participate in phase II. (For a list
of experts who participated in phase I and phase II, see appendix VI.)
The experts' responses to phase I were used to create the questions
for phase II. In phase II, we gathered more specific information on
ETA's research and dissemination practices. Phase II, which ran from
October 29, 2010, to December 14, 2010, consisted of 16 follow-up
questions where panelists were asked to either rank or rate the
responses from phase I. We pretested the questionnaire for the second
phase with three experts to ensure the clarity of the instrument. We
conducted two of our expert pretests in-person and one by phone.
Thirty-nine of the 41 experts completed phase II (about a 95 percent
response rate).
Site Visits to Workforce Agencies Implementing ETA-Funded Research
Studies:
To further enhance our understanding of how ETA conducts its research,
we visited two workforce agencies that are implementing ETA's ongoing
research studies. First, we visited the Lancaster County Workforce
Investment Board in Lancaster, Pa., which received funding from ETA to
implement the Young Parents Demonstration project.[Footnote 24] This
project provides educational and occupational skills training to
promote employment and economic self-sufficiency for mothers, fathers,
and expectant mothers ages 16 to 24. Second, we visited the Northern
Virginia Workforce Investment Board in Falls Church, Va., which
received funding from ETA to implement the second round of the Project
Growing America Through Entrepreneurship, also referred to as Project
GATE II.[Footnote 25] This grant helps dislocated workers aged 50 and
over obtain information, classroom training, one-to-one technical
assistance, counseling, and financial assistance to establish new
businesses in order to help them start and sustain successful self-
employment.
We selected these workforce agencies because they were identified by
ETA as having active research projects in the implementation stage.
These sites also required minimum travel expenditure. During our site
visits, we toured each workforce agencies' facilities and used a
semistructured interview protocol to interview the project director
and staff about their role and responsibilities, the extent to which
they communicate with ETA, and whether or not they face challenges
with regards to implementation. At the Lancaster County site, we
participated in an informal on-site lunch forum where local community
programs that the agency partners with talked with us about their
collaboration with the program. At the Northern Virginia GATE II site,
we observed a focus group operated by the program to facilitate
information-sharing among participants.
After our site visits, we conducted phone interviews with the
contractors that received funding from ETA to evaluate the outcomes of
two research projects. Specifically, we interviewed the Urban
Institute, which evaluates the Young Parents Demonstration project,
and IMPAQ International, which evaluates Project GATE II. Both
projects include an experimental component with control and comparison
groups to determine the effects of program interventions on
participants. During our interviews we used a semistructured
questionnaire and asked questions to better understand their roles and
responsibilities for the project, the extent to which they communicate
with ETA, and whether or not they experience methodological and
implementation challenges.
Analysis of Methodological Characteristics of ETA:
We reviewed the 58 research and evaluation reports that ETA
disseminated between January 2008 and March 2010 and assessed the
methodological soundness of 11 completed studies that cost $1 million
or more. In addition, we reviewed 10 ongoing studies costing $2
million or more to determine if research practices or the soundness of
research designs had changed over time. We categorized the 58 studies
disseminated between January 2008 and March 2010 by study type, cost,
and research area. For the larger studies costing $1 million or more,
we analyzed key characteristics including design features, scope,
generalizability, and the appropriateness of analytical approaches and
statistical procedures. These studies were analyzed independently by
two analysts and the agreement between their ratings was 100 percent.
(For results of this analysis, see appendix VI.)
Analysis of the Timeliness and Effectiveness of ETA's Dissemination
Activities:
To evaluate the availability of ETA's research, we measured the time
between when the final version of a research report was submitted to
ODPR and when it was posted on ETA's Web site. Specifically, we
measured the dissemination time frames for reports posted in 2008 and
compared that with the dissemination time frames for reports issued
between January 2009 through March 2010. In addition, we conducted a
series of systematic searches to test the reliability of ETA's Web-
based research database. To perform our searches, we selected a random
sample of 30 reports from the 312 reports available on ETA's research
database at the time of our review. Specifically, we tested a variety
of search functions available at the time of our review to determine
the extent to which research reports could be easily retrieved on
ETA's research database. These functions included searches by title,
keywords, author, and/or dates. We classified a report as retrievable
if it appeared anywhere in our search results. We conducted our
initial searches between June 30, 2010, and July 6, 2010. A second
round of searches was conducted between August 6, 2010, and August 10,
2010. Further, we interviewed Labor and ETA officials to learn more
about the search capabilities of ETA's research database and the
processes used to address errors and implement changes. Finally, we
interviewed officials to gather information about ETA's dissemination
methods, including its current techniques and future plans for
disseminating research reports.
Interviews with Labor and ETA Officials:
To better understand the agency's research capacity, we interviewed
ETA officials and reviewed relevant agency and budget documentation.
Similarly, to obtain information on ETA's research process and how
research findings are used to inform employment and training policy
and practice, we interviewed officials and reviewed agency
documentation, including relevant policies and procedures that guide
ETA's research. We also reviewed relevant federal laws.
We conducted this performance audit from March 2009 through March 2011
in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives.
[End of section]
Appendix III: The Panel's Ratings of Key Employment and Training
Issues, Populations, and Programs That ETA Should Address in Its
Future Research:
In our Delphi phase II Web-based questionnaire, we asked the panel of
experts to rate and rank the key employment and training issues,
populations, and programs that ETA should address in its future
research. These issues were identified by the panel during phase I.
For our analysis, we calculated basic descriptive statistics on these
issues, which are presented in tables 5 through 7.
Table 5: Descriptive Statistics on Key Employment and Training Issues
ETA Should Address in Its Future Research:
The responses in this table are based on the following questions:
Question 1: Taking into account ETA's limited resources, how
important, if at all, is it for ETA to address the following in future
research? (Q1):
Question 2: Among the areas you checked as at least moderately
important in question 1, which would you rank as the top 3 areas that
ETA should address in future research. (Please rank only 3 areas, with
1 as your top priority area, 2 as your second highest area, etc.)
Long-term outcomes of employment and training programs:
Responses to question 2: Ranked 1: 9;
Responses to question 2: Ranked 2: 3;
Responses to question 2: Ranked 3: 3;
Responses to question 2: Ranked 1, 2, or 3: 15;
Responses to question 1: Not at all important: 0;
Responses to question 1: Somewhat important: 4;
Responses to question 1: Moderately important: 4;
Responses to question 1: Very Important: 14;
Responses to question 1: Extremely important: 17.
Short-term outcomes of employment and training programs:
Responses to question 2: Ranked 1: 2;
Responses to question 2: Ranked 2: 0;
Responses to question 2: Ranked 3: 2;
Responses to question 2: Ranked 1, 2, or 3: 4;
Responses to question 1: Not at all important: 0;
Responses to question 1: Somewhat important: 5;
Responses to question 1: Moderately important: 12;
Responses to question 1: Very Important: 13;
Responses to question 1: Extremely important: 9.
Value of various credentials:
Responses to question 2: Ranked 1: 0;
Responses to question 2: Ranked 2: 1;
Responses to question 2: Ranked 3: 2;
Responses to question 2: Ranked 1, 2, or 3: 3;
Responses to question 1: Not at all important: 0;
Responses to question 1: Somewhat important: 7;
Responses to question 1: Moderately important: 12;
Responses to question 1: Very Important: 15;
Responses to question 1: Extremely important: 4.
Impact of long-term and/or short-term training:
Responses to question 2: Ranked 1: 5;
Responses to question 2: Ranked 2: 9;
Responses to question 2: Ranked 3: 4;
Responses to question 2: Ranked 1, 2, or 3: 18;
Responses to question 1: Not at all important: 0;
Responses to question 1: Somewhat important: 0;
Responses to question 1: Moderately important: 5;
Responses to question 1: Very Important: 18;
Responses to question 1: Extremely important: 16.
Job creation strategies, such as providing wage subsidies, tax
credits, or public service employment:
Responses to question 2: Ranked 1: 2;
Responses to question 2: Ranked 2: 9;
Responses to question 2: Ranked 3: 7;
Responses to question 2: Ranked 1, 2, or 3: 18;
Responses to question 1: Not at all important: 1;
Responses to question 1: Somewhat important: 2;
Responses to question 1: Moderately important: 5;
Responses to question 1: Very Important: 18;
Responses to question 1: Extremely important: 13.
Employment and training approaches that work and for whom:
Responses to question 2: Ranked 1: 9;
Responses to question 2: Ranked 2: 11;
Responses to question 2: Ranked 3: 8;
Responses to question 2: Ranked 1, 2, or 3: 28;
Responses to question 1: Not at all important: 0;
Responses to question 1: Somewhat important: 1;
Responses to question 1: Moderately important: 1;
Responses to question 1: Very Important: 18;
Responses to question 1: Extremely important: 19.
One-stop center management and operations:
Responses to question 2: Ranked 1: 0;
Responses to question 2: Ranked 2: 1;
Responses to question 2: Ranked 3: 2;
Responses to question 2: Ranked 1, 2, or 3: 3;
Responses to question 1: Not at all important: 0;
Responses to question 1: Somewhat important: 10;
Responses to question 1: Moderately important: 14;
Responses to question 1: Very Important: 12;
Responses to question 1: Extremely important: 3.
Effective performance measurement systems:
Responses to question 2: Ranked 1: 2;
Responses to question 2: Ranked 2: 3;
Responses to question 2: Ranked 3: 4;
Responses to question 2: Ranked 1, 2, or 3: 9;
Responses to question 1: Not at all important: 0;
Responses to question 1: Somewhat important: 4;
Responses to question 1: Moderately important: 10;
Responses to question 1: Very Important: 16;
Responses to question 1: Extremely important: 9.
Linkages between the public workforce system and economic development
entities:
Responses to question 2: Ranked 1: 4;
Responses to question 2: Ranked 2: 1;
Responses to question 2: Ranked 3: 1;
Responses to question 2: Ranked 1, 2, or 3: 6;
Responses to question 1: Not at all important: 2;
Responses to question 1: Somewhat important: 11;
Responses to question 1: Moderately important: 11;
Responses to question 1: Very Important: 8;
Responses to question 1: Extremely important: 7.
Needs of different labor markets or industries:
Responses to question 2: Ranked 1: 1;
Responses to question 2: Ranked 2: 1;
Responses to question 2: Ranked 3: 2;
Responses to question 2: Ranked 1, 2, or 3: 4;
Responses to question 1: Not at all important: 2;
Responses to question 1: Somewhat important: 6;
Responses to question 1: Moderately important: 15;
Responses to question 1: Very Important: 14;
Responses to question 1: Extremely important: 2.
Employment and training strategies for various economic conditions:
Responses to question 2: Ranked 1: 0;
Responses to question 2: Ranked 2: 0;
Responses to question 2: Ranked 3: 1;
Responses to question 2: Ranked 1, 2, or 3: 1;
Responses to question 1: Not at all important: 1;
Responses to question 1: Somewhat important: 4;
Responses to question 1: Moderately important: 13;
Responses to question 1: Very Important: 15;
Responses to question 1: Extremely important: 6.
Issues related to unemployment insurance:
Responses to question 2: Ranked 1: 5;
Responses to question 2: Ranked 2: 0;
Responses to question 2: Ranked 3: 3;
Responses to question 2: Ranked 1, 2, or 3: 8;
Responses to question 1: Not at all important: 2;
Responses to question 1: Somewhat important: 5;
Responses to question 1: Moderately important: 10;
Responses to question 1: Very Important: 12;
Responses to question 1: Extremely important: 10.
Source: GAO's survey of ETA's research priorities and dissemination
methods.
[End of table]
Table 6: Descriptive Statistics on Key Populations ETA Should Address
in Its Future Research:
The responses in this table are based on the following questions:
Question 3: How important, if at all, would it be for ETA to fund
research that focuses on what works for the following populations
given its resource constraints?
Question 4: Among the populations you checked as at least moderately
important in question 3, which would you rank as the top 5 populations
on which ETA should fund research in the future? (Please rank only 5
populations, with 1 as your top population, 2 as your second highest
population, etc.)
Short-term unemployed:
Responses to question 4: Ranked 1: 0;
Responses to question 4: Ranked 2: 2;
Responses to question 4: Ranked 3: 1;
Responses to question 4: Ranked 4: 0;
Responses to question 4: Ranked 5: 0;
Responses to question 4: Ranked 1,2, 3, 4, or 5: 3;
Responses to question 3: Not at all important: 3;
Responses to question 3: Somewhat important: 16;
Responses to question 3: Moderately important: 8;
Responses to question 3: Very Important: 7;
Responses to question 3: Extremely important: 2.
Long-term unemployed:
Responses to question 4: Ranked 1: 15;
Responses to question 4: Ranked 2: 10;
Responses to question 4: Ranked 3: 6;
Responses to question 4: Ranked 4: 1;
Responses to question 4: Ranked 5: 1;
Responses to question 4: Ranked 1,2, 3, 4, or 5: 33;
Responses to question 3: Not at all important: 0;
Responses to question 3: Somewhat important: 1;
Responses to question 3: Moderately important: 0;
Responses to question 3: Very Important: 19;
Responses to question 3: Extremely important: 19.
Dislocated workers:
Responses to question 4: Ranked 1: 5;
Responses to question 4: Ranked 2: 4;
Responses to question 4: Ranked 3: 4;
Responses to question 4: Ranked 4: 2;
Responses to question 4: Ranked 5: 3;
Responses to question 4: Ranked 1,2, 3, 4, or 5: 18;
Responses to question 3: Not at all important: 0;
Responses to question 3: Somewhat important: 4;
Responses to question 3: Moderately important: 7;
Responses to question 3: Very Important: 19;
Responses to question 3: Extremely important: 9.
Older workers:
Responses to question 4: Ranked 1: 0;
Responses to question 4: Ranked 2: 2;
Responses to question 4: Ranked 3: 2;
Responses to question 4: Ranked 4: 1;
Responses to question 4: Ranked 5: 2;
Responses to question 4: Ranked 1,2, 3, 4, or 5: 7;
Responses to question 3: Not at all important: 2;
Responses to question 3: Somewhat important: 11;
Responses to question 3: Moderately important: 9;
Responses to question 3: Very Important: 13;
Responses to question 3: Extremely important: 3.
Veterans:
Responses to question 4: Ranked 1: 1;
Responses to question 4: Ranked 2: 2;
Responses to question 4: Ranked 3: 2;
Responses to question 4: Ranked 4: 0;
Responses to question 4: Ranked 5: 0;
Responses to question 4: Ranked 1,2, 3, 4, or 5: 5;
Responses to question 3: Not at all important: 0;
Responses to question 3: Somewhat important: 9;
Responses to question 3: Moderately important: 13;
Responses to question 3: Very Important: 11;
Responses to question 3: Extremely important: 4.
Economically disadvantaged workers:
Responses to question 4: Ranked 1: 11;
Responses to question 4: Ranked 2: 8;
Responses to question 4: Ranked 3: 8;
Responses to question 4: Ranked 4: 0;
Responses to question 4: Ranked 5: 2;
Responses to question 4: Ranked 1,2, 3, 4, or 5: 29;
Responses to question 3: Not at all important: 0;
Responses to question 3: Somewhat important: 2;
Responses to question 3: Moderately important: 2;
Responses to question 3: Very Important: 22;
Responses to question 3: Extremely important: 13.
Racial and ethnic minorities:
Responses to question 4: Ranked 1: 0;
Responses to question 4: Ranked 2: 2;
Responses to question 4: Ranked 3: 0;
Responses to question 4: Ranked 4: 4;
Responses to question 4: Ranked 5: 2;
Responses to question 4: Ranked 1,2, 3, 4, or 5: 8;
Responses to question 3: Not at all important: 0;
Responses to question 3: Somewhat important: 8;
Responses to question 3: Moderately important: 13;
Responses to question 3: Very Important: 13;
Responses to question 3: Extremely important: 3.
Immigrants:
Responses to question 4: Ranked 1: 0;
Responses to question 4: Ranked 2: 2;
Responses to question 4: Ranked 3: 0;
Responses to question 4: Ranked 4: 0;
Responses to question 4: Ranked 5: 1;
Responses to question 4: Ranked 1,2, 3, 4, or 5: 3;
Responses to question 3: Not at all important: 2;
Responses to question 3: Somewhat important: 10;
Responses to question 3: Moderately important: 12;
Responses to question 3: Very Important: 11;
Responses to question 3: Extremely important: 2.
Adults with low basic skills:
Responses to question 4: Ranked 1: 3;
Responses to question 4: Ranked 2: 3;
Responses to question 4: Ranked 3: 6;
Responses to question 4: Ranked 4: 3;
Responses to question 4: Ranked 5: 7;
Responses to question 4: Ranked 1,2, 3, 4, or 5: 22;
Responses to question 3: Not at all important: 0;
Responses to question 3: Somewhat important: 2;
Responses to question 3: Moderately important: 6;
Responses to question 3: Very Important: 19;
Responses to question 3: Extremely important: 10.
Workers with physical and mental disabilities:
Responses to question 4: Ranked 1: 0;
Responses to question 4: Ranked 2: 0;
Responses to question 4: Ranked 3: 0;
Responses to question 4: Ranked 4: 3;
Responses to question 4: Ranked 5: 1;
Responses to question 4: Ranked 1,2, 3, 4, or 5: 4;
Responses to question 3: Not at all important: 0;
Responses to question 3: Somewhat important: 8;
Responses to question 3: Moderately important: 15;
Responses to question 3: Very Important: 13;
Responses to question 3: Extremely important: 1.
In-school youth:
Responses to question 4: Ranked 1: 0;
Responses to question 4: Ranked 2: 0;
Responses to question 4: Ranked 3: 3;
Responses to question 4: Ranked 4: 0;
Responses to question 4: Ranked 5: 2;
Responses to question 4: Ranked 1,2, 3, 4, or 5: 5;
Responses to question 3: Not at all important: 2;
Responses to question 3: Somewhat important: 14;
Responses to question 3: Moderately important: 12;
Responses to question 3: Very Important: 9;
Responses to question 3: Extremely important: 1.
Out-of-school youth:
Responses to question 4: Ranked 1: 4;
Responses to question 4: Ranked 2: 2;
Responses to question 4: Ranked 3: 4;
Responses to question 4: Ranked 4: 8;
Responses to question 4: Ranked 5: 2;
Responses to question 4: Ranked 1,2, 3, 4, or 5: 20;
Responses to question 3: Not at all important: 0;
Responses to question 3: Somewhat important: 3;
Responses to question 3: Moderately important: 10;
Responses to question 3: Very Important: 17;
Responses to question 3: Extremely important: 9.
Ex-offenders:
Responses to question 4: Ranked 1: 0;
Responses to question 4: Ranked 2: 2;
Responses to question 4: Ranked 3: 3;
Responses to question 4: Ranked 4: 3;
Responses to question 4: Ranked 5: 2;
Responses to question 4: Ranked 1,2, 3, 4, or 5: 10;
Responses to question 3: Not at all important: 0;
Responses to question 3: Somewhat important: 5;
Responses to question 3: Moderately important: 10;
Responses to question 3: Very Important: 17;
Responses to question 3: Extremely important: 6.
Source: GAO's survey of ETA's research priorities and dissemination
methods.
[End of table]
Table 7: Descriptive Statistics on Key Employment and Training
Programs ETA Should Address in Its Future Research:
The responses in this table are based on the following questions:
Question 6: How important, if at all, is it for ETA to evaluate the
following key employment or training programs (excluding UI)?
Question 7: Given a limited amount of resources, which three of the
key employment or training programs that you checked as at least
moderately important in question 6 should ETA address in future
research? (Please rank only 3 employment or training programs, with 1
as your top program, 2 as your second highest program, etc.)
WIA Adult:
Responses to question 7: Ranked 1: 12;
Responses to question 7: Ranked 2: 9;
Responses to question 7: Ranked 3: 7;
Responses to question 7: Ranked 1, 2, or 3: 28;
Responses to question 6: Not at all important: 0;
Responses to question 6: Somewhat important: 2;
Responses to question 6: Moderately important: 11;
Responses to question 6: Very Important: 11;
Responses to question 6: Extremely important: 13.
WIA Dislocated Worker:
Responses to question 7: Ranked 1: 8;
Responses to question 7: Ranked 2: 11;
Responses to question 7: Ranked 3: 3;
Responses to question 7: Ranked 1, 2, or 3: 22;
Responses to question 6: Not at all important: 0;
Responses to question 6: Somewhat important: 3;
Responses to question 6: Moderately important: 8;
Responses to question 6: Very Important: 13;
Responses to question 6: Extremely important: 13.
WIA Youth:
Responses to question 7: Ranked 1: 7;
Responses to question 7: Ranked 2: 6;
Responses to question 7: Ranked 3: 4;
Responses to question 7: Ranked 1, 2, or 3: 17;
Responses to question 6: Not at all important: 0;
Responses to question 6: Somewhat important: 4;
Responses to question 6: Moderately important: 11;
Responses to question 6: Very Important: 15;
Responses to question 6: Extremely important: 8.
Wagner-Peyser employment Service:
Responses to question 7: Ranked 1: 3;
Responses to question 7: Ranked 2: 3;
Responses to question 7: Ranked 3: 6;
Responses to question 7: Ranked 1, 2, or 3: 12;
Responses to question 6: Not at all important: 0;
Responses to question 6: Somewhat important: 7;
Responses to question 6: Moderately important: 13;
Responses to question 6: Very Important: 9;
Responses to question 6: Extremely important: 7.
Job Corps:
Responses to question 7: Ranked 1: 1;
Responses to question 7: Ranked 2: 3;
Responses to question 7: Ranked 3: 1;
Responses to question 7: Ranked 1, 2, or 3: 5;
Responses to question 6: Not at all important: 1;
Responses to question 6: Somewhat important: 11;
Responses to question 6: Moderately important: 12;
Responses to question 6: Very Important: 10;
Responses to question 6: Extremely important: 2.
Trade Adjustment Assistance:
Responses to question 7: Ranked 1: 2;
Responses to question 7: Ranked 2: 2;
Responses to question 7: Ranked 3: 3;
Responses to question 7: Ranked 1, 2, or 3: 7;
Responses to question 6: Not at all important: 2;
Responses to question 6: Somewhat important: 5;
Responses to question 6: Moderately important: 15;
Responses to question 6: Very Important: 11;
Responses to question 6: Extremely important: 5.
Veterans' Employment and Training:
Responses to question 7: Ranked 1: 1;
Responses to question 7: Ranked 2: 2;
Responses to question 7: Ranked 3: 4;
Responses to question 7: Ranked 1, 2, or 3: 7;
Responses to question 6: Not at all important: 1;
Responses to question 6: Somewhat important: 9;
Responses to question 6: Moderately important: 16;
Responses to question 6: Very Important: 6;
Responses to question 6: Extremely important: 6.
Apprenticeship program:
Responses to question 7: Ranked 1: 3;
Responses to question 7: Ranked 2: 5;
Responses to question 7: Ranked 3: 4;
Responses to question 7: Ranked 1, 2, or 3: 3;
Responses to question 6: Not at all important: 7;
Responses to question 6: Somewhat important: 14;
Responses to question 6: Moderately important: 1;
Responses to question 6: Very Important: 7;
Responses to question 6: Extremely important: 12.
Source: GAO's survey of ETA's research priorities and dissemination
methods.
[End of table]
[End of section]
Appendix IV: Characteristics of Research Studies Disseminated between
January 2009 and March 2010 That Cost $1 Million or More:
Report name: 1. Implementation Analysis of the High Growth Job
Training Initiative (HGJI) Programs;
ETA report number: 2008-10;
ETA research area: Integration of the workforce and regional economic
development;
Cost: $1,500,000;
Study type: Evaluation;
Study design: Descriptive;
Methods and techniques:
* Implementation review;
Consistency of analytical approach with research question: Yes;
Generalizability of findings: No.
Report name: 2. Workforce Investment Act Non-Experimental Net Impact
Evaluation: Final Report;
ETA report number: 2009-10;
ETA research area: Using state-level administrative data to measure
progress and outcomes;
Cost: $1,000,000;
Study type: Evaluation;
Study design: Quasi-experimental;
Methods and techniques:
* Other (comparison v. treatment group using propensity score
matching);
Consistency of analytical approach with research question: Yes;
Generalizability of findings: No.
Report name: 3. Evaluation of the Prisoner Re-Entry Initiative - Final
Report;
ETA report number: 2009-3;
ETA research area: Increasing the labor market participation of
underutilized populations;
Cost: $1,204,078;
Study type: Evaluation;
Study design: Descriptive;
Methods and techniques:
* Post-intervention only data collection;
* Other (some characteristics collected before the intervention, but
these were not used to make comparisons);
Consistency of analytical approach with research question: Yes;
Generalizability of findings: No.
Report name: 4. Implementing the National Fund for Workforce
Solutions: The Baseline Evaluation Report;
ETA report number: 2009-21;
ETA research area: Integration of the workforce and regional economic
development;
Cost: $1,000,000;
Study type: Evaluation;
Study design: Descriptive;
Methods and techniques:
* Implementation review;
* Secondary analysis;
Consistency of analytical approach with research question: No basis to
judge;
Generalizability of findings: No response.
Report name: 5. The Power of Partnership: American Regions
Collaborating for Economic Competitiveness (Generation I WIRED Interim
Eval);
ETA report number: 2009-18;
ETA research area: Integration of the workforce and regional economic
development;
Cost: $3,433,478[A];
Study type: Evaluation;
Study design: Descriptive;
Methods and techniques:
* Implementation review;
* Survey;
* Secondary analysis;
* Other (social network analysis);
Consistency of analytical approach with research question: No;
Generalizability of findings: No.
Report name: 6. Early Implementation of Generation I of the Workforce
Innovation in Regional Development (WIRED) Initiative, 2007 Interim
Evaluation Report;
ETA report number: 2008-03;
ETA research area: Integration of the workforce and regional economic
development;
Cost: [Empty];
Study type: Evaluation;
Study design: Descriptive;
Methods and techniques:
* Implementation review;
* Survey;
* Secondary analysis;
* Other (social network analysis);
Consistency of analytical approach with research question: No;
Generalizability of findings: No.
Report name: 7. Nurturing America's Growth in the Global Marketplace:
An Interim Report on the Evaluation of Generations II and III of WIRED;
ETA report number: 2009-19;
ETA research area: Integration of the workforce and regional economic
development;
Cost: $3,345,036;
Study type: Evaluation;
Study design: Quasi-experimental;
Methods and techniques:
* Survey;
* Secondary analysis;
* Other (comparison v. treatment group analysis );
Consistency of analytical approach with research question: Yes;
Generalizability of findings: No.
Report name: 8. Recent Changes in the Characteristics of Unemployed
Workers;
ETA report number: 2009-13;
ETA research area: Unemployment insurance;
Cost: $6,507,262[B];
Study type: Research;
Study design: Descriptive;
Methods and techniques:
* Prestest/post-test intervention data collection;
Consistency of analytical approach with research question: Yes;
Generalizability of findings: Yes.
Report name: 9. Trends in the Structure of the Labor Market and
Unemployment;
ETA report number: 2009-09;
ETA research area: Unemployment insurance;
Cost: [Empty];
Study type: Research;
Study design: Descriptive;
Methods and techniques:
* Secondary analysis;
Consistency of analytical approach with research question: Yes;
Generalizability of findings: Yes.
Report name: 10. Reemployment and Eligibility Assessment (REA) Study
FY 2005 Initiative;
ETA report number: 2008-02;
ETA research area: Unemployment insurance;
Cost: [Empty];
Study type: Research;
Study design:
* Experimental (randomized control trials;
* Descriptive;
Methods and techniques:
* Survey;
* Secondary analysis;
Consistency of analytical approach with research question: Yes;
Generalizability of findings: No.
Report name: 11. Growing America Through Entrepreneurship: Findings
from the Evaluation of Project GATE;
ETA report number: 2008-08;
ETA research area: Postsecondary education and job training;
Cost: $11,400,000;
Study type: Evaluation;
Study design: Experimental (randomized control trials);
Methods and techniques:
* Survey;
* Secondary analysis;
* Prestest/post-test intervention data collection;
Consistency of analytical approach with research question: Yes;
Generalizability of findings: No.
Report name: 12. Evaluation of Youth Build Offender Grants;
ETA report number: 2009-11;
ETA research area: Increasing the labor market participation of
underutilized populations;
Cost: $1,151,449;
Study type: Evaluation;
Study design: Descriptive;
Methods and techniques:
* Case studies;
* Other (data collected on characteristics upon entry and outcome
characteristics collected after completion);
Consistency of analytical approach with research question: Yes;
Generalizability of findings: No.
Report name: 13. Initial Implementation of the Trade Act;
ETA report number: 2009-14;
ETA research area: Methods of expanding U.S. workforce skills;
Cost: $10,453,957[C];
Study type: Evaluation;
Study design: Descriptive;
Methods and techniques:
* Case studies;
* Implementation review;
* Secondary analysis;
Consistency of analytical approach with research question: Yes;
Generalizability of findings: No.
Report name: 14. Assessment, Case Management, and Post-Training
Assistance for TAA Participants;
ETA report number: 2009-15;
ETA research area: Methods of expanding U.S. workforce skills;
Cost: [Empty];
Study type: Evaluation;
Study design: Descriptive;
Methods and techniques:
* Case studies;
Consistency of analytical approach with research question: Yes;
Generalizability of findings: No.
Report name: 15. Linkages Between TAA, One-Stop Career Center Partners
and Economic Development Agencies;
ETA report number: 2009-16;
ETA research area: Methods of expanding U.S. workforce skills;
Cost: [Empty];
Study type: Evaluation;
Study design: Descriptive;
Methods and techniques:
* Case studies;
Consistency of analytical approach with research question: Yes;
Generalizability of findings: No.
Report name: 16. Rapid Response and TAA;
ETA report number: 2009-17;
ETA research area: Methods of expanding U.S. workforce skills;
Cost: [Empty];
Study type: Evaluation;
Study design: Descriptive;
Methods and techniques:
* Case studies;
Consistency of analytical approach with research question: Yes;
Generalizability of findings: No.
Report name: 17. Youth Offender Demonstration Project Process
Evaluation Round Two;
ETA report number: 2004-10;
ETA research area: Increasing the labor market participation of
underutilized populations;
Cost: $1,734,393;
Study type: Evaluation;
Study design: Descriptive;
Methods and techniques:
* Case studies;
* Secondary analysis;
Consistency of analytical approach with research question: Yes;
Generalizability of findings: No.
Source: GAO analysis of ETA's disseminated studies between January
2008 and March 2010.
[A] Two of four reports produced or expected from the WIRED Generation
I Evaluation cost a total of $3,433,478 for the full evaluation.
[B] Three of five reports produced from the UI Benefits Study cost a
total of $6,507,262 for the full study.
[C] Four of approximately 10 reports produced or expected from the
National Evaluation of Trade Adjustment Assistance cost a total of
$10,453,957 for the full evaluation.
[End of table]
[End of section]
Appendix V: Delphi Phase I and Phase II Questionnaires:
GAO's Review of ETA's Research Priorities and Dissemination
Methods: Phase 1:
U.S. Government Accountability Office:
Introduction:
The Congress has asked the U.S. Government Accountability Office
(GAO), an independent agency that gathers information for the
Congress, to review the Employment and Training Administration's (ETA)
research activities.
As part of this review, we have asked you to participate in this
virtual expert panel to gather your input on national priorities in
the area of employment and training, and on ETA's research agenda and
dissemination practices. As you know, you have been identified as an
expert in employment and training services or programs, service
delivery approaches, and/or evaluation methodologies for these
programs. Your participation allows you an opportunity to assist us in
informing ETA's future work.
The virtual expert panel will consist of multiple phases. For the first
stage of the virtual panel, we ask that you complete a brief, five-
question survey. We will use the group's responses to these questions
to create a more detailed questionnaire in the second stage. This
questionnaire will allow you to consider information provided by other
members of the group. No information will be attributed to specific
individuals.
Thank you very much for agreeing to participate in the panel. Your
participation in our study is essential for us to provide requested
information to the Congress. Our work in this review will cover other
areas in addition to those addressed in the virtual panel. To learn
more about these additional areas,click here.
Contact:
If your response will be delayed or if you have any questions, please
call or e-mail Ashanta Williams or Kathy White. Click here for their
contact information.
Confidentiality of Responses:
We will not release individually identifiable information from this
survey while our work is in progress unless compelled by law or
required to do so by the Congress. Once the report is issued, we will
destroy the file that links names to individual responses. While the
results will generally be provided in summary form in our report,
individual answers may be discussed, but they will not include any
information that could be used to identify individual respondents.
Names of participants will be listed in our final report; however,
names will not be associated with responses. Furthermore, your
comments and answers will not be provided to other participants during
any stage in the process.
Navigation, Exiting and Printing the Survey:
To learn more about navigation, exiting and printing the survey,
please click here.
Key Employment and Training Issues:
1. In your opinion, what are the key employment and training issues that
ETA's research agenda should address? Please consider population
groups, programs, and/or methodological approaches in your response.
ETA's research agenda:
2. We understand that ETA is currently reviewing its research agenda
and that its research priorities may change moving forward. However,
we would like you to consider the priority areas addressed in the 2007-
2012 research agenda along with selected examples of the topics
covered in the agenda listed below.
Note: This is not a complete or comprehensive list of ETA's
disseminated research studies. We compiled this list to give you a
better sense of the types of research conducted by ETA.
ETA Research Priority Areas, with Example Topics Covered in ETA's
Research Agenda:
1. Integration of Workforce and Regional Economic Development:
* Job training initiatives to produce high-skill, high-wage jobs;
* Strategic partnerships between private business sector and public
entities;
* Regional and economic development.
2. Methods of Expanding U.S. Workforce Skills:
* Rapid response services for dislocated workers;
* Costs and benefits of apprenticeship;
* Trade adjustment assistance.
3. Increasing the Labor Market Participation of Underutilized
Populations:
* Enhanced services for the hard-to-employ;
* Employment-centered programs for ex-offenders;
* Current strategies to employ and retain older workers.
4. Using State-Level Administrative Data to Measure Progress and
Outcomes:
* Examination of local workforce investment areas o WIA and employment
outcomes.
5. Post-Secondary Education and Job Training:
* The role of community colleges in workforce development and training;
* Career advancement accounts;
* Growing America through entrepreneurship.
6. Unemployment Insurance:
* Unemployment insurance benefits
* Characteristics of unemployed workers
* Trends in the structure of the labor market and unemployment
Based on your knowledge, how, if at all, do you think ETA's 2007-2012
research agenda reflects the key employment and training issues you
identified in question one?
3. In your opinion, how well does ETA's research inform policy and
practice? Please consider ongoing research studies, of which you are
familiar, during the past 5 years.
4. What additional thoughts, if any, do you have about ETA's research or
research agenda?
ETA's Dissemination Methods:
5. What dissemination methods do you think would allow ETA to
effectively share its research findings?
Completed:
6. Have you finished this questionnaire?
(Check Only One Answer)
1. Yes.
2. No.
Print.
Submit.
Cancel.
[End of section]
Menu:
Display list of headings:
GAO's Review of ETA's Research Priorities and
Dissemination Methods: Phase 2:
Prioritization for Future Work:
Prioritization of ETA's Research:
Policy and Practice:
ETA's Dissemination Methods:
Demographics:
Summary:
View and print a summary of your responses.
Help:
If you have any questions about this GAO survey, please contact
Ashanta Williams at (202) 512-5110 (williamsar@qao.qoy) or Kathy White
at (202) 512-8512 (whitek@gao.gov).
Progress:
Navigate:
GAO's Review of ETA's Research Priorities and Dissemination Methods:
Phase 2:
U.S. Government Accountability Office:
In phase 1, we asked you to provide information on:
* key employment and training issues that ETA's research agenda should
address;
* the extent to which ETA's 2007-2012 research agenda addresses these
key employment and training issues; and
* dissemination methods that would allow ETA to effectively share its
research findings.
This is Phase II of our virtual panel. We developed our questions in
this phase using the responses of 36 experts who provided input in
Phase 1. Your responses will allow us to gather more specific
information for the Congress on ETA's research and dissemination
practices.
We thank you again for your participation in our panel. We appreciate
your time and found the information that you provided extremely
helpful.
Contact:
If your response will be delayed or if you have any questions, please
contact Ashanta Williams at (202) 512-5110 (williamsa@agao.gov) or
Kathy White at (202) 512-8512 (whitek@gao.gov).
Directions:
* Click the "Exit" button at the bottom of the screen if you wish to
save your responses and continue at a later time. Always use the
"Exit" button to close the survey to avoid losing data you have
entered.
Confidentiality of Responses:
We will not release individually identifiable information from this
survey while our work is in progress unless compelled by law or
required to do so by the Congress. Once the report is issued, we will
destroy the file that links names to individual responses. While the
results will generally be provided in summary form in our report,
individual answers may be discussed, but they will not include any
information that could be used to identify individual respondents.
Names of participants may be listed in our final report; however,
names will not be associated with responses. Furthermore, your
comments and answers will not be provided to other participants during
any stage in the process.
Prioritization for Future Work:
The questions in this section are based on respondents' answers to the
question:
In your opinion, what are the key employment and training issues that
ETA's research agenda should address?
1. Taking into account ETA's limited resources, how important, if at
all, is it for ETA to address the following in future research?
a. Long-term outcomes of Employment and Training Programs:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
b. Short-term outcomes of Employment and Training Programs:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
c. Value of various credentials:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
d. Impact of long-term and/or short-term training:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
e. Job creation strategies, such as providing wage subsidies, tax
credits, or public service employment:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
f. Employment and training approaches that work and for whom:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
g. One-stop center management and operations:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
h. Effective performance measurement systems:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
i. Linkages between the public workforce system and economic
development entities:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
j. Needs of different labor markets or industries:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
k. Employment and training strategies for various economic conditions:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
1. Issues related to Unemployment Insurance:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
2. Among the areas you checked at least moderately important in
question 1, which would you rank as the top 3 areas that ETA should
address in future research. (Please rank only 3 areas, with 1 as your
top priority area, 2 as your second highest area, etc.)
a. Long-term outcomes of Employment and Training Programs:
Rank:
b. Short-term outcomes of Employment and Training Programs:
Rank:
c. Value of various credentials:
Rank:
d. Impact of long-term and/or short-term training:
Rank:
e. Job creation strategies, such as providing wage subsidies, tax
credits, or public service employment:
Rank:
f. Employment and training approaches that work and for whom:
Rank:
g. One-stop center management and operations:
Rank:
h. Effective performance measurement systems:
Rank:
i. Linkages between the public workforce system and economic
development entities:
Rank:
j. Needs of different labor markets or industries:
Rank:
k. Employment and training strategies for various economic conditions:
Rank:
I. Issues related to Unemployment Insurance:
Rank:
3. How important, if at all, would it be for ETA to fund research that
focuses on what works for the following populations given its resource
constraints?
a. Short-term unemployed:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
b. Long-term unemployed:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
c. Dislocated workers:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
d. Older workers:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
e. Veterans:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
f. Economically disadvantaged workers:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
g. Racial and ethnic minorities:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
h. Immigrants:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
i. Adults with low basic skills:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
j. Workers with physical and mental disabilities:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
k. In-school youth (ages 14 - 21):
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
I. Out-of-school youth:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
m. Ex-offenders:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
4. Among the populations you checked as least moderately important in
question 3, which would you rank as the top 5 populations on which ETA
should fund research in the future, please rank only 5 populations,
with 1 as your top population, 2 as your second highest population,
etc.)
a. Short-term unemployed:
Rank:
b. Long-term unemployed:
Rank:
c. Dislocated workers:
Rank:
d. Older workers:
Rank:
e. Veterans:
Rank:
f. Economically disadvantaged workers:
Rank:
g. Racial and ethnic minorities:
Rank:
h. Immigrants:
Rank:
i. Adults with low basic skills:
Rank:
j. Workers with physical and mental disabilities:
Rank:
k. In-school youth (ages 14 - 21):
Rank:
I. Out-of-school youth:
Rank:
m. Ex-offenders:
Rank:
5. How important are each of the following Unemployment Insurance (UI)
system issues for ETA to include in future research efforts?
a. Examination of the incentives and disincentives in the UI system:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
b. Modernization of UI system (such as extending coverage to include
additional income groups):
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
c. Benefits and costs of UI system in different business cycles:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
d. Accountability mechanisms in the UI system (such as data accuracy,
enforcement of work search requirement):
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
e. Linkage between UI and employment and training programs or safety-
net programs (such as SNAP, TANF):
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
6. How important, if at all, is it for ETA to evaluate the following
key employment or training programs (excluding Up?
a. WIA Adult:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
b. WIA Dislocated Worker:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
c. WIA Youth:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
d. Wagner-Peyser Employment Service:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
e. Job Corps:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
f. Trade Adjustment Assistance:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
g. Veterans' Employment and Training:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
h. Apprenticeship program:
Extremely important:
Very important:
Moderately important:
Somewhat important:
Not at all important:
No response:
7. Given a limited amount of resources, which three of the key
employment or training programs that you checked as at least
moderately important in question 6 should ETA address in future
research. Please rank only 3 employment or training programs, with 1
as your top program, 2 as your second highest program, etc.)
a. WIA Adult:
Rank:
b. WIA Dislocated Worker:
Rank:
c. WIA Youth:
Rank:
d. Wagner-Peyser Employment Service:
Rank:
e. Job Corps:
Rank:
f. Trade Adjustment Assistance:
Rank:
g. Veterans' Employment and Training:
Rank:
h. Apprenticeship program:
Rank:
8. How important, if at all, is it for ETA to integrate the following
into its future research proposals?
a. Randomized experimental design:
Very important:
Somewhat important:
Neither important nor unimportant:
Somewhat unimportant:
Very unimportant:
No response:
b. Quasi-experimental design:
Very important:
Somewhat important:
Neither important nor unimportant:
Somewhat unimportant:
Very unimportant:
No response:
c. Descriptive studies (including program implementation and
operations):
Very important:
Somewhat important:
Neither important nor unimportant:
Somewhat unimportant:
Very unimportant:
No response:
d. Analysis of administrative data for research purposes:
Very important:
Somewhat important:
Neither important nor unimportant:
Somewhat unimportant:
Very unimportant:
No response:
e. Meta-analyses (including literature reviews or quantitative
analyses of prior research):
Very important:
Somewhat important:
Neither important nor unimportant:
Somewhat unimportant:
Very unimportant:
No response:
f. Secondary analysis of existing research data:
Very important:
Somewhat important:
Neither important nor unimportant:
Somewhat unimportant:
Very unimportant:
No response:
g. Evaluation of pilots and demonstrations:
Very important:
Somewhat important:
Neither important nor unimportant:
Somewhat unimportant:
Very unimportant:
No response:
Prioritization of ETA's Research:
9. Overall, to what extent do you feel ETA's 2007-2012 research agenda
reflect key employment and training issues? (For a reminder of the
topics covered during the 2007-2012 research agenda click here.)
* Great extent;
* Moderate extent;
* Some extent;
* Little extent;
* Not at all;
* No response.
Policy and Practice:
The questions in this section are based on respondents' answers to the
following questions:
"In your opinion, how well does ETA's research inform policy and
practice? Please consider ongoing research studies, of which you are
familiar, during the past 5 years."
"What additional thoughts, if any, do you have about ETA's research or
research agenda?"
10. In the past 5 years, to what extent has ETA's research informed
employment and training policy?
* Great extent;
* Moderate extent;
* Some extent;
* Little extent;
* Not at all;
* Don't know;
* No response.
11. In the past 5 years, to what extent has ETA's research informed
state and local employment and training practices?
* Great extent;
* Moderate extent;
* Some extent;
* Little extent;
* Not at all;
* Don't know;
* No response.
ETA's Dissemination Methods:
The questions in this section are based on respondents' answers to the
following question:
"What dissemination methods do you think would allow ETA to
effectively share its research findings?"
12. How effective, if at all, do you think the following methods would
be for ETA to disseminate its research findings?
a. Searchable database of papers on ETA's website:
Extremely effective:
Very effective:
Moderately effective:
Somewhat effective:
Not at all effective:
No response:
b. Distributing a compendium of ETA's research:
Extremely effective:
Very effective:
Moderately effective:
Somewhat effective:
Not at all effective:
No response:
c. Distributing a one-page summary of the research findings from each
report:
Extremely effective:
Very effective:
Moderately effective:
Somewhat effective:
Not at all effective:
No response:
d. Email notifications announcing newly released research:
Extremely effective:
Very effective:
Moderately effective:
Somewhat effective:
Not at all effective:
No response:
e. Publishing articles in journals:
Extremely effective:
Very effective:
Moderately effective:
Somewhat effective:
Not at all effective:
No response:
f. Publicizing research findings through press articles, industry
publications, etc.
Extremely effective:
Very effective:
Moderately effective:
Somewhat effective:
Not at all effective:
No response:
g. Contracting with relevant organizations and associations to
disseminate research:
Extremely effective:
Very effective:
Moderately effective:
Somewhat effective:
Not at all effective:
No response:
h. Webinars:
Extremely effective:
Very effective:
Moderately effective:
Somewhat effective:
Not at all effective:
No response:
i. Videos or podcasts:
Extremely effective:
Very effective:
Moderately effective:
Somewhat effective:
Not at all effective:
No response:
j. Social media (such as Twitter):
Extremely effective:
Very effective:
Moderately effective:
Somewhat effective:
Not at all effective:
No response:
k. National ETA-sponsored conferences:
Extremely effective:
Very effective:
Moderately effective:
Somewhat effective:
Not at all effective:
No response:
l. Regional ETA-sponsored conferences:
Extremely effective:
Very effective:
Moderately effective:
Somewhat effective:
Not at all effective:
No response:
m. Presenting at outside conferences:
Extremely effective:
Very effective:
Moderately effective:
Somewhat effective:
Not at all effective:
No response:
n. Briefings at ETA for external audiences (including stakeholders and
policymakers):
Extremely effective:
Very effective:
Moderately effective:
Somewhat effective:
Not at all effective:
No response:
13. Given a limited amount of resources, which five of the
dissemination methods that you checked as at least moderately
effective in question 12 should ETA focus on for disseminating
research in the future. Please rank only 3 methods, with 1 as your top
method, 2 as your second highest method, etc.).
a. Searchable data base of papers on ETA's website:
Rank:
b. Distributing a compendium of ETA's research:
Rank:
c. Distributing a one-page summary of the research findings from each
report:
Rank:
d. Email notifications announcing newly released research:
Rank:
e. Publishing articles in journals:
Rank:
f. Publicizing research findings through press articles, industry
publications, etc.
Rank:
g. Contracting with relevant organizations and associations to
disseminate research:
Rank:
h. Webinars:
Rank:
i. Videos or podcasts:
Rank:
j. Social media (such as Twitter):
Rank:
k. National ETA-sponsored conferences:
Rank:
l. Regional ETA-sponsored conferences:
Rank:
m. Presenting at outside conferences:
Rank:
n. Briefings at ETA for external audiences (including stakeholders and
policymakers):
Rank:
14. Do you have recent experience with any of the following ETA
research dissemination methods? If yes, how satisfied or dissatisfied
were you with each of the following ETA research dissemination methods?
Do you have recent experience using any of the following ETA research
dissemination methods?
If yes, how satisfied or dissatisfied were you with the following ETA
research dissemination methods?
a. Searchable database of research papers on ETA's website:
Do you have recent experience using any of the following ETA research
dissemination methods?
Yes: (Go to next column);
No.
If yes, how satisfied or dissatisfied were you with the following ETA
research dissemination methods?
Very satisfied:
Satisfied:
Neither satisfied nor dissatisfied:
Dissatisfied:
Very dissatisfied.
b. ETA's Training and Employment Notice (TENs) e-mail list:
Do you have recent experience using any of the following ETA research
dissemination methods?
Yes: (Go to next column);
No.
If yes, how satisfied or dissatisfied were you with the following ETA
research dissemination methods?
Very satisfied:
Satisfied:
Neither satisfied nor dissatisfied:
Dissatisfied:
Very dissatisfied.
c. Webinars:
Do you have recent experience using any of the following ETA research
dissemination methods?
Yes: (Go to next column);
No.
If yes, how satisfied or dissatisfied were you with the following ETA
research dissemination methods?
Very satisfied:
Satisfied:
Neither satisfied nor dissatisfied:
Dissatisfied:
Very dissatisfied.
d. National ETA-sponsored conferences:
Do you have recent experience using any of the following ETA research
dissemination methods?
Yes: (Go to next column);
No.
If yes, how satisfied or dissatisfied were you with the following ETA
research dissemination methods?
Very satisfied:
Satisfied:
Neither satisfied nor dissatisfied:
Dissatisfied:
Very dissatisfied.
e. Regional ETA-sponsored conferences:
Do you have recent experience using any of the following ETA research
dissemination methods?
Yes: (Go to next column);
No.
If yes, how satisfied or dissatisfied were you with the following ETA
research dissemination methods?
Very satisfied:
Satisfied:
Neither satisfied nor dissatisfied:
Dissatisfied:
Very dissatisfied.
f. Briefings at ETA for external audiences (including stakeholders and
policymakers):
Do you have recent experience using any of the following ETA research
dissemination methods?
Yes: (Go to next column);
No.
If yes, how satisfied or dissatisfied were you with the following ETA
research dissemination methods?
Very satisfied:
Satisfied:
Neither satisfied nor dissatisfied:
Dissatisfied:
Very dissatisfied.
15. Are there any additional thoughts you would like to share with us
about ETA's research and dissemination practices?
Demographics:
16. Which of the following best describe your recent professional
experience? (CHECK ALL THAT APPLY)
* Researcher
* Professor/academic
* Consultant;
* State Official;
* Local official;
* Other;
* No response.
17. Are you ready to submit your final completed survey to GAO? (This
is equivalent to mailing a completed paper survey to us. It tells us
that your answers are official and final.)
* Yes, my survey is complete - To submit your final responses, please
click on "Exit" below.
* No, my survey is not yet complete -To save your responses for later,
please click on "Exit" below.
You may view and print your completed survey by clicking on the
Summary link in the menu to the left.
Print:
Submit:
[End of section]
Appendix VI: Experts Who Agreed to Participate in GAO's Delphi Panel:
Expert: Burt S. Barnow;
Affiliation: Trachtenberg School of Public Policy and Public
Administration, George Washington University.
Expert: Jon Baron;
Affiliation: Coalition for Evidence-Based Policy.
Expert: Stephen Bell;
Affiliation: Abt Associates Inc.
Expert: Jacob Benus;
Affiliation: IMPAQ International, LLC.
Expert: Dan Bloom;
Affiliation: MDRC.
Expert: Gary Burtless;
Affiliation: Brookings Institution.
Expert: Paul Decker;
Affiliation: Mathematica Policy Research, Inc.
Expert: Randall Eberts;
Affiliation: W.E. Upjohn Institute for Employment Research.
Expert: Richard Freeman;
Affiliation: Harvard University;
and;
National Bureau of Economic Research.
Expert: Robert Giloth;
Affiliation: Annie E. Casey Foundation.
Expert: David Heaney;
Affiliation: Maximus.
Expert: Carolyn J. Heinrich;
Affiliation: Robert M. LaFollette School of Public Affairs, University
of Wisconsin-Madison.
Expert: Kevin Hollenbeck;
Affiliation: W.E. Upjohn Institute for Employment Research.
Expert: Louis Jacobson;
Affiliation: New Horizons Economic Research.
Expert: Richard Kazis;
Affiliation: Jobs for the Future.
Expert: Jacob Alex Klerman;
Affiliation: Abt Associates Inc.
Expert: Kathy Krepcio;
Affiliation: John J. Heldrich Center for Workforce Development,
Rutgers, The State University of New Jersey.
Expert: Robert L. Lerman;
Affiliation: Department of Economics, College of Arts and Sciences,
American University.
Expert: Alberto Martini;
Affiliation: Progetto Valutazione.
Expert: Sheena McConnell;
Affiliation: Mathematica Policy Research, Inc.
Expert: Rick McGahey;
Affiliation: Ford Foundation.
Expert: Peter Mueser;
Affiliation: Department of Economics, University of Missouri-Columbia.
Expert: Lee Munnich;
Affiliation: Humphrey School of Public Affairs, University of
Minnesota.
Expert: Burke Murphy;
Affiliation: Minnesota Department of Employment and Economic
Development.
Expert: Sigurd R. Nilsen;
Affiliation: Formerly of GAO.
Expert: Demetra Smith Nightingale;
Affiliation: Urban Institute.
Expert: Christopher O'Leary;
Affiliation: W.E. Upjohn Institute for Employment Research.
Expert: Marion Pines;
Affiliation: Institute for Policy Studies, Johns Hopkins University.
Expert: James Riccio;
Affiliation: MDRC.
Expert: Neil Ridley;
Affiliation: Center for Law and Social Policy.
Expert: Howard Rosen;
Affiliation: Peterson Institute for International Economics.
Expert: Peter Z. Schochet;
Affiliation: Mathematica Policy Research, Inc.
Expert: Kenneth R. Troske;
Affiliation: University of Kentucky, Department of Economics.
Expert: Jason Turner;
Affiliation: Heritage Foundation.
Expert: John Twomey;
Affiliation: New York Association of Training and Employment
Professionals.
Expert: Ray Uhalde;
Affiliation: Job for the Future.
Expert: Carl Van Horn;
Affiliation: John J. Heldrich Center for Workforce Development,
Rutgers, The State University of New Jersey.
Expert: John W. Wallace;
Affiliation: Formerly of MDRC.
Expert: Jeffrey B. Wenger;
Affiliation: Department of Public Administration and Policy, School of
Public and International Affairs, University of Georgia.
Expert: Michael Wiseman;
Affiliation: George Washington Institute of Public Policy, George
Washington University.
Expert: Steve A. Woodbury;
Affiliation: Department of Economics, Michigan State University;
and; W.E. Upjohn Institute for Employment Research.
Source: GAO.
[End of table]
[End of section]
Appendix VII: Comments from the Department of Labor:
U.S. Department of Labor:
Assistant Secretary for Employment and Training:
Washington, D.C. 20210
March 7, 2011:
Mr. George Scott:
Director, Education, Workforce and Income Security Issues:
U.S. Government Accountability Office:
441 G. Street, N.W.
Washington, D.C. 20548:
Dear Mr. Scott:
Thank you for the opportunity to review the Government Accountability
Office (GAO) draft report entitled: "Employment and Training
Administration: More Actions Needed to Improve Transparency and
Accountability of Its Research Program." We continue to look for ways
to strengthening the Employment and Training Administration's (ETA)
research program, we appreciate the information provided in the report
and the opportunity to comment.
The Department of Labor (DOL) agrees with all three recommendations.
GAO Recommendation 1:
Formally incorporate into its research process the routine involvement
of Chief Evaluation Office at key milestones, including at the
development of ETA's annual research agenda and spending priorities,
as well as at the early stages of developing specific research
projects.
DOL Response: The Department agrees with this recommendation and is in
the process of formally involving the Chief Evaluation Officer (CEO)
into its research process. In fact, ETA has worked closely with the
CEO since its establishment in spring 2010. Before setting formal
monthly meetings that began in fall 2010, ETA routinely sought
involvement of CEO to discuss research, demonstration projects, and
evaluations from the early development stages. Topics covered include
implications of research findings, strategic planning, and project and
evaluation development of the Green Jobs High Growth Health Care
Impact Evaluation, the Young Parents Demonstration and Impact
Evaluation, the Enhanced Transitional Jobs Demonstrations and Random
Assignment Evaluation, and the Unemployment Insurance-related American
Recovery and Reinvestment Act studies.
Currently, ETA also routinely involves the CEO in:
1) Reviewing research and demonstration statements of work;
2) Discussing research and evaluation methodologies with ETA and the
Office of Management and Budget and periodically, with other federal
agencies;
3) Discussing, on a monthly basis, on-going ETA research and
evaluation projects;
4) Reviewing research and evaluation design reports, interim reports
and final reports prior to their dissemination;
5) Participating in meetings with grantees involved in major research
demonstrations to help explain the importance of rigorous evaluations;
and;
6) Providing input on how to best incorporate research in ETA's
Solicitations for Grant Applications (SGAs) by strengthening the
evaluation and report sections of the SGAs and crafting language to
encourage applicants' use of evidence-based practices.
Going forward, ETA plans to continue its collaboration with the CEO
and involve the CEO in earlier stages of the research development
process, including meetings to discuss potential research,
demonstration and evaluation projects.
GAO Recommendation 2:
Develop a mechanism to enhance the transparency and accountability of
ETA's research program. For example, involve advisory bodies or other
entities outside ETA, in efforts to develop research policies and
processes.
DOL Response: The Department agrees with this recommendation and ETA
already has taken several steps to engage outside experts more fully
to enhance the transparency and accountability of its research
program. Specifically, ETA is: (1) including outside experts in the
development of its five-year research plan; (2) forming advisory and
peer review groups for major evaluations, (3) developing, with the
Coalition for Evidence-Based Policy, the capability to conduct peer
reviews on an ad hoc basis, and (4) having the CEO, who is in the
Department but independent of ETA, also review reports.
In addition, ETA is collecting and synthesizing feedback from a
variety of sources for developing its five-year research plan
including: (1) input received from experts and workforce system
practitioners at ETA's conferences, and (2) feedback solicited from
stakeholders in the workforce community. ETA also will analyze public
comments that it will solicit when the draft plan is announced in the
Federal Register and posted on the ETA Web site. ETA has strengthened
its collaboration with other Federal agency experts at the U.S.
Departments of Education and Health and Human Services in planning and
carrying out research and evaluation studies.
Moving forward, ETA plans to continue engaging outside experts in
developing research policies and processes.
GAO Recommendation 3:
Develop a formal plan for ensuring that ETA's research products are
easily accessible to stakeholders and to the general public through
its searchable database. Such a plan could involve requiring labor to
assess overall effectiveness of its Web based search page, including
user satisfaction with search features.
DOL Response: The Department agrees with this recommendation, and ETA
has taken steps to establish specific processes for disseminating
research and evaluation findings in an effort to ensure that its
research products are easily accessible to stakeholders and to
the general public. Specifically, ETA has improved the web-based
publication search engine, and featured research and evaluation
findings at agency-sponsored or other Federally-sponsored conferences.
ETA supports continuous improvement of its dissemination efforts to
ensure public accessibility. For example, last summer ETA upgraded the
web-based research database to improve its overall effectiveness and
user satisfaction by implementing the Google Search on all doleta.gov
sites and developing an interface between the Google Search Appliance
and ETA Research Publication Database. Since the improvements cited in
GAO's report, additional improvements have been made; when a keyword
search is performed in the search box, located in the top navigation
bar on doleta.gov, it returns a list of relevant publications housed
in the ETA Research Publication Database.
As part of ETA's plan to continue improving its dissemination
practices, we have linked to the customer satisfaction survey form
provided at the bottom of all Departmental Web pages. This survey
provides feedback from visitors to help ensure that the Web site is
useful, accurate and complete. ETA's Web Services team monitors the
system log daily and takes immediate action necessary to rectify any
issues. The Web Services team will periodically obtain feedback on
operation of the Research Publication Database and will gather new
requirements for further cost-effective enhancements that will be
specified in an improvement plan.
Additional Remarks:
The Department believes that without further clarifications on the
agency pilots, demonstration and research budget, the draft report
would be misleading as to the funds available for undesignated pilots,
demonstrations and research. We would encourage you to consider the
Department's technical comments on the draft report, enclosed for your
reference.
Again, thank you for the opportunity to review the draft report. If
you would like additional information, please do not hesitate to call
me at (202) 693-2700.
Sincerely,
Signed by:
Jane Oates:
Assistant Secretary:
Enclosure:
[End of section]
Appendix VIII: GAO Contact and Staff Acknowledgments:
Contact:
George A. Scott, (202) 512-7215 or scottg@gao.gov:
Staff Acknowledgments:
In addition to the contact listed above, Dianne Blank, Assistant
Director, and Kathleen White, analyst-in-charge, managed all phases of
the engagement. Ashanta Williams assisted in managing many aspects of
the work and was responsible for final report preparation. Lucas
Alvarez and Benjamin Collins made significant contributions to all
aspects of this report. In addition, Amanda Miller assisted with study
and questionnaire design; Joanna Chan performed the data analysis;
Stephanie Shipman advised on evaluation approaches; James Bennett
provided graphics assistance; David Chrisinger provided writing
assistance; Alex Galuten and Sheila McCoy provided legal support; and
Sheranda Campbell and Ryan Siegel verified our findings.
[End of section]
GAO Related Products:
Program Evaluation: Experienced Agencies Follow a Similar Model For
Prioritizing Research. [hyperlink,
http://www.gao.gov/products/GAO-11-176]. Washington, D.C.: January 14,
2011.
Employment and Training Administration: Increased Authority and
Accountability Could Improve Research Program. [hyperlink,
http://www.gao.gov/products/GAO-10-243]. Washington, D.C.: January 29,
2010.
Workforce Investment Act: Labor Has Made Progress in Addressing Areas
of Concern, but More Focus Needed on Understanding What Works and What
Doesn't. [hyperlink, http://www.gao.gov/products/GAO-09-396T].
Washington, D.C.: February 26, 2009.
Employment and Training Program Grants: Evaluating Impacts and
Enhanced Monitoring Would Improve Accountability. [hyperlink,
http://www.gao.gov/products/GAO-08-486]. Washington, D.C.: May 7, 2008.
Federal Research: Policies Guiding the Dissemination of Scientific
Research from Selected Agencies Should Be Clarified and Better
Communicated. [hyperlink, http://www.gao.gov/products/GAO-07-653].
Washington, D.C.: May 17, 2007.
Data Quality: Expanded Use of Key Dissemination Practices Would
Further Safeguard the Integrity of Federal Statistical Data.
[hyperlink, http://www.gao.gov/products/GAO-06-607]. Washington, D.C.:
May 31, 2006.
Workforce Investment Act: Substantial Funds Are Used for Training, but
Little Is Known Nationally about Training Outcomes. [hyperlink,
http://www.gao.gov/products/GAO-05-650]. Washington, D.C.: June 29,
2005.
Program Evaluation: An Evaluation Culture and Collaborative
Partnerships Help Build Agency Capacity. [hyperlink,
http://www.gao.gov/products/GAO-03-454]. Washington, D.C.: May 2, 2003.
Workforce Investment Act: Improvements Needed in Performance Measures
to Provide a More Accurate Picture of WIA's Effectiveness. [hyperlink,
http://www.gao.gov/products/GAO-02-275]. Washington, D.C.: February 1,
2002.
[End of section]
Footnotes:
[1] A program year begins on July 1 of a year and ends on June 30 of
the following year. Thus, program year 2010 began on July 1, 2010, and
ends on June 30, 2011. Of the $103 million appropriated for program
year 2010, Congress designated about $84 million for specific
projects, including $30 million for Transitional Jobs activities and
$5.5 million for the employment and training needs of young parents.
The remainder of the funds, approximately $18.7 million, was available
to ETA for undesignated pilots, demonstrations, and research.
[2] GAO, Workforce Investment Act: Labor Has Made Progress in
Addressing Areas of Concern, but More Focus Needed on Understanding
What Works and What Doesn't, [hyperlink,
http://www.gao.gov/products/GAO-09-396T] (Washington, D.C.: Feb. 26,
2009); Employment and Training Program Grants: Evaluating Impact and
Enhancing Monitoring Would Improve Accountability, [hyperlink,
http://www.gao.gov/products/GAO-08-486] (Washington, D.C.: May 7,
2008); and Employment and Training Program Grants: Labor Has Outlined
Steps for Additional Documentation and Monitoring but Assessing Impact
Still Remains an Issue, [hyperlink,
http://www.gao.gov/products/GAO-08-1140T] (Washington, D.C.: Sept. 23,
2008).
[3] GAO, Employment and Training Administration: Increased Authority
and Accountability Could Improve Research Program, [hyperlink,
http://www.gao.gov/products/GAO-10-243] (Washington, D.C.: Jan. 29,
2010).
[4] 29 U.S.C. § 2916(a).
[5] While ETA's program year 2010 began on July 1, 2010, ETA's funding
for pilots, demonstrations, and research was available starting on
April 1, 2010.
[6] American Evaluation Association, An Evaluation Roadmap for a More
Effective Government, (September 2010), available at [hyperlink,
www.eval.org/eptf.asp].
[7] Congress enacted WIA in part to increase employment, retention,
earnings, and occupational skill attainment of participants, thus
improving the quality of the workforce. WIA Adult primarily focuses on
disadvantaged adults and WIA Dislocated Workers focuses largely on
workers who have been laid off or have been notified that they will be
laid off. WIA Youth helps to prepare youth for employment and/or
postsecondary education through linkages between academic and
occupational learning.
[8] See [hyperlink, http://www.gao.gov/products/GAO-08-486].
[9] ETA now has a contract for $1 million with the firm that performed
the original GATE I evaluation to perform a 5-year follow-up to assess
long-term outcomes for Project GATE II.
[10] These studies should include designs that randomly assign
individuals to groups that receive enhanced program services and to
groups that do not.
[11] GAO, Program Evaluation: A Variety of Rigorous Methods Can Help
Identify Effective Interventions, [hyperlink,
http://www.gao.gov/products/GAO-10-30] (Washington, D.C.: Nov. 23,
2009).
[12] See [hyperlink, http://www.gao.gov/products/GAO-10-243].
[13] See [hyperlink, http://www.gao.gov/products/GAO-10-243].
[14] For more information on the criteria several other federal
agencies use to prioritize their research, see GAO, Program
Evaluation: Experienced Agencies Follow a Similar Model for
Prioritizing Research, [hyperlink,
http://www.gao.gov/products/GAO-11-176] (Washington, D.C.: Jan. 14,
2011).
[15] While ETA has a provision that allows contractors to publicly
release reports that have not been approved by the Assistant Secretary
after 9 months, researchers may be reluctant to do so out of concern
that this action may damage their credibility with the agency and
limit their ability to win future research contracts.
[16] See [hyperlink, http://www.gao.gov/products/GAO-10-243].
[17] A fourth study, the Impact Evaluation of the Senior Community
Service Employment Program, was also designed as a random assignment
study. We omitted it from our analysis because ETA was reconsidering
the funding of this evaluation at the time of our report.
[18] See [hyperlink, http://www.gao.gov/products/GAO-10-30].
[19] 29 U.S.C. § 2917(c).
[20] In addition to OMB's review, the evaluation contractor also
conducted a peer review of the study design.
[21] [hyperlink, http://www.gao.gov/products/GAO-10-243].
[22] See Interagency Committee on Government Information, Recommended
Policies and Guidelines for Federal Public Websites (2004). The
Interagency Committee on Government Information was established by OMB
in 2003 to improve the usability of federal public Web sites. In
particular, they recommend that "organizations should provide help,
hints, or tips, and include examples" for search users.
[23] The Delphi method, developed by the RAND Corporation in the
1950s, is most commonly applied in a group-discussion forum. We
modified the approach to have the group discussion take place in the
form of a Web-based forum.
[24] Labor awarded about $9.9 million to various entities to carry out
its Young Parents Demonstration for fiscal years 2008 and 2009.
Lancaster County Workforce Investment Board was 1 of 13 grantees that
received funding.
[25] In June 2008, ETA awarded GATE II grants to four states--one of
which was to Virginia--for the extension of the GATE model for helping
selected dislocated workers create their own business.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
