Employment and Training Administration
Increased Authority and Accountability Could Improve Research Program Gao ID: GAO-10-243 January 29, 2010With current rising unemployment rates and the need for a more skilled workforce, it is important for the Department of Labor's (Labor) Employment and Training Administration (ETA) to invest in sound research that identifies the most effective and efficient ways to train and employ workers for 21st century jobs. While ETA traditionally has played an important role in providing job training, employment assistance, and labor market information for the nation's workers, the current unemployment crisis has made this role more critical. The Bureau of Labor Statistics estimates that jobless rates have increased over the past year in all 50 states and the District of Columbia. In particular, several states have reported unemployment rates well over 10 percent. As ETA's new leadership works to help the nation meet these economic challenges, it must have solid information that is supported by sound research to guide decision-making. Since 2002, GAO and others have criticized ETA for not focusing sufficient attention on its research program, particularly with regard to complying with congressional mandates, conducting policy-relevant research, and disseminating key research findings in a timely way. In this context, we have examined the structure and processes of ETA's research and evaluation center in terms of the elements that leading national research organizations cite as essential to a sound program; that is, research independence, transparency and accountability, and policy relevance. Based on these elements, we addressed the following questions: (1) How does ETA's organizational structure provide for research independence? (2) What steps has ETA taken to promote transparency and accountability in its research program? (3) How does ETA ensure that its research is relevant to workforce development policy and practice?
In summary, we found that ETA's research center lacks independent authority for research, has limitations with regard to transparency and accountability processes, has not routinely involved stakeholders in developing its research agenda, and has been slow to evaluate the programs and activities carried out under the Workforce Investment Act of 1998 (WIA). (1) ETA is currently in the process of revising its organizational structure. Previously, its research and evaluation center, within the Office of Policy Development and Research (OPDR), was several levels removed from the Office of the Assistant Secretary and lacked independent authority for conducting research. Under ETA's revised structure, it is unclear whether OPDR will report directly to the Assistant Secretary or the Deputy Assistant Secretary. Furthermore, unlike the heads of some other research and evaluation centers, the head of ETA's research and evaluation center does not have the authority to set the research agenda, to approve requests for funded projects, or to disseminate research and evaluation reports. (2) While ETA has recently made improvements to its research program, some limitations remain with regard to its accountability processes and timely dissemination of research products. In 2007, ETA documented all stages of its research process from project selection to dissemination. ETA also began coordinating its research process with the Office of Management and Budget (OMB) and renewed its practice of sponsoring biennial research conferences to address a range of policy and program issues. Despite these recent changes, ETA's research processes lack specificity, including specific time frames for key milestones and an established criteria for peer review. In addition, ETA's research program lacks an information system to track its research from completion to dissemination and does not have a mechanism to ensure that research findings are disseminated in a timely manner. In 2008, ETA disseminated 34 research products to the public. However, 20 of these products, which had a combined cost of about $28 million, were delayed for between 2 and 5 years. (3) Labor, consistent with recent efforts of OMB, has taken steps to emphasize the value of research and evaluation, but ETA does not have sufficient mechanisms to ensure that its research can inform policy decisions. OMB currently has efforts underway to help agencies to enhance their research and evaluation programs. Labor also announced plans to create a position of chief evaluation officer and a new evaluation center within the Office of the Assistant Secretary for Policy to foster research relevant to policy. However, ETA has been slow to comply with a congressional mandate to evaluate WIA1 and it also lacks a standard process and advisory bodies to consistently involve stakeholders in the development of its research agenda. Similarly, ETA has no formal process in place to ensure that research findings are used to inform strategic planning and policy.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-10-243, Employment and Training Administration: Increased Authority and Accountability Could Improve Research Program
This is the accessible text file for GAO report number GAO-10-243
entitled 'Employment and Training Administration: Increased Authority
and Accountability Could Improve Research Program' which was released
on January 29, 2010.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Committees:
United States Government Accountability Office:
GAO:
January 2010:
Employment and Training Administration:
Increased Authority and Accountability Could Improve Research Program:
GAO-10-243:
Contents:
Letter:
Appendix I: Briefing Slides:
Appendix II: Dissemination Time Frames for ETA's Research Products
Published in 2008:
Appendix III: Comments from the Department of Labor:
Appendix IV: GAO Contact and Staff Acknowledgments:
Related GAO Products:
Table:
Table 1: ETA's 2008 Research Products by Dissemination Time Frames:
Abbreviations:
EPA: Environmental Protection Agency:
ERS: Economic Research Service:
ETA: Employment and Training Administration:
HUD: Department of Housing and Urban Development:
IES: Institute of Education Sciences:
NSF: National Science Foundation:
OMB: Office of Management and Budget:
OPDR: Office of Policy Development and Research:
USDA: U.S. Department of Agriculture:
WIA: Workforce Investment Act of 1998:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
January 29, 2010:
The Honorable Tom Harkin:
Chairman:
The Honorable Michael B. Enzi:
Ranking Member:
Committee on Health, Education, Labor, and Pensions:
United States Senate:
The Honorable Patty Murray:
Chairman:
The Honorable Johnny Isakson:
Ranking Member:
Subcommittee on Employment and Workplace Safety:
Committee on Health, Education, Labor, and Pensions:
United States Senate:
The Honorable Tom Harkin:
Chairman:
Subcommittee on Labor, Health and Human Services, Education and
Related Agencies:
Committee on Appropriations:
United States Senate:
With current rising unemployment rates and the need for a more skilled
workforce, it is important for the Department of Labor's (Labor)
Employment and Training Administration (ETA) to invest in sound
research that identifies the most effective and efficient ways to
train and employ workers for 21st century jobs. While ETA
traditionally has played an important role in providing job training,
employment assistance, and labor market information for the nation's
workers, the current unemployment crisis has made this role more
critical. The Bureau of Labor Statistics estimates that jobless rates
have increased over the past year in all 50 states and the District of
Columbia. In particular, several states have reported unemployment
rates well over 10 percent.
As ETA's new leadership works to help the nation meet these economic
challenges, it must have solid information that is supported by sound
research to guide decision-making. Since 2002, GAO and others have
criticized ETA for not focusing sufficient attention on its research
program, particularly with regard to complying with congressional
mandates, conducting policy-relevant research, and disseminating key
research findings in a timely way. In this context, we have examined
the structure and processes of ETA's research and evaluation center in
terms of the elements that leading national research organizations
cite as essential to a sound program; that is, research independence,
transparency and accountability, and policy relevance. Based on these
elements, we addressed the following questions: (1) How does ETA's
organizational structure provide for research independence? (2) What
steps has ETA taken to promote transparency and accountability in its
research program? (3) How does ETA ensure that its research is
relevant to workforce development policy and practice?
On December 3, 2009, we briefed your staff on the results of our
analysis. This report formally conveys the information provided during
this briefing (see app. I for the briefing slides). In summary, we
found that ETA's research center lacks independent authority for
research, has limitations with regard to transparency and
accountability processes, has not routinely involved stakeholders in
developing its research agenda, and has been slow to evaluate the
programs and activities carried out under the Workforce Investment Act
of 1998 (WIA).
* ETA is currently in the process of revising its organizational
structure. Previously, its research and evaluation center, within the
Office of Policy Development and Research (OPDR), was several levels
removed from the Office of the Assistant Secretary and lacked
independent authority for conducting research. Under ETA's revised
structure, it is unclear whether OPDR will report directly to the
Assistant Secretary or the Deputy Assistant Secretary. Furthermore,
unlike the heads of some other research and evaluation centers, the
head of ETA's research and evaluation center does not have the
authority to set the research agenda, to approve requests for funded
projects, or to disseminate research and evaluation reports.
* While ETA has recently made improvements to its research program,
some limitations remain with regard to its accountability processes
and timely dissemination of research products. In 2007, ETA documented
all stages of its research process from project selection to
dissemination. ETA also began coordinating its research process with
the Office of Management and Budget (OMB) and renewed its practice of
sponsoring biennial research conferences to address a range of policy
and program issues. Despite these recent changes, ETA's research
processes lack specificity, including specific time frames for key
milestones and an established criteria for peer review. In addition,
ETA's research program lacks an information system to track its
research from completion to dissemination and does not have a
mechanism to ensure that research findings are disseminated in a
timely manner. In 2008, ETA disseminated 34 research products to the
public. However, 20 of these products, which had a combined cost of
about $28 million, were delayed for between 2 and 5 years.
* Labor, consistent with recent efforts of OMB, has taken steps to
emphasize the value of research and evaluation, but ETA does not have
sufficient mechanisms to ensure that its research can inform policy
decisions. OMB currently has efforts underway to help agencies to
enhance their research and evaluation programs. Labor also announced
plans to create a position of chief evaluation officer and a new
evaluation center within the Office of the Assistant Secretary for
Policy to foster research relevant to policy. However, ETA has been
slow to comply with a congressional mandate to evaluate WIA[Footnote
1] and it also lacks a standard process and advisory bodies to
consistently involve stakeholders in the development of its research
agenda. Similarly, ETA has no formal process in place to ensure that
research findings are used to inform strategic planning and policy.
GAO is making several recommendations concerning the structure and
processes of ETA's research and evaluation center. Specifically, GAO
recommends that the Secretary of Labor:
* take steps to ensure that ETA clarifies its organizational structure
and OPDR reports directly to ETA's Assistant Secretary;
* provide sufficient decision-making authority to ETA's research and
evaluation center regarding its research;
* direct ETA's research and evaluation center to establish more
specific processes, including time frames for dissemination of
research;
* create an information system to track research projects; and:
* instruct ETA's research and evaluation center to develop processes
to involve outside experts in setting its research agenda.
We used several methodologies to develop our findings. To identify
elements of sound government-sponsored research, we reviewed relevant
guidelines established by the American Evaluation Association and the
National Research Council of the National Academy of Sciences. To
apply these guidelines to ETA's research and evaluation center, we
examined ETA's organizational structure and processes in terms of
three basic elements of sound research--independence, transparency and
accountability, and policy relevance. As part of this examination, we
analyzed information on ETA's disseminated research to determine
whether research products were released in a timely manner. (App. II
contains additional information regarding the time frames of these
research products.) In addition, we reviewed the research structures
and processes of five other federal agencies' research and evaluation
centers that had some of the characteristics that were identified by
the American Evaluation Association and the National Research Council
of the National Academy of Sciences to support sound research.
[Footnote 2] We did not evaluate the operation or quality of the
research produced by the centers but rather the structures and
policies supporting their research. We also reviewed relevant federal
laws.
We conducted this performance audit from October 2009 through December
2009 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on our audit objectives.
We believe that the evidence obtained provides a reasonable basis for
our findings and conclusions based on our audit objectives.
We provided a draft of this report to ETA for its review and comment.
ETA provided written comments, which are reproduced in appendix III.
In addition, ETA provided technical comments, which we incorporated
where appropriate.
In its response, ETA concurred with two of our recommendations and
cited ongoing activities addressing the areas covered in the other
three. Regarding our recommendation to clarify its organizational
chart to ensure that OPDR reports directly to ETA's Assistant
Secretary, ETA concurred and has since clarified this point in an
updated organizational chart that it has posted on its Web site. ETA
also concurred with our recommendation to create an information-
tracking system for its research products, noting that it has plans to
implement such a system to track product milestones. However,
officials did not provide specific time frames for implementation.
Because of the importance of developing such a system, we urge ETA to
give this effort high priority.
On the remaining three recommendations, ETA did not agree or disagree,
noting instead its relevant ongoing activities.
* Regarding our recommendation that ETA provide its research and
evaluation center with sufficient decision-making authority, officials
cited ongoing efforts to provide recommendations to the Assistant
Secretary and to collaborate with relevant program offices on ETA's
research plans. We considered these activities in our review and
discussed them in our report, but these efforts alone fall short of
providing OPDR with the necessary authority to make key decisions, and
they do not serve to insulate OPDR from undue political influence.
* Regarding our recommendation that ETA establish more specific
processes to guide research, officials cited their work with OMB to
establish time frames for disseminating research. However, specifying
time frames is but one component of ETA's research process that needs
to be addressed. Our report cites other steps that ETA should take
including specifying the criteria to be used in deciding whether a
report should be peer reviewed. In ETA's technical comments, officials
noted that they will be updating their process steps in early 2010. As
ETA reviews and updates its processes, we urge the agency to make the
steps more specific in order to help ensure transparency and
accountability.
* Regarding our recommendation to develop processes to routinely
involve outside experts in setting its research agenda, ETA officials
cited the various informal methods they currently use to gather input
from outside experts. These efforts may serve to increase outside
involvement in setting the agenda, but they are informal and ad hoc
and, therefore, do not provide ETA with a formal mechanism to ensure
outside experts--including researchers, policy makers, and
practitioners--consistently have a role in setting its research agenda.
We are sending copies of this report to relevant congressional:
committees. In addition, this report will be available at no charge on
GAO's Web site at [hyperlink, http://www.gao.gov].
If you or your staff have any questions about this report, please
contact me at (202) 512-7215 or scottg@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. GAO staff members who made key
contributions to this report are listed in appendix IV.
Signed by:
George A. Scott:
Director, Education, Workforce, and Income Security Issues:
[End of section]
Appendix I: Briefing Slides:
Employment and Training Administration: Increased Authority and
Accountability Could Improve Research Program:
Briefing to Congressional Committees:
December 3, 2009:
Overview:
* Introduction:
* Research Objectives:
* Scope and Methodology:
* Summary of Findings:
* Background:
* Findings:
* Conclusions and Recommendations:
Introduction:
A sound program of research and evaluation is essential to finding
effective ways for training and employing workers for 21st century
jobs.
Since 2002, we and others have criticized the Employment and Training
Administration (ETA) for not focusing sufficient attention on its
research program, particularly with regard to:
* complying with statutory research mandates;
* conducting research to help policy makers understand what works and
what does not; and;
* disseminating key research findings in a timely way.
* Most recently, we have been asked to review the extent to which
ETA‘s structure and processes promote sound research;
* At the time of our review, ETA was transitioning from one
administration to another, and;
* ETA‘s new leadership is formulating new plans on how best to carry
out its research and evaluations.
Leading national research organizations, including the American
Evaluation Association and the National Research Council of the
National Academy of Sciences”have identified a number of elements of a
sound research and evaluation program.
Three of the elements of sound research are:
* Research Independence;
* Transparency and Accountability;
* Policy Relevance.
[End of Introduction]
Objectives:
Based on these elements, we examined ETA‘s research program to answer
the following questions:
1. How does ETA‘s organizational structure provide for research
independence?
2. What steps has ETA taken to promote transparency and accountability
in its research program?
3. How does ETA ensure that its research is relevant to workforce
development policy and practice?
[End of Objectives]
Scope and Methodology:
To identify basic principles of sound government-sponsored research,
we reviewed relevant guidelines as developed by the American
Evaluation Association and the National Research Council of the
National Academy of Sciences.
These guidelines are suitably applicable to a full range of government
research and evaluation centers.
To examine ETA‘s organizational structure and processes, we:
* Interviewed agency officials, stakeholders, and experts.
* Examined agency documents, policies, and procedures.
To examine the structure and processes of other government research
and evaluation centers, we:
* Interviewed research experts.
* Examined agencies‘ documents and external reports.
* Obtained information from GAO officials with knowledge of these
centers.
In addition to ETA, we present information for comparative purposes on
the following research centers:
* The Department of Agriculture‘s (USDA) Economic Research
Service(ERS) and National Institute of Food and Agriculture;
* The Department of Education‘s (Education) Institute of Education
Sciences (IES);
* The Department of Housing and Urban Development‘s (HUD) Office of
Policy Development and Research;
* The Environmental Protection Agency‘s (EPA) Office of Research and
Development; and;
* The National Science Foundation (NSF).
We did not evaluate the operation or quality of the research produced
by the centers but rather the structures and policies supporting their
research.
We selected these centers as examples because they had some of the
characteristics that support sound research as identified by the
American Evaluation Association and the National Research Council of
the National Academy of Sciences guidelines.
We conducted this performance audit from October 2009 to December 2009
in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives.
Key Terminology:
* Governing or advisory board: This term generally refers to an
independent body that provides advice to senior leadership on a range
of issues, including recommending research priorities, helping to
safeguard the independence of the agency, and providing critical links
to practice and policy communities.
* Peer review: A quality assurance process that involves selecting
independent and knowledgeable individuals to review and provide
objective feedback on an agency‘s research.
* Research and evaluation center: The entity within an agency
responsible for planning and conducting research and evaluations.
[End of Scope and Methodology]
Summary of Findings:
ETA is currently in the process of revising its organizational
structure. Previously, its research and evaluation center, within the
Office of Policy Development and Research (OPDR), was several levels
removed from the Office of the Assistant Secretary and lacked
independent authority for conducting research.
While ETA has recently taken steps to improve its research program,
some limitations remain with regard to its accountability processes
and timely dissemination of research products.
The Office of Management and Budget (OMB) and Labor have both
emphasized the value of research and evaluation. However, ETA has not
routinely involved stakeholders in developing its research agenda and
has also been slow to evaluate the Workforce Investment Act of 1998
(WIA).
[End of Summary of Findings]
Background:
Some Essential Elements of a Sound Government Research and Evaluation
Program:
Some elements of sound research practices that can help agencies‘
strengthen their research and evaluation function, regardless of key
differences related to the agency‘s overall mission and the size of
its research and evaluation center:
Research Independence;
Transparency and Accountability;
Policy Relevance.
Research Independence: Some Essential Elements of Sound Practices:
According to leading experts, to achieve research independence,
research and evaluation centers should be:
* Located at a level of influence within the agency or department.
That is, research centers should be located at a level that allows its
leaders to report directly to the senior executive of the organization.
* Responsible for key aspects of its research program, including the
design, conduct, and dissemination of research studies.
Transparency and Accountability: Some Essential Elements of Sound
Practices:
To achieve transparency and accountability, research and evaluation
centers should:
* Release their research agendas to the public;
* Document research policies and procedures; and;
* Publicly disseminate research findings in a timely and accessible
manner.
Policy Relevance: Some Essential Elements of Sound Practices:
To achieve policy relevance, research and evaluation centers should:
* Use research results to both inform and respond to agency strategic
planning;
* Focus research agenda on issues of importance to policy and
practice; and;
* Involve key stakeholders (researchers, policy makers, and
practitioners) in planning, designing, and conducting research.
Objective 1: Research Independence:
Previously, ETA‘s OPDR Did Not Report Directly to the Assistant
Secretary:
Figure: Organization chart:
[Refer to PDF for image]
Research and evaluation center was several levels below the
Assistant Secretary:
Top level:
Secretary of Labor;
* Deputy Secretary.
Second level: Report to Deputy Secretary;
* Assistant Secretary: Administration and Management;
* Bureau of Labor Statistics;
* Assistant Secretary: ETA;
* Women's Bureau;
* Occupational Safety & Health Administration.
Third level: Report to Assistant Secretary: ETA;
* Deputy Assistant Secretary: Workforce Investment System;
* Deputy Assistant Secretary: Administration and National Activity.
Fourth level: Reporting to Deputy Assistant Secretary, Administration
and National Activity:
* Trade Adjustment Assistance;
* Foreign Labor Certification;
* Apprenticeship;
* Policy Development and Research;
* Financial and Administrative Management;
* Performance and Technology.
Fifth level: Reporting to Policy Development and Research:
* Policy Legislation and Dissemination;
* Research and Evaluations.
Source: Department of Labor's Employment and Training Administration
and GAO analysis.
[End of figure]
ETA‘s Emerging Organizational Structure Does Not Ensure OPDR is
Located at a Level of Influence:
At the time of our review, ETA was revising its organizational
structure.
Under the revised structure, OPDR is placed at the same level as
several program offices such as the Offices of Apprenticeship and
Trade Adjustment Assistance.
It is unclear from ETA‘s revised organizational chart whether OPDR
reports directly to the Assistant Secretary or the Deputy Assistant
Secretary.
It is also unclear whether, as part of OPDR, ETA‘s research and
evaluation center is located at a level of sufficient influence within
ETA.
Alternative Example: Education‘s Research Center (IES) Reports
Directly to the Secretary:
Figure: Organization chart:
[Refer to PDF for image]
Top level:
* Secretary of Education.
Second level: Reporting to Secretary of Education:
* Planning, Evaluation and Policy Development;
* Communications and Outreach;
* General Counsel;
* Institute of Education Sciences:
- National Board for Education Sciences.
Third level: Reporting to Institute of Education Sciences:
* Administration and Policy:
- Management and Operations;
- Outreach and Communications;
- Grants Administration;
* Science:
- Standards and Review.
Fourth level: Reporting to Institute of Education Sciences:
* National Center for Education Research;
* National Center for Education Statistics;
* National Center for Education Evaluation and Regional Assistance;
* National Center for Special Education Research.
Source: Department of Education and GAO analysis.
[End of figure]
ETA‘s Research and Evaluation Center Lacks Sufficient Independence:
The head of ETA‘s research and evaluation center lacks sufficient
authority to approve:
* its research agenda;
* requests for funded proposals, grant announcements, and statements
of work; or;
* dissemination of research and evaluation publications.
According to ETA officials, these authorities reside with ETA‘s Office
of the Assistant Secretary.
Neither ETA‘s research and evaluation center, nor the Office of Policy
Development and Research of which it is a part, has a specific
mechanism that could provide insulation from undue influence.
Other Agencies Have Mechanisms Designed to Promote Research
Independence:
For example, the directors of IES and NSF take the following actions:
* Implement their research agenda;
* Design funded projects;
* Submit reports for peer review and publish research, statistics,and
evaluations with or without the approval of any other office within
their organizations; and;
* Work with governing boards on research agenda and policies.
[End of Objective 1]
Objective 2: Transparency and Accountability:
ETA Has Recently Documented Its Research Process:
In 2007, ETA began documenting its research process.
* This documentation details the eight steps ETA uses to conduct its
research, from project selection to dissemination, and the expected
time frames for each step.
* ETA developed this documentation at the request of OMB.
* Prior to this, ETA did not have a documented process.
However, ETA has not made this documentation available to the public
and includes little information regarding its processes on its Web
site.
ETA‘s Research Process Involves Coordination with OMB:
In September 2007, ETA agreed to inform OMB of sizable research
projects and evaluations.
* OMB reviews the statement of work for all projects that exceed
$250,000.
* OMB informs ETA if it wants to discuss the statement of work.
* OMB has requested that ETA alert it of research reports submitted to
ETA for dissemination and not approved by the Office of the Assistant
Secretary within 9 months.
ETA Now Allows Contractors to Release Products Delayed for More Than 9
Months:
Since January 2008, ETA has included a provision in its research
contracts that allows the contractor to release products submitted to
ETA but not approved by the Assistant Secretary within 9 months.
However, ETA does not broadly advertise the availability of these
products on its Web site to the public.
ETA Has Also Renewed Its Practice of Sponsoring National Conferences
to Highlight Research Findings:
In the past, ETA has sponsored biennial research conferences to
address a range of policy and program issues and document promising
program practices. This practice ended in 2003.
In September 2009, ETA renewed this practice by sponsoring a national
research conference on ’Recovery and Reemployment“ that highlighted
ETA sponsored research.
* The conference also assembled leading researchers from a variety of
government and private entities and fields of research and practice.
* ETA has made the PowerPoint presentations of conference sessions
available on the Internet and is in the process of making the audio
and visual transcripts available to the public.
Despite Recent Changes, ETA‘s Research Processes Lack Specificity:
Time frames for ETA‘s key milestones are stated loosely.
For example:
* During the final review and approval process for dissemination,
project managers are instructed to inform management when products
have not been approved by the Office of the Assistant Secretary,and as
a result, have stalled in the process for an ’excessive“ amount of
time.
* However, ETA has not explained what constitutes an "excessive“
amount of time.
ETA Does Not Have Specific Guidance for Peer Review:
ETA‘s research process provides for peer review but does not specify
the criteria that will be used to determine whether a report should be
peer reviewed.
ETA officials told us that the scope of a project determines whether
it will be peer reviewed, but did not provide specific criteria.
ETA does not provide information on or guidelines for the peer review
process, including how comments will be used and who will be selected
to serve as peer reviewers.
ERS Reported That It Has Specific Policies and Procedures for Peer
Review:
According to USDA information:
* ERS conducts peer reviews of its reports and has established review
guidelines, including criteria for selecting qualified peer reviewers.
* Each report is subjected to both internal and formal, independent,
external peer review.
* ERS requires thorough review of data products by knowledgeable staff
prior to dissemination.
ETA Lacks a System to Track Research Progress:
ETA has not established an information system to track its research
products from completion to dissemination.
* As a result, ETA is unable to track the status of products and
determine which products are delayed and where in the process they are
delayed.
* Without a tracking system, it is difficult for ETA to determine if
it is following its processes or complying with established time
frames.
ETA Has Not Consistently Disseminated Research Findings in a Timely
Manner:
ETA disseminated 34 research products to the public in 2008.
* Twenty of these products had been waiting 2 to 5 years for approval.
(See appendix II for information on report dissemination time frames.)
* ETA officials attributed the delays to inadequate methodologies and
poorly written products. However, ETA had previously agreed to these
methodologies at design and reviewed and approved earlier drafts of
these reports.
* Presently ETA is reviewing its procedures to identify areas where it
can obtain organizational buy-in earlier in the process.
Dissemination Delays Could Have Cost, Policy, and Practice
Implications:
Most of ETA‘s sponsored research whose publication was delayed until
2008 were evaluations of workforce development strategies.
The combined cost of the 20 delayed research products was about $28
million.
Due to the delays, findings from these studies could have a limited
impact in influencing policy and practice.
* Delayed products covered a range of topics including evaluations of
labor exchanges in the one-stop delivery system, the youth offender
demonstration project, and workforce development in rural areas.
ETA‘s Dissemination of Research Findings Is Limited:
ETA‘s dissemination of research findings is largely limited to
postings on its Web site and consists of research and evaluation
reports that it has sponsored.
ETA also issues e-mail advisories to notify its subscribers of newly
released research publications.
ETA has not used dissemination strategies, such as the use of
clearinghouses, to more broadly disseminate its research.
HUD Disseminates Its Research, including Data Sets, through
Clearinghouses and a Dedicated Web Site:
According to department information:
* HUD uses a dedicated Web site to disseminate information that it and
others have collected on housing issues, including housing market
trends, economic and demographic data, housing finance, and energy and
resource-efficient housing designs and construction.
* It has two clearinghouses that provide (1) information on state and
local regulatory reform strategies to support affordable housing and
(2) access to HUD-sponsored research and publications.
* HUD also makes data sets collected for both research and
administrative purposes routinely available to the public.
EPA Disseminates Its Research Products through Partnerships and a
Range of Media:
According to department information:
* EPA partners with other federal agencies and with states to
disseminate and publicize its research.
* EPA‘s Web site contains a searchable database of EPA science
activities and scientific and technical products produced by EPA and
through EPA-funded assistance agreements.
- It also provides public access to its data sets.
- Users can receive e-mail notifications of new products based on key
words, topics, or general research.
* EPA issues press releases and holds research and press conferences.
[End of Objective 2]
Objective 3: Policy Relevance:
OMB Has Taken Steps to Promote and Strengthen Evaluation Efforts:
Current efforts are underway at OMB to encourage agencies to increase
their emphasis on the importance of research and evaluations and
strengthen evaluation methods.
OMB plans to work with agencies to:
* Expand information about program evaluations that are made public.
* Establish an interagency working group to build evaluation capacity
and create effective evaluations that work.
* Provide additional funding to agencies for high-priority evaluation
activities.
Labor Has Taken Recent Steps to Foster Research Relevant to Policy:
In September 2009, Labor announced plans to create a position of chief
evaluation officer and a new evaluation center within the Office of
the Assistant Secretary for Policy.
* This position is designed to link evaluation with policy development
and strategic planning.
* However, the linkage between the new position and ETA‘s research and
evaluation center is unclear.
Recently, ETA‘s Assistant Secretary has acknowledged the importance of
research-driven policy and the need to conduct rigorous evaluations.
ETA has taken steps to increase its evaluation activities and has
applied to OMB for additional funds to conduct rigorous evaluations.
However, ETA Has Been Slow to Evaluate WIA Program Activities:
Congress required an impact evaluation of WIA activities by 2005.
[Footnote 3]
* However, ETA did not award a contract to conduct the evaluation
until 2008.
* As of mid-November 2009, the evaluation‘s research design has not
yet been finalized.
* Officials told us that they plan to implement the evaluation in
January 2010.
Key Stakeholders Have Not Been Consistently Involved to Ensure ETA‘s
Research Is Relevant:
ETA has had no standard process for selecting stakeholders to review
its research agenda or incorporating their comments.
ETA‘s documents indicate that, in recent years, ETA‘s Assistant
Secretary had set and implemented the agenda with limited involvement
from external stakeholders.
* For example, ETA did not solicit public comment on its finalized2007
research plan.
* Moreover, ETA never finalized its 2003 and 2005 revisions to its 5-
year research plan or made the plans available to the public.
* However, in 1999, ETA advertised the availability of a draft of its
plan through a Federal Register notice and solicited public comments
for consideration before the plan was finalized.
Unlike Other Agencies, ETA Does Not Have Advisory Bodies to Involve
Stakeholders in the Development of Its Research Agenda:
For example:
* EPA‘s Board of Scientific Counselors provides advice, information,
and recommendations about its research program.
* IES‘ National Board for Education Sciences approves or disapproves
the priorities for the Institute proposed by the Director and ensures
that its priorities are consistent with its mission.
* NSF‘s National Science Board establishes its research policies and
approves its strategic budget direction.
Labor Is Currently Assisted by 14 Advisory Boards That Perform a
Variety of Functions:
Currently, 14 advisory boards assist Labor and serve a variety of
purposes. For example:
* The Advisory Committee on Apprenticeship provides assistance and
advice to the Assistant Secretary of ETA on policies and programs
regarding apprenticeship.
* The Bureau of Labor Statistics Data Users Committee provides advice
on the collection and analysis of the Bureau's statistics, its
published reports, and its overall mission.
ERS Routinely Consults with Stakeholders to Ensure Its Research Is
Relevant and Useful:
According to USDA, ERS‘ procedures call for consulting with
stakeholders”including policy makers and key institutions that
influence public policy.
* ERS involves stakeholders in discussions of past research
accomplishments, and program effectiveness and impact.
* ERS also consults with and works across department offices and
programs in devising cross-cutting research.
While Advisory Committees Can Promote Research Relevance, They Involve
Some Trade-Offs:
* Involving advisory committees requires additional time and effort,
as well as some minimal expense.
* In creating advisory committees, federal agencies are generally
required to follow the provisions of the Federal Advisory Committee
Act, which include provisions limiting when such committees can be
established.
* Most of the agencies we reviewed had advisory committees.For
example, during FY 2008, the General Services Administration reported
a total of 917 active committees with a total of nearly 64,000 members
who provided advice and recommendations to 50 federal agencies.
ETA Does Not Have a Formal Process in Place to Ensure Strategic
Planning Is Informed by Research:
Although ETA coordinates with its strategic planning office, there is
no formal process in place to ensure that research findings are used
to inform strategic planning and policy.
For example:
* Officials told us that they respond to requests from other
departments to provide information on research and evaluation findings.
* However, there is no formal process in place to ensure that all
research and evaluation findings are shared on a consistent basis.
In Contrast, EPA‘s Structure Has the Potential to Encourage
Coordination Between Research and Strategic Planning:
According to agency documents, EPA has established two offices within
its research office to help ensure its policy and planning are
informed by its research.
* Office of Science Advisor: establishes specific mechanisms for
ensuring that scientific results and hypotheses, with technical
evaluation and peer-review, play a prominent role in all regulatory
decisions.
* Office of Science Policy: ensures that the scientific information
generated through EPA and other research is used to inform EPA
decisions.
USDA‘s Advisory Board Reported That It Serves as a Link to Strategic
Planning:
According to department information, USDA‘s advisory board plays a key
role in coordinating strategic planning for the National Institute of
Food and Agriculture.
The board is to help this institute to:
* establish policies and priorities on a quarterly basis,
* evaluate the effectiveness of those policies and priorities, and,
* develop its 5-year strategic plan.
In addition, according to the National Institute of Food and
Agriculture, it revises its strategic plan on an ongoing basis so that
its research, education, and extension strategies align with
Agriculture‘s goals and objectives.
Additionally, IES‘ Board Facilitates Stakeholder Involvement:
The National Board for Education Services is required by law to:
* Solicit advice and information from those in the field of education
to recommend topics that require research, and;
* Recommend ways to enhance partnerships and collaboration among other
federal and state research agencies.
IES also involves stakeholders by establishing:
* Fellowships for education research at universities, and;
* Standing review panels of scientists from university and industry
settings who review grant applications.
[End of Objective 3]
Conclusions:
A variety of elements”including research independence, transparency
and accountability, and policy relevance”may foster conditions that
can help achieve sound and relevant research.
ETA‘s OPDR and its component research center have limited decision-
making authority and lack formal access to executive leadership
leaving it potentially vulnerable to undue political influence.
Moreover, lacking specific processes and a tracking system to monitor
progress through review inhibits transparency and may lead to
continued delays in approving and disseminating research products.
The involvement of stakeholders and advisory bodies can help ensure
that decisions to conduct research are grounded in policy
implications. Such involvement also serves to foster public
accountability. While deciding to involve advisory bodies comes at
some cost in time and resources, such a move may lend credibility to
the research program.
[End of Conclusions]
Recommendations:
To improve ETA‘s research program, we recommend that the Secretary of
Labor:
* Take steps to clarify ETA‘s revised organizational structure and
ensure that OPDR reports directly to ETA‘s Assistant Secretary;
* Provide sufficient authority to ETA‘s research and evaluation center
to plan, conduct, and disseminate research;
* Direct ETA‘s research and evaluation center to establish more
specific processes, including time frames for dissemination of
research to promote transparency and accountability;
* Create an information system to track research projects at all
phases to ensure timely completion and dissemination; and;
* Instruct ETA‘s research and evaluation center to develop processes
to routinely involve outside experts in setting its research agenda
and to the extent required, do so consistent with the Federal Advisory
Committee Act.
[End of Briefing Slides]
Appendix II: Dissemination Time Frames for ETA's Research Products
Published in 2008:
Table 1: ETA's 2008 Research Products by Dissemination Time Frames:
1;
Publication title: Current Strategies to Employ and Retain Older
Workers;
ETAOP#: 2008-01;
Date of final submission to ETA/OPDR: 1/1/2008;
Date posted on ETA publication database/Web site: 3/3/2008;
Time frames between submission and dissemination[A]: 2.2 months.
2;
Publication title: Reemployment and Eligibility Assessment (REA) Study;
ETAOP#: 2008-02;
Date of final submission to ETA/OPDR: 3/1/2008;
Date posted on ETA publication database/Web site: 3/12/2008;
Time frames between submission and dissemination[A]: 11 days.
3;
Publication title: Assessment of Strategies to Retain Experienced
Technical and Professional Healthcare Personnel After Retirement Age:
Mature Healthcare Workers Focus on Group Research - Final Report;
ETAOP#: 2008-04;
Date of final submission to ETA/OPDR: 1/22/2008;
Date posted on ETA publication database/Web site: 5/8/2008;
Time frames between submission and dissemination[A]: 4 months.
4;
Publication title: Early Implementation of Generation I of the
Workforce Innovation in Regional Economic Development (WIRED)
Initiative: 2007 Interim Evaluation Report;
ETAOP#: 2008-03;
Date of final submission to ETA/OPDR: 5/16/2008;
Date posted on ETA publication database/Web site: 5/21/2008;
Time frames between submission and dissemination[A]: 5 days.
5;
Publication title: Evaluation of State Implementation of Section 303
(K), Social Security Act, "SUTA Dumping" - Final Report;
ETAOP#: 2008-05;
Date of final submission to ETA/OPDR: 5/7/2008;
Date posted on ETA publication database/Web site: 6/12/2008;
Time frames between submission and dissemination[A]: 1 month.
6;
Publication title: Evaluation of Prisoner Re-Entry Initiative- Interim
Report;
ETAOP#: 2008-06;
Date of final submission to ETA/OPDR: 9/28/2007;
Date posted on ETA publication database/Web site: 6/16/2008;
Time frames between submission and dissemination[A]: 8.5 months.
7;
Publication title: Unemployment Insurance: Assessment of the Impact of
the 2002 Reed Act Distribution;
ETAOP#: 2004-11;
Date of final submission to ETA/OPDR: 12/1/2004;
Date posted on ETA publication database/Web site: 7/30/2008;
Time frames between submission and dissemination[A]: 3.8 years.
8;
Publication title: Responses to Personal Remployment Accounts (PRAs):
Findings from the Demonstration States --Final Evaluation Report;
ETAOP#: 2008-07;
Date of final submission to ETA/OPDR: 6/6/2008;
Date posted on ETA publication database/Web site: 8/11/2008;
Time frames between submission and dissemination[A]: 2 months.
9;
Publication title: Navigating the U.S. Labor Market: Trends and
Prospects for Workers;
ETAOP#: 2003-09;
Date of final submission to ETA/OPDR: 9/22/2003;
Date posted on ETA publication database/Web site: 8/14/2008;
Time frames between submission and dissemination[A]: 4.10 years.
10;
Publication title: Review of Recent Pilot, Demonstration, Research,
and Evaluation Initiatives to Assist in the Implementation of Programs
under the Workforce Investment Act;
ETAOP#: 2003-10;
Date of final submission to ETA/OPDR: 9/22/2003;
Date posted on ETA publication database/Web site: 8/14/2008;
Time frames between submission and dissemination[A]: 4.11 years.
11;
Publication title: Net Impact Estimates for Services Provided through
the Workforce Investment Act;
ETAOP#: 2005-06;
Date of final submission to ETA/OPDR: 10/1/2005;
Date posted on ETA publication database/Web site: 8/14/2008;
Time frames between submission and dissemination[A]: 2.10 years.
12;
Publication title: Review of Alternative Methodologies for Employment
and Training Research;
ETAOP#: 2003-11;
Date of final submission to ETA/OPDR: 9/22/2003;
Date posted on ETA publication database/Web site: 8/15/2008;
Time frames between submission and dissemination[A]: 4.9 years.
13;
Publication title: Anatomy of a One-Stop: Baltimore City East Side
Career Center;
ETAOP#: 2006-07;
Date of final submission to ETA/OPDR: 8/20/2008;
Date posted on ETA publication database/Web site: 8/29/2008;
Time frames between submission and dissemination[A]: 9 days.
14;
Publication title: Anatomy of Two One-Stops: Camdenton and Columbia,
Missouri;
ETAOP#: 2006-08;
Date of final submission to ETA/OPDR: 8/20/2008;
Date posted on ETA publication database/Web site: 8/29/2008;
Time frames between submission and dissemination[A]: 9 days.
15;
Publication title: Youth Offender Demonstration Project Evaluation
Final Report - Volume One;
ETAOP#: 2006-06;
Date of final submission to ETA/OPDR: 6/1/2006;
Date posted on ETA publication database/Web site: 9/4/2008;
Time frames between submission and dissemination[A]: 2.3 years.
16;
Publication title: Workforce Development in Rural Areas - Changes in
Access, Service Delivery and Partnerships;
ETAOP#: 2005-07;
Date of final submission to ETA/OPDR: 6/30/2005;
Date posted on ETA publication database/Web site: 9/4/2008;
Time frames between submission and dissemination[A]: 3.2 years.
17;
Publication title: Unemployment Insurance and Reemployment among Older
Workers;
ETAOP#: 2006-09;
Date of final submission to ETA/OPDR: 7/1/2006;
Date posted on ETA publication database/Web site: 9/4/2008;
Time frames between submission and dissemination[A]: 2.2 years.
18;
Publication title: Youth Offender Demonstration Project Process
Evaluation Final Report - Round Two;
ETAOP#: 2004-10;
Date of final submission to ETA/OPDR: 1/30/2004;
Date posted on ETA publication database/Web site: 9/8/2008;
Time frames between submission and dissemination[A]: 4.2 years.
19;
Publication title: Evaluation of Labor Exchange in the One-Stop
Delivery System Environment;
ETAOP#: 2004-09;
Date of final submission to ETA/OPDR: 2/1/2004;
Date posted on ETA publication database/Web site: 9/11/2008;
Time frames between submission and dissemination[A]: 4.7 years.
20;
Publication title: Use of Experimental Methods in Workforce
Evaluations;
ETAOP#: 2005-08;
Date of final submission to ETA/OPDR: 4/18/2005;
Date posted on ETA publication database/Web site: 9/16/2008;
Time frames between submission and dissemination[A]: 3.5 years.
21;
Publication title: On the Use of Administrative Data for Workforce
Development Program Evaluations;
ETAOP#: 2005-09;
Date of final submission to ETA/OPDR: 4/18/2005;
Date posted on ETA publication database/Web site: 9/16/2008;
Time frames between submission and dissemination[A]: 3.5 years.
22;
Publication title: Immigration and the Effects on the US Labor Market;
ETAOP#: 2005-10;
Date of final submission to ETA/OPDR: 4/18/2005;
Date posted on ETA publication database/Web site: 9/16/2008;
Time frames between submission and dissemination[A]: 3.5 years.
23;
Publication title: The Labor Market Effects of Globalization and TAA;
ETAOP#: 2005-11;
Date of final submission to ETA/OPDR: 4/18/2005;
Date posted on ETA publication database/Web site: 9/16/2008;
Time frames between submission and dissemination[A]: 3.5 years.
24;
Publication title: Systemic Disincentive Effects of the UI Program;
ETAOP#: 2005-12;
Date of final submission to ETA/OPDR: 4/18/2005;
Date posted on ETA publication database/Web site: 9/16/2008;
Time frames between submission and dissemination[A]: 3.5 years.
25;
Publication title: Older workers and the Labor Market/Labor Market
Policies for Older Workers;
ETAOP#: 2005-13;
Date of final submission to ETA/OPDR: 4/18/2005;
Date posted on ETA publication database/Web site: 9/16/2008;
Time frames between submission and dissemination[A]: 3.5 years.
26;
Publication title: Programs to Support Out-of-School Youth;
ETAOP#: 2005-14;
Date of final submission to ETA/OPDR: 4/18/2005;
Date posted on ETA publication database/Web site: 9/16/2008;
Time frames between submission and dissemination[A]: 3.5 years.
27;
Publication title: Community College Training and the Workforce
Investment System;
ETAOP#: 2005-15;
Date of final submission to ETA/OPDR: 4/18/2005;
Date posted on ETA publication database/Web site: 9/16/2008;
Time frames between submission and dissemination[A]: 3.5 years.
28;
Publication title: Project GATE;
ETAOP#: 2008-08;
Date of final submission to ETA/OPDR: 5/1/2008;
Date posted on ETA publication database/Web site: 9/17/2008;
Time frames between submission and dissemination[A]: 4.5 months.
29;
Publication title: Flexible Learning Options for Adult Students;
ETAOP#: 2008-09;
Date of final submission to ETA/OPDR: 4/1/2008;
Date posted on ETA publication database/Web site: 10/16/2008;
Time frames between submission and dissemination[A]: 6.5 months.
30;
Publication title: Implementation Analysis of High Growth Job Training
Initiative (HGJTI);
ETAOP#: 2008-10;
Date of final submission to ETA/OPDR: 6/1/2008;
Date posted on ETA publication database/Web site: 10/16/2008;
Time frames between submission and dissemination[A]: 4.5 months.
31;
Publication title: Literature Review: Intermediaries and FBCOs Working
Together;
ETAOP#: 2008-11;
Date of final submission to ETA/OPDR: 10/2/2008;
Date posted on ETA publication database/Web site: 10/29/2008;
Time frames between submission and dissemination[A]: 27 days.
32;
Publication title: Youth Opportunity Grants Initiative (YO);
ETAOP#: 2008-12;
Date of final submission to ETA/OPDR: 12/1/2005;
Date posted on ETA publication database/Web site: 12/23/2008;
Time frames between submission and dissemination[A]: 3.0 years.
33;
Publication title: What's Known About the Effects of Publicly-Funded
Employment and Training Programs;
ETAOP#: 2006-10;
Date of final submission to ETA/OPDR: 4/21/2006;
Date posted on ETA publication database/Web site: 12/15/2008;
Time frames between submission and dissemination[A]: 2.8 years.
34;
Publication title: Competency Models - A Review of the Literature and
the Role of the Employment and Training Administration (ETA);
ETAOP#: 2008-13;
Date of final submission to ETA/OPDR: 1/29/2008;
Date posted on ETA publication database/Web site: 12/23/2008;
Time frames between submission and dissemination[A]: 11 months.
Source: Department of Labor's Employment and Training Administration
and GAO analysis of data.
Note: The information presented in this table was provided by ETA
officials.
[A] Time frames were calculated using the date each research product
was submitted to ETA and the date it was posted on ETA's Web site.
[End of table]
[End of section]
Appendix III: Comments from the Department of Labor:
U.S. Department of Labor:
Assistant Secretary for Employment and Training:
Washington, D.C. 20210:
January 12, 2010:
Mr. George A. Scott:
Director:
Education, Workforce, and Income Security Issues:
U.S. Government Accountability Office:
441 G. Street, N.W.
Washington, D.C. 20548:
Dear Mr. Scott:
The Employment and Training Administration (ETA) is in receipt of the
draft Government Accountability Office (GAO) draft report number 10-
243 entitled, "Employment and Training Administration: Increased
Authority and Accountability Could Improve Research Program." We
appreciate the opportunity to comment on the draft.
Responses to Recommendations:
On page 5 and slide 50, GAO makes five recommendations regarding ETA's
research program. Following is ETA's response.
Recommendation #1: Organizational Structure:
Take steps to clarify ETA revised organizational structure and ensure
that OPDR reports directly to ETA's Assistant Secretary.
The most recent organizational structure for the Employment and
Training Administration (ETA) has the Office of Policy Development and
Research (OPDR) reporting directly to the Assistant Secretary for
Employment and Training. See [hyperlink,
http://www.doleta.gov/pdf/OrgChart.pdf]. Consistent with the Essential
Elements of a Sound Research and Evaluation Program,[Footnote 4] under
the most recent organizational structure, the Administrator for ETA's
research and evaluation center has direct access to and reports to the
Assistant Secretary for Employment and Training. The center, lead by
the OPDR Administrator, is placed organizationally within ETA's
leadership structure to have influence and input to the organization's
ultimate decision maker, the Assistant Secretary.[Footnote 5]
Recommendation #2: Decision-Making Authority:
Provide sufficient decision-making authority to ETA's research and
evaluation center to plan, conduct, and disseminate research.
ETA's research and evaluation center currently provides
recommendations to the Assistant Secretary regarding plans for
conducting and disseminating research. Meetings with ETA's decision-
makers are scheduled to discuss such plans and decisions are
communicated with all relevant parties within the agency for
implementation. ETA's research and evaluation center develops the
agency's plans for conducting research collaboratively with the
relevant program offices. This ensures that research and evaluation
efforts are coordinated and aligned with agency programmatic
priorities. In addition, program offices participate in the
dissemination decision-making process by reviewing and commenting on
final draft reports.
Recommendation #3: Establish Specific Time Frames:
Direct ETA's research and evaluation center to establish more specific
processes, including time frames for dissemination of research to
promote transparency and accountability.
ETA's process improvement steps, developed in 2007 and finalized in
2008 in agreement with the Office of Management and Budget, provide
set time frames for the dissemination of research. The time frames arc
based on ranges in order to accommodate the variance in reviewing and
finalizing reports. This iterative process involves the Federal
project officer, OPDR staff, the relevant program offices, and
contractor staff. The time frames can range from one week to one
month, or one month to several months to show the ideal review
schedule but allows for a realistic schedule that takes into
consideration the workload of staff serving as project officer, the
availability of program office staff and other OPDR staff in reviewing
reports, and the contractor's completion schedule of a final draft
report.
ETA is committed to publishing reports that disseminate evidence-based
knowledge of what works and what does not on a timely basis to assist
policymakers with policy and programmatic decisions. ETA has shared
and disseminated research and evaluation findings at the agency's
national conferences that showcases research. Recently, ETA's
commitment to dissemination was evidenced at the recent Recovery and
Reemployment Research Conference which featured findings from many
Department-funded research and evaluation projects and engaged a broad
spectrum of communities, including workforce investment, research,
education, oversight agencies, non-profit organizations, and public
policy makers.
In addition, ETA has sponsored a series of briefings titled Coffee
House Briefing Series that featured findings and results from ETA-
funded research. These briefings are announced on the agency's
Intranet website. The briefing announcements have also been shared
with colleagues in other Department of Labor agencies such as the
Office of the Assistant Secretary for Policy and the Veterans
Employment and Training Service as well as with other Federal
agencies, including the Departments of Health and Human Services and
Education. These agencies and departments have attended several of the
briefing sessions. These briefing sessions are well attended by ETA
staff from different program offices.
Lastly, in keeping with the Department's commitment to the President's
Open Government philosophy, ETA's research will also be disseminated
as part of the Department's efforts to make all DOL agencies' research
products available through its Open Government Web site.
Recommendation #4: Track Research:
Create an information system to track research projects at all phases
to ensure timely completion and dissemination.
Currently, ETA research projects are tracked by individual project
officers who report any significant delays to their Team Leaders and
the Division Director. A departmental research inventory is produced
semi-annually which updates the status of all projects, however, this
inventory has only limited project management value. Therefore, ETA is
planning to implement a centralized, electronic tracking system for
its research projects. The new system will track major project
milestones during the course of the study and steps in the review and
dissemination of reports, once reports have been completed. The
project officers will be responsible for updating the system as events
occur.
Recommendation #5: Involve Outside Experts:
Instruct ETA's research and evaluation center to develop processes to
routinely involve outside experts in setting its research agenda and
to the extent required, do so consistent with the Federal Advisory
Committee Act.
ETA has in the past, and most recently, increased its efforts to
involve outside experts in setting its research agenda. Under Section
171 of the Workforce Investment Act, ETA is required to develop a Five-
Year Strategic Research Plan in consultation with states, localities
and other interested parties. The strategic plan provides guidance on
projects for pilots, demonstrations, research and evaluations. The
plan provides information on the types of research topics and projects
that should be taken into consideration. The first plan transmitted to
the Congress covered the years 2000-2005. For the years covering 2002-
2007 and 2004-2009, the strategic plans remained in draft form and
were not transmitted to the Congress since they did not pass
Departmental clearance as they were compendiums of various research
papers rather than a plan to help guide the agency's pilots,
demonstrations, research and evaluations. The most recent strategic
research plan for 2007-2012, was transmitted to the Congress on July
20, 2007. All of these plans were developed in consultation with
outside experts.
In addition, ETA has engaged informally in dialogues with outside
experts through conferences as well as through interagency
collaborations in the development of the research and evaluation
agenda. For instance, ETA has collaborated with the research and
evaluation centers of the Departments of Education and Health and
Human Services on areas of shared interest. These efforts are designed
to identify ways each agency's resources and expertise can be
leveraged in support of each agency's research agenda. Another
informal, however routine, method for leveraging input from
programmatic and research experts on ETA's research agenda has been
through the sponsorship of the Research Showcase annually at the past
Workforce Innovations Conferences and the most recent Recovery and
Reemployment Research Conference. These venues afford ETA
opportunities to engage in a dialogue with research and program
experts on completed research and evaluation projects and discuss
opportunities for future research.
Thank you for the opportunity to comment on this report. Attached
you'll find technical comments on report and slides.
Sincerely,
Signed by:
Jane Oates:
Assistant Secretary:
[End of section]
Appendix IV: GAO Contact and Staff Acknowledgments:
GAO Contact:
George A. Scott, Director (202) 512-7215 or scottg@gao.gov:
Acknowledgments:
In addition to the contact listed above, Dianne Blank (Assistant
Director) and Kathleen White (Analyst-in-Charge) supervised the
development of this product. Linda Stokes, Amy Sweet, and Ashanta
Williams made significant contributions to all aspects of this report.
In addition, Stephanie Shipman assisted with the conceptual
development of our study; Amanda Miller assisted with design and
analyses; James Bennett provided graphic assistance; Susan Bernstein
provided writing assistance; Alex Galuten provided legal support; and
Karen Brown and Melissa Jaynes verified our findings.
[End of section]
Related GAO Products:
Workforce Investment Act: Labor Has Made Progress in Addressing Areas
of Concern, but More Focus Needed on Understanding What Works and What
Doesn't. GAO-09-396T. Washington, D.C.: February 26, 2009.
Federal Research: Policies Guiding the Dissemination of Scientific
Research from Selected Agencies Should Be Clarified and Better
Communicated. GAO-07-653. Washington, D.C.: May 17, 2007.
Data Quality: Expanded Use of Key Dissemination Practices Would
Further Safeguard the Integrity of Federal Statistical Data. GAO-06-
607. Washington, D.C.: May 31, 2006.
Workforce Investment Act: Substantial Funds Are Used for Training, but
Little Is Known Nationally about Training Outcomes. GAO-05-650.
Washington, D.C.: June 29, 2005.
Program Evaluation: An Evaluation Culture and Collaborative
Partnerships Help Build Agency Capacity. GAO-03-454. Washington, D.C.:
May 2, 2003.
Workforce Investment Act: Improvements Needed in Performance Measures
to Provide a More Accurate Picture of WIA Effectiveness. GAO-02-275.
Washington, D.C.: February 1, 2002.
[End of section]
Footnotes:
[1] 29 U.S.C. § 2917(c).
[2] In addition to ETA, we present information for comparative
purposes on the following research centers: the Department of
Agriculture's (USDA) Economic Research Service (ERS) and National
Institute of Food and Agriculture; the Department of Education's
(Education) Institute of Education Sciences (IES); the Department of
Housing and Urban Development's (HUD) Office of Policy Development and
Research; the Environmental Protection Agency's (EPA) Office of
Research and Development; and the National Science Foundation (NSF).
[3] 29 U.S.C. § 2917(c).
[4] GAO Draft Report titled "Employment and Training Administration:
Increase Authority and Accountability Could Improve Research Program",
[hyperlink, http://www.gao.gov/products/GA0-10-243], page 21 (slide
15).
[5] Ibid; page 25 (slide 19).
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
