Youthbuild Program
Analysis of Outcome Data Needed to Determine Long-Term Benefits Gao ID: GAO-07-82 February 28, 2007Since 1993, the Department of Housing and Urban Development (HUD) has provided funding for Youthbuild, a competitive grant program that trains and educates disadvantaged youth and helps build low-income housing. In 2006, Youthbuild was transferred to the Department of Labor (Labor) to better align the Youthbuild program with existing youth workforce and training programs. In response to concerns about the overall quality of Youthbuild, a Senate report directed GAO to assess the program. GAO's objectives included (1) evaluating how HUD assessed and oversaw the program, (2) determining what results the program achieved, and (3) assessing how successful grantees were in obtaining outside funding. GAO analyzed Youthbuild performance data, visited Youthbuild sites, and interviewed agency officials.
While HUD requires grantees to report basic performance data, such as the number of program participants and graduates and job placements, HUD has not aggregated or analyzed the data and conducted limited oversight of grantees. According to HUD officials, they did not have staff available to analyze the closeout reports that grantees must submit, and a lack of resources also limited oversight of grantees. The monitoring HUD did primarily focused on compliance with program requirements such as documentation rather than on performance. As a result, HUD largely was unable to tell how the individual Youthbuild grantee programs performed. Limited outcome data preclude any overall assessment of the performance of the Youthbuild program; further, the few other analyses available such as the one GAO did in this study to augment limited existing data cannot be generalized programwide. GAO analyzed 245 closeout reports, representing 46 percent of the grantees or 12,863 participants. While GAO could determine percentages of participants who received high school diplomas or were placed in jobs, GAO could not determine outcomes over time, partly because the reports lacked baseline information and grantees were not required to and generally did not follow participants after graduation. Further, while closeout reports include information about impediments to program success and "best practices," HUD did not systematically review this information or share it with its primary technical assistance contractor. Consequently, the lack of programwide evaluations, follow-up data, and dissemination of best practices make it very difficult to assess the performance of Youthbuild over time and determine which programs and strategies have worked best. Reporting on post-program performance outcomes, such as the number of participants placed and retained in construction-related employment, could increase the value of the closeout reports and better measure program results. Labor officials indicated that they would consider including such measures for program reporting. Grantees had varying success in obtaining funds from outside sources, but YouthBuild USA data suggest that continued (multiyear) HUD funding was critical to sustaining grantee operations and attracting leveraged funds. Grantees' success in obtaining additional funds varied widely, from 21 grants reporting no additional funding sources to 40 reporting more than $1 million. While most grantees have generated outside funding, YouthBuild USA reported that most grantees have had difficulty continuing operations without continued HUD funding. Their data show that 90 percent of grantees ceased operations if not funded for 3 consecutive years by HUD. Further, YouthBuild USA also noted that grantees with follow-on HUD funding achieved better performance outcomes, such as higher rates of job placements, than grantees that did not receive subsequent HUD funding.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-07-82, Youthbuild Program: Analysis of Outcome Data Needed to Determine Long-Term Benefits
This is the accessible text file for GAO report number GAO-07-82
entitled 'Youthbuild Program: Analysis of Outcome Data Needed to
Determine Long-Term Benefits' which was released on February 28, 2007.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Committees:
United States Government Accountability Office:
GAO:
February 2007:
Youthbuild Program:
Analysis of Outcome Data Needed to Determine Long-Term Benefits:
GAO-07-82:
GAO Highlights:
Highlights of GAO-07-82, a report to congressional committees
Why GAO Did This Study:
Since 1993, the Department of Housing and Urban Development (HUD) has
provided funding for Youthbuild, a competitive grant program that
trains and educates disadvantaged youth and helps build low-income
housing. In 2006, Youthbuild was transferred to the Department of Labor
(Labor) to better align the Youthbuild program with existing youth
workforce and training programs. In response to concerns about the
overall quality of Youthbuild, a Senate report directed GAO to assess
the program. GAO‘s objectives included (1) evaluating how HUD assessed
and oversaw the program, (2) determining what results the program
achieved, and (3) assessing how successful grantees were in obtaining
outside funding. GAO analyzed Youthbuild performance data, visited
Youthbuild sites, and interviewed agency officials.
What GAO Found:
While HUD requires grantees to report basic performance data, such as
the number of program participants and graduates and job placements,
HUD has not aggregated or analyzed the data and conducted limited
oversight of grantees. According to HUD officials, they did not have
staff available to analyze the closeout reports that grantees must
submit, and a lack of resources also limited oversight of grantees. The
monitoring HUD did primarily focused on compliance with program
requirements such as documentation rather than on performance. As a
result, HUD largely was unable to tell how the individual Youthbuild
grantee programs performed.
Limited outcome data preclude any overall assessment of the performance
of the Youthbuild program; further, the few other analyses available
such as the one GAO did in this study to augment limited existing data
cannot be generalized programwide. GAO analyzed 245 closeout reports,
representing 46 percent of the grantees or 12,863 participants. While
GAO could determine percentages of participants who received high
school diplomas or were placed in jobs, GAO could not determine
outcomes over time, partly because the reports lacked baseline
information and grantees were not required to and generally did not
follow participants after graduation. Further, while closeout reports
include information about impediments to program success and ’best
practices,“ HUD did not systematically review this information or share
it with its primary technical assistance contractor. Consequently, the
lack of programwide evaluations, follow-up data, and dissemination of
best practices make it very difficult to assess the performance of
Youthbuild over time and determine which programs and strategies have
worked best. Reporting on post-program performance outcomes, such as
the number of participants placed and retained in construction-related
employment, could increase the value of the closeout reports and better
measure program results. Labor officials indicated that they would
consider including such measures for program reporting.
Grantees had varying success in obtaining funds from outside sources,
but YouthBuild USA data suggest that continued (multiyear) HUD funding
was critical to sustaining grantee operations and attracting leveraged
funds. Grantees‘ success in obtaining additional funds varied widely,
from 21 grants reporting no additional funding sources to 40 reporting
more than $1 million. While most grantees have generated outside
funding, YouthBuild USA reported that most grantees have had difficulty
continuing operations without continued HUD funding. Their data show
that 90 percent of grantees ceased operations if not funded for 3
consecutive years by HUD. Further, YouthBuild USA also noted that
grantees with follow-on HUD funding achieved better performance
outcomes, such as higher rates of job placements, than grantees that
did not receive subsequent HUD funding.
What GAO Recommends:
GAO recommends that (1) HUD analyze closeout reports by grant and share
information on identified problems and best practices with its
technical assistance contractor and Labor, (2) Labor develop post-
program performance outcomes, and (3) Labor consider multiyear funding
of grants for the program. HUD and Labor provided comments on a draft
of this report. Labor generally agreed with the recommendations. HUD
agreed but cited resource constraints in implementing its
recommendation.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-82].
To view the full product, including the scope and methodology, click on
the link above. For more information, contact William B. Shear at (202)
512-4325 or shearw@gao.gov.
[End of Section]
Contents:
Letter:
Results in Brief:
Background:
HUD's Grant-Making Procedures Incorporated Legislative Criteria:
HUD Oversight of Grantees' Performance Has Been Limited:
Incomplete Data on Participants' Outcomes Precludes an Overall
Assessment of Program's Success:
Youthbuild Grantees Have Experienced Varied Success in Obtaining
Outside Funding Sources, and Many Cited Continued HUD Funding as
Crucial:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Comments from the U.S. Department of Labor:
Appendix II: GAO Contact and Staff Acknowledgments:
Figures:
Figure 1: Youthbuild Appropriations, Fiscal Years 1993-2006:
Figure 2: Characteristics of Participants Entering YouthBuild USA-
Affiliated Programs, 1997-2005:
Figure 3: NOFA Point-Scoring System for Fiscal Year 2005 Applications:
Figure 4: HUD Awards by Funding Category for Fiscal Years 2001-2006:
Figure 5: Youthbuild Grant Applications and the Number Funded by Fiscal
Year (1996-2006):
Figure 6: Based on Closeout Reports, Youthbuild Performance Measures
Indicate Varied Outputs:
Figure 7: Outside Funding Amounts Varied Widely for Grants We Reviewed:
Abbreviations:
CPD: Community Planning and Development:
GED: General Equivalency Diploma:
GPRA: Government Performance and Results Act:
HUD: Department of Housing and Urban Development:
NOFA: Notice of Funding Availability:
OMB: Office of Management and Budget:
WIA: Workforce Investment Act:
United States Government Accountability Office:
Washington, DC 20548:
February 28, 2007:
The Honorable Patty Murray:
Chairman:
The Honorable Christopher Bond:
Ranking Minority Member:
Subcommittee on Transportation, Housing and Urban Development, and
Related Agencies:
Committee on Appropriations:
United States Senate:
The Honorable John W. Olver:
Chairman:
The Honorable Joe Knollenberg:
Ranking Minority Member:
Subcommittee on Transportation, Housing and Urban Development, and
Related Agencies:
Committee on Appropriation:
House of Representatives:
Since 1993, the Department of Housing and Urban Development (HUD) has
provided funding for Youthbuild, a "second chance" program that trains
and educates disadvantaged youth and helps build housing for low-and
very low-income families across the United States. As stated in the
authorizing legislation, the purposes of the Youthbuild program are (1)
expanding the supply of affordable housing by utilizing economically
disadvantaged young adults, (2) providing economically disadvantaged
young adults with opportunities for meaningful work and service to
their communities, (3) enabling economically disadvantaged young adults
to obtain the education and employment skills necessary to achieve
economic self-sufficiency, and (4) fostering the development of
leadership skills and commitment to community development among young
adults in low-income communities.[Footnote 1] Although a few studies
have shown instances where Youthbuild graduates credit the program for
helping them overcome difficult circumstances, gain critical job
skills, and continue their education, concerns have been expressed
about whether HUD's program oversight has been sufficient to adequately
measure program outcomes.
The Housing and Community Development Act of 1992 authorized the
Youthbuild program. The program was based on the Youth Action Program
developed in East Harlem, New York, in the late 1970s in which high
school dropouts received academic training and learned job skills by
renovating abandoned buildings. The YouthBuild Transfer Act shifted
program oversight of grants issued beginning in fiscal year 2007 to the
Department of Labor (Labor)--in part, to better align the program with
existing federal workforce development and youth training
programs.[Footnote 2] The Transfer Act also expanded the program's
scope to allow construction and rehabilitation of community facilities
in addition to housing. HUD will continue to oversee Youthbuild grants
funded through fiscal year 2006. That is, HUD's oversight
responsibility could continue for several more years since grants can
remain active for 3 years or more.
Noting concerns regarding the overall quality of the Youthbuild
program, the Senate report accompanying the fiscal year 2006
Transportation, Treasury, Housing and Urban Development, the Judiciary,
the District of Columbia, and Independent Agencies Appropriations Act
directed us to comprehensively assess the Youthbuild program's overall
success, including its ability to attract local support and funding.
Specifically, the objectives of this report are to (1) describe how HUD
makes Youthbuild program grants; (2) evaluate how HUD assesses and
oversees the Youthbuild program; (3) determine how successful the
Youthbuild program has been in achieving desired results, such as
providing the education and job training needed to achieve economic
self-sufficiency and supplying affordable housing; and (4) assess how
successful local Youthbuild organizations have been in obtaining
private funding and support.
We focused our analysis on reviewing HUD's grant making and oversight
processes and evaluating available data for measuring program success.
To determine how HUD makes annual program grants, we reviewed the
application and review process HUD uses to identify and select
organizations for grant awards.[Footnote 3] To evaluate how HUD
assesses and oversees the program, we interviewed HUD officials at the
national and field offices. We also obtained information on the extent
to which grantees are monitored and the results of the monitoring.
Further, we obtained information on the type of performance data
grantees submit to HUD and HUD's role in reviewing the information. We
also obtained funding and grant award information from HUD. We assessed
the reliability of these data by comparing the information with other
data sources to identify obvious problems with completeness or
accuracy, and interviewing knowledgeable agency officials about the
data. We determined that the data were sufficiently reliable for the
purposes of this report. To develop information on how successful the
Youthbuild program has been, we consulted relevant literature,
including three studies of the Youthbuild program's effect on
participants' lives.[Footnote 4] However, the studies were limited in
scope and methodology and constrained by the absence of a database
containing performance information from all the grantees in the
Youthbuild program. We also interviewed officials and received
information from HUD's primary technical assistance contractor,
YouthBuild USA. Further, because HUD lacked a centralized database on
grantees' program performance results, we analyzed 245 grant closeout
reports submitted by grantees to HUD from April 2000 to August
2006.[Footnote 5] We also visited nine Youthbuild programs in
California, Illinois, Massachusetts, Pennsylvania, Rhode Island, and
Washington, D.C.[Footnote 6] The sites were chosen in part because the
programs varied in size, leadership style, and teaching approach,
including the charter school format. The closeout reports we reviewed
represented about 46 percent of the closed grants awarded from 1997
through 2003. We were unable to review the remainder of the reports
either because HUD was not able to provide a copy of the closeout
reports, the grantee had not submitted a closeout report, or the report
did not have sufficient information for us to use. In addition, we
reviewed monitoring reports covering 16 grantees prepared by HUD field
office staff as part of their program oversight duties. To assess how
successful local Youthbuild organizations have been in obtaining
private funding and support, we analyzed the closeout reports to
determine the extent to which the individual grantees obtained
additional funding. From a list of 100 grantees from YouthBuild USA
that closed their operations, we obtained information on a
nonprobability selection of 23 grantees by directly contacting them or
from other knowledgeable sources to determine how HUD funding decisions
contributed to their closure. We selected and contacted grantees
primarily based on those that had received more than one grant and
which we were able to locate because they were still providing other
social services. We also reviewed literature and reports on Labor's
youth employment and training programs and met with officials from
Labor, where oversight of the Youthbuild program will be transferred.
We conducted our work in Boston and Somerville, Massachusetts;
Washington, D.C; and various Youthbuild locations in the states noted
above from March 2006 through February 2007 in accordance with
generally accepted government auditing standards.
Results in Brief:
Each year, HUD staff scored grant applications based on factors
outlined in the enacting legislation for serving economically
distressed areas and detailed in an annual Notice of Funding
Availability (NOFA). The Housing and Community Development Act of 1992
required HUD to establish selection criteria such as the qualifications
or potential capabilities of applicants, the feasibility of the
proposed Youthbuild program, the potential for developing a successful
Youthbuild program, and the applicants' commitment to obtain outside
resources. HUD incorporated these criteria in an annual NOFA, which
directed applicants and detailed the scoring process for grant awards.
The award process has been competitive, with HUD having sufficient
resources to fund less than one-third of the applicants since 1996.
While HUD requires grantees to submit closeout reports with basic
performance data, HUD has not verified, aggregated, or analyzed the
data and conducts limited oversight of grantees. The data in the
closeout reports were intended for performance assessments;
specifically, in its instructions to grantees, HUD states that closeout
reports will be used to help evaluate the grantees' programs. Further,
in accordance with federal performance management standards, accurate
and complete performance data are needed to inform Congress and the
public about the effectiveness of programs and help agencies
effectively manage and oversee programs.[Footnote 7] HUD officials said
that they did not have staff available to verify, aggregate, or analyze
the closeout reports that they required Youthbuild grantees to submit;
instead, they contracted with YouthBuild USA to collect a set of
performance outputs on a quarterly basis, which provides a more limited
perspective on grantee performance because the data have not been
aggregated for the term of the grants. While HUD has recognized the
importance of performance information by planning for a computerized
data collection system to capture performance information on Youthbuild
and other competitive grant programs, the implementation has been
delayed since 2003 due to funding and technical problems. In addition,
according to HUD officials, a lack of resources also limited assessment
and oversight of Youthbuild grantees. For example, HUD field offices,
which are responsible for overseeing the Youthbuild program, typically
were limited to visiting and reviewing one or two grantees annually.
HUD's monitoring primarily focused on compliance with program
requirements such as documentation rather than on performance. As a
result of a lack of verified and aggregated data and analysis and
limited oversight, HUD largely was unable to tell how well the
individual Youthbuild grantee programs performed.
In addition, lack of programwide performance data and analysis limited
any overall assessment of the success of the Youthbuild program, and
limited available analyses cannot be used for programwide assessments.
Since 2002, a few studies have provided some evidence of the success of
individual Youthbuild programs. While limited in scope, the studies
have shown that, at a small number of grantees, Youthbuild has been
successful in assisting youth to enter higher education programs and
obtain job skills. The value of the data that YouthBuild USA collects
in its Web-based system is limited as well; the data are self-reported
and contain little information on participant outcomes, partly because
grantees are not required to follow participants after graduation. To
augment limited existing data, we analyzed the performance information
available in 245 closeout reports that HUD supplied to us, which
covered 12,863 program participants. The results cannot be generalized
to the entire program. For each of the grantees that reported
information, we calculated percentages of participants who completed or
placed in an education program or in employment. The median percentage
of participants who obtained a general equivalency or high school
diploma across all the programs that reported information was 28,
compared with 36 percent who were placed in jobs, and 7 percent who
went on to higher education. However, because the reports lack some
baseline information and grantees generally did not follow participants
after program completion, we were unable to determine long-term
outcomes such as percentages or number of participants receiving
degrees or retaining employment. Moreover, while closeout reports do
include information about impediments to program success and "best
practices," HUD did not systematically review this information or share
it with YouthBuild USA, which is the primary technical assistance
contractor for the program with responsibilities that include
identifying new strategies for operating successful programs. As a
result, the lack of follow-up data and programwide evaluations make it
very difficult to determine which programs and strategies have worked
best and assess the performance of Youthbuild over time. As Labor
assumes program oversight responsibilities, officials said they would
consider that the closeout reports include program participant data,
such as degrees obtained prior to program entry and details on
participant outcomes such as the types of construction jobs obtained
and retained upon program completion. The officials also told us that
they believe impediments and best practices should be evaluated and
disseminated to grantees, but they had not yet determined who would
have primary responsibility for doing so.
Grantees had varying success in obtaining funds from outside sources,
but grant officials told us that HUD funding was critical to attracting
the leveraged funds. In assessing applications, HUD gave consideration
to program applicants that were able to secure outside funding.
Applicants receive points based on the ratio of committed leveraged
funds compared with the amount of Youthbuild funds requested in the
application. Data from the closeout reports showed the median amount
spent by HUD Youthbuild grantees was $400,000 in HUD funds and an
additional $430,000 from outside sources. However, success in obtaining
additional funds varied widely, from 21 grants reporting no additional
funding sources to 40 reporting more than $1 million. While most
grantees have been successful in generating outside funding, several
grant officials told us that many existing and potential contributors
would not be willing to continue funding if they perceived that the
program would not be able to continue without the HUD grant. YouthBuild
USA reported that most grantees have had difficulty continuing
operations without continued HUD funding. Further, YouthBuild USA also
noted that grantees with follow-on HUD funding achieved better
performance outcomes than grantees that did not receive subsequent HUD
funding.
This report contains three recommendations for the Secretaries of HUD
and Labor to analyze closeout reports and share the best practices
identified, develop post-program performance outcomes and share the
data with grantees, and consider whether multiyear funding could be
useful in helping Youthbuild grantees attract additional outside
funding. We provided a draft of this report to HUD and Labor for their
review and comments. Officials from HUD's Office of Community Planning
and Development (CPD) provided comments. They stated that they did not
disagree with the report's overall findings, but added that HUD's
oversight activity was limited by resource constraints. Labor's
Assistant Secretary for Employment and Training wrote that she agreed
with our recommendation to develop and monitor post program performance
outcome measures and share data with grantees. She also wrote that
Labor agreed that there could be value in multiyear funding and that
the department was considering such an approach for existing grantees.
Background:
CPD manages the Youthbuild program. CPD programs generally fall into
three categories: (1) formula grants, of which the largest are the
Community Development Block Grant and the HOME Investment Partnership
Program, which account for the majority of CPD funding; (2) competitive
homeless programs; and (3) competitive nonhomeless programs, including
Youthbuild. While no staff are dedicated solely to Youthbuild
oversight, CPD monitors the program as part of its oversight
responsibilities for 22 programs and initiatives, which 829 staff (233
in headquarters and 596 in field offices) administered as of the
beginning of fiscal year 2007. In fiscal year 2006, Congress
appropriated $19.1 billion for the CPD programs and initiatives, of
which Youthbuild represents a small portion.[Footnote 8] The first
program appropriation for Youthbuild was $40 million for fiscal year
1993. Appropriations reached a high of $65 million in fiscal year 2002,
but dropped to $49.5 million in fiscal year 2006, the last year in
which HUD was authorized to award grants (see fig. 1).[Footnote 9]
Figure 1: Youthbuild Appropriations, Fiscal Years 1993-2006:
[See PDF for image]
Source: HUD.
[End of figure]
HUD has provided program funding for Youthbuild through an annual
competitive grant application process. Any applicant can receive up to
a $700,000 grant; however, new applicants and rural/underserved
applicants primarily receive grants up to $400,000. Grant awards can
last up to 30 months (or longer if the grantee is given an extension),
but some grantees use the funds at a much quicker rate and usually
reapply every year, while others may take the full 30 months.
Prospective grantees are state, private, nonprofit, and local entities
that implement the Youthbuild program by serving low-income youth, ages
16 to 24, who demonstrate educational need--either by virtue of being
high-school dropouts or by justifying their need for inclusion in the
program. According to program regulations, at least 75 percent of the
participants must come from very low-income families and have dropped
out of high school.[Footnote 10] The program allows up to 25 percent of
the youth to have a high-school diploma or have a slightly higher
income; however, data developed by YouthBuild USA show that most
organizations report that less than 25 percent of participants have
high-school diplomas or General Equivalency Diplomas (GED) upon
entering their programs. Also, according to data from YouthBuild USA, a
number of the youths have been adjudicated (judicially determined to be
delinquent), were on welfare upon entering the program, or lived in
public housing (see fig. 2). Other studies we reviewed identified
similar participant characteristics. Program regulations require
participants to spend 50 percent of their program time receiving job
training on construction sites, building affordable housing, and 50
percent receiving academic training. Recently, some of the Youthbuild
grantees have added nonconstruction programs to their offerings, such
as in the areas of computer repair and health care. Funding for the
additional programs has come from AmeriCorps.[Footnote 11]
Figure 2: Characteristics of Participants Entering YouthBuild USA-
Affiliated Programs, 1997-2005:
[See PDF for image]
Source: YouthBuild USA.
Note: Participants could have one or more of depicted characteristics.
[End of figure]
In 1994, YouthBuild USA, located in Somerville, Massachusetts, became
the primary technical assistance contractor for the Youthbuild program.
YouthBuild USA is a national support center and intermediary for the
program. As such, it provides training, develops materials and
handbooks to help grantees replicate the program, conducts peer-to-peer
seminars, and disseminates materials on best practices. YouthBuild USA
also has established an affiliated network that grantees can choose to
join. As of September 2006, 107 of 173 Youthbuild grantees with active
grants were affiliates. Since 2002, YouthBuild USA has been using a Web-
based system, WebSTA, to collect participant characteristic and
performance data from its affiliates.
Among the purposes of transfering the YouthBuild program to Labor was
better alignment of existing federal workforce and youth training
programs. Labor's Employment and Training Administration offers
programs that assist disadvantaged youths, similar to the programs that
Youthbuild grantees offer. The Workforce Investment Act (WIA) of 1998
established a variety of programs to serve low-income youths who face
barriers to employment. Labor allocates funds for the youth service
programs to state and local areas based on a distribution formula. The
grants are made to states, and the states in turn allocate funds to
approximately 600 local workforce investment boards. The amount of
funding available to states is determined by the distribution of
unemployed individuals and disadvantaged youths by state. From 2000
through 2005, Labor administered the Youth Opportunity Grant program,
which also was authorized by WIA. This grant program focused on
improving education and employment opportunities for at-risk youths,
ages 14 to 21, in high-poverty areas. Funding for the program was
eliminated beginning in fiscal year 2004. In addition, Labor
administers the Job Corps program, which began in 1964, and provides
education, training and support services to economically challenged
youths, ages 16 through 24, who face multiple barriers to employment.
Private companies--chosen through competitive contracting processes--
and other federal agencies operating under interagency agreements with
Labor, operate the Job Corps centers (located throughout the United
States and Puerto Rico) on behalf of Labor.
HUD's Grant-Making Procedures Incorporated Legislative Criteria:
Each year, HUD staff rated and ranked grant applications for serving
economically distressed areas based on factors outlined in the enacting
legislation and detailed in an annual NOFA. The Housing and Community
Development Act of 1992 required HUD to establish selection criteria
such as the qualifications or potential capabilities of applicants, the
feasibility of the proposed Youthbuild program, the potential for
developing a successful Youthbuild program, and the applicants'
commitment to obtain outside resources.[Footnote 12]
Accordingly, HUD annually issued a NOFA that described the grant
application process, outlined criteria for grant awards, and included
scoring factors (see fig. 3). The major factors considered were the
capacity and relevant experience of the organization (performance),
need and extent of the problem, applicant's soundness of approach,
leveraging of nonhousing funds, and ability to show how it would
achieve and measure results. Under these factors, grant applicants had
to consider area poverty and high-school dropout rates, describe their
management and training capabilities, and offer proof of access to
property and the existence of partners to build low-income housing. HUD
issued its last NOFA for the Youthbuild program on March 8, 2006, and
announced the winning grantees in November 2006.
Figure 3: NOFA Point-Scoring System for Fiscal Year 2005 Applications:
[See PDF for image]
Source: HUD.
[End of figure]
Since fiscal year 2001, HUD has grouped applicants into three separate
scoring categories--new, general (that are typically awarded to
established Youthbuild organizations), and rural/underserved area
grantees. Figure 4 shows the awards by category. The total numbers of
grants as well as the grants by the individual categories have varied
by year.
Figure 4: HUD Awards by Funding Category for Fiscal Years 2001-2006:
[See PDF for image]
Source: HUD.
[End of figure]
The awards process also has been competitive. HUD historically funded
less than one-third of all the applicants (see fig. 5). Since 1996, the
number of applicants funded ranged from about 7 to 44 percent.
Figure 5: Youthbuild Grant Applications and the Number Funded by Fiscal
Year (1996-2006):
[See PDF for image]
Source: HUD.
[End of figure]
HUD Oversight of Grantees' Performance Has Been Limited:
Although HUD requires grantees to submit closeout reports, which are
intended to help evaluate the programs, it has not verified,
aggregated, or analyzed data from those reports. Further, the closeout
reports do not include other data that could help determine participant
characteristics upon entry or the types of jobs they received after
completing the program. YouthBuild USA, the technical services provider
for HUD, collects quarterly information, but it has not aggregated the
information to obtain performance information over the span of a grant.
HUD also has encountered significant delays in developing a database
intended to gather additional performance information from grantees.
Finally, citing resource constraints, HUD has conducted limited on-site
oversight of grantees. As a result of a lack of verified and aggregated
data or analysis and limited oversight, HUD largely was unable to tell
how well the individual Youthbuild grantees performed.
HUD Has Not Collected or Analyzed Data That Could Help It Assess
Grantees:
HUD requires grantees to submit performance reports, including closeout
reports, but HUD has not developed or fully utilized information--
particularly from the closeout reports--that could help it assess
Youthbuild grantee programs and has lacked resources to conduct
comprehensive oversight of grantees. Such information serves as the
basis for management and oversight of programs. For instance, HUD
states in the instructions for filling out the closeout reports that
the reports will be used to monitor and evaluate progress of grantees
and programs. Further, the Government Performance and Results Act of
1993 (GPRA) shifted the focus of federal managers from the number of
tasks completed or services provided to a more direct consideration of
the outcomes or results of the program. GPRA provides a performance-
based management framework for agencies to set goals; measure progress
toward those goals; deploy strategies and resources to achieve them;
and ultimately, use performance information to make the programmatic
decisions necessary to improve performance. Accurate and complete
performance data that document outcomes are needed to inform Congress
and the public about the effectiveness of programs and help agencies
conduct effective oversight. The combination of baseline information
and intermediate outputs or outcomes can be used to show progress or
contribution toward final outcomes, that is, the intended results of
carrying out the program.[Footnote 13] For example, in Youthbuild,
baseline information such as the number of participants entering a
program with and without GEDs or high-school diplomas could be combined
with intermediate outputs and outcomes such as graduation rates and
jobs attained, which could then be linked to desired program outcomes
such as the number of Youthbuild participants attaining economic self-
sufficiency.
However, HUD has not set up a system to verify, aggregate, or analyze
the data from closeout reports--information and analyses that would
form the basis for assessments of grantee performance. Citing resource
constraints, HUD filed the reports in the individual grant files in
headquarters and field offices and did not further review them. In
addition, some basic baseline data are missing from the closeout
reports, including the number of participants who enter the programs
with a GED or high-school diploma. Such information is necessary to get
an accurate count of the number of participants receiving a GED degree
or diploma through Youthbuild programs and also provides a basis for
comparison with which performance can be measured. Furthermore, while
closeout reports require that some performance outputs be collected,
including the number of participants receiving GEDs or high-school
diplomas, placed in jobs, or entering higher education programs, the
reports do not collect other output information such as how many
participants were placed in construction-related employment.
Citing resource constraints and a need to comply with the Office of
Management and Budget (OMB) performance reporting requirements, HUD
contracted with YouthBuild USA in 2004 to collect quarterly grantee
performance information for all Youthbuild grantees. Programs
administered by six federal agencies, including HUD and Labor, are
subject to "common measures" reporting which OMB established to provide
agencies with the ability to describe, in a similar manner, their core
performance outputs across different grantee programs.[Footnote 14] The
common measures include employment or education placements, degrees or
certificates attained, and literacy and numeracy skills increased
(increases in educational functioning). Consequently, YouthBuild USA
provides HUD with program information including participant gender and
ethnicity data, GED attainment statistics, grade level improvements,
job or school placements, and the number of new and rehabilitated
housing units that were completed. HUD uses the reports to provide
performance information to Congress and meet OMB reporting
requirements. However, because the information is captured on a
quarterly basis, it only represents the performance of a program at a
particular point in time and YouthBuild USA has not aggregated the
information to obtain data for the term of individual grants. As a
result, HUD has been largely unable to track program outcomes, but its
data will be needed for ongoing program assessments.
Moreover, HUD has encountered significant delays in developing a
database intended to provide additional performance outcomes on grantee
programs. Recognizing this deficiency, HUD officials said that they
have been planning for about 3 years to implement an electronic data
collection system to help HUD measure the success of Youthbuild
programs, along with its other nonhomeless competitive grant programs.
However, the system's implementation has been delayed because of
funding and technical problems. As a result of not systematically
collecting or analyzing needed information from closeout reports,
relying predominately on limited point-in-time program information, and
not being able to utilize a planned database to aggregate performance
outcomes, HUD is limiting its ability to assess grantees. Further,
although the YouthBuild Transfer Act shifted oversight of grants issued
in fiscal year 2007 to Labor, HUD will be required to continue
monitoring existing Youthbuild grants for approximately the next 3
years. Therefore, the data HUD has at its disposal or could develop
will continue to be valuable for ongoing assessments of the program.
Citing Resource Constraints, HUD Performed Limited Oversight of
Grantees:
In addition to not having or utilizing available performance data, HUD
has exercised limited on-site review and oversight of grantee
performance. According to HUD's Office of Inspector General, on-site
monitoring is an essential tool for HUD (through CPD) to assess program
performance and identify and address potential program problems. Such
monitoring is also useful in motivating grantees to exercise sound
judgment in carrying out their grant activities. GAO's Standards for
Internal Control in the Federal Government note that controls generally
should be in place to ensure that ongoing monitoring occurs during
normal operations. Monitoring should be performed continually, be
ingrained in an entity's operations, be used to assess the quality of
performance over time, and support the prompt resolution of any
identified problems.[Footnote 15]
CPD goals require on-site monitoring of only a small percentage of
grants. For fiscal year 2003, the plan called for the field offices to
monitor 35 percent of formula grants; 20 percent of competitive
homeless grants; and 7 percent of competitive nonhomeless grants,
including Youthbuild. HUD has since changed its baseline monitoring
goals (requiring monitoring of 20 percent of grantees with active, open
grants established since 2004) to give its field offices more
flexibility to meet monitoring priorities based on local grantee
knowledge. As a result, goals are no longer set by program. Field
offices decide on the number of formula versus competitive grantees to
be monitored based on risk analyses that evaluate each grantee on
factors such as the size of the grant, perceived project management
capacity, grantee responsiveness, and the extent of citizen complaints.
HUD officials cited a lack of resources as the primary reason for not
requiring the monitoring of a larger percentage of grantees. In 2006,
according to HUD officials, field offices typically had one person
working part time responsible for overseeing the individual Youthbuild
grantees in the state, visiting no more than one or two grantees per
year. As a result, field staff have not been able to comprehensively
monitor grantees and verify the accuracy of the output data submitted
by the grantees.
In general, we found that the monitoring reports that we reviewed
(which HUD field offices produced) on grantees primarily addressed
various compliance issues, rather than reporting on the overall
performance of a particular grant. For example, several reports
identified grantees' failure to receive prior approval from HUD for
changing proposed job training construction sites. Other reports
identified inadequate record keeping, insufficient documentation, and
lack of sufficient financial internal controls on the part of grantees.
However, although a few reports did comment on some individual
performance outputs, such as job placements or GEDs attained, most did
not.
The lack of analysis of performance information of grantees also has
implications for assessing the Youthbuild program overall, particularly
because HUD has devoted few staff to overall program management and
oversight. Since 2000, in headquarters, Youthbuild had a program
director with one or two staff managing the program each year. During
fiscal year 2006, with the expected program transfer to Labor, the HUD
program manager was assigned to other duties, and a part-time program
manager was assigned to oversee the 2006 grant process, in which HUD
received 308 applications and awarded 74 grants. Although they are
aware of the limited oversight of Youthbuild, CPD division officials
said it has provided appropriate staff resources for program oversight
of Youthbuild grantees, commensurate with the program's size--the
annual appropriation for which represents less than 1 percent of CPD's
usual annual budget of about $8 billion. However, without sufficient
oversight HUD was unable to tell how the individual Youthbuild grantee
programs performed.
Incomplete Data on Participants' Outcomes Precludes an Overall
Assessment of Program's Success:
HUD has not performed the programwide analysis needed to determine
overall program success, and weaknesses in other assessments of the
programs and data preclude an overall assessment of Youthbuild success.
While limited in scope, three studies suggest Youthbuild helped youths
achieve economic self-sufficiency. And, although YouthBuild USA tracks
performance data, grantees do not have the follow-up information needed
to measure performance outcomes. To augment limited existing data, we
analyzed information in 245 closeout reports, but the results cannot be
generalized to the entire program. We found a wide range of results in
outputs such as job placements; however, because grantees generally did
not follow up on participants, we were unable to determine long-term
outcomes such as percentages of participants retaining employment.
Moreover, while closeout reports include impediments to success and
"best practices," HUD did not systematically review this information or
share it with YouthBuild USA. As a result, the lack of follow-up data
and programwide evaluations make it very difficult to determine which
programs and strategies have worked best and assess the performance of
Youthbuild over time. Labor officials said they would consider that the
closeout reports include outcomes such as jobs obtained and retained
and said that they believe impediments and best practices should be
evaluated and disseminated to grantees.
A Few Studies Document Individual Youthbuild Program Successes but Are
Limited in Scope:
A few studies have provided some evidence of the success of individual
Youthbuild programs. Although subject to many limitations, the studies
suggest that identified programs have achieved certain Youthbuild
objectives. However, without comparable information for the entire
population of Youthbuild grantees, it is not possible to determine the
extent to which these outcomes and reported successes are reflective of
what was achieved by the grantees not included in the studies. More
specifically:
* In June 2004, researchers from Brandeis and Temple Universities
published a study entitled, "Life after YouthBuild - 900 YouthBuild
Graduates Reflect on Their Lives, Dreams, and Experiences," in
collaboration with YouthBuild USA. The study was based on a survey of
graduates from 11 Youthbuild grant sites and included in-depth
interviews with 57 graduates from eight of these sites. The researchers
determined that, for those 882 participants (for a 23 percent response
rate) who responded to the survey, many are doing well and were helped
considerably by the Youthbuild program. Many participants found jobs or
furthered their education and have volunteered in community activities.
In addition, a number of participants surveyed indicated they also
obtained intangible benefits from the program, such as increased self-
esteem and had developed positive attitudes about themselves and their
community. A limited nonresponse analysis was conducted by YouthBuild
USA to compare the 882 survey responses and the 57 interviews with one
site that had a 98 percent response rate. However, using one site that
had a high response rate is not representative and cannot be
generalized to the whole sample. As such, it is not known if the 77
percent who did not respond were doing more or less well than those who
responded to the survey.
* In August 2003, HUD issued a study entitled, "Evaluation of the
Youthbuild Program," in which a HUD contractor compared program results
and costs per participant from the Youthbuild Program and four other
federal employment and training programs--Job Corps, JOBSTART, both
designed to serve disadvantage youth, and the youth components of the
Job Training Partnership Act and Supported Work Demonstration
programs[Footnote 16]. The researchers conducted their work at 20
Youthbuild sites, selected in part because they had received funding
for at least two grants, obtained a variety of organizational
sponsorships, built new or rehabilitated housing, and covered a variety
of funding category sizes. The study found that 36 percent of the
Youthbuild participants at these sites obtained jobs after enrolling
and 29 percent obtained either a GED or high-school diploma. In
addition, 12 percent of the Youthbuild graduates in the 20 sites
reportedly went onto higher education. However, results at the five
federal programs were not directly comparable because researchers
obtained data from Youthbuild program participants at the time they
exited the program while data from the other programs' were obtained
from tracking participants up to 3 years after leaving the program. In
addition, the limited scope of this study precludes an overall
assessment of the program's success.
* In February 2003, the state of Minnesota issued a study entitled,
"Minnesota Youthbuild Program: A Measurement of Costs and Benefits to
the State of Minnesota." This study analyzed the costs and benefits of
the state-funded Youthbuild program in Minnesota. The researchers
measured post-program participant performance outcomes by tracking
program year participants beyond graduation. They found that Youthbuild
graduates earned more than double the minimum wage, whereas youth with
similar characteristics who had not been through the program typically
earned only the minimum wage. In addition, the graduates who had been
adjudicated had a considerably lower rate of recidivism (rearrest,
reconviction, or return to prison) than other youth involved with the
correctional system.[Footnote 17] Using state administrative data to
estimate the amount of revenues and savings generated through increased
tax revenues and reduced prison costs, the study estimated first-year
benefits to the state of about $1.5 million dollars versus costs of
about $877,000. Over a 4-year period, the study estimated the total net
benefit to the state at about $7.3 million (by 2006). The study was
limited to costs and benefits directly related to state programs and
did not include potential benefits such as reductions to welfare rolls,
state-sponsored medical care, or other reductions in human services
costs.
YouthBuild USA WebSTA Data Have Limitations, Particularly for Assessing
Outcomes:
Since 2002, YouthBuild USA has been using a Web-based system--WebSTA--
to collect participant characteristics and performance data from its
affiliates. According to YouthBuild USA officials, as of December 2006,
about 90 percent of its affiliates were entering data into WebSTA to
some extent. However, the value of YouthBuild USA's WebSTA data is
limited, largely because grantees self-report the data, not all
grantees are meeting the requirement to send in information, and the
information is not reported at the grant level. That is, if affiliates
have more than one Youthbuild grant, the data for each grant are not
reported separately. As a result, the data are not verified,
comprehensive, or comparable. Also, because the grantees submit the
information quarterly, the data provide only a point-in-time snapshot
of the program's performance and cannot be summed to provide annual
data. Recognizing these limitations, according to YouthBuild USA's
President, upgrades are planned to be implemented in early 2007 that
will allow grantees to enter data in WebSTA by individual grant, which
in turn will allow YouthBuild USA to analyze grant-level performance.
We also found that the limited depth of the information collected
restricts the value of WebSTA information for performance assessments.
For example, WebSTA collects information on participant attendance,
retention and graduation rates, and initial job and college placements
and participant characteristics (such as welfare, public housing, and
adjudication status) when participants enter Youthbuild programs.
However, many grantees do not follow up on participants once they leave
the programs. As a result, they often do not know if students placed in
jobs were still working at those jobs weeks or months later. Also,
several program directors we visited said that most of their program
graduates who go on to higher education sign up for free remedial-level
courses at nearby colleges. Although they have not had the funding to
systematically follow the students' progress, some said they generally
have found that the students have had varying success in advancing to
college-level credit courses.
Without grantee-level follow-up systems, WebSTA cannot track program
outcomes that would reflect whether the graduates are achieving
economic self-sufficiency--such as the percentage of program graduates
who remove themselves from the welfare rolls, move out of public
housing, or stay out of the court system. The system also does not
collect follow-up information on the type of jobs attained (including
construction), pay raises, benefits received, and college or trade
degrees or certifications received. Analyzing such performance outcomes
could provide, HUD, Labor, and other interested parties with the
measures needed to evaluate overall program performance and success.
Closeout Reports We Analyzed Provided Little Detail on Long-Term
Outcomes Because Grantees Are Not Required to Follow up on Graduates:
Because HUD had not aggregated or analyzed data for Youthbuild, we
obtained and analyzed 245 closeout reports that grantees submitted to
HUD. Figure 6 provides selected performance information that we
summarized from the closeout reports. However, because HUD has not
established objective performance goals for the program, which would
serve as baseline criteria against which to judge performance, the
information related to the experience of the 12,863 participants who
entered the program under the 245 grants gave us limited perspectives
about how individual programs were working. In addition, because we had
information on only 245 (or 46 percent) of the grants, the results
cannot be generalized to the entire program.
As shown in figure 6 the percentages for participants graduating,
receiving a GED, or being placed in employment varied widely. As
previously discussed, HUD has not set benchmarks, and critical data
elements (baseline data) are missing in the closeout reports. For
instance, it is not possible to obtain an accurate number of diplomas
and GEDs attained from the closeout reports because the reports do not
contain data on the number of participants that enter the program with
a high-school diploma or GED. As result, we could not determine how
well the programs (except for those with the very highest percentages)
performed individually or in comparison with each other.
Figure 6: Based on Closeout Reports, Youthbuild Performance Measures
Indicate Varied Outputs:
[See PDF for image]
Source: GAO analysis of grantee closeout reports.
[A] Not all grantees provided information for each category.
[B] Includes 1,239 participants that were reported as still active
because they were carried over to a subsequent grant or allowed to stay
on the books to receive continuing education and other support
services. In addition, the drop outs, graduates, and still active
categories do not sum to the number of participants category because
all grantees did not report information for each category.
[C] Regulations allow up to 25 percent of the participants who enroll
to already have a GED or high-school diploma. As a result, this number
may be underrepresentative since the data do not specify the number of
participants that entered the program with a GED or high-school
diploma.
[D] The same participants can be included in the employment and higher
education categories.
[End of figure]
The closeout reports also do not provide consistent and reliable data
on the number of units rehabilitated or built by Youthbuild
participants. A basic purpose for which Youthbuild was authorized was
to expand the supply of affordable housing. According to the closeout
reports, 158 grantees reported that their participants built a total of
1,483 new units, while 139 grantees reported that their participants
rehabilitated a total of 2,238 units.[Footnote 18] Some of the
Youthbuild program directors with whom we met said that during a
typical reporting period they would complete from one to four new or
gut-rehabilitated housing units. Other directors said that when a
program reports about 200 units rehabilitated, it generally means the
students are performing minor repetitive tasks such as changing locks
or painting a limited number of interior walls. As a result, the number
of units rehabilitated likely is overstated in the closeout reports.
Although construction training is a major part of the Youthbuild
program, HUD's closeout reports also do not track the number of
graduates obtaining construction-related jobs or specify the types of
jobs obtained (construction or otherwise).
In addition, HUD officials told us that grantees are not required to
follow up on participants after they leave the program. While some
grantee officials with whom we met recently had attempted to establish
follow-up systems, most grantees cited funding constraints as the
reason for not following up on graduates. Several officials further
noted that it was difficult to locate many participants after they had
left the program. However, as a result, it is not possible to obtain
comprehensive post-program results.
Finally, some program directors with whom we spoke cautioned that any
follow-up system should not be strictly focused on students obtaining
higher education degrees or obtaining high-paying jobs. They emphasized
that such goals were beyond the abilities of some of their
participants, but that moving from a fifth-grade to a ninth-grade
reading level or obtaining a service sector job sometimes was a real
victory and should be counted as a success.
Limited Information We Developed from Site Visits Pointed to Varied
Results for Programs, and Also Provided Little Detail on Long-Term
Outcomes:
The information we obtained from our visits also demonstrates varied
results for continued employment and academic success, and little
detail on long-term success. As with our closeout report analysis, the
information cannot be generalized to the universe of Youthbuild
grantees--although YouthBuild USA officials told us that they believe
we visited some of the more innovative grantees. For example, although
not required, some of the nine grantees we visited used other funding
sources, such as United Way or AmeriCorps funds, to establish a
participant follow-up system. As a result, some of the grantees were
able to provide us with varying amounts of information on the status of
program graduates. For example:
* One grantee, which was able to track individuals who completed the
program since 2001, provided us with data that showed 70 of the 82
program participants were either employed or pursuing higher education.
Seven of the 70 participants were working and going to school. The
hourly wages of those employed ranged from $7 to $13.75. However, the
grantee did not have information on the types of employment.
* Another grantee that recently established a follow-up system showed
that 11 of the 26 program graduates who started in 2004 had obtained
employment. The positions ranged from pizza delivery person to union
carpenter. However, only the union carpenter and a graduate hired by
the program as a construction trainer were working in the construction
field. The hourly wages of the graduates ranged from $5.50 to more than
$16 per hour for a graduate hired by the grantee.
* Another grantee used WEBSTA to track continuing education and work
history, including each job held. The program director told us that
such a follow-up system allows the program to spot trends, which in
turn can be used to fine-tune the program. Of 60 graduates from 2002 to
2005, the program continually has tracked 53 and reported that 8 are
receiving higher education, 24 are working in nonconstruction jobs, and
16 are working in construction jobs. The pay for the graduates ranged
from $6.50 to $15 per hour. The grantee continues to keep in touch with
the graduates by helping them with housing, legal aid, school advice,
tutoring, and help toward achieving a GED.
Grantees are not required to collect such post-program follow-up
information. Yet, both grantees and program participants emphasized
that the Youthbuild program created a very supportive atmosphere that
many participants had never experienced prior to joining a program.
During our visits, we met with students who credited the program with
improving their lives by either helping get them off drugs, get out of
the gang lifestyle, or disengage from other activities that likely
could have resulted in incarceration or worse. The studies we mentioned
previously had similar findings. However, it is difficult to gain
insight into how many program participants remain drug free, crime
free, in advanced degree programs, or working in construction or other
trades without post-program performance data. It is also difficult to
gain insight into which program strategies are most effective for
serving these youth.
HUD Does Not Review or Disseminate Impediments and Best Practices
Identified in Closeout Reports That Could Help Improve Program
Performance:
Citing resource constraints, HUD did not systematically review the
impediments and best practices identified in closeout reports or share
the information in them with YouthBuild USA. In particular, identifying
best practices (or the converse, impediments) can provide a model for
improvement for other organizations with similar missions. The closeout
reports contain a section addressing what the grantees considered to be
best practices. For example:
* Several grantees set-up educational opportunities, such as GED
preparation, at local colleges for participants. Two grantees noted
that being on a college campus provided a positive experience for the
participants. Two grantees noted that they made an arrangement with a
local technical college to offer free enrollment to all participants
that successfully completed the Youthbuild program. Another grantee
noted that working through community college GED programs allowed them
to lower their per-student cost and serve more students.
* One grantee established an alumni council consisting of elected
representatives from each of its seven graduating classes. All
graduates receive quarterly mailings that include an alumni newsletter,
new job and educational opportunities, and a quarterly calendar of
events.
* Another grantee has maintained a program serving graduates for up to
12 months to provide on-going employment support, including assistance
in securing a second job in the event that the initial placement did
not work out.
* One grantee started a "drug free" club and found that students really
wanted to participate and were anxious to become drug free and would
publicly announce their success.
* Another grantee worked extensively recruiting area college students
to volunteer to help with tutoring and program community service
activities.
Closeout reports also contain information on impediments that grantees
encountered in implementing and operating the programs. Impediments to
program success varied in scope and severity. Some of the more
significant impediments mentioned by numerous grantees included the
following.
* Turnover of key staff, especially on-site construction managers;
* Negative and apathetic attitudes among participants;
* Alcohol and drug abuse issues among participants;
* Difficulties in dealing with municipal organizations to obtain
building permits and contractual difficulties, which delayed
construction training activities; and:
* Acceptance of students with very low educational levels, which made
obtaining high GED levels very difficult.
HUD's contract with YouthBuild USA specifically calls for the
contractor to assist HUD in the management, supervision, and
coordination of Youthbuild programs by strengthening Youthbuild program
design and disseminating information on best program practices.
Although analyzing the impediment and best practices falls within the
scope of HUD's contract with YouthBuild USA, two CPD program officials
said that HUD never considered sharing the reports with its technical
services provider. YouthBuild USA's chief operating officer said that
the organization would welcome access to the closeout reports. The
officer said his staff already spent many hours each month trying to
identify best practices from Youthbuild grantees and would be anxious
to analyze the closeout reports for both the problems and successes
identified and forward the information to interested grantees.
YouthBuild USA's President told us that she not only agreed that
reviewing the closeout reports for problems and "best practices" would
be beneficial, but that she believes the closeout reports needed to "be
analyzed to get to the bottom line because the total impact (outcomes)
of a grant could not be determined from data provided on a rolling
quarterly basis."[Footnote 19] As a result of not sharing the closeout
reports with YouthBuild USA, HUD may have missed opportunities to
improve the program as a whole and help grantees improve performance.
Labor Officials Indicated They Will Consider Using Information from
Closeout Reports to Assess and Improve YouthBuild Performance:
According to Labor officials, they will consider analyzing information
from the closeout reports to assess and improve the performance of the
YouthBuild program. Until 2006, Labor assessed past performance of its
youth employment and training grantees through measures required under
WIA. For example, under the WIA Youth and Job Corps programs, Labor
tracked and analyzed 6-month retention and earnings wage change
information. Labor officials told us that they would use OMB's common
measures to track and analyze the performance of YouthBuild grantees
but said that they have not finalized the information they would
require YouthBuild grantees to collect. The officials indicated to us
that they would consider developing post-program performance outcomes
such as the types of employment graduates attained and retained
(including construction), wage rates, and college or trade degrees or
certifications received. Although Labor may incur some additional costs
in developing such information, the ability to measure program success
would improve. Labor officials also noted that the information
currently contained in the closeout reports would be considered with
other factors they deem as pertinent to assessing grantees'
performance. However officials said they would also likely consider
modifying the closeout reports to include other program participant
data, such as degrees obtained prior to program entry and details on
the types of construction jobs obtained upon program completion.
Labor officials also acknowledged that information on program
impediments and best practices should be evaluated and passed to
grantees and that performance data needed to be aggregated, verified,
and analyzed to determine which programs and strategies worked best. As
of October 2006, Labor had not determined whether its Employment and
Training Administration or an outside contractor would have primary
responsibility for collecting, aggregating, and disseminating the
information.
Youthbuild Grantees Have Experienced Varied Success in Obtaining
Outside Funding Sources, and Many Cited Continued HUD Funding as
Crucial:
Our analysis of available closeout reports showed that the grantees
have had varying success in obtaining additional funding sources. In
assessing grant applications, HUD gave consideration to program
applicants that were able to secure outside funding. Applicants
received points for leveraging nonhousing resources compared with the
amount of Youthbuild funds requested in the application. Overall, the
median amount spent by a HUD Youthbuild grantee was $400,000 in HUD
funds; the HUD grant represented about 48 percent of the average total
spending per grantee, which included additional funding sources of
about $430,000. However, success in obtaining additional funds varied
widely, from 21 grants reporting no additional funding sources to 40
reporting more than $1 million (see fig. 7).
Figure 7: Outside Funding Amounts Varied Widely for Grants We Reviewed:
[See PDF for image]
Source: GAO analysis of HUD Closeout Reports.
[End of figure]
According to information in the closeout reports and the officials from
the grantees that we visited, the additional funding they obtained
largely came from federal, state, and local agencies, businesses, and
charitable organizations. According to a YouthBuild USA official,
states have provided funding to local Youthbuild grantees over the past
several years. For instance, the Massachusetts State Department of
Education has provided funding in its budget for the state's Youthbuild
program. The Massachusetts funds are provided to existing Youthbuild
programs, based on a formula the state developed in conjunction with a
state coalition of Youthbuild grantees. In Minnesota, 10 Youthbuild
programs in 2005 shared $754,000 in state Youthbuild program funding,
which the state supplemented with more than $3 million in matching
funds from local partners. In Minnesota as elsewhere, urban grantees
have had greater opportunities to obtain funding or in-kind support
from local businesses and charitable organizations than rural grantees
that typically have had access to fewer businesses and might not be a
focus of charitable activity.
Regardless of their success in obtaining outside funds, several
grantees we visited stated that their Youthbuild program could not be
sustained without the HUD grant because the HUD funding allowed them to
attract the leveraged funds. Also, several grant officials said that
many existing and potential contributors would not be willing to
continue funding if they perceived that the program would not be able
to continue without the HUD grant. One exception (which we did not
visit) was in Minnesota where the state Youthbuild program director
said that only 4 of the state's 10 programs have had HUD funds in the
past, including 2 programs which received HUD funding in 2004 through a
HUD grant administered by the state. The program director added that
the structure of the Minnesota state Youthbuild program is similar to
that of HUD's, although the programs are typically smaller, having from
10 to 25 students.
Information from YouthBuild USA generally agrees with grantees'
contention about the need for continuous HUD funding. Overall,
YouthBuild USA found that 60 percent of programs that HUD did not fund
for 2 consecutive years ceased operations, and 90 percent of programs
not funded for 3 consecutive years ceased operations. According to
YouthBuild USA, 173 of 462 grantees funded since 1994 currently are
operating with active HUD grants. YouthBuild USA, also found that
grantees with continuous HUD funding achieved higher outputs, such as
GED attainments and job placements, than the ones that did not receive
one or more follow-on grants. According to some grantee officials, that
success is partly due to continuous funding that allows grantees to
develop infrastructure and maintain experienced staff to establish a
better program. In its fiscal year 2005 Performance and Accountability
Report, HUD noted the benefits of making grants to previous awardees:
"Having established programs and experience running a Youthbuild
program, these grantees are more efficient in enrolling students,
resulting in a greater number of youth trained." Furthermore,
Youthbuild program directors with whom we met emphasized the intangible
benefit that such established programs provide such as a family
atmosphere that allows graduates to return for advice and support as
needed--a benefit that goes away when programs close.
HUD has not kept statistics to find out why some grantees did not
reapply for additional grants or why grantees did not qualify for
follow-up grants. Through discussions with a nonprobability selection
of 23 of the 100 grantees that closed programs, we found that some did
not reapply because they were no longer interested in dealing with at-
risk youth and others closed due to poor management. But most of the
closed grantees with whom we spoke said they closed because they lost
HUD funding for one or more years and were unable to obtain sufficient
private funds to make up for the lost funding. Of the sites we
contacted, two were still in operation after ceasing to receive HUD
funds--but they appeared to be the exception. Both were already
operating youth technical training programs prior to their first HUD
Youthbuild grant.
In addition, officials from several grantees that also received funding
from AmeriCorps stated that they preferred the AmeriCorps concept of
providing dedicated funds for 3 years. These officials stated that
knowing that funding levels were longer-term provided more continuity
to their program, enabled better planning, and allowed them to build a
cushion through leveraged funding in the event that they did not
receive a HUD grant for a year or two. Finally, when the question of
multiyear funding was raised at a Web-based listening session in
December 2006, designed to help Labor identify Youthbuild program
administration issues and concerns, most participating grantees spoke
out in favor of multiyear funding.
Conclusions:
HUD has not taken advantage of several opportunities to use existing
information for purposes of program assessment and oversight. Such
opportunities should not be ignored, particularly because HUD has
limited resources with which to assess and oversee current grantees.
Specifically, HUD has not verified, aggregated or analyzed closeout
reports, which grantees are required to submit. HUD also has conducted
limited oversight of grantees, citing resource constraints; the
impending transfer of oversight to Labor bodes no change to this
specific situation. However, given these conditions, analysis of
existing closeout reports would help provide crucial information on the
performance of the Youthbuild programs, particularly in the absence of
a centralized HUD database that could be used to track program
performance. At a minimum, the closeout reports form a logical basis
for providing useful information that HUD, Labor, and other interested
parties can use to judge the performance of Youthbuild. HUD still will
maintain an active involvement for several more years, so coordination
between Labor and HUD will be critical to producing reliable
assessments of how successful the program has been and will be. HUD
also does not share reported constraints and best practices with the
technical services contractor it pays for performing such work and
disseminating best practices. By not utilizing performance data and
sharing information, HUD has missed opportunities to determine whether
the overall Youthbuild program has achieved its intended results, among
them enabling disadvantaged young adults to gain economic self-
sufficiency. Therefore, Congress and the public also lack the
information needed to make such determinations.
Labor has experience in managing youth training and education programs
and collecting performance information, and Congress transferred
Youthbuild to Labor in part to better align the program with existing
federal workforce development and youth training programs. In offering
perspectives for Labor to consider, we stress the value that closeout
reports have in the absence of more comprehensive and programwide data,
although the existing reports lack certain elements that could help
program managers determine and report on outcomes. The reports do
contain valuable information on impediments and best practices, which
could provide both program managers and grantees with much broader
perspectives on how successful the program has been and suggestions for
improvements. Labor has indicated that it would consider developing
post-program performance outcomes such as the types of jobs graduates
attained and retained (including construction), wage rates, and college
or trade degrees or certifications received. As a result, while there
may be some additional costs involved, the ability to measure program
success would improve. And, because alternatives to the HUD data are
limited, the grant-level assessment discussed above could act as the
building blocks of an overall program assessment. Labor also has agreed
that information on impediments and best practices would be valuable
and ought to be passed on to grantees, but has not yet determined how
this might be done.
Finally, data on grantee success in attracting other funding, although
limited, offer more perspectives for Labor to consider because outside
funding helps leverage limited federal dollars. While many grantees
have obtained extensive outside funding and support, YouthBuild USA has
found that very few operated for more than a few years without
continued HUD funding. Tellingly, their data show that 90 percent of
programs not funded by HUD for 3 consecutive years ceased operations.
According to YouthBuild USA officials, their data indicate that
grantees with repeat awards also produced better outcomes such as a
higher level of job placements. Further, when HUD funding stopped,
potential contributors became concerned that grantees would no longer
stay in business and were reluctant to jeopardize their own funds.
These data, while not definitive, do suggest that multiyear funding has
the potential to produce dividends for the program. Many grantees
expressed similar belief in the benefits of longer term funding at a
Department of Labor YouthBuild program listening session.
Recommendations for Executive Action:
To improve the reporting and assessment of performance for Youthbuild
grantee programs and develop the bases for an overall assessment of the
program, we are making the following three recommendations:
* that the Secretary of HUD analyze closeout reports by grant and share
information on identified problems and "best practices" with its
technical services contractor and Labor;
* that the Secretary of Labor develop and monitor post-program
performance outcome measures for the YouthBuild program, such as the
types of employment graduates attained and retained, wage rates, and
degrees or certifications received, and share the data with the
grantees; and:
* that the Secretary of Labor consider whether multiyear funding could
be useful in helping YouthBuild grantees attract additional outside
funding.
Agency Comments and Our Evaluation:
We provided HUD and Labor with a draft of this report for review and
comment. HUD's comments are summarized below. Labor's Assistant
Secretary for Employment and Training provided written comments that
are presented in appendix I.
Officials from HUD's Office of Community and Planning and Development
provided comments. They stated that they did not disagree with GAO's
overall findings, but added that HUD's oversight responsibility was
limited by resource constraints. They cited the fact that the
Youthbuild program represents 0.63 percent of the Community Planning
and Development Division's entire portfolio. Nevertheless, because HUD
will have monitoring responsibilities for existing grants for 3 years
or more, we believe it is important that HUD analyze the closeout
reports in order to provide Labor with information about how current
grantees are performing.
Labor's Assistant Secretary for Employment and Training wrote that she
believes that the report will be very useful to Labor as it assumes
responsibility for administering the YouthBuild program. She noted that
Labor agreed with our recommendation to develop and monitor post
program performance outcome measures for the YouthBuild program and
share these data with grantees. She added that Labor was building a Web-
based Management Information System to collect and report on
participant information. Finally, she wrote that Labor agreed that
there could be value on multiyear funding and that the Department was
considering such an approach for existing grantees.
We are sending copies of this report to other interested congressional
committees and the Secretaries of Housing and Urban Development and
Labor. We will also make copies available to others upon request. The
report will also be available at no charge on GAO's Web site at
http://www.gao.gov.
If you or your staff have questions regarding this report, please
contact me at (202) 512-4325 or shearw@gao.gov. Contact points for our
Offices of Public and Congressional Affairs may be found on the last
page of this report. GAO staff who made major contributions to this
report are listed in appendix II.
Signed by:
William B. Shear:
Director, Financial Markets and Community Investment:
[End of section]
Appendix I: Comments from the U.S. Department of Labor:
U.S. Department of Labor:
Assistant Secretary for Employment and Training:
Washington, D,C. 20210:
Feb 20 2007:
Mr. William B. Shear:
Director, Financial Markets and Community Investment:
Government Accountability Office:
441 G Street NW:
Washington, D.C. 20548:
Dear Mr. Shear:
Thank you for the opportunity to comment on your Government
Accountability Office (GAO) draft report, "YouthBuild Program: Analysis
of Outcome Data Needed to Determine Long Term Benefits" GAO-07-82. This
is a valuable report that will be very useful to the Department of
Labor (DOL) as we assume responsibility for administering the
YouthBuild program. I will focus my comments on the two recommendations
addressed to the Department of Labor.
The Department of Labor agrees with the GAO recommendation to develop
and monitor post-program performance outcome measures for the
YouthBuild program and share these data with grantees. The Department
of Labor is building a web-based Management Information System (MIS)
that will collect individual record data for all YouthBuild
participants enrolled in the DOL-funded YouthBuild programs. At this
time, we intend to require that YouthBuild grantees collect and report
participant information on demographics, assessment information (such
as educational attainment upon program enrollment), services, and
outcomes. The grantees will also be required to provide one year of
follow-up services and collect one year of post-program follow-up data.
DOL intends to collect quarterly performance reports, generated by the
web-based MIS, which will include the three youth common performance
measures (placement in employment/ education, attainment of a degree/
certificate, and literacy/ numeracy gains) as well as a six month
retention rate, a recidivism rate, and additional data, such as wage
rates at placement and types of employment placements. DOL has the
ability to aggregate individual participant data to create grantee and
national level reports along with additional management reports for
grantees and DOL to support performance and improvement.
As for the second recommendation to consider multi-year funding of
grants for the program, the Department of Labor is currently evaluating
the most effective way to manage the grant funding cycle for
YouthBuild. We agree that there may be value in developing a multi-year
grant cycle for a variety of reasons. Therefore, we are giving
consideration to a two tiered approach, which would combine both multi-
year cycles to support continuity of funding and service delivery and a
single-year funding cycle to promote opportunities for new grantees.
If you would like additional information, please don't hesitate to call
me at (202) 693-2700:
Sincerely,
Signed by:
Emily Stover DeRocco:
[End of section]
Appendix II: GAO Contact and Staff Acknowledgments:
GAO Contact:
William B. Shear, (202) 512-4325, shearw@gao.gov:
Staff Acknowledgments:
In addition to the individual named above, Andy Finkel, Assistant
Director; Rich LaMore, John McGrail, Marc Molino, Luann Moy, Barbara
Roesmann, and Tom Taydus made key contributions to this report.
FOOTNOTES
[1] Youthbuild was originally authorized by provisions contained in the
Housing and Community Development Act of 1992. The YouthBuild Transfer
Act repealed the original authorizing legislation, effective September
2006, when the Youthbuild program was transferred to the Department of
Labor (Labor). The original authorizing legislation was in effect
during the time period covered by our report.
[2] HUD spells Youthbuild with a lower case "b," as written in the
Housing and Community Development Act of 1992. As such, when referring
to the HUD program, we spell Youthbuild with a "b." The YouthBuild
Transfer Act used the upper case "B" spelling. To be consistent with
the legislation, Labor plans to use the capitalized spelling when it
takes over program administration.
[3] Grantees include public or private nonprofit organizations, states
or units of local government, or any entity eligible to provide
education and employment training under other federal employment
training programs.
[4] Andrew Hahn, Thomas Leavitt, Erin McNamara Horvat, and James Earl
Davis, "Life after YouthBuild, 900 YouthBuild Graduates Reflect on
Their Lives, Dreams, and Experiences" (YouthBuild USA: June 2004);
Maxine V. Mitchell, CRE et al., "Evaluation of the Youthbuild Program
(U.S. Department of Housing and Urban Development, Office of Policy
Development and Research: August 2003); and Minnesota Department of
Economic Security, "Minnesota Youthbuild Program: A Measurement of
Costs and Benefits to the State of Minnesota" (Commissioned by the
Minnesota Youthbuild Coalition, Prepared by MDES staff: February 2003).
[5] HUD identifies the reports as Final Performance Evaluation Reports.
However, in the report we use the term "closeout reports."
[6] Youthbuild locations we visited included San Francisco, California;
Bloomington and Rockford, Illinois; Brockton, Lowell, and Springfield,
Massachusetts; Philadelphia, Pennsylvania; Providence, Rhode Island;
and Washington, D.C.
[7] Government Performance and Results Act of 1993 and S. Rep. No. 58,
103rd Cong., 1st Sess. (1993); Office of Management and Budget Circular
(OMB) A-11, Part 6, "Preparation and Submission of Strategic Plans,
Annual Performance Plans, and Annual Program Performance Reports" (June
2006).
[8] Appropriations for CPD programs and initiatives in fiscal year 2006
were significantly higher than the $8 billion in prior fiscal years
because of the approximately $11.5 billion Congress added for disaster
assistance to five Gulf Coast states affected by hurricanes.
[9] HUD completed the grant awards process in November 2006.
[10] Very low-income families are those with incomes 50 percent or less
of area median income.
[11] The Corporation for National and Community Service, a public-
private partnership, administers AmeriCorps. The program allows
participants to earn education awards to help pay for postsecondary
education in exchange for performing community service.
[12] HUD program officials also complied with the fairness,
transparency, and nondisclosure requirements of sections 102 and 103 of
the HUD Reform Act of 1989.
[13] GPRA; S. Rep. No. 58, 103rd Cong., 1st Sess. (1993); and OMB
Circular A-11.
[14] In addition to HUD and Labor, the Departments of Education,
Health, and Human Services, Veterans Affairs, and Interior are subject
to the common measures.
[15] GAO issued these standards as required by 31 U.S.C. §3512(c). Also
see GAO, Standards for Internal Control in the Federal Government, GAO/
AIMD-00-21.3.1 (Washington, D.C.: November 1999) and Internal Control
Management and Evaluation Tool, GAO-01-1008G (Washington, D.C.: August
2001).
[16] The Job Training Partnership Act program was replaced by the
Workforce Investment Act, which was implemented in 2000. This
evaluation included data from 1996 through 1999 when the Job Training
Partnership Act programs were still in effect.
[17] Minnesota Youthbuild program participants entered unsubsidized
employment with an average starting wage of $11.60 an hour. The minimum
wage of $5.15 an hour was used as a baseline for the comparison group.
Fourteen (5 percent) of the participants with one or more offenses
prior to enrollment were rearrested, reconvicted, or returned to a
state correctional facility within 2 years after enrollment in the
Minnesota Youthbuild program, compared with a 20 percent rate of return
for juveniles within 2 years of release from Minnesota correctional
facilities.
[18] Some grantees reported that they used their program to both
construct new units as well as rehabilitated others under the same
grant.
[19] As we previously discussed, the quarterly data cannot be summed to
provide an annualized report.
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site (www.gao.gov). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site.
To have GAO e-mail you a list of newly posted products every afternoon,
go to www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125
Washington, D.C. 20548:
Public Affairs:
Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548:
