Small Business Innovation Research
SBA Should Work with Agencies to Improve the Data Available for Program Evaluation Gao ID: GAO-11-698 August 15, 2011Federal agencies with a budget of at least $100 million for research and development (R&D) conducted by others must participate in the Small Business Innovation Research (SBIR) program. SBIR has four purposes: meet federal R&D needs; stimulate technological innovation; increase commercialization (e.g., sales) of innovations based on federal R&D; and encourage participation in innovation by small businesses owned by disadvantaged individuals and women. The Small Business Administration (SBA) oversees the efforts of participating agencies, which make awards to small businesses using SBIR funds. Congress directed SBA to develop a database with commercialization data for government use in evaluating the program. GAO was asked to determine (1) how agencies have addressed SBIR's purposes and (2) the extent of data available to evaluate progress in increasing commercialization. GAO analyzed program documents and interviewed officials at SBA and five agencies that accounted for about 96 percent of SBIR awards.
For fiscal years 2008 through 2011, the participating agencies GAO reviewed--the Department of Defense (DOD), the Department of Energy (DOE), the National Aeronautics and Space Administration, the Department of Health and Human Services' National Institutes of Health, and the National Science Foundation (NSF)--addressed SBIR's purposes through solicitations for award applications, technical assistance or matching funds programs, and outreach. In particular, the agencies addressed SBIR purposes related to meeting federal R&D needs and stimulating technological innovation through their solicitations, which included research topics that were designed to meet agencies' respective R&D or mission needs. Agencies also addressed commercialization of innovations through solicitations, as well as through technical assistance for award recipients or through matching funds programs. To provide technical assistance, the agencies contracted with vendors and consultants for help in developing business plans and identifying potential customers for SBIR award recipients, among other things. Agency matching funds programs provided additional SBIR funds to award recipients that obtained commitments from outside investors. Agencies generally addressed the remaining SBIR purpose, encouraging participation by small businesses owned by disadvantaged individuals and women, through outreach activities aimed at a broader audience, such as sharing information on Web sites. However, the effectiveness of these efforts is difficult to evaluate, in part because SBA does not collect data on the number of SBIR applications submitted by such businesses, thus hindering analysis of trends in their submission of applications. Comparable data are not available across participating agencies to evaluate progress in increasing commercialization of SBIR technologies. SBA has not yet expanded an existing database to include commercialization data for program evaluation, but the agency has hired a contractor and allocated funds to develop the expanded database by August 2011. SBA has also worked with participating agencies to develop common metrics for commercialization. In the absence of the expanded database, agencies have independently gathered commercialization data for their own use that are not comparable. In collecting these data, agencies differed in the types of data collection instruments used, dates the instruments were administered, award recipient populations queried, and types of data requested. Furthermore, with the exception of DOD, agencies that GAO reviewed did not generally take steps to verify commercialization data they collected from award recipients, so the accuracy of the data is largely unknown. SBA has worked with SBIR agencies to identify best practices in other areas of program management but has not identified best practices for agencies to use in verifying the accuracy of commercialization data. Implementing the expanded database should improve the comparability of commercialization data available, but a lack of consistent practices for verifying the accuracy of these data may limit their usefulness for programwide evaluation. GAO recommends that SBA work with participating agencies to (1) collect data on applications from small businesses owned by disadvantaged individuals and women and (2) identify best practices for verification of commercialization data. SBA, DOE, and NSF generally agreed with these recommendations; the other agencies GAO reviewed neither agreed nor disagreed.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Franklin W. Rusco Team: Government Accountability Office: Natural Resources and Environment Phone: (202) 512-4597GAO-11-698, Small Business Innovation Research: SBA Should Work with Agencies to Improve the Data Available for Program Evaluation
This is the accessible text file for GAO report number GAO-11-698
entitled 'Small Business Innovation Research: SBA Should Work with
Agencies to Improve the Data Available for Program Evaluation' which
was released on September 14, 2011.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
United States Government Accountability Office:
GAO:
Report to Congressional Requesters:
August 2011:
Small Business Innovation Research:
SBA Should Work with Agencies to Improve the Data Available for
Program Evaluation:
GAO-11-698:
GAO Highlights:
Highlights of GAO-11-698, a report to congressional requesters.
Why GAO Did This Study:
Federal agencies with a budget of at least $100 million for research
and development (R&D) conducted by others must participate in the
Small Business Innovation Research (SBIR) program. SBIR has four
purposes: meet federal R&D needs; stimulate technological innovation;
increase commercialization (e.g., sales) of innovations based on
federal R&D; and encourage participation in innovation by small
businesses owned by disadvantaged individuals and women. The Small
Business Administration (SBA) oversees the efforts of participating
agencies, which make awards to small businesses using SBIR funds.
Congress directed SBA to develop a database with commercialization
data for government use in evaluating the program. GAO was asked to
determine (1) how agencies have addressed SBIR‘s purposes and (2) the
extent of data available to evaluate progress in increasing
commercialization. GAO analyzed program documents and interviewed
officials at SBA and five agencies that accounted for about 96 percent
of SBIR awards.
What GAO Found:
For fiscal years 2008 through 2011, the participating agencies GAO
reviewed”-the Department of Defense (DOD), the Department of Energy
(DOE), the National Aeronautics and Space Administration, the
Department of Health and Human Services‘ National Institutes of
Health, and the National Science Foundation (NSF)-”addressed SBIR‘s
purposes through solicitations for award applications, technical
assistance or matching funds programs, and outreach. In particular,
the agencies addressed SBIR purposes related to meeting federal R&D
needs and stimulating technological innovation through their
solicitations, which included research topics that were designed to
meet agencies‘ respective R&D or mission needs. Agencies also
addressed commercialization of innovations through solicitations, as
well as through technical assistance for award recipients or through
matching funds programs. To provide technical assistance, the agencies
contracted with vendors and consultants for help in developing
business plans and identifying potential customers for SBIR award
recipients, among other things. Agency matching funds programs
provided additional SBIR funds to award recipients that obtained
commitments from outside investors. Agencies generally addressed the
remaining SBIR purpose, encouraging participation by small businesses
owned by disadvantaged individuals and women, through outreach
activities aimed at a broader audience, such as sharing information on
Web sites. However, the effectiveness of these efforts is difficult to
evaluate, in part because SBA does not collect data on the number of
SBIR applications submitted by such businesses, thus hindering
analysis of trends in their submission of applications.
Comparable data are not available across participating agencies to
evaluate progress in increasing commercialization of SBIR
technologies. SBA has not yet expanded an existing database to include
commercialization data for program evaluation, but the agency has
hired a contractor and allocated funds to develop the expanded
database by August 2011. SBA has also worked with participating
agencies to develop common metrics for commercialization. In the
absence of the expanded database, agencies have independently gathered
commercialization data for their own use that are not comparable. In
collecting these data, agencies differed in the types of data
collection instruments used, dates the instruments were administered,
award recipient populations queried, and types of data requested.
Furthermore, with the exception of DOD, agencies that GAO reviewed did
not generally take steps to verify commercialization data they
collected from award recipients, so the accuracy of the data is
largely unknown. SBA has worked with SBIR agencies to identify best
practices in other areas of program management but has not identified
best practices for agencies to use in verifying the accuracy of
commercialization data. Implementing the expanded database should
improve the comparability of commercialization data available, but a
lack of consistent practices for verifying the accuracy of these data
may limit their usefulness for programwide evaluation.
What GAO Recommends:
GAO recommends that SBA work with participating agencies to (1)
collect data on applications from small businesses owned by
disadvantaged individuals and women and (2) identify best practices
for verification of commercialization data. SBA, DOE, and NSF
generally agreed with these recommendations; the other agencies GAO
reviewed neither agreed nor disagreed.
[End of section]
Contents:
Letter:
Background:
Agencies Addressed SBIR's Purposes through Solicitations, Technical
Assistance, Matching Funds Programs, and Outreach:
Comparable Data Are Not Available across Participating Agencies to
Evaluate Progress in Increasing Commercialization of SBIR Technologies:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Comments from the Small Business Administration:
Appendix III: Comments from the Department of Energy:
Appendix IV: Comments from the National Science Foundation:
Appendix V: GAO Contact and Staff Acknowledgments:
Table:
Table 1: Summary of Five SBIR Agencies' Efforts to Gather
Commercialization Data:
Abbreviations:
DOD: Department of Defense:
DOE: Department of Energy:
NASA: National Aeronautics and Space Administration:
NIH: National Institutes of Health:
NSF: National Science Foundation:
R&D: Research and Development:
Recovery Act: American Recovery and Reinvestment Act:
SBA: Small Business Administration:
SBIR: Small Business Innovation Research:
STTR: Small Business Technology Transfer:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
August 15, 2011:
The Honorable Ben Quayle:
Chairman:
Subcommittee on Technology and Innovation:
Committee on Science, Space, and Technology:
House of Representatives:
The Honorable Adrian Smith:
House of Representatives:
Small businesses have been collectively described as an engine for
economic growth and innovation. According to the Small Business
Administration (SBA), such businesses have been responsible for
creating 64 percent of net new jobs over the past 15 years and have
produced 13 times more patents per employee than larger firms. To
assist small businesses in undertaking and obtaining the benefits of
research and development (R&D), Congress first passed legislation
authorizing the Small Business Innovation Research (SBIR) program in
1982. SBA's Office of Technology administers the program, which
presently has four overarching purposes: to use small businesses to
meet federal R&D needs, stimulate technological innovation, increase
commercialization of innovations derived from federal R&D efforts,
[Footnote 1] and encourage participation in technological innovation
by small businesses owned by disadvantaged individuals[Footnote 2] and
women.[Footnote 3] Since the program was first authorized in 1982,
there has been increased congressional interest in its results,
particularly the commercialization aspect.
Every federal agency with a budget of $100 million or more for
extramural R&D--which is conducted by nonfederal employees outside
federal facilities--is required to establish and operate an SBIR
program funded by 2.5 percent of that budget. Eleven agencies
currently participate in the program.[Footnote 4] In fiscal year 2009
(the most recent year for which SBA data are available), the 11
agencies reported more than $2.2 billion in SBIR awards. Although each
of these agencies manages its own program, SBA oversees and
coordinates agency efforts by setting policy, collecting program data,
reviewing agency progress, and reporting annually to Congress.
Since the creation of the SBIR program 29 years ago, it has been
reauthorized, modified, and extended by Congress at various times. The
Small Business Reauthorization Act of 2000 extended the program
through September 30, 2008. This act was succeeded by a series of
temporary extensions, the most recent of which extended the program
through September 30, 2011.
In this context, you asked us to examine the practices of
participating agencies. Accordingly, we agreed to determine (1) how
participating agencies have addressed the SBIR program's four
overarching purposes when implementing their programs and (2) the
extent of SBIR program data available to evaluate progress in
increasing commercialization of SBIR technologies.
Our review of the program focused on SBA and five agencies that
accounted for about 96 percent of the total dollars awarded by the
program in 2009: the Department of Defense (DOD), the Department of
Energy (DOE), the National Aeronautics and Space Administration
(NASA), the Department of Health and Human Services' National
Institutes of Health (NIH),[Footnote 5] and the National Science
Foundation (NSF).[Footnote 6] At two of these agencies--DOD and NIH--
we also reviewed program activities conducted by the three
subcomponent agencies with the largest SBIR budgets because some key
activities are carried out at that level. Specifically, for DOD, we
examined the SBIR programs of the Army, Air Force, and Navy and for
NIH, we examined the programs of the National Institute of Allergy and
Infectious Diseases; the National Cancer Institute; and the National
Heart, Lung, and Blood Institute. In conducting our review, we
analyzed documentation from these agencies for fiscal years 2008
through 2010 and, when possible, for fiscal year 2011. The
documentation we reviewed included, among other things, policy
guidance, solicitations for applications for SBIR awards, and
descriptions of commercialization assistance provided to SBIR
awardees. We also reviewed applicable laws and regulations. In
addition, we interviewed SBIR program officials at the agencies, as
well as inspector general staff at NSF and knowledgeable stakeholders,
including staff of the National Academy of Sciences' National Research
Council and representatives of trade associations. We selected the
trade associations based on their familiarity with the program, the
technologies on which they focus, and whether their membership
includes small businesses owned by disadvantaged individuals and
women.[Footnote 7] To obtain further context for our review, we
attended two national conferences on the SBIR program, as well as a
workshop conducted by the National Research Council. We conducted this
performance audit from June 2010 to August 2011, in accordance with
generally accepted government auditing standards. Those standards
require that we plan and perform the audit to obtain sufficient,
appropriate evidence to provide a reasonable basis for our findings
and conclusions based on our audit objectives. We believe that the
evidence we obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives. A more detailed description
of the scope of our review and the methods we used is contained in
appendix I.
Background:
SBA coordinates and oversees the efforts of the 11 agencies currently
participating in the SBIR program. SBA coordinates the agencies'
schedules for issuing solicitations--announcements of opportunities
for small businesses to apply for awards--and provides access to these
solicitations through its Web site. As part of its oversight effort,
SBA collects SBIR data from the participating agencies, aggregates the
data, and uses the data to, among other things, monitor the program
and report to Congress. SBA also provides guidance to participating
agencies on the general conduct and operation of the program, which it
periodically updates, for example, in response to changes in the
program's authorizing legislation. Under the legislation and SBA's
guidance, agencies have considerable flexibility to design their
programs. For example, each agency determines, in consultation with
SBA, such items as the number of solicitations to be issued during a
fiscal year and the dates applications are due. Agencies also have
discretion to determine what type of research to include in their
solicitations, how to review applications for technical and scientific
merit, which applications to fund, and the size of the award, among
other things.
The Small Business Innovation Development Act of 1982 provided for a
competitive three-phased SBIR program. In phase I, participating
agencies award up to $150,000 for a period of about 6 to 9 months to
small businesses to conduct experimental or theoretical R&D. Small
businesses whose phase I projects demonstrate scientific and technical
merit, in addition to commercial potential, may compete for phase II
awards of up to $1 million to continue the R&D for an additional
period, normally not to exceed 2 years.[Footnote 8] Phase I and II
award funds may be used for costs related to conducting the research,
such as salaries, fringe benefits, equipment, and consulting services,
as well as for profits and fees. To be eligible for a phase I or II
SBIR award, a business must have 500 or fewer employees, be organized
for profit with a place of business in the United States, and operate
primarily in the United States or make a significant contribution to
the U.S. economy. Generally, a business must also be at least 51
percent owned and controlled by one or more individuals who are U.S.
citizens or permanent resident aliens.[Footnote 9] These eligibility
requirements apply at the time that a phase I or II award is made.
During phase III, businesses must secure non-SBIR funding to develop
the commercial potential of the innovative technologies resulting from
their SBIR projects;[Footnote 10] such funding may come from the
private sector, federal agencies, or other sources.
As the program has been reauthorized over the years, legislation has
established a number of requirements related to the program's
purposes. For example, the Small Business Research and Development
Enhancement Act of 1992 directed SBA to make more information
available about the SBIR program, particularly about participation by
small businesses owned by disadvantaged individuals and women, and
required that agencies increase their outreach to such businesses. In
addition, the Small Business Reauthorization Act of 2000 directed that
applicants for phase II SBIR awards be required to submit
commercialization plans[Footnote 11], and it mandated that SBA
develop, maintain, and make available to the public a database that
contained SBIR award data. In addition, the act required SBA to
develop and maintain, by June 2001, a restricted government-use
database that would contain award-related data from the public
database, as well as additional confidential data that would be
accessible only to government agencies and other authorized users. The
act stated that this database would be used exclusively for program
evaluation--which, as we have noted in past work, involves the
systematic collection and analysis of accurate, comparable, and
complete data on program result[Footnote 12]s. The act required the
government-use database to contain, among other things, data that
applicants for phase II awards would be required to supply on the
commercialization success of any prior phase II awards, such as data
on sales of or additional investment in the technologies funded under
the awards. The act further specified that the government-use database
would contain annual updates to these data, which phase II award
recipients would be requested to voluntarily provide for 5 years after
the period covered by the award.
To accomplish this mandate, SBA envisioned expanding an electronic
database, known as Tech-Net, which it had developed in the late 1990s,
into two sections: a public-use portion and a government-use portion
containing commercialization data. The public-use portion of the
database has been available since 2000, according to SBA, and it
contains such award-related data as the phase of the award, amount of
the award, name and location of the business receiving the award, an
abstract of the work to be conducted under the award, and whether the
business is categorized as owned by disadvantaged individuals or
women. In October 2006, however, we reported that some SBIR agencies
did not consistently provide or correctly format the awards-related
data for several fields in the public-use portion of the database.
[Footnote 13] For example, two of the eight agencies we reviewed had
not consistently provided data on whether the businesses receiving the
awards were categorized as owned by disadvantaged individuals or
women.[Footnote 14] At that time, we also reported that SBA had not
implemented the government-use portion of the database, primarily,
according to SBA officials, because of increased security requirements
for the database, agency management changes, and budgetary
constraints. Additionally, we reported that while five of the agencies
we reviewed had systematically collected commercialization data, their
data collection efforts differed in ways that made it challenging to
evaluate the program across agencies. In August 2009, we testified
before Congress that SBA said the database would no longer accept
incorrectly formatted awards-related data from participating
agencies.[Footnote 15]
A committee of the National Academy of Sciences' National Research
Council has conducted a series of assessments of the SBIR program,
both within and across agencies, as part of a legislatively mandated
study. The results were summarized in a single report, in which the
committee stated that SBIR is making significant progress in achieving
congressional goals.[Footnote 16] The study concluded that the SBIR
program is "sound in concept and effective in practice." The study
also recommended changes that could make the program more effective.
Among other things, the study recommended that SBA and participating
agencies improve the collection of data that track participation in
the SBIR program by businesses owned by disadvantaged individuals and
women, develop targeted outreach to such businesses that is based on
an analysis of factors that affect their participation, and improve
documentation of commercialization success. The National Research
Council is now undertaking another round of assessments to provide a
second snapshot of the program's progress in achieving its legislative
purposes.
Agencies Addressed SBIR's Purposes through Solicitations, Technical
Assistance, Matching Funds Programs, and Outreach:
For fiscal years 2008 through 2011, the five participating agencies we
reviewed addressed the SBIR purposes of using small business to meet
federal R&D needs and stimulating technological innovation through
their solicitations. Agencies also used solicitations, as well as
technical assistance or matching funds programs, to address the SBIR
purpose of increasing commercialization of innovations derived from
federal R&D efforts. To address the remaining program purpose--
encouraging participation in technological innovation by small
businesses owned by disadvantaged individuals and women--agencies
relied mainly on outreach activities aimed at a broader audience.
Agencies Addressed the SBIR Purposes Related to Meeting Federal
Research Needs and Stimulating Innovation through Their Solicitations:
All of the participating agencies that we reviewed designed the SBIR
solicitations that they issued for fiscal years 2008 through 2011 to
meet federal R&D or mission needs and stimulate technological
innovation.[Footnote 17] All of these agencies selected research
topics for their solicitations that were designed to meet their
respective R&D or mission needs and specified that applications would
be evaluated on the basis of responsiveness to those topics. The
agencies that purchase SBIR-funded technologies for their own use--
DOD, DOE, and NASA--tended to select solicitation topics that met
specific agency needs for R&D. For example, in fiscal year 2011, DOD
solicited applications to develop a fuel cell system capable of
converting ethanol into electricity in an efficient, small,
lightweight, portable power system. According to the solicitation,
such advanced fuel cell systems could provide soldiers power to
complement batteries and to charge rechargeable batteries, reducing
the number of batteries required for extended time in the field. In
contrast, NIH and NSF, which generally do not purchase SBIR-funded
technologies, tended to issue solicitations for a broader spectrum of
R&D to support their missions of advancing biomedical and other
scientific and engineering disciplines. Among the agencies we
reviewed, NIH and its components gave applicants the most leeway in
addressing agency needs: rather than limiting applications to specific
research topics identified in solicitations, NIH and its components
usually listed suggested topics and encouraged applicants to propose
innovative projects that fit the agency's mission.
Concerning innovation, each of the agencies included instructions in
its SBIR solicitations about the type of information applicants had to
provide about the innovativeness of the proposed work. For example,
NASA informed phase I and II applicants that a competitive application
would describe the proposed innovation relative to state-of-the-art
knowledge in the field, among other things. In addition, these
agencies explained to applicants how reviewers would consider evidence
of the innovativeness of the applicants' proposed research approaches.
For example, in its fiscal year 2010 solicitation, NSF stated that
applications would be evaluated, in part, on the basis of whether they
reflected state-of-the-art knowledge in the major research activities
proposed and whether the work was likely to advance state-of-the-art
knowledge.
Agencies Worked to Increase Commercialization of Innovations through
Solicitations, Technical Assistance, and Matching Funds Programs:
The participating agencies we reviewed addressed the SBIR purpose of
increasing commercialization of innovations through solicitations, as
well as through technical assistance or matching funds programs.
Solicitations. Of the five agencies we reviewed, all but NIH required
in their solicitations for fiscal years 2008 through 2011 that
applicants for phase I awards submit a commercialization strategy
demonstrating that the applicants had taken steps such as identifying
a market for their SBIR technologies, planning to secure financing,
and estimating expected future sales. For phase II awards, all of the
agencies we reviewed required that applicants submit a
commercialization plan.[Footnote 18] In general, the solicitations we
reviewed required that phase II commercialization plans discuss the
potential market and competitors; the qualifications of key management
and technical personnel; as well as financing, marketing, and
manufacturing plans, among other things.
The agencies we reviewed differed in their stated processes for
evaluating the commercial potential of applications. For example, DOD
guidance to applicants outlined a systematic process for how the
agency would consider commercialization potential when evaluating
applications submitted by small businesses that had received multiple
prior awards. DOD indicated that, under this process, it would assign
a commercialization achievement score to applicants that had completed
the work for four or more phase II awards from any agency;[Footnote
19] this score would reflect how the applicants' commercialization
experience compared with historical averages. Applicants whose scores
fell within the lowest 20 percent would not be allowed to receive more
than half the maximum number of points possible for commercialization
potential, which was to be assessed on the basis of several factors,
including the commercialization strategy or plan. DOD guidance stated
that businesses with fewer than four completed phase II awards would
not be affected by the absence of a commercialization achievement
score. Although the other four agencies we reviewed did not outline as
systematic a process for evaluating past commercialization success as
a gauge of commercialization potential, they still indicated that
commercialization potential would be taken into account in reviewing
applications. For example, DOE's solicitation instructions encouraged
phase I applicants to seek firm commitments for private-sector or non-
SBIR federal funding prior to applying for a phase II award. The
instructions further stated that phase II applicants that obtained
such commitments were more likely to receive full credit for
commercialization planning during the evaluation of their
applications. In the case of NSF, solicitation instructions stated
that proposals are usually reviewed by 3 to 10 outside experts in
fields related to the proposal; according to NSF officials, these
reviewers have business experience. NSF's solicitation instructions
further stated that the agency would not review applications that
lacked sufficient information on commercial potential.
In 2010, two agencies we reviewed also issued SBIR solicitations under
new programs that were explicitly oriented toward increasing
commercialization. Specifically, in July 2010, DOE launched a program
under which it solicited applications for phase III of SBIR, the
commercialization phase. DOE documents indicated that the agency would
make available approximately $30 million, including funding from the
American Recovery and Reinvestment Act (Recovery Act),[Footnote 20]
for phase III awards, which are intended to allow businesses to pursue
commercial applications of work performed under phase I and II awards.
In addition, NIH's National Cancer Institute began a program under
which it solicited phase I applications to continue development of
technologies that have originated in its laboratories, with the goal
of advancing these technologies toward commercial products.[Footnote
21] SBA has designated the use of the SBIR program to encourage
commercialization of agencies' internal research as a best practice on
its SBIR Web site.
Technical assistance. All five agencies included in our review
provided technical assistance to help award recipients build their
capacity to commercialize their technologies. To provide the
assistance, the agencies contracted with vendors and consultants who
have experience in bringing technologies to market. With the exception
of NASA, the agencies supported the technical assistance at least in
part through the use of SBIR funds.[Footnote 22]
In fiscal years 2008 to 2010, DOD, DOE, NIH, and NSF spent SBIR funds
on technical assistance for phase I award recipients. Some of the
assistance was in the form of interactive training Webinars or online
tools directed toward a broad spectrum of SBIR applicants and award
recipients. For example, the Navy offered phase I award recipients the
use of a software tool, known as WebTRIMS, that helps identify,
quantify, and track risks associated with SBIR technology development
and covers topics such as contracting strategies, business and
transition planning, and manufacturing readiness. Other phase I
assistance was more customized. For example, DOE offered phase I award
recipients customized technical assistance designed to help them
develop a commercialization plan complete with an implementation
schedule and suggestions for product design. Similarly, on a first-
come, first-served basis, NIH offered phase I award recipients
assessments of their SBIR-funded technologies' likely niche in the
existing commercial market, which could help recipients develop
commercialization plans for phase II applications.[Footnote 23]
Additionally, NSF offered phase I award recipients personalized
mentoring and coaching sessions with an advisor. According to NSF
officials, 92 percent of phase I recipients chose to participate in
the technical assistance program in 2010. NASA did not provide
technical assistance for phase I award recipients; NASA officials told
us they believed technical assistance would have the most utility for
phase II NASA award recipients.
During at least a portion of the period we reviewed, all five agencies
offered individualized technical assistance for phase II award
recipients, although DOE curtailed such assistance in 2010, and NASA
discontinued its assistance in 2008. Award recipients were selected
for assistance on the basis of factors such as recommendations from
SBIR program staff and the award recipients' potential for rapidly
moving their technologies to phase III. The assistance consisted of in-
depth training and one-on-one assistance from advisors and industry
experts. For example, as part of its Commercialization Pilot Program,
[Footnote 24] the Army assisted selected phase II SBIR award
recipients in assessing commercialization potential, developing
business plans, and matching their technologies with potential
government and industry customers. At DOE, staff could nominate phase
II award recipients for assistance in preparing to negotiate business
deals, for example, including joint ventures and licensing agreements
for use of their technologies. DOE officials told us that in 2010 the
agency curtailed its use of SBIR funds for phase II technical
assistance, spending such funds on assistance only for award
recipients that had specifically budgeted for it in their
applications. In 2007 and 2008, NASA partnered with the Navy to pilot
a technical assistance program for NASA phase II recipients. The
program was designed to help SBIR businesses develop a plan for
transitioning to phase III, among other things. In 2007, 17 phase II
companies with 19 SBIR projects participated in the program, and in
the following year, 19 phase II companies with 20 SBIR projects
participated. The program was not renewed for 2009; NASA officials
told us that they believed the program was generally successful, but
that they preferred to use SBIR funds to make larger awards.[Footnote
25] NIH offered selected current or past phase II award recipients the
opportunity to work one-on-one with an advisor over a 9-month period
to develop business plans to commercialize their technologies, as well
as to prepare materials to help attract potential investors or
partners. Since 2004, almost 700 award recipients have received the
assistance, including the 80 award recipients currently participating,
according to NIH officials. Finally, NSF offered customized assistance
to phase II award recipients through its Innovation Accelerator
Initiative. According to NSF officials, through this initiative, award
recipients received help in connecting with potential investors and
negotiating company acquisitions and mergers. NSF officials told us
that, in 2010, approximately 33 percent of NSF's phase II recipients
received this assistance.
In some cases, agencies that we reviewed used non-SBIR funds to
broaden the scope of the technical assistance they provided to help
award recipients commercialize their technologies.[Footnote 26] For
example, DOD used non-SBIR funds to host its annual Beyond Phase II
Conference and Technology Showcase, a 3-day event that features
matchmaking sessions with SBIR award recipients and prime contractors.
Similarly, the Navy used non-SBIR funds to maintain databases with
advanced searching capability to help award recipients identify
potential business partners. The Navy also used non-SBIR funds for its
Transition Assistance Program, which provides individualized help with
commercialization planning, culminating in a conference designed to
facilitate interaction with potential business partners. Moreover, in
2011, the National Cancer Institute launched its Regulatory Assistance
Program using non-SBIR funds. According to agency officials and
information from the agency's Web site, this program provides SBIR
award recipients time with consultants experienced in various
regulatory requirements--such as those for anticancer therapies,
imaging technologies, and medical devices--to prepare strategies for
obtaining regulatory approvals required before the technologies can be
commercialized. The National Cancer Institute also used non-SBIR funds
to support its Investor Forum, which provides competitively selected
SBIR award recipients an opportunity to showcase their technologies
and enter into discussions with the biotech investment community. In
2010, 14 award recipients that were selected on the basis of strength
of research, impact on cancer, product development, and market
potential participated in the forum along with more than 175 potential
investors, according to the agency's Web site.
Matching funds programs. Through matching funds programs, agencies
provide additional SBIR funds to award recipients that obtain monetary
commitments above certain thresholds from outside investors.[Footnote
27] SBA has designated matching funds programs as a best practice on
its SBIR Web site, and all of the agencies we reviewed except DOE have
established such programs.[Footnote 28] For example, for award
recipients that obtain a minimum of $100,000 from an outside investor,
NSF will match up to 50 percent of the outside investment for a
maximum of $500,000 in NSF matching funds. NASA and NIH officials said
that matching funds programs encourage outside investment during the
early stages of R&D--a time when many investors are reluctant to
invest. In particular, officials at the National Cancer Institute said
that matching funds can help attract outside investment because they
can be used as leverage to increase investors' potential returns. DOD
and NSF offer matching funds to award recipients at the end of phase I
and during phase II, while NASA and the National Cancer Institute
offer matching funds during phase II. DOE has not established a
matching funds program for its SBIR program. DOE officials told us,
however, that they are exploring whether to do so and have held
discussions with other SBIR participating agencies about their
matching funds programs.
Officials at DOD, NASA, and NIH said they have not collected data to
compare the commercialization success of recipients that received
matching funds with the success of those that did not.[Footnote 29]
NSF conducted a study to assess the effect of the $18 million in
matching funds it invested in fiscal year 2006 for 48 phase II award
recipients that had raised a total of $58 million from outside
investors[Footnote 30]. According to NSF officials, results of this
study showed that, in the 5 years following the start of these phase
II projects, 70 percent of recipients that had received matching funds
achieved commercial success compared with a 30 percent success rate
for recipients that had not received such funds.
Agencies Generally Encouraged Participation by Disadvantaged
Individuals and Women through Outreach Activities Directed toward a
Broader Audience:
SBA's guidance states that small businesses owned by disadvantaged
individuals and women must compete for SBIR awards on the same basis
as all other small businesses. However, to meet requirements for
greater outreach to small businesses owned by disadvantaged
individuals and women, SBA has encouraged participating SBIR agencies
to reach out to such businesses and to develop methods that encourage
their participation. SBA has also raised the topic of outreach during
recent quarterly meetings of agency SBIR program managers.[Footnote
31] Officials at all of the agencies we reviewed told us they
generally reach out to such businesses through activities directed
toward a broader audience, such as by attending SBIR national
conferences and industry-sponsored events and by sharing information
via Web sites or e-mail lists. Agency officials also noted that they
try to accommodate requests for speakers at events sponsored by, or
likely to be attended by, small businesses owned by disadvantaged
individuals and women--for example, events sponsored by trade
organizations for minority-or women-owned businesses. However,
officials from some trade organizations for businesses owned by
disadvantaged individuals and women told us that the outreach of
agencies we reviewed was often ineffective in educating the
organizations' members about the SBIR program.
Of the agencies we reviewed, NIH and NSF have made specific efforts,
including the following, to improve their outreach:
* For fiscal years 2010 and 2011, NIH developed a goal to increase
awareness of its SBIR program among businesses owned by disadvantaged
individuals and women, and it outlined specific activities aimed at
reaching this goal.
* Both NIH and NSF offered various fellowships for postdoctoral
research conducted by disadvantaged individuals and women; these
fellowships were available to support SBIR projects, as well as other
research.
* In 2010, NSF assigned a full-time staff member to help it develop a
plan to increase participation in SBIR by businesses owned by
disadvantaged individuals and women in response to a recommendation
from its SBIR advisory committee.
* Through a review of academic literature, as well as informal polling
of NSF applicants and award recipients, NSF has identified barriers to
SBIR participation by small businesses owned by disadvantaged
individuals and women, NSF officials told us. These barriers include
disparities in the owners' levels of education and access to capital
compared with those of other entrepreneurs. To address identified
barriers, NSF is, among other things, establishing partnerships with
industry and academia to expose African American, Latino, and other
college students to entrepreneurship in scientific and technical
fields, according to NSF officials.
Evaluation of the effectiveness of agencies' outreach efforts is
hindered by a lack of accurate and complete data. Although SBA
collects data on the number and dollar value of awards to small
businesses owned by disadvantaged individuals and women, SBA officials
told us that they cannot accurately tabulate data on such awards,
particularly awards to women-owned businesses, because of
inconsistencies in the data on business ownership. According to the
officials, SBA has taken steps to correct the inconsistencies for data
submitted after 2006 but has not done so for earlier years. Moreover,
SBA does not collect data on the number of applications submitted by
businesses owned by disadvantaged individuals and women. As a result,
SBA's data do not allow for an examination of trends in the submission
of applications from such businesses, analysis of the percentage of
applications from these businesses that lead to awards, or correlation
of these trends and percentages with outreach efforts. SBA officials
told us in March 2011 that they were considering whether their
database should include information on the numbers of applications
submitted by these businesses.
Comparable Data Are Not Available across Participating Agencies to
Evaluate Progress in Increasing Commercialization of SBIR Technologies:
SBA has not yet developed the government-use portion of its database
for collecting comparable commercialization data on SBIR technologies,
but it is taking steps to do so. In the interim, agencies have, for
their own purposes, independently gathered commercialization data that
are not comparable; the accuracy of these data is largely unknown.
Implementing the government-use portion of the database should improve
the comparability of the data. However, programwide evaluation of
progress in increasing commercialization may continue to be impaired
by long-standing challenges.
SBA Has Not Implemented the Government-Use Portion of Its Database
Intended to Collect Comparable Commercialization Data for Program
Evaluation but Is Taking Steps to Do So:
As of June 2011, SBA had not met the legislative mandate to develop
and implement, by June 2001, a government-use database that can
provide data on commercialization for evaluating the SBIR program.
[Footnote 32] However, the agency's efforts to develop such a database
recently gained additional prominence and resources. Specifically, SBA
linked development of the government-use portion of its database to
one of the agency's high-priority performance goals for fiscal years
2011 and 2012. Additionally, in September 2010, SBA allocated $1.4
million in Recovery Act funds to hire a new contractor to develop the
government-use portion's capacity to accept commercialization data
submitted by participating SBIR agencies and award recipients, as well
as to make other improvements to the database. For example, SBA said
that it has been working with the contractor to consolidate data on
previous awards. SBA officials said that past award recipients have
been assigned unique identifiers that will be used to track awards
issued to those recipients over the lifetime of the SBIR program;
unique identifiers are also to be assigned to small businesses newly
entering the program. In the future, SBA intends for the unique
identifiers to allow agencies to validate business information by
comparing it against information in other federal databases such as
the Central Contractor Registration database, which contains
information on businesses that want to contract with the federal
government.[Footnote 33] SBA officials told us that they expect to
implement the government-use portion of SBA's database by August 2011
and to provide for its basic maintenance and support despite
reductions in the agency's overall budget.[Footnote 34]
The government-use portion is intended to allow both participating
agencies and award recipients to enter commercialization data in a
comparable format to assist in program evaluation. SBA officials told
us that they have worked with participating agencies to develop common
metrics for commercialization data, as well as a standardized data
collection instrument that will accommodate the various types of SBIR
technologies the agencies fund to meet their different missions. These
metrics, which will correspond to fields in the database, include the
following:
* indication of whether an award resulted in a commercialized
technology and whether other SBIR awards contributed to
commercialization of the technology;
* estimated sales;
* estimated investment (other than SBIR funding);
* any patents applied for or received related to the award; and:
* any initial public offering, merger, or sale of the business that
resulted, at least in part, from the award.
SBA officials told us in May 2011 that they plan to implement the
metrics and data collection instrument in August 2011.[Footnote 35]
SBA is requesting that participating agencies voluntarily begin
entering historical commercialization data into the government-use
database before August 2011. To facilitate this process, SBA is
working with its contractor to ensure that historical agency data can
be matched to fields in the new database. Nevertheless, officials from
SBA and participating SBIR agencies said that some agencies may not
enter historical data or may be delayed in doing so because they
either did not collect such data or do not have the data in electronic
form. For example, NASA officials stated that much of their
commercialization data are stored in paper format and expressed doubt
that the agency would be able to convert the data into the required
format for entering by SBA's August deadline.
SBA officials also told us that, after the government-use portion of
the database is available, some agencies may instruct applicants and
award recipients to submit their commercialization data directly into
the database. Other agencies, such as DOD, may continue to require
applicants and recipients to submit commercialization data directly to
the agencies, which would then upload the data into the database. As
of May 2011, SBA officials were unsure which approach agencies would
take, noting that agencies may wait to see how the database works
before making a decision.
Without the Government-Use Portion of the SBA Database, Agencies Have
Independently Collected Commercialization Data That Are Not Comparable:
In the absence of the government-use portion of SBA's database, the
five participating SBIR agencies we reviewed have independently
collected commercialization data that are not comparable. The agencies
collected these data using various methods for their own purposes, as
summarized in table 1.
Table 1: Summary of Five SBIR Agencies' Efforts to Gather
Commercialization Data:
Agency: DOD;
Data collection instrument: Commercialization report submitted as a
component of applications for SBIR awards[A];
Dates administered: 2000 to present;
Population queried: Applicants for phase I and phase II awards;
Purpose of data collection: To assess an applicant's ability to
commercialize SBIR technologies, which DOD considers when selecting
projects for funding, and to assess the extent to which DOD's SBIR-
funded technologies are commercialized.
Agency: DOE;
Data collection instrument: Annual survey;
Dates administered: 1986-2007[B];
Population queried: Award recipients with active phase II awards[C];
Purpose of data collection: To assess the commercialization success of
DOE's SBIR-funded projects.
Agency: NASA;
Data collection instrument: Annual survey;
Dates administered: 1997-2002[D];
Population queried: Award recipients that received a phase II award
from 1983 through 1996;
Purpose of data collection: To assess the effect of changes that NASA
made in 1995 to its SBIR program management and application review
process--i.e., the effect of these changes on the success of award
recipients in commercializing technologies and in integrating those
technologies into the agency's mission programs.
Agency: NIH;
Data collection instrument: Periodic survey;
Dates administered: 2002 (with follow-up surveys of 2002 respondents
in 2004, 2005, and 2007);
2008; Population queried: 2002 and follow-up surveys: award recipients
that received phase II awards from 1992 through 2001; 2008: award
recipients that received phase II awards from 2002 through 2006;
Purpose of data collection: To evaluate the extent to which NIH's
program met the overall program goals, including the extent to which
it increased commercialization of SBIR technologies that the agency
funded.
Agency: NSF;
Data collection instrument: Periodic survey;
Dates administered: 2005-present;
Population queried: Phase II award recipients marking the 3rd, 5th,
and 8th anniversary of the receipt of their awards;
Purpose of data collection: To help assess the commercialization
success of NSF's SBIR program.
Source: GAO analysis of agency data.
[A] Small businesses applying for DOD SBIR awards must report on the
commercialization history of any prior phase II SBIR awards received.
[B] According to DOE officials, they discontinued the annual survey
for a combination of reasons, including contractor staff changes and
resource constraints. However, DOE has indicated that it expects to
resume surveying in fall 2011 using an improved survey instrument.
[C] DOE award recipients were surveyed during the years in which their
phase II awards were active and for up to 5 years after the awards
ended.
[D] According to NASA officials, the agency has not conducted a follow-
up survey owing to a lack of resources, although the agency has
continued to link its survey instrument to its SBIR solicitations for
applicants to complete voluntarily. However, agency officials noted
that response rates have been low.
[End of table]
In conducting their data collection efforts, agencies differed in the
extent to which they asked award recipients to do the following, among
other things:
* Identify the type of customer and the amount of sales or further
investment for SBIR-funded technologies. For example, most agencies
asked award recipients to report federal and nonfederal sales
separately, but NIH and NSF asked award recipients to report combined
sales.
* Account for indirect sales and nonfinancial indicators of
commercialization. NASA, NIH, and NSF asked award recipients to
indicate whether an SBIR-funded technology had resulted in licensing
agreements with other businesses to sell the technology, while DOD and
DOE did not ask that question. NASA further asked award recipients to
estimate the financial value of such agreements, while the other
agencies did not. Similarly NASA, NIH, and NSF asked award recipients
to indicate whether specific SBIR-funded technologies had resulted in
patents, while DOD and DOE asked award recipients to report the total
number of patents resulting from all their SBIR awards.
* Quantify the dollar values of cumulative sales. While most agencies
asked award recipients to report a specific dollar amount in
cumulative sales resulting from their SBIR-funded technologies over a
period of time, NIH asked award recipients to report such sales by
choosing among ranges, beginning with "$50,000 or less" and extending
to "$50,000,000 or more." Because NIH has reported cumulative sales in
ranges rather than specific dollar amounts, comparing its results with
those reported by other agencies is difficult.
While each agency's data collection efforts resulted in, among other
information, estimates of total or average sales of SBIR technologies,
differences in the agencies' data collection efforts make it difficult
to compare results across agencies. The following are examples of
commercialization data reported by agencies:
* DOD estimated that commercialization of SBIR technologies that it
funded generated federal and nonfederal sales and non-SBIR funding of
$22 billion on a program investment of $11 billion from 2000 through
March 2010.[Footnote 36]
* DOE estimated that, from 1986 through 2007, SBIR technologies
developed by recipients of phase II awards resulted in a total of $2.4
billion in federal and nonfederal sales and $1.6 billion in non-SBIR
investment. On average, award recipients reported receiving more than
$3 million in sales related to SBIR-funded technologies. During the
same period, DOE reported that it had invested $1.6 billion in phase I
and II SBIR awards.
* NASA estimated that, as of 2002, SBIR technologies developed by
award recipients that received a phase II award from 1983 through 1996
had generated approximately $2.8 billion in federal and nonfederal
sales and non-SBIR funding compared with $1.1 billion in SBIR
investment from NASA.
* In NIH's 2002 survey, which covered 1992 through 2001, 27 percent of
respondents reported an estimated total of $821 million in sales of
SBIR technologies; the other respondents did not report any sales.
[Footnote 37] NIH estimated that it invested $2.2 billion in phase I
and phase II awards from 1992 through 2001. For the 2008 survey, which
covered 2002 through 2006, 33 percent of respondents reported an
estimated total of $396 million in federal and nonfederal sales of
SBIR technologies.[Footnote 38] NIH estimated that it invested $2.7
billion in phase I and phase II awards from 2002 through 2006. NIH was
the only agency we reviewed that reported sales lower than its SBIR
investment for the periods it examined. According to NIH officials,
many of the technologies that the agency supports through its SBIR
program, such as drugs and medical devices, take longer to
commercialize than those funded by other agencies because of the need
for extensive clinical testing and regulatory approval.
* NSF officials estimated that recipients marking the eighth
anniversary of the receipt of their awards from July 2005 through May
2010 had realized a total of $1.05 billion in commercial revenue.
[Footnote 39] NSF estimated that it invested $628 million in SBIR
awards during roughly the same period.[Footnote 40]
Further, with the exception of DOD, agencies we reviewed generally did
not take steps to verify commercialization data that they received
from award recipients, so the accuracy of the data is largely unknown.
As officials from some of the agencies in our review noted, award
recipients may have an incentive to overstate their commercialization
success in the hope of improving their prospects of receiving future
SBIR awards. While SBA has worked with SBIR agencies to identify best
practices in other areas of SBIR program management, it has not
identified best practices for agencies to use in verifying the
accuracy of commercialization data. Without consistent practices for
verifying the accuracy of these data, the usefulness of the government-
use portion of SBA's database as a tool for evaluating the SBIR
program's success in increasing commercialization may be limited.
To verify the accuracy of award recipients' commercialization data,
DOD performs an annual review of all projects in its Company
Commercialization Database, which contains the commercialization data
it gathers from award recipients. This review includes checks to
ensure that prior award recipients applying for new awards are not
reporting the same project results more than once, substituting the
results of one project for that of another, or incorrectly reporting
sales to third parties. According to DOD officials, after its 2010
review, the agency sent approximately 300 e-mail queries to applicants
whose reported commercialization data were identified as having
potential problems. The officials said that applicants that do not
respond to such queries are blocked from submitting further
applications until concerns related to their commercialization reports
are addressed. Even with these verification activities, however, Army
officials expressed concern to us about the accuracy of the
applicants' self-reported commercialization data; these officials
stated their preference for using data from the Federal Procurement
Data System, which contains government information on federal
contracts, including sales.[Footnote 41] Moreover, a Navy official
acknowledged the possibility that additional verification activities,
such as selective spot visits to SBIR award recipients, could further
deter recipients from misrepresenting their commercialization success,
although he noted that such activities would compete with other
administrative priorities. Similarly, officials from DOE and NIH
stated that additional verification activities would be useful but
also said that they needed to devote program administration resources
to higher priority activities, such as preparing solicitations and
supporting review panels for applications.[Footnote 42]
Implementing the Government-Use Portion of the Database Should Improve
the Comparability of Commercialization Data, but Long-standing
Challenges May Still Impair Evaluation of Commercialization Progress:
SBA's implementation of the government-use portion of its database
should improve the comparability of commercialization data available
for programwide evaluation. Nevertheless, long-standing challenges may
continue to impair programwide evaluation of progress in increasing
commercialization of SBIR-funded technologies. As we reported in
October 2006, notable among these challenges is that prior award
recipients that are no longer participating in the SBIR program are
not required to provide updated commercialization data and may prefer
not to do so.[Footnote 43] For example, DOD indicated in written
comments to us that, from 2008 to 2010, 46 percent of nonparticipating
prior phase II award recipients did not provide updates despite DOD's
request that they update commercialization data annually after their
awards ended.[Footnote 44] Similarly, in a report on its 2002 survey,
NASA observed that many recipients of multiple awards elected not to
respond to its survey despite "extensive telephone follow-up" and that
many recipients that ultimately responded "would likely have preferred
not to."[Footnote 45] Some award recipients may be reluctant to
provide commercialization data because the data are business-
sensitive. SBA officials told us that mechanisms to require or
encourage nonparticipating recipients to report their data need to be
explored. A NASA official told us that effective incentives to
encourage wider voluntary reporting might include publicizing
commercial success or giving monetary prizes for success.
The difficulties agencies face in persuading prior award recipients to
volunteer commercialization information can be compounded by
challenges in maintaining contact with them. Specifically, prior award
recipients can change names or personnel, go out of business, or be
sold during the 10 or more years that it can take for an SBIR-funded
technology to reach the marketplace. In NIH's 2002 survey of award
recipients, for example, the portion of the sample that was
"unusable"--a group that consisted primarily of recipients that no
longer existed or could not be found--increased from 2 percent in the
first year after the end of the award to 52 percent in the tenth year.
Programwide evaluation--particularly efforts to compare
commercialization success across agencies--can also be complicated by
differences in the time required to commercialize various types of
SBIR-funded technologies. Comparing agencies' commercialization
results at a given point in time may not present a true picture of
each agency's success because some agencies fund technologies that are
relatively close to being market-ready while others fund technologies
that need more extensive development or regulatory approval.
Furthermore, as we have previously reported, the SBIR program's other
goals remain important, and comparisons that focus on
commercialization may not adequately take into account progress toward
these goals.[Footnote 46] For example, one agency official told us
that some SBIR-funded technologies, such as those related to national
security, may never have great commercial potential but are important
to the agency's mission.
Conclusions:
DOD, DOE, NASA, NIH, and NSF have designed their SBIR solicitations to
address the program's purposes of using small businesses to meet
federal R&D needs, stimulating technological innovation, and
increasing commercialization of innovations derived from federal R&D
efforts, and they have further addressed commercialization by
providing technical assistance or matching funds to award recipients.
These agencies have also conducted outreach and other activities to
address the SBIR purpose of encouraging participation in technological
innovation by small businesses owned by disadvantaged individuals and
women. However, evaluation of progress in achieving the program's
purposes is impeded by a lack of accurate, comparable, and complete
data on program results. For example, it is difficult to evaluate the
program's effectiveness in encouraging small businesses owned by
disadvantaged individuals or women to participate in technological
innovation because SBA does not collect data on the number of
applications submitted by such businesses. It is also difficult to
evaluate the program's effectiveness in increasing commercialization
of SBIR-funded technologies because, although agencies participating
in the program have gathered commercialization data for their own
purposes, comparable data on commercialization are not available
across agencies. SBA's planned implementation of a government-use
portion of its database should go some way toward improving the
comparability of the commercialization data as they are systematically
collected using common metrics. However, the commercialization data
that the database is intended to contain are largely self-reported by
award recipients that may have an incentive to overstate their
commercialization success. DOD has adopted practices for verifying the
accuracy of commercialization data it collects from prior award
recipients, but most of the participating agencies we reviewed did not
verify the accuracy of commercialization data from their prior award
recipients, and SBA has not identified best practices for
participating agencies to use in doing so. As long as participating
agencies do not consistently verify the accuracy of commercialization
data, the usefulness of the government-use portion of SBA's database
as a tool for evaluating the SBIR program's success in increasing
commercialization may be limited.
Recommendations for Executive Action:
To build upon efforts to implement a government-use database for
program evaluation, we recommend that the Administrator of the Small
Business Administration work with participating SBIR agencies to take
the following two actions:
* collect data on the number of applications submitted by small
businesses owned by disadvantaged individuals and women, and:
* identify best practices for verifying the accuracy of data related
to progress in increasing commercialization.
Agency Comments and Our Evaluation:
We provided a draft of this report to SBA, the Departments of Defense
and Energy, the National Aeronautics and Space Administration, the
National Institutes of Health, and the National Science Foundation for
review and comment. SBA generally agreed with our findings as well as
our recommendations, which it offered an action plan to address.
Specifically, with respect to our first recommendation, SBA stated
that, beginning in fiscal year 2012, it plans to use its database to
collect information from agencies about applicants that did not
receive awards--information that could include whether the applicants
were small businesses owned by disadvantaged individuals or women.
Further, SBA indicated that it plans to hold a workshop in fall 2011
for participating SBIR agencies to share best practices for reaching
out to small businesses owned by disadvantaged individuals. According
to SBA, the workshop should result in a commitment from agencies to
develop baselines for numbers of applications from such businesses.
Regarding our second recommendation, SBA indicated that it will seek
to identify best practices and methods for verifying the accuracy of
commercialization data and will work with agencies toward
implementation of those practices and methods. SBA also noted that its
effort to collect commercialization data is intended to establish a
baseline, against which SBA can review progress in increasing
commercialization. SBA's letter conveying its comments is contained in
appendix II.
Among the SBIR participating agencies that we reviewed, DOE and NSF
concurred with our recommendations and provided general comments,
which are included in appendixes III and IV, respectively. Both DOE
and NSF also made technical comments, which we have incorporated into
our report as appropriate. In its general comments, DOE stated that it
collects information on the number of applications submitted by small
businesses owned by disadvantaged individuals and women and is willing
to report the data to SBA. DOE further stated that it does not verify
commercialization data because of resource limitations--not a belief
that verification is of limited value--and it expressed an interest in
learning about best practices for verification of these data. In
addition, DOE commented that, until universal metrics are identified
for measuring the success of SBIR programs across agencies, the
compatibility of available data among agencies will remain a secondary
concern. NSF stated that it concurs with the underlying goals of our
recommendations. Moreover, NSF affirmed its commitment to
implementation of a government-use database for program evaluation,
collection of data on participation in small business innovation, and
identification of best practices for verification of commercialization
data. The remaining agencies--DOD, HHS, and NASA--neither agreed nor
disagreed with our recommendations but provided technical comments,
which we have incorporated into our report as appropriate.
As agreed with your offices, unless you publicly announce the contents
of this report earlier, we plan no further distribution until 30 days
from the report date. At that time, we will send copies to the
appropriate congressional committees, Administrators of SBA and NASA,
Secretaries of Defense and Energy, Directors of NIH and NSF, and other
interested parties. In addition, this report will be available at no
charge on the GAO Web site at [hyperlink, http://www.gao.gov].
If you or your staff members have any questions about this report,
please contact me at (202) 512-3841 or ruscof@gao.gov. Contact points
for our Offices of Congressional Relations and Public Affairs may be
found on the last page of this report. Key contributors to this report
are listed in appendix V.
Signed by:
Frank Rusco:
Director, Natural Resources and Environment:
[End of section]
Appendix I: Scope and Methodology:
In conducting this study, we reviewed Small Business Innovation
Research (SBIR) program-related activities of the Small Business
Administration (SBA) and 5 of the 11 SBIR participating agencies--the
Department of Defense (DOD), Department of Energy, National
Aeronautics and Space Administration, Department of Health and Human
Services' National Institutes of Health (NIH),[Footnote 47] and
National Science Foundation (NSF). For the two agencies with the
largest SBIR budgets--DOD and NIH--we reviewed program activities
conducted by the three participating subcomponent agencies with the
largest SBIR budgets because some key activities are carried out at
that level. Specifically, for DOD, we examined the SBIR programs of
the Army, Air Force, and Navy, and for NIH, we examined the programs
of the National Institute of Allergy and Infectious Diseases; the
National Cancer Institute; and the National Heart, Lung, and Blood
Institute. The five participating agencies we reviewed accounted for
about 96 percent of the total dollars awarded by the program in fiscal
year 2009. We reviewed applicable laws and regulations and literature
on the SBIR program, including our prior reports and assessments by a
committee of the National Academy of Sciences' National Research
Council. To obtain further context for our review, we attended two
national conferences and a National Research Council workshop on the
SBIR program, and we interviewed National Research Council staff with
program expertise.
More specifically, to determine how participating agencies have
addressed the SBIR program's four overarching purposes when
implementing their programs, we reviewed SBA documents and data,
including SBA's policy directive on implementation of the SBIR
program, minutes from selected meetings of SBA and SBIR program
directors, SBA's SBIR annual report for fiscal year 2008 (the latest
year for which an annual report was available), and data on the dollar
value of SBIR awards by participating agencies in fiscal year 2009
(the latest year for which SBA could provide the data). We examined
relevant documents from participating agencies for fiscal years 2008
through 2010, and for fiscal year 2011 when possible. Documents we
reviewed included solicitations for applications issued by each of
these agencies, instructions to applicants, minutes from meetings of
SBA and SBIR program directors, performance plans and reports,
descriptions of commercialization assistance provided to SBIR
awardees, and minutes from meetings of agency advisory committees. In
addition, we also identified and interviewed SBIR program officials at
each agency and officials responsible for implementing programmatic
goals. For these interviews, we asked a standard set of questions to
help ensure that we obtained consistent information about the SBIR
programs at each of the agencies. We also interviewed inspector
general staff at NSF, which facilitated SBIR-related activities
conducted by the Council of Inspectors General on Integrity and
Efficiency. Finally, we interviewed representatives of trade
associations about their views of the SBIR program. We selected the
trade associations on the basis of their familiarity with the program,
the technologies on which they focus, and whether their membership
includes small businesses owned by disadvantaged groups and women. The
views of the representatives of these associations cannot be
generalized to other associations.
To determine the extent of SBIR program data available to evaluate
progress in increasing commercialization of SBIR technologies, we
reviewed documents related to SBA's SBIR database, including terms of
work, work schedules, and proposed guidance related to the development
of the government-use portion of the database. For the five SBIR
participating agencies whose programs we reviewed, we examined
documents dating from 2002 through 2011; these documents reflected
commercialization data for SBIR award recipients that had received
awards from 1983 (the first year in which agencies issued SBIR awards)
through 2010. The documents we reviewed included surveys and other
data collection instruments that the agencies used to gather
commercialization information from award recipients; reports on data
collection results, including any information on SBIR award spending
during the years corresponding to those covered in each of the
commercialization data collection efforts; and anecdotal descriptions
of commercialization success. We also reviewed agency solicitations
from fiscal years 2008 through 2010--and for fiscal year 2011 when
possible--that contained reporting requirements for award recipients.
We interviewed officials at SBA and each of the five participating
agencies included in our review to obtain information on the specific
commercialization metrics they use to monitor the commercialization
experience of award recipients, the history of each agency's data
collection efforts, and the agencies' experience in obtaining such
information from current and past award recipients.
We conducted this performance audit from June 2010 to August 2011, in
accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence we obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives.
[End of section]
Appendix II: Comments from the Small Business Administration:
U.S. Small Business Administration:
Washington, DC 20416:
July 28, 2011:
Mr Frank Rusco:
Director, Natural Resources and Environment:
Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear MI Rusco:
Thank you for your letter dated June 30, 2011, seeking the U.S. Small
Business Administration's (SBA's) comment on the U S Government
Accountability Office (GAO) draft report entitled: "Small Business
Innovation Research: SBA Should Work with Agencies to Improve the Data
Available for Program Evaluation." We do not have any issue with the
facts and findings presented in the draft report. In addition, we
generally agree with the recommendations, and offer the following plan
of action in response.
With respect to the first recommendation: "Collect data on the number
of applications submitted by small businesses owned by disadvantaged
individuals and women," we plan the following actions:
* The SBA currently collects the number of awards and aggregate amount
of awards to women, minority-owned, and HUBZone small business
concerns for the SBA-issued Annual Report to Congress.
* In the new SBA database at SBIR goy, we plan on collecting non-
awardee information at an applicant-level from agencies. This
information could include designations of women-owned, minority-owned,
and HUBZone small business concerns. Additionally, this information on
non-awardees”-as stated in the report”-can be compared with
information in the Central Contractor Registration database in order
to determine whether the applicant is a women-owned, minority-owned or
HUBZone small business concern We anticipate collecting this
information in FY2012, to allow agencies to organize their databases
in line with our new data collection schema.
* This fall, the SBA plans to hold a workshop to share best practices
in outreach to socially and economically-disadvantaged groups to apply
to the SBIR program. The outcome of the workshop should be a
commitment from agencies to baseline their current level of applicant
and awardee activity from socially and economically-disadvantaged
groups. If appropriate, agencies should make commitments to obtain
applicant numbers from these groups.
With respect to the second recommendation: "Identify best practices
for verifying the accuracy of data related to progress in increasing
commercialization," we plan the following actions:
* We will look to identify Best Practices and verification methods and
work with agencies toward their implementation.
* While our plan is to collect commercialization metrics, the
recommendation also addresses the tracking of such information to
determine progress in "increasing commercialization" The current
effort for data collection is to establish a baseline of such
information for the program. Once that it is established, we can
review the progress of increasing commercialization.
SBA looks forward to working with you and the SBIR participating
agencies.
If you have any questions, please contact Shawn McKeehan at 202-205-
7729.
Sincerely,
Signed by:
Sean J. Greene:
Associate Administrator for Investment:
[End of section]
Appendix III: Comments from the Department of Energy:
Department of Energy:
Office of Science:
Washington, DC 20585:
July 27, 2011:
Mr Franklin Rusco:
Director, Natural Resources and Environment:
Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Mr. Rusco,
Thank you for the opportunity to comment on the draft Government
Accountability Office (GAO) report entitled, "Small Business
Innovation Research: SBA Should Work with Agencies to improve the Data
Available for Program Evaluation" (GAO-11-698). We have reviewed the
draft report and provide general comments below. The comments provided
here have been coordinated with other relevant offices of the
Department of Energy (DOE).
The GAO was charged by the Energy and Environment and the Technology
and Innovation Subcommittees of the House Committee on Science and
Technology "to determine (1) how participating agencies have addressed
the SBIR program's four overarching purposes when implementing their
programs and (2) the extent of SBIR program data available to evaluate
progress in increasing commercialization of SBIR technologies." We
recognize that conducting a review of the five-noted SBIR agencies and
associated activities is an enormous undertaking, and in particular,
we appreciate the time and thoroughness the GAO took to review the
Department's SBIR program.
With regard to the two specific recommendations provided by GAO, we
concur with these recommendations. DOE does collect data on the number
of applications submitted by small businesses owned by disadvantaged
individuals and women and is willing to include this data in its
reporting to SBA. DOE is also interested in learning about the best
practices for verifying the accuracy of data related to progress in
increasing commercialization. It should be noted that the reason DOE
does not currently verify commercialization data stems from
limitations on resources and not from a belief that verification is of
limited value.
In addition, we would like to provide one comment regarding the second
charge to GAO. The GAO report discusses the variations in
commercialization data collected by the agencies and the challenges
that result when trying to do program-wide assessment. Implicit in
this discussion is the idea that there is one set of metrics that can
be used to compare program success across all participating agencies.
Agencies have different missions, and as noted in the report, also
include non-financial factors, (e.g. national security) in assessing
their SBIR programs. The GAO report does not address the underlying
questions of whether (I) there exist a universal set of metrics for
measuring the success of SBIR programs across multiple agencies and
(2) what those universal metrics should be. Until these questions are
addressed, the compatibility of the available data among agencies will
remain a secondary concern.
With regard to the draft report's content, please find attached
specific comments and suggested edits for your consideration. Thank
you, again, for the opportunity to provide comments on this draft
report. If you have any questions or concerns, please call Manny
Oliver at (301) 903-0309. We look forward to receiving your final
report.
Sincerely,
Signed by:
Patricia M. Dehmer:
Deputy Director for Science Programs: DOE Office of Science:
Attachment:
[End of section]
Appendix IV: Comments from the National Science Foundation:
National Science Foundation:
Office Of The Director:
4201 Wilson Boulevard, Room 1270:
Arlington, Virginia 22230:
Tel. 703-292-8040:
Fax. 703-292-9040:
July 29, 2011:
Mr, Franklin Rusco:
Director, Natural Resources and Environment:
United States Government Accountability Office:
Washington, DC 20548:
Dear Mr. Rusco:
The National Science Foundation (NSF) appreciates the opportunity to
review and comment on the Government Accountability Office's (GAO's)
draft report entitled Small Business Innovation Research: SBA Should
Work with Agencies to Improve the Data Available for Program
Evaluation (GAO-11-698).
NSF has a long history of participation in the Small Business
Innovation Research (SBIR) program dating back to the 1980s and has
strongly supported the program's overarching purposes (i.e., to meet
federal R&D needs; to stimulate technological innovation; to increase
commercialization of federal R&D-based innovations; and to encourage
participation by businesses owned by women and disadvantaged
individuals). NSF has been proactive in addressing the program's
objectives through:
* broad topical solicitations that emphasis commercial potential;
* mentorship of its small business clients by program directors with
extensive business and technical backgrounds;
* technical assistance in commercialization planning;
* matching funds to stimulate investment in innovative projects;
* supplemental awards and outreach activities to encourage members of
underrepresented groups to participate in the small business
innovation process; and;
* a proactive process to capture the commercial outcome of NSF Phase
II awards over time.
NSF concurs with the underlying goals of the GAO's recommendations”to
enhance the collection and analysis of SBIR program data”and endorses
the importance of sharing best practices for verifying data accuracy.
NSF is committed to collaborate with the Small Business Administration
and the other SBIR-participating agencies in the:
* implementation of a government-use data base for program evaluation
particular focused on commercial outcomes;
* collection of data on the participation of underrepresented
individuals in the small business innovation process; and;
* identification of best practices for verification of data related to
commercialization outcomes.
NSF commends the GAO audit team for an outstanding job of capturing
the characteristics and features of the SBIR process at NSF, as
reflected in the GAO-developed statement of facts and in this draft
report. In reviewing the draft report, a few inaccuracies were
identified; therefore, the following amendments to the draft report
are recommended.
[Page and footnote numbers in the draft report may differ from those
in this report]
Page 7, Footnote 17. The report on the NRC Assessment of the SBIR
Program at NSF, published in 2008, should be noted (consistent with
similar reports for other agencies that are cited in the footnote).
Propose the following, "...prepublication in November 2007, the DOE
and NASA reports were released in prepublication in June 2008 and
December 2008, respectively, and the NSF report was issued in 2008."
Page 23, Line 8. The provided dollar figure of "$1.5 billion" that
appears in the draft is incorrect. It should read..."evaluated from
July 2005 through May 2010 had realized a total of $1.05 billion in
commercial revenue."
Page 23, Footnote 39. As written, the footnote introduces several
inaccuracies. A revised version is offered. "NSF's broad estimate of
commercial revenue includes sales revenue from products and/or
services that are a consequence of the award, licensing revenue,
revenue from mergers or sales of award recipients' businesses that
resulted, at least in part from the awards. NSF's estimate does not
include venture capital funding."
Thank you for the opportunity to comment on this draft report. If you
have any questions regarding this response, please contact Kathryn
Sullivan at 703-292-7375. We look forward to receiving your final
report.
Sincerely,
Signed by:
Subra Suresh:
Director:
[End of section]
Appendix V: GAO Contact and Staff Acknowledgments:
GAO Contact:
Frank Rusco (202) 512-3841 or ruscof@gao.gov:
Staff Acknowledgments:
In addition to the individual named above, key contributors to this
report include Cheryl Williams, Assistant Director; Antoinette
Capaccio; Stephen Carter; Nancy Crothers; Laurie Ellington; Cindy
Gilbert; Cynthia Norris; Christine Senteno; and Kiki Theodoropoulos.
[End of section]
Footnotes:
[1] For the SBIR program, SBA has defined commercialization as the
"process of developing marketable products or services and producing
and delivering products or services for sale (whether by the
originating party or by others) to Government or commercial markets."
Small Business Innovation Research Program Policy Directive, 67 Fed.
Reg. 60,072, 60,083 (Sept. 24, 2002).
[2] Generally, socially disadvantaged individuals are those who have
been subjected to racial or ethnic prejudice or cultural bias within
American society because of their identities as members of groups and
without regard to their individual qualities. Economically
disadvantaged individuals are socially disadvantaged individuals whose
ability to compete in the free enterprise system has been impaired by
diminished capital and credit opportunities as compared with that of
others in the same or similar line of business who are not socially
disadvantaged. Throughout this report, we refer to socially and
economically disadvantaged individuals as "disadvantaged individuals."
To be considered owned by disadvantaged individuals, businesses must
be at least 51 percent owned by one or more disadvantaged individuals
or, in the case of any publicly owned business, at least 51 percent of
the stock must be owned by disadvantaged individuals, and those who
are disadvantaged must control the management and daily business
operations.
[3] Encouraging participation by women-owned businesses was not
included among the program's original purposes but was added by the
Small Business Research and Development Enhancement Act of 1992. Pub.
L. No. 102-564, § 102, 106 Stat. 4249, 4250. To be women-owned, a
small business must be at least 51 percent owned by one or more women
or, in the case of any publicly owned business, at least 51 percent of
the stock must be owned by women, and women must control the
management and daily business operations.
[4] The 11 agencies are the U.S. Department of Agriculture, Department
of Commerce, Department of Defense, Department of Education,
Department of Energy, Department of Health and Human Services,
Department of Homeland Security, Department of Transportation,
Environmental Protection Agency, National Aeronautics and Space
Administration, and National Science Foundation.
[5] The National Institutes of Health accounted for more than 98
percent of the SBIR expenditures of the Department of Health and Human
Services from fiscal year 2000 through fiscal year 2010. Four other
agencies in the department--the Centers for Medicaid and Medicaid
Services, the Centers for Disease Control and Prevention, the
Administration for Children and Families, and the Food and Drug
Administration--also issued SBIR solicitations during that period.
[6] While the agencies that we reviewed account for the vast majority
of SBIR program expenditures, the results of our review cannot be
generalized to all participating agencies.
[7] The views expressed by representatives of these trade associations
cannot be generalized to other trade associations.
[8] Prior to 2010, the award ceiling for phase I was $100,000, and the
award ceiling for phase II was $750,000. According to SBA guidance,
agencies may exceed these ceilings with appropriate justification.
[9] In addition, small businesses that are majority-owned or -
controlled by another business or joint venture that meets the 51
percent requirement are also eligible to compete for SBIR awards. 13
C.F.R. § 121.702(a).
[10] In this report, we are using "technologies" to refer to products,
systems, methods, services or other results of SBIR-funded R&D.
[11] The requirement specifically applies to technologies that are
moving toward commercialization.
[12] See GAO, Small Business Innovation Research: Agencies Need to
Strengthen Efforts to Improve the Completeness, Consistency, and
Accuracy of Awards Data, [hyperlink, http://www.gao.gov/products/GAO-
07-38] (Washington, D.C.: Oct. 19, 2006) and GAO, Managing for
Results: Measuring Program Results That Are under Limited Federal
Control, [hyperlink, http://www.gao.gov/products/GGD-99-16]
(Washington, D.C.: Dec. 11, 1998).
[13] [hyperlink, http://www.gao.gov/products/GAO-07-38].
[14] For our 2006 report, we reviewed the SBIR programs of the
Department of Agriculture, Environmental Protection Agency, and
National Institute of Standards and Technology in addition to those
programs of the agencies we reviewed for this report.
[15] GAO, Small Business Innovation Research: Observations on
Agencies' Data Collection and Eligibility Determination Efforts,
[hyperlink, http://www.gao.gov/products/GAO-09-956T] (Washington,
D.C.: Aug. 6, 2009).
[16] National Research Council, An Assessment of the SBIR Program
(Washington, D.C.: National Academies Press, 2008). This report is
part of a series published by the National Academies in response to
the mandate.
[17] Our review did not include NASA's 2011 SBIR solicitation, which
was released in July 2011.
[18] NSF does not issue separate solicitations for phase II projects
and considers phase II proposals only from recipients of NSF phase I
awards.
[19] DOD officials told us that for fiscal years 2008-2010, such
scores were calculated for 42-45 percent of applicants.
[20] American Recovery and Reinvestment Act of 2009, Pub. L. No. 111-
5, 123 Stat. 115.
[21] According to NIH officials, the National Cancer Institute based
its program on one originated by the National Institute of Standards
and Technology.
[22] Participating agencies have authority to spend or authorize
expenditure of a portion of their SBIR funds for technical assistance
to SBIR awardees. The value of the technical assistance provided with
or authorized from SBIR funds can be up to $4,000 per award in phase I
and up to $4,000 per award per year in phase II.
[23] According to NIH officials, since the agency began offering such
assistance in 2004, NIH has provided the assistance to nearly 700 of
the 5,090 businesses that received phase I SBIR awards.
[24] The National Defense Authorization Act for Fiscal Year 2006, Pub.
L. No. 109-163, § 252, 119 Stat. 3136, 3178 (codified as amended at 15
U.S.C. § 638(y)) authorized DOD's Commercialization Pilot Program,
which allows DOD components to use a portion of their SBIR funds for
program administration activities that support accelerated transition
of SBIR-funded technologies into phase III. DOD allows its components
discretion to use these funds for their own specific needs.
[25] NASA did not use SBIR funds to support the technical assistance
pilot program; however, a NASA official indicated that, if the agency
had continued the program, it likely would have done so using SBIR
funds. NASA officials indicated that the agency has relied on other
resources to help increase commercialization, such as national SBIR
conferences hosted by states; industry-sponsored technology summits;
the Space Alliance Technology Outreach Program, which is designed to
speed the transfer of space technology to the private sector; non-SBIR-
specific agency publications; and the agency's Web site.
[26] In this report, we are using "non-SBIR funds" to refer to funds
other than the 2.5 percent of the participating agencies' extramural
R&D budgets that must be set aside for SBIR.
[27] Outside investment can be in the form of capital, liquid assets,
or convertible debt. SBA has identified outside (non-SBIR) investment
as a metric for commercialization, as discussed later in this report.
[28] NIH's National Cancer Institute refers to its matching funds
awards as bridge awards. The National Cancer Institute is the only NIH
component included in our review that offered matching funds.
[29] NASA and the National Cancer Institute established their matching
funds programs relatively recently--in 2010 and 2009, respectively.
DOD began matching funds for phase I award recipients in 1995 and for
phase II award recipients in 2000.
[30] NSF established a matching funds program for its phase II award
recipients in 1998 and later established a similar program for phase I
award recipients in 2007.
[31] In addition, in 2010 SBA made 20 awards to state organizations
though the Federal and State Technology Partnership Program. Under the
program, SBA, with the concurrence of NSF and DOD officials, makes
awards to provide outreach, financial support, or technical assistance
to small businesses participating in or interested in participating in
the SBIR program. In making such awards, the agencies are required to
consider whether the applications being funded address the needs of
small businesses owned by minorities or women, among other things.
[32] 15 U.S.C. § 638(k)(2) (2006).
[33] Businesses are required to provide information for the Central
Contractor Registration database such as their Employer Identification
Number, legal business name, and the goods and services they provide.
[34] SBA officials noted that budgetary uncertainties may affect
related efforts. For example, the officials said that SBA has delayed
plans to develop a commercialization survey for award recipients that
do not apply for a new SBIR award within 1 year after their phase II
award ends.
[35] SBA officials said that the agency had previously obtained
approval from the Office of Management and Budget, as required by the
Paperwork Reduction Act, for data collection instruments that are
substantially similar to the ones proposed, and they plan to rely on
this approval. Paperwork Reduction Act, 44 U.S.C. §§ 3501-3520 (2006).
[36] DOD's estimate of returns on investment also includes the
commercialization results of awards made through DOD's Small Business
Technology Transfer (STTR) program. The STTR program resembles the
SBIR program in some respects but provides funding for research
proposals that are developed and executed cooperatively between small
businesses and research organizations.
[37] For NIH's 2002 survey, 768 (73 percent) of the 1,052 recipients
of phase II awards from 1992 through 2001 were located and responded
to the survey. Follow-up surveys experienced declining responses from
recipients. Of the 768 respondents to the 2002 survey, 275 (36
percent) were located and responded to the final follow-up survey in
2007. In reporting survey results, NIH calculated its response rate
from the sample of award recipients that it deemed usable and
eligible--that is, those award recipients that, among other criteria,
could be located and were still operating in the United States. For
the 2002 survey, NIH used the size of this sample--905 contacts out of
the 1,052 award recipients--to determine that the 768 respondents
represented a survey response rate of 85 percent.
[38] For NIH's 2008 survey, 719 (69 percent) of the 1,037 recipients
of phase II awards from 2002 through 2006 were located and responded
to the survey. For the 2008 survey, NIH used the size of the sample
deemed usable and eligible--918 out of the 1,037 award recipients--to
determine that the 719 respondents represented a response rate of 78
percent.
[39] NSF's estimate of commercial revenue includes sales revenue from
SBIR-funded technologies; licensing revenue; and revenue from mergers
or sales of the award recipients' businesses that resulted, at least
in part, from the awards. NSF's estimate does not include venture
capital funding.
[40] NSF's estimated investment covers the period from October 2004
through October 2010.
[41] As we reported in 2010, commercialization data are not always
correctly entered into this database by DOD procurement officers. See
GAO, Space Acquisitions: Challenges in Commercializing Technologies
Developed under the Small Business Innovation Research Program,
[hyperlink, http://www.gao.gov/products/GAO-11-21] (Washington, D.C.:
Nov. 10, 2010). However, DOD officials said that they are improving
data in this system by training officers to correctly account for SBIR
contracts.
[42] Agencies are prohibited from using SBIR funds for program
administration with the exception of certain administrative activities
authorized as part of DOD's Commercialization Pilot Program.
[43] [hyperlink, http://www.gao.gov/products/GAO-07-38].
[44] According to DOD, this percentage does not necessarily represent
historic averages.
[45] NASA's survey, conducted from 1997 through 2002, achieved a 78
percent response rate.
[46] [hyperlink, http://www.gao.gov/products/GAO/RCED-99-114].
[47] NIH accounted for more than 98 percent of the SBIR expenditures
of the Department of Health and Human Services from fiscal year 2000
through fiscal year 2010. Four other agencies in the department--the
Centers for Medicaid and Medicaid Services, the Centers for Disease
Control and Prevention, the Administration for Children and Families,
and the Food and Drug Administration--also issued SBIR solicitations
during that period.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the
performance and accountability of the federal government for the
American people. GAO examines the use of public funds; evaluates
federal programs and policies; and provides analyses, recommendations,
and other assistance to help Congress make informed oversight, policy,
and funding decisions. GAO's commitment to good government is
reflected in its core values of accountability, integrity, and
reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black
and white. Pricing and ordering information is posted on GAO‘s Web
site, [hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
