Small Business Innovation Research
Agencies Need to Strengthen Efforts to Improve the Completeness, Consistency, and Accuracy of Awards Data Gao ID: GAO-07-38 October 19, 2006The Small Business Innovation Research (SBIR) program was created to increase the use of small businesses to meet federal research needs and commercialize the results of this research. To monitor the program, the Small Business Administration (SBA) requires participating agencies to provide, in a standard format, specific data on all SBIR awards they make. SBA then compiles these data into a database known as Tech-Net. Congress also required SBA to create, by 2001, a restricted and more comprehensive database that would provide information for government agencies to use in evaluating the program. GAO was asked to identify the (1) types of data that agencies report to SBA for inclusion in the Tech-Net database, (2) extent to which these data are provided in a standard format, (3) extent to which SBA has established the government-use database, and (4) extent to which SBIR agencies have developed and implemented techniques to track commercialization of SBIR projects. GAO reviewed 8 of the 11 agencies participating in SBIR.
Federal agencies participating in the SBIR program annually submit over 40 data elements to SBA for each award they make. These data include information on the award, such as value and a descriptive abstract; information on the recipient, such as name and gender; and information about the firm receiving the award, such as number of employees and location. Participating agencies submit most of the information required by SBA, but they are not consistently providing all required data elements, including the number of employees in the firm, and the gender and socio-economic status of the award recipient, resulting in incomplete sections of the database. Agencies stated that this happens because they do not collect all of the information that SBA wants and because SBA's requirements change regularly. Some participating agencies are not submitting SBIR award data in the standard format required by SBA, and although SBA's quality assurance processes correct most of these problems, they do not correct all of them. In 2004 and 2005, about 25 percent of the data provided by five participating agencies did not comply with SBA's format. Formatting inconsistencies occur because the template SBA has provided agencies for reporting data can be edited. According to SBA, identifying and correcting inconsistently formatted data involves considerable resources, therefore the agency has focused its quality assurance efforts only on key data elements needed to track awards; other fields, such as those containing demographic data, are generally not corrected. As a result, comprehensive program evaluations may be limited by the quality of the data in these fields. SBA officials expect this problem to be resolved by fiscal year 2007, when all data will be submitted via an Internet interface that will not allow changes in the format. SBA is 5 years behind schedule in meeting the congressional mandate to implement a restricted government-use database for the SBIR program. SBA had planned to meet this requirement by expanding its Tech-Net database to include a restricted government-use section. SBA officials attributed the delay in meeting the 2001 deadline primarily to increased security requirements needed for the database, agency management changes, and budgetary constraints. SBA officials expect the government-use section of Tech-Net to be operational by October 1, 2006, when safeguards to protect the proprietary commercialization information in the database are in place. Most agencies GAO reviewed systematically gather data on the commercialization success of SBIR-funded projects. Five of these eight agencies regularly survey all awardees to gather information on program participation, including commercial success, and one agency is about to start a similar survey. In contrast, two agencies only gather anecdotal success stories from a small sample of SBIR awardees. SBA and agency officials generally agree that despite their best efforts, obtaining commercialization information from awardees remains a major challenge.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-07-38, Small Business Innovation Research: Agencies Need to Strengthen Efforts to Improve the Completeness, Consistency, and Accuracy of Awards Data
This is the accessible text file for GAO report number GAO-07-38
entitled 'Small Business Innovation Research: Agencies Need to
Strengthen Efforts to Improve the Completeness, Consistency, and
Accuracy of Awards Data' which was released on November 21, 2006.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Committees:
United States Government Accountability Office:
GAO:
October 2006:
Small Business Innovation Research: Agencies Need to Strengthen Efforts
to Improve the Completeness, Consistency, and Accuracy of Awards Data:
Small Business Innovation Research:
GAO-07-38:
GAO Highlights:
Highlights of GAO-07-38, a report to congressional committees:
Why GAO Did This Study:
The Small Business Innovation Research (SBIR) program was created to
increase the use of small businesses to meet federal research needs and
commercialize the results of this research. To monitor the program, the
Small Business Administration (SBA) requires participating agencies to
provide, in a standard format, specific data on all SBIR awards they
make. SBA then compiles these data into a database known as Tech-Net.
Congress also required SBA to create, by 2001, a restricted and more
comprehensive database that would provide information for government
agencies to use in evaluating the program. GAO was asked to identify
the (1) types of data that agencies report to SBA for inclusion in the
Tech-Net database, (2) extent to which these data are provided in a
standard format, (3) extent to which SBA has established the government-
use database, and (4) extent to which SBIR agencies have developed and
implemented techniques to track commercialization of SBIR projects. GAO
reviewed 8 of the 11 agencies participating in SBIR.
What GAO Found:
Federal agencies participating in the SBIR program annually submit over
40 data elements to SBA for each award they make. These data include
information on the award, such as value and a descriptive abstract;
information on the recipient, such as name and gender; and information
about the firm receiving the award, such as number of employees and
location. Participating agencies submit most of the information
required by SBA, but they are not consistently providing all required
data elements, including the number of employees in the firm, and the
gender and socio-economic status of the award recipient, resulting in
incomplete sections of the database. Agencies stated that this happens
because they do not collect all of the information that SBA wants and
because SBA‘s requirements change regularly.
Some participating agencies are not submitting SBIR award data in the
standard format required by SBA, and although SBA‘s quality assurance
processes correct most of these problems, they do not correct all of
them. In 2004 and 2005, about 25 percent of the data provided by five
participating agencies did not comply with SBA‘s format. Formatting
inconsistencies occur because the template SBA has provided agencies
for reporting data can be edited. According to SBA, identifying and
correcting inconsistently formatted data involves considerable
resources, therefore the agency has focused its quality assurance
efforts only on key data elements needed to track awards; other fields,
such as those containing demographic data, are generally not corrected.
As a result, comprehensive program evaluations may be limited by the
quality of the data in these fields. SBA officials expect this problem
to be resolved by fiscal year 2007, when all data will be submitted via
an Internet interface that will not allow changes in the format.
SBA is 5 years behind schedule in meeting the congressional mandate to
implement a restricted government-use database for the SBIR program.
SBA had planned to meet this requirement by expanding its Tech-Net
database to include a restricted government-use section. SBA officials
attributed the delay in meeting the 2001 deadline primarily to
increased security requirements needed for the database, agency
management changes, and budgetary constraints. SBA officials expect the
government-use section of Tech-Net to be operational by October 1,
2006, when safeguards to protect the proprietary commercialization
information in the database are in place.
Most agencies GAO reviewed systematically gather data on the
commercialization success of SBIR-funded projects. Five of these eight
agencies regularly survey all awardees to gather information on program
participation, including commercial success, and one agency is about to
start a similar survey. In contrast, two agencies only gather anecdotal
success stories from a small sample of SBIR awardees. SBA and agency
officials generally agree that despite their best efforts, obtaining
commercialization information from awardees remains a major challenge.
What GAO Recommends:
GAO recommends that SBA and SBIR participating agencies work together
to improve the quality of the data in SBA‘s Tech-Net database. SBA and
the SBIR participating agencies included in this report generally
agreed with our recommendation.
[Hyperlink: http://www.gao.gov/cgi-bin/getrpt?GAO-07-38.]
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Anu K. Mittal at (202)
512-3841 or mittala@gao.gov.
Contents:
Letter:
Results in Brief:
Background:
Agencies Provide Most of the Required SBIR-Award Data Elements to SBA,
but Some Data Submitted are Incomplete:
Agencies Generally Comply with SBA's Formatting Guidance, but Key Data
on SBIR Award Recipients May Be Inconsistent or Inaccurate:
SBA Is Five Years Behind Schedule in Meeting its Obligation to
Implement a Government-Use SBIR Database:
Tracking Data on the Commercialization Success of SBIR Projects Varies
Among Agencies and Remains a Challenge:
Conclusions:
Recommendation for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Objectives, Scope and Methods:
Appendix II: Comments from the Small Business Administration:
GAO Comments:
Appendix III: Comments from the Department of Defense:
Appendix IV: Comments from the National Aeronautics and Space
Administration:
Appendix V: Comments from the National Institutes of Health:
Appendix VI: GAO Contact and Staff Acknowledgments:
Abbreviations:
USDA: Department of Agriculture:
DOD: Department of Defense:
DOE: Department of Energy:
EPA: Environmental Protection Agency:
FDA: Food and Drug Administration:
NASA: National Aeronautics and Space Administration:
NIST: National Institute of Standards and Technology:
NIH: National Institutes of Health:
NSF: National Science Foundation:
R&D: research and development:
SBA: Small Business Administration:
SBIR: Small Business Innovation Research:
United States Government Accountability Office:
Washington, DC 20548:
October 19, 2006:
The Honorable Sherwood Boehlert:
Chairman:
The Honorable Bart Gordon:
Ranking Minority Member:
Committee on Science:
House of Representatives:
The Honorable Vernon J. Ehlers:
Chairman:
The Honorable David Wu:
Ranking Minority Member:
Subcommittee on Environment, Technology and Standards:
Committee on Science:
House of Representatives:
Recognizing the potential of small businesses to be a source of
significant innovation, the Congress established the Small Business
Innovation Research (SBIR) program in 1982. From fiscal year 1983
through fiscal year 2004, federal agencies that participated in the
SBIR program awarded over $17 billion in grants, contracts, or
cooperative agreements to over 82,000 projects. The primary goals of
the SBIR program are to stimulate technological innovation, meet
federal research and development (R&D) needs, foster participation by
minority and disadvantaged persons in technological innovation, and
increase the commercial success (commercialization) of innovation that
is derived from federally funded R&D.[Footnote 1] Because the Congress
did not define what constitutes commercial success of federally funded
R&D or how best to measure it, agencies have used different
commercialization outcomes for the SBIR program, such as the sale of
the resulting SBIR-funded product or process, the extent to which SBIR
firms have received non-SBIR funding, or the creation of new jobs or
products. The SBIR program is currently scheduled to expire on
September 30, 2008.
Every federal agency with an R&D budget of $100 million or more is
required to establish and operate an SBIR program funded by 2.5 percent
of the agency's budget for research conducted by others, called
extramural research. Currently, 11 federal agencies participate in the
SBIR program.[Footnote 2] Each agency manages its own program,
including targeting research areas, reviewing proposed projects, and
making research awards through grants, contracts, or cooperative
agreements. The Small Business Administration (SBA) plays a central
administrative role by, for example, issuing policy directives to the
participating federal agencies, collecting data from participating
agencies on awards and recipients, and reporting program results
annually to the Congress.
Over the last 24 years, the SBIR program has been reauthorized and
modified by the Congress at various times. For example, the Small
Business Research and Development Enhancement Act of 1992 directed SBA
and participating agencies to, among other things, emphasize the goal
of increasing commercialization of research results and to improve the
government's dissemination of program-related data.[Footnote 3] As a
result, agencies were required to include commercialization potential
as a criterion for selecting award recipients. During this same period,
SBA began to develop a publicly available database, known as Tech-Net,
that contained information on all awards made through the SBIR
program.[Footnote 4] The Tech-Net database is intended to be, among
other things, an electronic gateway of technology information and
resources for researchers, scientists, and government officials about
federally-funded leading edge technology research. The Small Business
Innovation Research Program Reauthorization Act of 2000 formalized this
database by requiring SBA to develop, maintain, and make available to
the public a searchable, up-to-date, electronic database that contained
SBIR award information.[Footnote 5] The 2000 reauthorization act also
required SBA to develop and maintain another restricted government
database that would contain additional information on commercialization
not contained in the public Tech-Net database, thereby allowing better
evaluations of the SBIR program on an ongoing basis. This database was
to be established by mid-2001 and made available only to government
agencies and certain other authorized users.
As the program has evolved over time, congressional direction has
focused, among other things, on the ability of SBIR award recipients to
commercialize the results of their research, as evidenced by the
increased emphasis on commercialization in the 1992 and 2000
reauthorizations of the program. At various points in the life of the
program, we have reported that the SBIR program has been successful in
increasing commercialization of research results and that agencies have
used various methods to measure the commercial success of the projects
they fund.[Footnote 6] Largely these methods have consisted of surveys
of award recipients to obtain data on indicators of commercial success
and soliciting "success stories" voluntarily provided by SBIR award
recipients. However, we also reported that these methods provided only
"snapshots" of commercial success and did not allow for a systematic
demonstration of changes in program commercialization rates over time.
We reported that the lack of clarity on how much emphasis agencies
should place on commercialization versus other SBIR program goals had
also created challenges for assessing the program's results.
In the context of efforts to monitor and evaluate the success of the
SBIR program, you requested that we identify the (1) types of data that
participating SBIR agencies are reporting to SBA for inclusion in the
Tech-Net database; (2) extent to which agencies provide data for the
Tech-Net database in a standard format to enable program evaluation;
(3) extent to which SBA has met the mandate to establish, by mid-2001,
a government-use database that can be used for program evaluation; and
(4) extent to which participating SBIR agencies have developed and
implemented techniques to track the commercialization success of SBIR
projects.
In conducting this study, we reviewed the SBIR-related activities at 8
of the 11 SBIR participating agencies, which account for over 98
percent of the total dollars awarded by the program in fiscal year
2005.[Footnote 7] To determine the types of data these participating
agencies are reporting to SBA and the extent to which SBA has complied
with the requirement to establish a government-use database that can be
used for program evaluation purposes, we compared the provisions in the
Small Business Innovation Research Program Reauthorization Act of 2000,
SBA's amended Policy Directive implementing the Act issued in September
2002, and other guidance with agency SBIR Tech-Net database reports. To
determine the extent to which data for the Tech-Net database are
provided in a standard format to enable program evaluation, we compared
the data from participating agencies with data in SBA's Tech-Net
database for fiscal years 2004 and 2005, the 2 most recent years for
which data were available. Our data reliability review focused on SBA's
data system and internal controls, rather than on the systems and
internal controls agencies use to create the data provided to SBA. To
assess the reliability of the data in SBA's Tech-Net database, we
interviewed SBA officials about the database and reviewed related
documentation. We determined that the data are sufficiently reliable
for our purpose, which is to report on SBA's efforts to ensure
consistency and completeness of the data it receives. We used GAO's
data reliability guidance to identify key attributes of data quality
that can facilitate program evaluation. We also interviewed SBA and
agency officials to determine the extent to which the government-use
database requirements have been implemented. To determine the extent to
which participating agencies and SBA have developed and implemented
techniques to evaluate commercialization success of SBIR projects, we
reviewed agency documentation and interviewed SBA and agency officials.
Appendix I contains a detailed discussion of the scope and methodology
of our review. We conducted our review from April 2006 through
September 2006 in accordance with generally accepted government
auditing standards.
Results in Brief:
Federal agencies participating in the SBIR program annually submit to
SBA a wide range of descriptive information about each award they make.
Each year SBA requires participating agencies to provide over 40 data
elements for each SBIR award made. These data include information about
the award itself, such as the award amount, a descriptive abstract of
the project, and a unique tracking number; information about the award
recipient such as the name, gender, and socio-economic status; and
information about the type of firm that received the award, such as
number of employees and geographic location. However, we determined
that although participating agencies are providing most of the
information requested by SBA, they are not consistently providing the
full range of required information. For example, two of the eight
agencies we reviewed had not consistently provided SBA data on the
gender and socio-economic status of SBIR award recipients, and five did
not provide data on the number of employees working at the firms that
received SBIR awards. As a result, certain sections of the database
needed for comprehensive program evaluation are incomplete. Agencies
cited a variety of reasons for not providing the data requested by SBA,
including frequent changes in SBA's data requirements and differences
in the types of data agencies collect versus the types of data that SBA
wants to collect.
Some participating agencies are not submitting SBIR award data in the
standard format required by SBA, and while some of these problems are
corrected by SBA's quality assurance processes, others are not. Since
2000, SBA officials have worked with participating agencies to ensure
greater consistency in how the data required for the Tech-Net database
are formatted and have developed a reporting template that includes the
required data fields and instructions for appropriate data entry.
However, almost a quarter of the data provided by five participating
agencies in 2004 and 2005 still did not comply with SBA's formatting
guidance. As a result, SBA officials said that they have to spend a
considerable amount of time and resources correcting these data, and
are unable to correct all of them. SBA's current quality assurance
efforts therefore focus on obtaining complete and accurate data only
for those fields essential to tracking specific awards, such as the
tracking numbers and award amounts, and not correcting data in fields
that contain demographic information about award recipients. The
primary reason for the inconsistently formatted data is that the
template SBA has provided for reporting data can be edited, and some
agencies have done so and introduced different formats. According to
SBA, by fiscal year 2007, this issue should be resolved because
agencies will have to submit their data through an Internet interface
that will no longer allow the agencies to change the format of titles
and data. In light of the problems we identified with the Tech-Net
database and the implications for these errors to limit evaluations of
the SBIR program, we are recommending that SBA work with participating
agencies to strengthen efforts to improve the quality of the data.
SBA has not met the congressional mandate to develop and implement, by
June 2001, a government-use database that can be used for evaluating
the SBIR program. To meet the congressional requirement to develop and
implement a government-use database, SBA planned to expand the existing
Tech-Net database and include a restricted government-use section that
would be accessible only to government agencies and other authorized
users. This government-use section of the Tech-Net database would rely
on information already gathered for the Tech-Net database and
supplemented by information on the commercialization outcomes for SBIR
awards. However, SBA officials told us that they have been unable to
meet the requirement to implement a government-use database by 2001
because of management changes that have occurred at the agency and
because of budgetary constraints. To date, with the help of two
contractors, SBA has developed the framework for (1) importing data
into the government-use section of the database, and (2) an Internet-
based interface that would allow agencies and award recipients to
access and enter commercialization information. According to SBA, the
government-use section of the database will not be fully operational
until the agency resolves certain outstanding issues, such as making
the commercialization information secure because this information is
considered proprietary and confidential. SBA officials expect that the
government-use section of the database will be operational by October
1, 2006; however, they also recognize that additional enhancements to
improve the efficiency of the database may still be needed after the
system is operational.
Seven of the eight agencies participating in the SBIR program that we
reviewed are gathering data on the commercialization success of SBIR-
funded projects; however, the methodological rigor of their methods
varies significantly. Under the program's authorizing legislation and
implementation guidance, agencies have been given considerable
flexibility in tracking the commercial success of their SBIR-funded
projects. Specifically, five of the eight agencies systematically and
periodically survey SBIR award recipients to gather a range of
information on program participation, including commercialization
experience. For example, DOD annually surveys SBIR award recipients to
gather data on (1) sales of SBIR-funded research results, (2)
commitments SBIR recipients have received for additional development
funding, and (3) whether SBIR-funded research results have been used by
other federal programs. Two of the eight agencies we reviewed use less
systematic data gathering efforts and instead focus on gathering
success stories and conducting periodic follow-up with a small sample
of SBIR award recipients. For example, USDA periodically contacts a
small group of award recipients to update and obtain information on
their commercialization experiences. The remaining agency--NIST--has
developed a survey that it plans to send to all of its SBIR award
recipients in 2006. Regardless of the methods used to gather
commercialization information, SBA and agency officials believe that
several factors complicate their efforts to obtain this information in
a standardized and complete manner. For example, these officials told
us that many years may elapse from the time an SBIR award is granted to
the time a product or process achieves commercial success and
maintaining contact with award recipients during this time period is
often difficult. Moreover, during this time, firms may change their
names, be purchased by other firms, or start new businesses to pursue
the project's commercial potential, making it even more difficult for
the agencies to track these firms and link them to their original SBIR
awards.
Background:
The Small Business Innovation Development Act of 1982 established the
SBIR program and identified four program goals: technological
innovation, commercialization of the research results, the use of small
businesses to meet agencies' R&D needs, and participation in federal
R&D by minorities and disadvantaged persons. The legislation provided
for a competitive three-phased program: phase I to determine the
technical and scientific merit and feasibility of a proposed research
idea; phase II to further develop the idea, taking into account its
commercial potential; and phase III to commercialize the resulting
product or process with private or federal investment but no additional
SBIR funding.
Under the SBIR program provisions, federal agencies that have external
R&D budgets of $100 million or more are required to use 2.5 percent of
these budgets to establish and operate an SBIR program. SBA oversees
and coordinates the efforts of the eleven agencies currently
participating in the program. In this capacity, SBA coordinates the
participating agencies' schedules to announce opportunities for firms
to apply for awards, called a solicitation, and provides access to
these solicitations through its Web site. As part of its oversight
effort, SBA collects SBIR data from the participating agencies,
aggregates the data, and uses the data to, among other things, monitor
the program and report annually to the Congress.
In reauthorizing the SBIR program in 1992, the Congress stated its
intention to expand and improve the program by emphasizing its goal of
increasing private sector commercialization, increasing participation
in federal R&D by small businesses, and improving the government's
dissemination of program-related information. One of the new provisions
under the 1992 legislation requires agencies, when evaluating phase II
proposals, to consider their commercial potential, including the
recipients' experiences commercializing the results from previous SBIR
awards, commitments accompanying the proposals for developmental
funding from sources other than the SBIR program, and other factors.
The SBIR program was again reauthorized in 2000 by the Small Business
Innovation Research Program Reauthorization Act of 2000. The 2000
legislation directed SBA and participating agencies to, among other
things, expand the scope of publicly available information on specific
awards and to annually report data on their SBIR programs to SBA. The
act required that SBIR phase II award recipients be requested to
voluntarily provide information to the agencies describing the outputs
and outcomes of their SBIR award. The act also required SBA to
establish, by mid-2001, a searchable and up-to-date electronic database
available for public use, and a restricted government-use database. To
accomplish this mandate, SBA envisioned expanding the electronic
database, known as Tech-Net, that it had developed in the late 1990s,
into two sections: a public-use section and a restricted government-use
section. The public-use section of the Tech-Net database would provide
access to nearly all of the statutorily-required award information for
SBIR awards gathered by the agencies. The public-use section was
intended to be an electronic gateway of technology information and
resources for researchers, scientists, and government officials who are
seeking information on potential small business partners, contractors,
or leading edge technology research. The government-use section would
be solely used for program evaluation purposes accessible only to
government agencies and other authorized users, and would contain
commercialization data voluntarily supplied by SBIR recipients upon
completion of their phase II SBIR funding agreement, such as revenue
from the sale of new products or services resulting from the research
undertaken with the award. In addition, applicants for phase II awards
would be required to update information on the commercialization
success of any prior SBIR awards they had received. Currently, SBA has
created and is maintaining the public-use section of the Tech-Net
database, which is available on the Internet to the general public.
Agencies Provide Most of the Required SBIR-Award Data Elements to SBA,
but Some Data Submitted are Incomplete:
While federal agencies participating in the SBIR program submit a wide
range of descriptive award information to SBA annually, these agencies
are not consistently providing all of the required information. As
outlined in SBA's guidance, each year SBIR participating agencies are
required to collect and maintain information from recipients and
provide it to SBA so that it can be included in the Tech-Net database.
Specifically, participating agencies are required to provide over 40
data elements for each SBIR award they make. These data include award-
specific information, such as the date and amount of the award, an
abstract of the project funded by the award, and a unique tracking
number for each award. Participating agencies are also required to
provide data about the award recipient, such as gender and socio-
economic status, and information about the type of firms that received
the awards, such as number of employees and geographic location. Much
of the data collected by participating agencies are provided by the
SBIR applicants at the time they apply for an award. Agencies provide
additional information, such as the grant/contract number and the
dollar amount of the award, after the award is made.
For the most part, all of the eight agencies we reviewed provided the
majority of the SBIR award data requested by SBA, including the program
identification number; company name, address, and contact information;
award year and amount; a unique tracking number that will stay with the
award through both phase I and phase II; and the title and abstract of
the project funded. However, we also determined that some of the
agencies are not providing the full range of information required by
SBA. For example, two of the eight agencies we reviewed had not
provided SBA data on the gender or socio-economic status of SBIR award
recipients in 2004 and 2005. Similarly, in 2005, five of the eight
agencies failed to provide data on the number of employees working at
the firms that received the awards. As a result, SBA does not have
information on the number of employees at SBIR awardee firms for about
one-third of all the awards reported by these agencies in 2004 and 2005.
SBA officials acknowledged that agencies do not routinely provide all
of the information requested because either they do not capture the
information in their agency databases or they are not requesting the
information from SBIR applicants. Officials at the participating
agencies also cited other reasons for the incomplete data they provided
to SBA. For example, NIH officials stated that for the past several
years, the SBA Tech-Net annual reporting requirements have changed each
year. At the end of calendar year 2003, SBA changed the field
description from "minority" to "socially and economically disadvantaged
small business." According to NIH officials, because the SBIR
information is captured by the agency at the time the application is
submitted/received, there is a lag in time between when application
data is input into the NIH database and when the agency receives new
SBA data field requirements. According to these officials, responding
to SBA's changes in field names presents significant challenges to NIH
for collecting the data needed to complete the Tech-Net reports,
especially if the requirements change annually. As a result, these
officials stated that complying with the changing requirements
necessitates significant NIH resources and efforts to research and
identify the information needed for the new data fields, not all of
which can be provided. In commenting on a draft of this report, SBA
stated that it has only requested minor clarifications of data
requirements and has not made frequent changes as stated by the
agencies. SBA believes that this may have been caused by lack of clear
communication to the agencies.
The agencies also noted that data for certain Tech-Net fields will be
absent from their reports to SBA if the data fields do not exist in the
NIH application or in its awards database. Similarly, USDA officials
stated that although they try to keep their records as up-to-date as
possible, problems occur when company or contact information changes
and the SBIR recipient fails to provide updated information to the
agency, such as the e-mail address for the central contact person.
Additionally, like NIH, USDA officials stated that certain information
requested by SBA is not collected by their agency. For example, as long
as SBIR applicants have certified that they meet the criterion of being
a small business (under 500 employees), they do not ask for nor do they
record information on the number of employees in the firm.
Agencies Generally Comply with SBA's Formatting Guidance, but Key Data
on SBIR Award Recipients May Be Inconsistent or Inaccurate:
Participating agencies are providing some data that do not comply with
SBA's formatting guidance, and while some of these inconsistencies are
corrected by SBA's quality assurance processes, others are not. As a
result, some data elements in the Tech-Net database may be inconsistent
or inaccurate thereby, compromising the value of these data for program
evaluation. SBA's quality assurance efforts focus on obtaining complete
and accurate data for those fields essential to tracking specific
awards, such as the tracking number and award amount, rather than on
those fields that contain demographic information about the award
recipient. Because the data contained in the public-use section of Tech-
Net will be incorporated into the government-use section of the
database, inaccurate data in one section of the database will be
replicated in the other, and these inaccurate data will limit
evaluations of the SBIR program.
Both SBA and agency officials acknowledge that SBA has worked with
participating agencies since 2000 to help ensure greater consistency in
the formatting of the SBIR award data reported by agencies each year.
To assist agencies in formatting the data and to minimize the number of
inconsistencies in the data reported to SBA, SBA has taken a number of
steps to improve the data formatting process. Specifically, SBA
provided all of the participating agencies specific guidance on its
requirements for the data, including its preferred program application
for submitting the data, the length of each data field, and whether
data should be entered as numbers, characters, or a mix of both. SBA
has also included discussions of the Tech-Net database as an agenda
item at its quarterly meetings with SBIR program managers from each of
the participating agencies. Additionally, SBA developed a reporting
template for agencies to use that includes the required data fields and
instructions for appropriate data entry.
Under the current process, participating agencies aggregate all of the
data provided by SBIR recipients in their award applications with
additional information on the award amounts and submit the combined
data to SBA by March 15 of each year. SBA then electronically checks
the data to locate and reformat inconsistencies, and adds the data to
the Tech-Net database. SBA officials told us that when they detect
inconsistencies in data fields essential to tracking a specific award,
such as the award tracking number, contact information for the
recipient or principal investigator, or awarding agency, they contact
the agency to obtain the correct information. However, SBA does not
currently take steps to ensure that agency-provided data are accurate
and complete. For example, SBA does not require agencies to submit a
random sample of applications so that it can compare the data submitted
by agencies with the original applicant information to ensure that the
submitted data contain all the relevant application information.
Instead, SBA relies on the agencies to fully report all the required
application information on the awards they make. SBA officials told us
that they believe that over the past two years the quality and
consistency of the data received from participating agencies had
greatly improved.
In reviewing the SBIR Tech-Net data that the eight agencies reported to
SBA in 2004 and 2005, we determined that almost a quarter of the data
provided by five agencies was incorrectly formatted for one or more
fields. For example, phone number and award amount fields contained
both characters and numbers and first and last names of principal
investigators were combined into a single word that was used as both
the first and last names. Moreover, we found that agency-provided data
on gender and socio-economic status for over half of the awards
reported in 2004 were incorrectly formatted. SBA officials said that
detecting and correcting some of these formatting errors required a
considerable amount of time and effort, and that their electronic check
does not detect all of the errors that we identified.
The formatting inconsistencies that we identified arise primarily
because the SBA reporting template used by agencies to submit required
data can be edited. Consequently, agencies can and do edit the
template. Agencies can change the names of various data fields in SBA's
template, delete fields altogether, and enter data as numbers,
characters, or both, regardless of what SBA has specified. One of the
data format issues identified by SBA was the deletion of fields that
agencies do not consider relevant or necessary. For example, DOD is the
only agency that uses the field called "branches" to specify which
component of DOD, such as the Army, made the award; other agencies have
deleted this field because it does not relate to the structure of their
agency. Similarly, we found instances where agencies had entered data
on the gender or socio-economic status of the recipient and award
amount in a format that differed from the numerical format specified by
SBA. For example, one agency entered "Y" and "N" rather than "0" and
"1." According to SBA officials, by fiscal year 2007, this issue should
be resolved because agencies will have to submit their data through an
Internet interface that will contain edit checks and should eliminate
many of these problems.
We also determined that inconsistencies or inaccuracies can arise in
certain data fields because SBA interprets the absence of certain data
elements as a negative entry without confirming the accuracy of such an
interpretation with the agency. In other words, if an agency did not
provide information on whether the recipient is a woman or a member of
a socio-economically disadvantaged group, SBA has entered a "no" into
the database. SBA stated that they generally do not contact agencies to
obtain correct information on data elements that are not used to track
specific awards, such as gender or socio-economic status of the
recipient. However, this inaccurate data on the award recipients could
limit efforts to use these fields for comprehensive program evaluation.
Information in the Tech-Net database will be used to populate the
government-use section of the database that SBA is developing (as
discussed below) for the purpose of supporting SBIR program
evaluations. However, SBA has no plans to correct any of the errors or
inconsistencies in the database that relate to the historical data
already collected. As a result, the errors in the existing database
will migrate to the government-use section of the database and we
believe will compromise the usefulness of the government-use database
for program evaluation purposes.
SBA Is Five Years Behind Schedule in Meeting its Obligation to
Implement a Government-Use SBIR Database:
SBA has not met its obligation to implement a restricted government-use
database that would allow SBIR program evaluation as directed by the
2000 SBIR reauthorization act. As outlined in the legislation, SBA, in
consultation with federal agencies participating in the SBIR program,
was to develop a secure database by June 2001 and maintain it for
program evaluation purposes by the federal government. SBA planned to
meet this requirement by expanding the existing Tech-Net database to
include a restricted government-use section that would be accessible
only to government agencies and other authorized users. In constructing
the government-use section of the database, SBA planned to supplement
existing data already gathered for the public-use section of the Tech-
Net database with information from SBIR recipients and from
participating agencies on commercialization outcomes for phase II SBIR
awards. However, according to SBA officials, the agency has been unable
to meet this requirement, primarily because of increased security and
other information technology project requirements, agency management
changes, and budgetary constraints. SBA's current goal for having the
government-use section of the Tech-Net database operational is October
1, 2006. In commenting on a draft of this report, SBA modified this
date, stating that they anticipate having the government-use section of
the Tech-Net database operational early in fiscal year 2007.
To date, with the help of two contractors, SBA has developed the
framework for importing data into the government-use section of the
Tech-Net database and for an Internet-based interface that would allow
agencies and award recipients to access the database and enter
commercialization information. According to SBA officials, as currently
envisioned, the government-use section of the Tech-Net database will
include the records of all applicants, including those that did not
receive SBIR awards. Participating SBIR agencies will be asked to
provide a unique business identification number, called the Data
Universal Numbering System or DUNS number, for each award recipient,
information about SBIR applicants that were not funded, and any
historical data they have obtained about the commercialization of SBIR
funded technologies. SBA has developed a standardized electronic
commercialization questionnaire to gather data for the government-use
section of the database from applicants and award recipients.
Information that will be captured in the questionnaire will include the
number of SBIR awards the company has received, the number of patents
or copyrights that have resulted from the award, sales revenue realized
as a result of the SBIR award, and sources of additional investment
funding. SBA officials told us that the commercialization questionnaire
will become an integral part of the SBIR application process in the
future, and any company applying for an SBIR award will be required to
complete and update the relevant information on phase II SBIR awards
previously received by the company at the time the application is
submitted. According to SBA officials, applications will not be
accepted until this information is completed, and failure to submit the
information may affect an applicant's ability to receive an award. In
addition, SBIR award recipients will be requested to voluntarily update
the commercialization information in the government-use section of the
Tech-Net database annually for a minimum period of five years following
the completion of their SBIR-funded project.
Although SBA has developed the majority of the functions needed to
populate the government-use section of the Tech-Net database with the
data currently gathered for the public-use section of the database, it
can not be made operational until certain security issues are resolved.
For example, because the government-use section of the database will
contain information from recipients that is considered proprietary and
confidential, SBA needs to ensure that adequate security measures are
in place to prevent unauthorized access to the data. This entails the
successful completion of a series of security and development checks to
ensure that the database system is operating as designed. While SBA
officials expect the government-use section of the database to be
operational by October 1, 2006, they also recognize that additional
enhancements, such as improving the user-friendliness of the interface
for online submission of SBIR data by participating agencies and
recipients, will be needed after the system is made operational.
According to SBA officials, the agency's priority is to get the
government-use section of the database up and running before
considering further improvements to the database.
Tracking Data on the Commercialization Success of SBIR Projects Varies
Among Agencies and Remains a Challenge:
Seven of the eight participating agencies we reviewed have implemented
techniques to track the commercialization success of their SBIR-funded
projects, and the eighth is planning to do so, although the
methodological rigor of these techniques varies significantly. Under
the program's authorizing legislation and SBA's implementation
guidance, agencies have been given considerable flexibility to design,
monitor, and evaluate the extent to which their SBIR programs have
achieved commercialization success. For example, while some agencies
use more systematic approaches to gathering data, such as periodically
surveying SBIR award recipients, other agencies are less methodical,
choosing instead to follow up periodically with a relatively small
sample of SBIR award recipients. Regardless of how they track
commercialization success, both SBA and agency officials generally
agree that several factors complicate their efforts to obtain this
information.
Agencies' Efforts to Track SBIR Commercialization Success Vary:
Of the eight agencies we reviewed, five systematically and periodically
survey SBIR recipients to gather a variety of data on program
participation, including the recipients' commercialization experiences.
Specifically,
* Since 2000, DOD has systematically gathered information
electronically on the commercialization of phase II awards from all
phase I and phase II applicants and award recipients and maintains the
information in a commercialization database. Commercialization outcomes
that DOD monitors include such measures as (1) sales revenue from new
products and non-R&D services resulting from the phase II technology;
(2) additional investment from sources other than the federal SBIR
program in activities that further the development, commercialization,
or both of the phase II technology; (3) whether the phase II technology
has been used in a DOD system or acquisition program, and if so, which
system or program; (4) the number of patents resulting from the
contractor's participation in the SBIR program; (5) growth in the
number of employees at the firm; and (6) whether the firm has completed
an initial public offering of stock resulting, in part, from the phase
II project. DOD uses the accumulated data to assign a commercialization
score to applicants that have received four or more prior SBIR awards
based on a comparison of their commercialization experience with the
average experience of other comparable applicants and uses the score to
help select proposed projects for funding. In addition, recipients of
phase II awards are required to update the information one year after
the start of the project, at the completion of the project, and
subsequently when the recipient submits a new SBIR application to DOD.
Firms that do not submit a new SBIR application are asked to
voluntarily provide updates on an annual basis after the completion of
their phase II project. According to DOD officials, 66 percent of phase
II award recipients updated their commercialization information when
they submitted a new application, 11 percent provided the information
without submitting a new application, and 23 percent did not update
their information.
* For over 23 years, DOE has conducted an annual survey of SBIR phase
II recipients, both active and inactive, to track the commercialization
success of its SBIR-funded projects. The survey requests recipients to
(1) list all products and services derived from their SBIR projects;
(2) report on sales, phase III investment related to these products and
service, or both; and (3) identify which phase II projects contributed
to the development of the products and services. According to DOE,
approximately 90 percent of its phase II recipients respond to these
annual surveys.
* NASA has systematically gathered information on the commercialization
of SBIR awards through annual surveys of phase II award recipients from
1997 to 2002. In these surveys, NASA obtained data on various
commercial outcomes, such as sales to nongovernment markets of the SBIR-
funded research results, procurement of the research results by NASA or
other federal agencies, cumulative private capital funding, royalty and
licensing revenue from nongovernment sources based on the SBIR-funded
research results, creation of new business ventures based on the SBIR-
funded research results, and number of patents and patent applications
resulting from these awards. According to NASA, approximately 91
percent of its phase II award recipients responded to these annual
surveys.
* NIH also surveys SBIR award recipients to gather commercialization
information. Specifically, in 2002, NIH conducted a comprehensive
evaluation of its SBIR program. As part of this evaluation, NIH
surveyed recipients of phase II awards between 1992 and 2001 to obtain
data relating to the range of SBIR program goals, including
commercialization of research results. According to NIH officials, the
2002 survey results formed a baseline that NIH staff could use to
systematically monitor and evaluate the program. In 2004 and 2005, NIH
again contacted award recipients to update the information obtained in
the 2002 survey. Commercialization outcomes tracked by NIH include (1)
sales realized for a product or service that resulted from the SBIR-
funded research; (2) status of the Food and Drug Administration (FDA)
approval process for the SBIR-funded research results; (3) receipt of
FDA approval for SBIR-funded research results; and (4) receipt of
additional non-SBIR funding or capital.
* Since about 1998, NSF has collected historical commercialization
information from all phase II award recipients at the time an SBIR
application is submitted. This information is used in the proposal
review process to help select proposed projects for funding. In 2005,
NSF developed an annual survey of phase II award recipients that will
be used to gather information three, five, and eight years following
their awards. Specific outcomes on which NSF will gather data include
sales revenue based on the SBIR-funded research results, growth of
overall company sales and employment, receipt of additional non-SBIR
funding, and patents related to the SBIR funded research.
In contrast, two of the remaining three agencies we reviewed have
focused their efforts on gathering anecdotal success stories and
conducting periodic follow-up with a relatively smaller sample of SBIR
award recipients. For example, over the past 7 years, EPA has contacted
all phase II award recipients after their projects end to learn about
their commercial successes. Based on these contacts, officials estimate
that approximately 25 percent of the phase II projects funded by EPA
have been commercialized. EPA defines "success" as the receipt of more
than $300,000 in revenue from sales of the SBIR-funded project, an
amount greater than the SBIR funds awarded by EPA. Similarly, USDA has
periodically contacted a sample of about 20 to 25 percent of award
recipients to obtain information about sales of their SBIR-funded
research results. USDA last surveyed its phase II award recipients in
1997. USDA publishes the success stories on its Web site and in an
agency newsletter. About 2,500 people receive the newsletter and USDA
makes copies of the success stories available at SBIR conferences.
According to USDA officials, in the future they hope to gather data
more systematically and conduct site visits to the SBIR firms. The
eighth agency we reviewed, NIST, has recently developed a Web-based
survey that it plans to send in 2006 to all of its SBIR-award
recipients.
Tracking the Commercialization Success of the Overall SBIR Program
Remains a Challenge:
Although each of the eight agencies we reviewed has implemented or
plans to implement a method for gathering commercialization data,
agency officials identified several factors that complicate their
efforts. First, agency officials stated that it is difficult to track
commercialization because it can take years before companies achieve
commercial success. For example, USDA officials stated that, even over
the short term, the effort to contact past award recipients consumes
considerable effort. During this time, companies may move, change
names, start a new business, or be purchased by other firms, all of
which make it difficult for the agencies to track and link companies to
the original SBIR awards.
Second, because the authorizing legislation lacks a clear definition of
what constitutes "commercialization" success, agencies not only
differed on the types of commercialization outcomes they captured, but
also in how they tracked commercial success. SBA officials acknowledged
that its guidance has provided considerable latitude to agencies on
this issue in light of the wide range of industries represented by the
participating agencies. Commercialization outcomes captured by the
participating agencies included sales revenue based on the SBIR-funded
research results, receipt of additional non-SBIR funding to further
develop the research results, marketing activities ongoing or
completed, and public offering of company stock. However, not all
agencies are tracking all of these outcomes; therefore, assessing
overall commercial success of the SBIR program across the various
agencies remains a challenge.
Third, agency officials stated that SBIR award recipients may be
reluctant to provide information related to their trade and business
operations, which they consider proprietary and sensitive. Companies
are often not willing to provide comprehensive data on their sales and
particularly the investments they receive due to competitive concerns.
Finally, agency officials told us that past recipients have no
incentive to voluntarily complete commercialization surveys and update
the information on their commercial experience unless they are applying
for a new SBIR award. As a result, they do not expect that a large
percentage of recipients will complete the information needed for the
government-use section of the Tech-Net database. Agency officials
believe that despite their best efforts, the data needed to conduct
evaluations of the SBIR program are likely to be incomplete.
Conclusions:
In the last 5 years, SBA has been unable to meet the congressional
directive to develop a government-use database that would provide
better information on the SBIR program and allow for program
evaluation. Although it has established a public-use Tech-Net database
and has worked with participating agencies to achieve greater
consistency in the data submitted for the database, the quality of the
data remains a concern. The steps on which SBA relies to ensure that
data are complete and accurate are inadequate and it has no plans to
correct errors or supply missing data associated with the historical
data already in the database. We believe that unless necessary controls
are established and implemented to ensure the completeness,
consistency, and accuracy of the SBIR data reported to SBA by
participating agencies, the government-use section of the Tech-Net
database, which depends on the public-use Tech-Net database, will be of
limited use for program evaluation purposes when it becomes operational.
Recommendation for Executive Action:
We recommend that the Administrator, SBA, and the SBIR participating
agencies work together to strengthen efforts to ensure that the data
collected for SBA's Tech-Net database are complete, consistent, and
accurate.
Agency Comments and Our Evaluation:
We provided SBA and the eight SBIR participating agencies included in
this review a draft of this report for their review and comment. SBA
and the eight agencies generally agreed with the report's findings and
SBA and five of the eight agencies also stated their concurrence with
the recommendation. Three agencies--EPA, NIST, and NSF--did not
indicate whether they agreed or disagreed with the recommendation.
In addition, SBA stated that it was concerned that our conclusions did
not reflect the fact that it plans to have the government-use section
of the Tech-Net database operational by early fiscal year 2007. We have
not modified our conclusions because the fact remains that SBA has not
met the congressional directive to establish a government-use database
during the last five years. Moreover, throughout this review, SBA
officials told us that the database would be operational by October 1,
2006. However, in its official comments, the agency has again modified
this date to some time early in fiscal year 2007. SBA also provided us
with technical comments that we have incorporated as appropriate. SBA's
letter is included in appendix II and the letters that we received from
DOD, NASA, and NIH are included in appendix III through V.
As agreed with your office, unless you publicly announce the contents
of this report earlier, we plan no further distribution until 30 days
from the report date. At that time, we will send copies of this report
to interested congressional committees; the Administrators of the
Environmental Protection Agency, National Aeronautics and Space
Administration, and Small Business Administration; the Directors of the
National Institutes of Health, National Institute of Standards and
Technology, and National Science Foundation; the Secretaries of
Agriculture, Defense, and Energy; and other interested parties. We will
also make copies available to others upon request. In addition, the
report will be available at no charge on the GAO Web site at http://
www.gao.gov.
If you or your staff have any questions about this report, please
contact me at (202) 512-3841 or mittala@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. GAO staff who made contributions to this
report are listed in appendix VI.
[Signed by]:
Anu K. Mittal:
Director, Natural Resources and Environment:
[End of section]
Appendix I: Objectives, Scope and Methods:
Our objectives for this review were to identify (1) the types of data
that participating Small Business Innovation Research (SBIR) program
agencies are reporting to the Small Business Administration (SBA) for
inclusion in the Tech-Net database, (2) the extent to which agencies
provide data for the Tech-Net database in a standard format to enable
program evaluation, (3) the extent to which SBA has met the mandate to
establish by early 2001 a government-use database that can be used for
program evaluation, and (4) the extent to which participating SBIR
agencies have developed and implemented techniques to track the
commercialization success of SBIR projects.
In conducting this study, we reviewed SBA and the SBIR-related
activities of 8 of the 11 SBIR participating agencies--Department of
Defense (DOD), Department of Energy (DOE), Environmental Protection
Agency (EPA), National Aeronautics and Space Administration (NASA),
National Institutes of Health (NIH), National Institute of Standards
and Technology (NIST), National Science Foundation (NSF), and
Department of Agriculture (USDA). These eight agencies account for over
98 percent of the total dollars awarded by the program in fiscal year
2005. To determine the types of data these participating agencies are
reporting to SBA and the extent to which SBA has complied with the
requirement to establish a government-use database that can be used for
program evaluation purposes, we compared the provisions in the Small
Business Innovation Research Program Reauthorization Act of 2000,
relevant legislative histories, SBA's Policy Directive implementing the
act issued in September 2002, and other guidance with agency SBIR Tech-
Net database reports. We identified and interviewed SBIR program
officials at each agency and officials responsible for submitting
program data to SBA. For these interviews, we used a protocol guide to
obtain information on program operations, data reporting, data quality,
and the Tech-Net database, including development of the government-use
section of the database. We also reviewed documentation from each of
the eight agencies and SBA regarding data elements and formatting
instructions.
To determine the extent to which data for the Tech-Net database are
provided in a standard format, enabling program evaluation, we compared
data provided to SBA by the eight participating agencies with data in
SBA's Tech-Net database for fiscal years 2004 and 2005, the 2 most
recent years for which data were available. The agency data included
information about the award itself, such as the award amount, a
descriptive abstract of the project, and a unique tracking number;
information about the award recipient, such as gender and socio-
economic status; and information about the type of firm that received
the award, such as number of employees and geographic location. To
assess the reliability of the data provided by the agencies, we
reviewed SBA's data system, internal controls, and related guidance,
rather than the systems and internal controls participating agencies
use to create the data provided to SBA. To assess the reliability of
the data in SBA's Tech-Net database, we interviewed SBA officials about
the database and reviewed related documentation. We also conducted
tests of the data themselves. We used analytic software to compare the
data provided by participating agencies with the data maintained by
SBA. We focused our review on data fields considered critical by SBA
officials, such as the awarding agency, the date of award, the award
recipient, the amount of the award, and the purpose of the award. We
also reviewed data fields related to SBIR program goals, such as gender
and socio-economic status of the recipient, and data on the number of
employees at the recipient firm. We used GAO's data reliability
guidance to identify key attributes of data quality that facilitate
program evaluation. These attributes include completeness, accuracy,
and consistency in format. Finally, we reviewed internal quality
control procedures. We determined that the data are sufficiently
reliable for our purpose, which is to report on SBA's efforts to ensure
consistency and completeness of the data it receives.
To determine the extent to which the government-use database
requirements have been implemented, we interviewed SBA officials, and
reviewed related documentation, such as minutes from meetings of SBA
and SBIR program directors, and overviews of the existing and planned
data systems. In addition, we attended a demonstration of the proposed
Internet interface for the government-use section and interviewed the
current contractor assisting SBA about implementation progress. We also
reviewed a contractor-prepared analysis of the functional and data
requirements for the integration of the public-and government-use
sections of the Tech-Net database. At each of the eight participating
agencies, we interviewed SBIR officials regarding the extent to which
SBA had consulted them in the development of the government-use
database.
To determine the extent to which participating agencies have developed
and implemented techniques to evaluate commercialization success of
SBIR projects, we reviewed agencies' documentation about their
commercialization assistance and monitoring efforts. Specifically, we
reviewed surveys that agencies had administered to award recipients,
resulting reports on survey results, and anecdotal descriptions of
commercialization success. We also reviewed provisions in SBIR
legislation, relevant legislative histories, and SBA's Policy Directive
regarding commercialization of SBIR-funded projects, as well as past
GAO reports. In addition, we interviewed officials at each of the eight
participating agencies to obtain information on the specific
commercialization outcomes they monitor, the history of each agency's
data collection efforts, and the agencies' experience in obtaining such
information from current and past award recipients.
We conducted our work from April 2006 through September 2006 in
accordance with generally accepted government auditing standards.
[End of section]
Appendix II: Comments from the Small Business Administration:
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
Page numbers in draft report may differ from those in this report.
[See PDF for image]
[End of figure]
U.S. Small Business Administration;
Washington, DC 20416:
October 6, 2006:
Ms. Anu Mittal:
Director, Natural Resources and Environment:
U.S. Government Accountability Office:
441 G St., NW:
Washington, DC 20548:
Dear Ms. Mittal:
Thank you for the opportunity to review and comment on the Government
Accountability Office (GAO) draft report entitled, "Small Business
Innovation Research: Agencies Need to Strengthen Efforts to Improve the
Completeness, Consistency, and Accuracy of Awards Data," GAO-07-38. We
have reviewed the draft report and have the following comments:
* Footnote 2 of the report is confusing. While it does indicate there
are eleven SBIR participating agencies, the language of the footnote is
uneven.
* On page 4, the report mentions that the SBIR Policy Directive (PD)
was issued in September 2002. This should be reworded to state that the
amended PD was issued in September 2002, otherwise, someone reading it
might think 2002 was the first time SBA issued a PD.
* Footnote 7 is confusing - read literally, it looks like the SBA is an
SBIR participating agency.
* Pages 5 and 11 of the report state that the SBIR participating
agencies
have cited a "variety" of reasons for not providing the data requested
including "frequent" changes in SBA's data requirements. SBA contends
that there have not been "frequent" changes, but rather minor
clarifications of data requirements.
For example, NIH cited a change in SBA's reporting requirements as a
reason for submission of incomplete data. SBA changed the field name
from "Minority" to "Socially and Economically Disadvantaged" because it
is the accepted description today. The content of that field was not
expected to change. Perhaps our communication about that was not clear,
but it is not a new field.
* Page 16 discusses the SBIR participating agencies' efforts in
tracking the commercialization success of the SBIR awardee. The report
discusses the differences in how the agencies define commercialization
success. The SBA looks to assist in resolving this problem by using the
Government Database to assess commercialization program-wide.
As a whole, the general findings and recommendations of the report
appear reasonable. We do, however, take exception to the tone and
content of the "Conclusions" section of the report. Specifically, we
are concerned that while the "Conclusion" indicates that SBA has not
developed the Government-Use section of the TechNet Database in over
six years, it does not mention that SBA anticipates having this section
of the database operational early in FY 2007.
Additionally, while we concur with the finding that the quality of the
data remains a concern, SBA has taken significant steps to help improve
the quality of data. SBA is working with the agencies to improve the
controls on the propriety of the data submitted.
In the section labeled "Conclusions," the report states "The steps that
SBA relies on to ensure that data are complete and accurate are
inadequate and it has no plans to correct errors or supply missing data
associated with historical data already in the database." SBA is aware
of records in the database that are incomplete. However, this can be
rectified by using a combination of electronic and manual processes to
gather the missing data elements. The collection of DUNS numbers in the
database would allow SBA to access information provided by many
awardees to the Central Contractor Registration (CCR) database and the
Dynamic Small Business (DSBS) database. DUNS matching would all ow the
collection of current firm-level information. SBA and the participating
agencies will have to work together to correct some of the historical
data. SBA has spoken with one agency to date about working
cooperatively to provide historical data for older awards and has a
commitment from that agency. Similar dialog and success can be expected
with other agencies.
Should you have questions on these comments, please contact Edsel
Brown, Assistant Administrator, Office of Technology at (202) 205-7343.
Sincerely,
[Signed by]:
Arthur Collins:
Acting Associate Administrator:
Office of Government Contracting:
The following are GAO‘s comments on the Small Business Administration‘s
(SBA) letter dated October 6, 2006.
GAO Comments:
1. We revised footnote 2 to more clearly reflect the 11 SBIR
participating agencies.
2. We revised the text to clarify this statement.
3. We deleted reference to SBA in footnote 7.
4. We added SBA‘s position to the report.
5. We added SBA‘s position to the report. However, because SBA has not
met the congressional deadline to develop a government-use database, we
did not modify our conclusions.
[End of section]
Appendix III: Comments from the Department of Defense:
[See PDF for image]:
[End of image]:
Office of the Under Secretary of Defense
3000 Defense Pentagon
Washington, DC 2031-3000
Acquisition:
Technology:
And Logistics:
Oct 06 2006
Ms. Anu Mittal:
Director, Natural Resources and Environment:
U.S. Government Accountability Office:
441 G St., NW:
Washington, DC 20548:
Dear Ms. Mittal:
This is the Department of Defense (DoD) response to GAO Draft Report
"Small Business Innovation Research: Agencies Need to Strengthen
Efforts to Improve the Completeness, Consistency, and Accuracy of
Awards Data," (GAO Code 360677/ GAO-07-38), dated September 20, 2006.
The GAO recommends that the Administrator, the Small Business
Administration (SBA) and agencies participating in the SBIR Program
work together to strengthen efforts to ensure that the data collected
for SBA's Tech-Net database are complete, consistent, and accurate. The
Department concurs with the recommendation recognizing that complete,
high-integrity data is essential for program monitoring, assessment and
evaluation.
We appreciate the opportunity to comment on the draft report and look
forward to working in concert with the SBA and the other participating
agencies to improve the quality of Tech-Net data.
Sincerely,
[Signed by]:
Linda B. Oliver for:
Frank M. Ramos:
Director, Small Business Programs:
Enclosure:
As stated:
GAO Draft Report--Dated September 20, 2006:
GAO Code 360677/ GAO-07-38:
"Small Business Innovation Research: Agencies Need to Strengthen
Efforts to Improve the Completeness, Consistency, and Accuracy of
Awards Data"
Department of Defense Comments:
To the Recommendation:
Recommendation: The GAO recommends that the Administrator, the Small
Business Administration (SBA) and agencies participating in the SBIR
Program work together to strengthen efforts to ensure that the data
collected for SBA's Tech-Net database are complete, consistent, and
accurate.
Response: Concur. Complete, high-integrity data is essential for SBIR
program monitoring, assessment and evaluation. The Department fully
supports SBA annual data calls and looks forward to working together
with the SBA and the other participating agencies to improve the
quality of Tech-Net data.
[End of section]
Appendix IV: Comments from the National Aeronautics and Space
Administration:
National Aeronautics and:
Space Administration:
Headquarters:
Washington, DC 20546-0001:
[See PDF for image]:
[End of image]
October 6, 2006:
Innovative Partnerships Program Office:
Reply to Attn of:
Ms. Anu K. Mittal:
Director, Natural Resources and Environment:
United States Government
Accountability Office Washington, D.C. 20548:
Dear Ms. Mittal:
NASA appreciates the opportunity to comment on the draft Government
Accountability Office (GAO) report, "Small Business Innovation Research
Agencies Need to Strengthen Efforts to Improve the Completeness,
Consistency, and Accuracy of Awards Data" (GAO-07-3 8).
The draft report makes a single recommendation, addressed to all Small
Business Innovation Research (SBIR) participating agencies, including
NASA and the Small Business Administration (SBA), which is responsible
for providing policy direction for the program across all agencies. The
report recommends "that the Administrator, SBA, and the SBIR
participating agencies work together to strengthen efforts to ensure
that the data collected for SBA's Tech-Net database are complete,
consistent, and accurate.":
NASA concurs with this recommendation. The SBIR program executive and
program manager responsible for the execution of the program at NASA
will contact Mr. Edsel M. Brown Jr., Assistant Administrator, Office of
Technology, U.S. Small Business Administration, to affirm NASA's
commitment to participate in joint discussions with all Federal
agencies involved for the purposes of considering GAO's findings and
recommendation, as well as determining and implementing an appropriate
course of action.
Thank you for the opportunity to respond to this draft report.
Sincerely,
[Signed by]:
Jack Yadvish, Acting Director:
Innovative Partnerships Program Office:
[End of section]
Appendix V: Comments from the National Institutes of Health:
[See PDF for image]:
[End of image]:
Appendix V: Comments from the National Institutes of Health:
Department Of Health & Human Services:
Public Health Service:
National Institutes of Health:
Bethesda, Maryland 20892:
Oct 6, 2006:
Ms. Anu K. Mittal:
Director, Natural Resources and Environment:
U.S. Government Accountability Office:
441 G Street, N.W.
Washington, D.C. 20548:
Dear Ms. Mittal:
We appreciate the opportunity to review and comment on the U.S.
Government Accountability Office (GAO) report, Small Business
Innovation Research: Agencies Need to Strengthen Efforts to Improve the
Completeness, Consistency, and Accuracy of Awards Data (GAO-07-38).
The National Institutes of Health supports the recommendation that the
Small Business Administration (SBA) and Small Business Innovation
Research participating agencies work together to strengthen efforts to
ensure that the data collected for SBA's Tech-Net database are
complete, consistent, and accurate. We recognize the importance of the
collection of program results and look forward to working
collaboratively with the SBA to improve the quality of the data in the
Tech-Net database.
Sincerely,
Elias A. Zerhouni, M.D.
Director:
[End of section]
Appendix VI: GAO Contact and Staff Acknowledgments:
GAO Contact: Anu K. Mittal, (202) 512-3841:
Acknowledgments: In addition to the contact named above, Cheryl
Williams, Assistant Director; Bernice H. Dawson; Vondalee R. Hunt; and
Marcus L. Oliver made key contributions to this report. Also
contributing to this report were Nancy Crothers, Grant Mallie, Gregory
Marchand, and Rebecca Shea.
[End of section]
FOOTNOTES
[1] Small Business Innovation Development Act of 1982, Pub. L. No. 97-
219 (1982).
[2] The eleven SBIR participating agencies are the departments of
Agriculture (USDA), Commerce, Defense (DOD), Education, Energy (DOE),
Health and Human Services, Homeland Security, and Transportation; the
Environmental Protection Agency (EPA); the National Aeronautics and
Space Administration (NASA); and the National Science Foundation (NSF).
[3] Pub. L. No. 102-564 (1992).
[4] The Tech-Net database also contains award information on SBA‘s
Small Business Technology Transfer (STTR) program. However, the focus
of this report is on the SBIR program and not the STTR program. As
such, any discussion of the Tech-Net database only refers to the
information in the database related to the SBIR awards.
[5] Pub. L. No. 106-554 (2000).
[6] See GAO, Federal Research: Observations on the Small Business
Innovation Research Program, GAO-05-861T, June 28, 2005; Federal
Research: Observations on the Small Business Innovation Program, GAO/T-
RCED-98-218, June 4, 1998; Federal Research: Small Business Innovation
Research Shows Success but Can be Strengthened, GAO/RCED-92-37, Mar.
30, 1992; and Federal Research: Assessment of Small Business Innovation
Research Programs, GAO/RCED-89-39, Jan. 23, 1989.
[7] The SBIR participating agencies included in this review are DOD,
DOE, EPA, NASA, National Institutes of Health (NIH) within the
Department of Health and Human Services, National Institute of
Standards and Technology (NIST) within the Department of Commerce, NSF,
and USDA.
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
