Small Business Administration
Model for 7(a) Program Subsidy Had Reasonable Equations, but Inadequate Documentation Hampered External Reviews Gao ID: GAO-04-9 March 31, 2004The Small Business Administration (SBA) approved about $8.6 billion in loan guarantees through its 7(a) loan program in fiscal year 2003. SBA must estimate the subsidy cost of this program. Since fiscal year 2003, SBA has been using econometric modeling to estimate the subsidy. This report reviews SBA's estimation methodology and equations, assesses the default and recovery rates the model produced, identifies ways to enhance the estimates' reliability, describes the process for developing the model, and analyzes SBA's data.
From an economics perspective, SBA's econometric equations were reasonable, and its model produced estimated default and recovery rates that were in line with historical experience. However, from an audit perspective, SBA's lack of documentation of the model development process precluded GAO, and others, from independently evaluating the model's development and determining if SBA used a sound and consistently applied method to select and reject model variables. Taking into account economic reasoning and research, SBA's econometric equations for estimating defaults, prepayments, and recoveries were reasonable. SBA's equations used a limited set of variables; equations using other variables could also be reasonable but would produce different estimates. Since an estimate is an approximation, no one estimate can be considered accurate, and reasonable estimates can fall within a range of values. The model's estimated default and recovery rates were in line with recent historical experience. SBA could improve its estimation methodology by periodically checking for and correcting errors and should consider adding more borrower information, such as credit scores. Some errors in the model resulted in understating the estimated program costs. SBA used the expertise of other agencies and a contractor to develop its model and worked closely with the Office of Management and Budget (OMB), which must approve the methodology agencies use to estimate subsidies. OMB officially approved the model in the fall of 2002. SBA did not adequately document its model development process, including alternative variables considered and rejected, to enable external reviewers to assess the process that was used. Further, GAO and two other independent reviewers could not determine whether a bias existed in the model by systematically excluding variables to influence the subsidy rate in a particular direction. Adequate documentation, a key internal control, would enable SBA and other agencies to demonstrate the rationale and basis for key aspects of the model that provide important cost information for budgets, financial statements, and congressional decision makers and facilitate SBA's annual financial statement audit. Current OMB and other guidance is either silent or unclear about the level of documentation necessary for credit subsidy model development. SBA had a process to help ensure data integrity and data consistency in the equations with the loan-level data in its databases. Although errors existed in SBA's data systems, the magnitude and nature of these errors were not likely to significantly affect the subsidy rate.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-04-9, Small Business Administration: Model for 7(a) Program Subsidy Had Reasonable Equations, but Inadequate Documentation Hampered External Reviews
This is the accessible text file for GAO report number GAO-04-9
entitled 'Small Business Administration: Model for 7(a) Program Subsidy
Had Reasonable Equations, but Inadequate Documentation Hampered
External Reviews' which was released on March 31, 2004.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Committees:
March 2004:
SMALL BUSINESS ADMINISTRATION:
Model for 7(a) Program Subsidy Had Reasonable Equations, but Inadequate
Documentation Hampered External Reviews:
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-9]:
GAO Highlights:
Highlights of GAO-04-9, a report to Chairman and Ranking Minority
Member, House Committee on Small Business; Ranking Minority Member,
Senate Committee on Small Business and Entrepreneurship
Why GAO Did This Study:
The Small Business Administration (SBA) approved about $8.6 billion in
loan guarantees through its 7(a) loan program in fiscal year 2003. SBA
must estimate the subsidy cost of this program. Since fiscal year 2003,
SBA has been using econometric modeling to estimate the subsidy. This
report reviews SBA‘s estimation methodology and equations, assesses the
default and recovery rates the model produced, identifies ways to
enhance the estimates‘ reliability, describes the process for
developing the model, and analyzes SBA‘s data.
What GAO Found:
From an economics perspective, SBA‘s econometric equations were
reasonable, and its model produced estimated default and recovery rates
that were in line with historical experience. However, from an audit
perspective, SBA‘s lack of documentation of the model development
process precluded GAO, and others, from independently evaluating the
model‘s development and determining if SBA used a sound and
consistently applied method to select and reject model variables.
Taking into account economic reasoning and research, SBA‘s econometric
equations for estimating defaults, prepayments, and recoveries were
reasonable. SBA‘s equations used a limited set of variables; equations
using other variables could also be reasonable but would produce
different estimates. Since an estimate is an approximation, no one
estimate can be considered accurate, and reasonable estimates can fall
within a range of values. The model's estimated default and recovery
rates were in line with recent historical experience. SBA could improve
its estimation methodology by periodically checking for and correcting
errors and should consider adding more borrower information, such as
credit scores. Some errors in the model resulted in understating the
estimated program costs.
SBA used the expertise of other agencies and a contractor to develop
its model and worked closely with the Office of Management and Budget
(OMB), which must approve the methodology agencies use to estimate
subsidies. OMB officially approved the model in the fall of 2002.
SBA did not adequately document its model development process,
including alternative variables considered and rejected, to enable
external reviewers to assess the process that was used. Further, GAO
and two other independent reviewers could not determine whether a bias
existed in the model by systematically excluding variables to influence
the subsidy rate in a particular direction. Adequate documentation, a
key internal control, would enable SBA and other agencies to
demonstrate the rationale and basis for key aspects of the model that
provide important cost information for budgets, financial statements,
and congressional decision makers and facilitate SBA‘s annual financial
statement audit. Current OMB and other guidance is either silent or
unclear about the level of documentation necessary for credit subsidy
model development.
SBA had a process to help ensure data integrity and data consistency in
the equations with the loan-level data in its databases. Although
errors existed in SBA‘s data systems, the magnitude and nature of these
errors were not likely to significantly affect the subsidy rate.
What GAO Recommends:
SBA should (1) determine whether to include in the model other
information from its new loan monitoring system, (2) periodically
evaluate and update the model, and (3) document the model development
process. OMB should require agencies to document the basis and process
for developing their credit subsidy models.
SBA agreed with recommendations to improve the final model but SBA and
OMB disagreed that the model development was inadequately documented
and disagreed with our recommendations to improve such documentation
and guidance.
However, given the difficulty experienced by reviewers due to
inadequate documentation, we continue to recommend that SBA document
the basis and process for developing its model and that OMB require
this documentation.
www.gao.gov/cgi-bin/getrpt?GAO-04-9.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Davi D'Agostino at (202)
512-8678 or dagostinod@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
SBA's Equations Were Reasonable and Estimated Default, and Recovery
Rates Were in Line with Historical Experience:
SBA's Model Could Be Enhanced by Adding Information on Borrowers,
Correcting Errors, and Updating Some Data:
SBA Collaborated with OFHEO and OMB to Develop the Model:
Lack of Adequate Model Documentation Hampered Independent Reviews of
SBA's Model:
SBA Had a Process to Help Ensure Data Quality and the Data Used in the
Model and SBA's Loan Level Databases Were Consistent:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendixes:
Appendix I: Objectives, Scope, and Methodology:
Assessing the Reasonableness of the Model's Econometric Equations and
Evaluating the Model's Estimated Default, Prepayment, and Recovery
Rates:
Identifying Additional Steps SBA Could take to Further Enhance the
Reliability of the Model:
Reviewing SBA's Process of Developing the Subsidy Model:
Evaluating the Model's Supporting Documentation, Including Its
Discussion of What Variables Were Tested and Rejected:
Determining What Steps SBA Took to Ensure the Integrity of the Data
Used in the Model and Whether These Data Were Consistent with
Information in Its Databases:
Appendix II: Analysis of Default, Prepayment, and Recoveries
Econometric Equations:
SBA's Default and Prepayment Equations:
Effects of Including Additional Variables:
SBA's Recovery Equation:
Appendix III: Comments from the Small Business Administration:
GAO Comments:
Appendix IV: Comments from the Office of Management and Budget:
GAO Comments:
Appendix V: GAO Contacts and Staff Acknowledgments:
GAO Contacts:
Staff Acknowledgments:
Tables:
Table 1: Variable Names and Descriptions:
Table 2: Multinomial Logistic Regression Coefficient Estimatesa:
Table 3: Names and Descriptions of Additional Variables:
Table 4: Multinomial Logistic Regression Coefficient Estimatesa:
Table 5: Distribution of SIC Industry Codes in SBA's Loan Database
Distribution of SIC Industry Codes in SBAs Loan Database:
Table 6: Variable Names and Descriptions:
Table 7: Recovery Model:
Figures:
Figure 1: Major Segments of the Model to Estimate 7(a) Subsidy Rate:
Figure 2: Estimated Default Rates Compared with Average Default
Experience from 1992 through 2001:
Figure 3: Estimated Default Rates Compared with Fiscal Year 2001 Actual
Default Experience:
Abbreviations:
CFO: Chief Financial Officer:
FCRA: Federal Credit Reform Act of 1990:
GDP: gross domestic product:
NAIC: North American Industrial Classification:
OFHEO: Office of Federal Housing Enterprise Oversight:
OMB: Office of Management and Budget:
SBA: Small Business Administration:
SIC: Standard Industrial Classification:
Letter March 31, 2004:
The Honorable Donald A. Manzullo:
Chairman, Committee on Small Business:
House of Representatives:
The Honorable Nydia M. Velazquez:
Ranking Minority Member:
Committee on Small Business:
House of Representatives:
The Honorable John F. Kerry:
Ranking Minority Member:
Committee on Small Business and Entrepreneurship:
United States Senate:
The 7(a) program is the Small Business Administration's (SBA) largest
lending program for small businesses. SBA reported that it approved
about $8.6 billion in loan guarantees in fiscal year 2003. The program
provides loan guarantees of up to 85 percent for loans made to small
businesses that are unable to obtain financing on reasonable terms in
the private credit markets. Like most federal loan or loan guarantee
programs, SBA's 7(a) program is subject to the Federal Credit Reform
Act of 1990 (FCRA). FCRA requires most agencies with government lending
programs to estimate annually the cost to the federal government of
extending or guaranteeing credit over the life of the loans (the
subsidy cost). Since an estimate is an approximation, no one estimate
can be considered accurate with certainty, and reasonable estimates can
fall within a range of values. Changes in estimation methodologies,
variables, or data used to calculate an estimate, are likely to result
in differences in the estimate. In fiscal year 2003, SBA implemented a
new methodology to estimate the subsidy cost of the 7(a) program that
is based on econometric modeling.[Footnote 1] SBA officials told us
that the new 7(a) model was the first step in a long-term effort to
develop and implement new econometric models for their credit programs.
Although this allowed SBA to build a model that responds to the need
for greater sensitivity to a wider variety of factors than a model
based on historical averages, SBA believes that this approach may not
be appropriate for all its credit programs.
In order to calculate the subsidy cost of their programs, agencies must
estimate the present value of future cash flows over the life of the
program, which for the 7(a) program are principally affected by
defaulted loans, prepayments of outstanding loans, recoveries on
defaulted loans, and fees. The revised method SBA adopted for the
subsidy calculation has four segments: (1) the econometric equations
that are used to estimate the likelihood of defaults and prepayments,
(2) the equations used to estimate the extent of recoveries, (3) the
cash flow module, and (4) the Office of Management and Budget (OMB)
Credit Subsidy Calculator, as shown in figure 1. The results of the
first and second segments--the econometric equations--are a key input
into the third. The third segment--the cash flow module--uses these
results, along with OMB forecasts of interest rates, unemployment
rates, and gross domestic product growth rates to estimate cash inflows
from fees and recoveries on defaulted loans and outflows from claim
payments on defaulted loans. The resulting cash flows are entered into
the fourth segment, the OMB Credit Subsidy Calculator, which calculates
the (1) present values of the cash flows and (2) the subsidy rate.
Figure 1: Major Segments of the Model to Estimate 7(a) Subsidy Rate:
[See PDF for image]
[End of figure]
This report responds to your November 26, 2002, and December 11, 2002,
requests that we review the methodology that SBA developed to estimate
the subsidy costs of its 7(a) loan program for the fiscal year 2004
budget. As agreed with your staff, we (1) assessed the reasonableness
of the model's econometric equations and evaluated the model's
estimated default and recovery rates based on the 7(a) program's recent
historical loan experience; (2) identified any additional steps SBA
could take to further enhance the reliability of its subsidy estimate
produced by the model; (3) described SBA's process for developing the
subsidy model; (4) evaluated the model's supporting documentation
including its discussion of what variables were tested and rejected;
and (5) determined what steps SBA takes to ensure the integrity of the
data used in the model and determined whether these data are consistent
with information in SBA's databases. We did not, however, validate
SBA's model.
First, to analyze the model, we obtained from SBA copies of the model
as approved by OMB in 2002, along with the loan-level data that were
used to develop the subsidy estimates. We analyzed the econometric
equations to determine whether they were reasonable based on the
variables they included, the statistical techniques used, and the
results obtained. For example, we determined whether the econometric
equations included appropriate variables and whether the variables used
in the equations were statistically significant. To evaluate the
model's estimated default and recovery rates, we compared these rates
with recent historical loan experience of the 7(a) program provided by
SBA. Using SBA's data, we also calculated what SBA would have estimated
for default and recovery rates based on the estimation methodology it
used prior to its fiscal year 2003 budget submission. Second, to
identify any additional steps SBA could take to enhance the reliability
of its model, we considered additional types of data that SBA might
collect and consider including in its econometric equations. As part of
this analysis, we reviewed the academic literature on default modeling
and interviewed officials with several banks engaged in similar
efforts. Third, to describe SBA's process for developing the model we
met with SBA and OMB officials. Fourth, to evaluate the model's
supporting documentation, including its discussion of what variables
were tested and rejected, we obtained and analyzed available relevant
documents and met with SBA officials and their contractor who developed
the model. We compared the information presented in SBA's model
documentation with existing credit subsidy guidance. Finally, to
determine what steps SBA took to ensure the integrity of the data used
by the model and to determine whether these data were consistent with
information in its databases, we assessed SBA's processes for ensuring
data reliability. We examined the type and level of errors and
evaluated the likelihood that they would significantly affect the
credit subsidy estimates. We also compared the loan-level data used in
the model with the data contained in SBA's databases. Appendix I
discusses the details of our methodology.
We conducted our work in Washington, D.C. from December 2002 to March
2004 in accordance with generally accepted government auditing
standards.
Results in Brief:
Overall, we found that from an economics perspective, SBA's econometric
equations were reasonable, and the SBA model produced estimated default
and recovery rates that were in line with historical experience.
However, from an audit perspective, SBA's lack of adequate
documentation of the model development process precluded us from (1)
independently evaluating the model's development; (2) determining
whether SBA used a sound and consistently applied method to select and
reject variables to be included in the model; and (3) determining
whether a bias from selecting variables existed in the model.
We found that SBA's econometric equations for estimating defaults,
prepayments, and recoveries were reasonable. SBA's equations used a
limited set of variables; equations using other variables could also be
reasonable but would produce different estimates. We also found that
the model's estimated default and recovery rates were in line with
recent historical experience. SBA's econometric equations related the
likelihood of defaults and/or prepayments to several variables that
economic reasoning and prior research suggested were appropriate to
this type of model, and, at the time of our review, SBA used
appropriate statistical techniques to identify the nature of these
relationships. In addition, SBA's equations produced estimated
relationships for defaults and prepayments that were consistent with
expectations based on economic reasoning. For example, the likelihood
of default was estimated to be higher when unemployment was higher.
SBA's equations used a limited set of variables, and we found that
equations using additional variables available to SBA that it did not
include, such as measures of interest rates and the businesses'
industry type, would also be reasonable. If SBA had used these
alternative equations, it might have estimated a higher or lower
subsidy rate. SBA did not include any economic variables in its
equation for estimating recoveries, so that forecasted recovery amounts
were not dependent on expected economic conditions. According to
documentation provided by SBA of the work done to develop this
equation, adding economic variables would not have increased the
precision of the recovery rate estimates.
SBA could enhance the model and the reliability of the subsidy estimate
produced by the model by including additional information that SBA
expects to have in the future and by correcting errors. SBA intends to
collect new business and business-owner information to determine how it
affects loan performance and such information may suggest variables
that can be useful in the model. SBA's econometric equations used
variables from its current databases and economic indicators, such as
gross domestic product (GDP) growth rates and unemployment rates, to
forecast future defaults and prepayments. However, at the time of our
review, SBA's current database did not include other information on
businesses or business owners, such as information on borrowers' credit
that is often used by private sector lenders to determine potential
defaults and losses. Academic literature on default models suggests
that such information is predictive of defaults. SBA has recently
contracted to develop a loan monitoring system that is intended to
track this information and allow the agency to determine how it affects
loan performance. During our review of the model, we identified some
errors that resulted in underestimates of the program costs of around
$6.5 million or about 6.8 percent of the estimated cost of the program
for fiscal year 2004.
To develop its subsidy model, SBA drew on the expertise of other
government agencies and consulted with OMB officials. In February 2002,
SBA entered into an arrangement with the Office of Federal Housing
Enterprise Oversight (OFHEO), which has staff with expertise in
econometric modeling, to assist in the development of the 7(a) subsidy
model.[Footnote 2] OMB also played a key role in the development of the
model because FCRA requires OMB to approve the methodology that each
federal agency uses to estimate the subsidy costs of its loan programs.
Thus, SBA consulted with OMB officials during the model's development,
and OMB officially approved the model in the fall of 2002. OMB
officials said that their role in reviewing the model was primarily to
provide oversight and ensure compliance with the law. Because at the
time of our review, SBA routinely had its cash flow models reviewed by
an independent third party, it hired an outside consultant to conduct
limited reviews of the econometric equations and cash flow segment. The
consultant identified some errors that SBA corrected.
SBA did not prepare adequate supporting documentation to enable us and
other independent reviewers to understand and evaluate the process that
SBA used to develop the model. While SBA provided some general
documentation of its model development process, the documentation
lacked adequate discussion of alternative variables or combinations of
variables that SBA considered, which variables were rejected for which
reasons, and specific examples based on results of earlier regressions.
As a result, we were unable to determine whether a bias in selecting
variables existed in the model. SBA officials told us that they did not
prepare this type of documentation because they believed that there was
no specific requirement to do so. Current guidance is either silent or
unclear about supporting documentation needed to explain the
development of econometric models used to generate credit subsidy
estimates for the budget and financial statements. However, maintaining
adequate documentation on how such models were developed is a sound
internal control practice that would provide SBA and other agencies the
opportunity to more fully demonstrate and explain the rationale and
basis for key aspects of their models that provide important cost
information for budgets, financial statements, and congressional
decision makers. This documentation would also help facilitate SBA's
annual financial statement audit.
SBA hired a private contractor to reconcile the information submitted
to it by 7(a) program lenders with the data stored in SBA's loan-level
databases on a monthly basis and, at the time of our review, had an
ongoing process to correct any errors that were found. Although errors
existed in SBA's data systems at the time of our review, we determined
that the magnitude and nature of these errors were not likely to
significantly affect the subsidy rate. In addition, SBA officials told
us that they performed various ad hoc reviews of the information in
SBA's loan-level databases to assess its accuracy and were currently
assessing various alternatives to further enhance its data integrity.
On the basis of our analysis of a statistical sample of defaulted,
prepaid, and active loans, as well as recoveries from defaulted loans,
we found that the data SBA used to calculate the subsidy costs were
consistent with the loan level data contained in SBA's actual databases
at the time of our review.
This report contains three recommendations to SBA and one
recommendation to OMB. We recommend that SBA (1) determine how best to
include in the model borrower-specific information that it intends to
collect in its new loan monitoring system; (2) establish a process for
periodically revising the model to correct errors and to reflect any
changes in the 7(a) program or other factors that could affect the
subsidy estimate; and (3) prepare adequate documentation of the model
development process including a detailed discussion of alternative
variables or combinations of variables that were considered, tested,
and rejected and criteria for doing so. We also recommend that OMB
require that agencies document the basis for credit subsidy estimates
and reestimates, including the process followed for selecting model
methodologies over alternatives and variables tested and rejected with
the basis for excluding them.
We received comments on a preliminary draft of this report from SBA and
OMB. SBA agreed with the findings and the first two recommendations
related to the final model. OMB had no comments. While a draft of this
report was at the agencies for comment, we continued to pursue
additional documentation that SBA had that might further explain its
7(a) model development process, including what variables were selected
and rejected and why. This final report discusses the lack of adequate
documentation and recommends improvements in SBA's documentation of the
development process for its credit subsidy models and in OMB's Circular
A-11 guidance. SBA generally disagreed with our findings and
recommendations related to the lack of adequate documentation
supporting the model's development process. OMB disagreed with our
recommendation that it revise Circular A-11. However, in light of the
consistent difficulty experienced by three independent reviewers of
SBA's 7(a) credit subsidy model, including SBA's financial statement
auditors, we continue to recommend that SBA enhance its credit subsidy
model documentation and that OMB require agencies to document the basis
and process used to develop credit subsidy models, including
understanding the model's basis and the variables that were selected
and rejected.
Background:
FCRA was enacted, among other reasons, to provide more accurate
measures of the costs of federal loan programs and to more accurately
compare costs among credit programs and between credit and noncredit
programs. FCRA requires agencies with loan guarantee programs to
estimate the subsidy cost, or the cost to the government, of their loan
guarantees over the life of the loan. To calculate the subsidy costs,
agencies must calculate, on a cohort[Footnote 3] basis, the net present
value of the forecasted cash flows for the program, which for SBA
included estimated defaults, recoveries, and fees related to the 7(a)
program. In addition, as part of this process, SBA must determine the
effects of loan prepayments on the cash flows. Under FCRA, SBA provides
information that generates a single subsidy rate and does not provide
information about any uncertainty in its estimate of the rate or other
factors affecting the rate, such as prepayments or defaults.
Prior to its 2003 budget submission, SBA's methodology for estimating
the subsidy on its 7(a) loans used historical averages for defaults and
recoveries based on loan data going back to 1986 as the basis for
estimates of future defaults and recoveries. This approach resulted in
fairly stable subsidy estimates on a yearly basis as it included a
sufficient volume of historical information that smoothed out
fluctuations in economic conditions from year to year. However, this
approach resulted in SBA consistently overestimating defaults and
recoveries. In previous work, we found that SBA overestimated defaults
by about $2 billion from fiscal years 1992 to 2000.[Footnote 4]
In an effort to improve the accuracy of its subsidy estimate, SBA
implemented a new methodology based on econometric modeling to estimate
the subsidy cost for the fiscal year 2003 and 2004 budget submissions.
Econometric modeling has advantages over historical averaging. For
example, to the extent that data are available, it can take into
account the effects of changes in such factors as economic conditions,
program rules, and loan types on defaults and prepayments.
All forecasts are uncertain, and this uncertainty has multiple causes.
When relationships among economic variables are estimated, uncertainty
may arise from the choice of variables used in the model, from the
degree of precision with which the strength of the relationships is
estimated, and from uncertainty about the future values of the
independent variables used in the forecasting equation. Excluding a
variable that should be in a forecasting model can reduce the quality
of the model. For example, if some industries have high default rates,
then excluding industry variables will tend to underestimate default
costs in years when many loans go to high risk industries and overstate
default costs in years when many loans go to low risk industries. The
choice of variables to be used in a model results from a process of
professional judgment and balancing the risks of including too many or
too few variables. Economic theory and statistical tests play an
important role in these decisions. The remaining sources of
uncertainty, the precision of the estimated relationships and
uncertainty about future values of independent variables, are often
beyond the control of those building the model. The precision of the
effects of the independent variables is determined largely by the
amount of data available to the analyst, and uncertainty about future
values of independent variables is inherent in any forecast.
Internal control is a major part of managing an organization and this
includes controls over data gathering and processing, such as SBA's
data on 7(a) loans. As mandated by the Federal Managers' Financial
Integrity Act of 1982, the Comptroller General issues standards for
internal control in the federal government.[Footnote 5] These standards
provide the overall framework for establishing and maintaining internal
control and for identifying and addressing major performance and
management challenges and areas at greatest risk of fraud, waste,
abuse, and mismanagement. According to these standards, internal
control comprises the plans, methods, and procedures used to meet
missions, goals, and objectives. Control activities are the policies,
procedures, techniques, and mechanisms that enforce management's
directives and help ensure that actions are taken to address risks.
Control activities are an integral part of an entity's planning,
implementing, reviewing, and accounting for government resources and
achieving effective results. They include a wide range of diverse
activities including controls over information processing. These
controls are established to ensure that all data inputs are received,
are valid, and outputs are correct. Agency management should design and
implement internal control based on the related costs and benefits. No
matter how well designed and operated, internal control cannot provide
absolute assurance that all agency objectives will be met and, thus,
once in place, internal control provides reasonable, not absolute,
assurance of meeting an agency's objectives.
SBA's Equations Were Reasonable and Estimated Default, and Recovery
Rates Were in Line with Historical Experience:
We found that the econometric equations that SBA used to estimate
defaults, prepayments, and recoveries were reasonable, although other
equations could also be reasonable. SBA uses an appropriate statistical
technique for identifying the nature of these relationships. In
addition, SBA's equations produced estimated relationships for defaults
and prepayments that were consistent with expectations based on
economic reasoning. We found that there were additional variables
available to SBA that it did not include in its equations, such as
measures of interest rates and the borrower's industry type that would
also be reasonable and would produce different subsidy rates. In
addition, SBA did not include any economic variables in its equation
for estimating recoveries. According to documentation provided by SBA
to estimate recoveries on defaulted loans, adding economic variables
would not have increased the precision of the recovery rate estimates.
Finally, we found that the new model's estimated default and recovery
rates were in line with recent historical experience.
Variables in SBA's Default and Prepayment Equations Were Appropriate:
The econometric equations that SBA used at the time of our review
related the likelihood that a borrower would either default on or
prepay a loan to several variables that economic reasoning and prior
research suggested were appropriate to include in these types of
equations. These variables included: (1) characteristics of the
borrower's business, such as whether it was a sole proprietorship,
partnership, or corporation; (2) characteristics of the loan, such as
the amount borrowed; and (3) two measures of economic conditions, the
unemployment rate in the state where the loan was made and the GDP
growth rate. Economic reasoning and prior research suggested that
differences in borrower and loan characteristics and economic
conditions were likely to influence defaults and prepayments. For
example, prior research suggested that new businesses were less likely
to survive than were established businesses and thus were more likely
to default.[Footnote 6] Prior research also suggested that the
likelihood of default on loans made to partnerships or corporations
should be less than it was for loans made to sole proprietors, while
the likelihood of prepayment should be greater. Details about SBA's
econometric equations are found in appendix II.
SBA's Statistical Technique and Estimated Relationships for Prepayments
and Defaults Were Appropriate:
At the time of our review, SBA used an appropriate technique known as
multinomial logistic regression[Footnote 7] to identify whether the
variables included in its model were important influences on the
likelihood that a borrower would either default on or prepay a loan and
to estimate the magnitude of these relationships. This technique, which
has been used in other models of this type, was appropriate because it
corresponded to the decision-making process that borrowers faced. When
deciding whether to default on the loan, prepay the loan, or keep it
active, using this technique, SBA produced estimates of both the
probability of default and the probability of prepayment.[Footnote 8]
The relationships that SBA's equations estimated between different
variables and the likelihood of defaults and prepayments were
consistent with economic reasoning. For example, SBA's default equation
suggested that defaults were more likely when unemployment was higher,
and the rate of increase in gross domestic product was lower. Both of
these estimated relationships were consistent with economic reasoning
because it was less likely borrowers would continue paying their debts
when more people are out of work, and the economy was growing less
rapidly or in decline.
SBA's prepayment equation also suggested that prepayments were more
likely when loans were made under the SBA Express Program, for which
SBA guaranteed a smaller percentage of the loan amount than it did
under the regular 7(a) business loan program. This result was
consistent with our expectations because the smaller guarantee was
likely to make lenders more cautious in making lending decisions, such
that firms borrowing through this program may have been more
creditworthy than firms borrowing through the regular program. In turn,
the businesses' enhanced creditworthiness may have led to more
prepayments because these businesses may have been relatively more
financially stable and may have been more likely to pay off their loans
early. The details of SBA's default and prepayment equations, which
show these relationships, are in appendix II.
Other Default and Prepayment Equations Would Also Be Reasonable and
Lead To Different Subsidy Rate Estimates:
We identified additional variables available to SBA, but not included
in the model, that also influenced the likelihood of defaults and
prepayments. The choice of variables included in a model reflects the
modelers' professional judgment and different equations using different
sets of variables can all be considered reasonable. To analyze the
effect of adding additional variables, we tested SBA's model to
estimate the 2003 subsidy cost using additional variables that (1)
measured the current interest rate on 1-year U.S. Treasury bills and
(2) considered the industry in which the borrowing firm operates. The
interest rate could be important as either another measure of general
business conditions or as a specific measure of the cost of capital.
The industry in which the borrowing firm operates could be important if
default and/or prepayment rates vary among industries, and the
distribution of loans among industries varies over time. In addition,
banks have traditionally recognized that the financial performance of a
borrower depends on the nature of the business supporting the loan, the
structure of the loan, and the financial condition of the firm. At the
time of our review, SBA's econometric equations contain information on
the loan and the firm but did not include information on the firm's
business.
The estimates produced by our testing suggest that these variables also
influenced the likelihood of defaults and prepayments occurring and,
therefore, that equations using these variables could also be
reasonable.[Footnote 9] However, there are additional considerations
that could be important in deciding whether to include a measure of
interest rates in the default and prepayment equations. Specifically,
including an interest rate variable would mean that forecasted interest
rates would be used with the results of the econometric equations (and
forecast values of other economic variables) to forecast future
defaults and prepayments. The fact that forecasting interest rates is
difficult may be a reason for not including an interest rate variable,
even if the variable appears to be significantly related to the
historical likelihood of default or prepayment. Furthermore, at
present, forecasted interest rates are low relative to the interest
rates that prevailed over most of the period from which the data were
drawn to develop SBA's equations, potentially limiting the usefulness
of including an interest rate variable.
We found that including either the interest rate on 1-year Treasury
bills or the industry in which the borrowing firm operates as a
variable in the default and prepayment equations changed the estimated
cost of the program. (See app. II.) According to SBA's model, the
estimated subsidy rate for loans disbursed in 2003 was 1.04 percent.
This estimate increased to 1.13 percent with the industry identifiers
included and decreased to 0.76 percent with the inclusion of the
interest rate on 1-year Treasury bills. In addition, when we included
both the interest rate variable and the industry identifiers, we
estimated a subsidy rate of 0.83 percent. Because interest rates are
difficult to predict and have recently been quite low, we conducted
tests to determine how sensitive the estimate was to small changes in
forecasted interest rates. We found that it is not very sensitive to
such changes. For example, when we increased the forecasted values
above those included in the official OMB forecast by 10 percent, we
estimated a subsidy rate of 0.80 percent while when we decreased the
forecasted values by 10 percent we estimated a subsidy rate of 0.73
percent.
The range of estimated subsidy rates that result from including
additional variables was roughly comparable to the range that resulted
from using different economic assumptions. We tested the sensitivity of
SBA's estimated subsidy rate to small changes in the forecast values of
the GDP growth rate and the unemployment rate by reestimating the
subsidy rate with SBA's model but used both more optimistic and more
pessimistic assumptions about future economic conditions.[Footnote 10]
With the more optimistic assumptions, we estimated the subsidy rate
decreased to 0.81 percent while with the more pessimistic assumptions
we estimated that it increased to 1.28 percent.
Estimates of Recoveries Depended Only on Age of Loan, Not Economic
Conditions:
SBA's model also included a separate econometric equation for
estimating recoveries, which are the amounts of defaulted loans that
were eventually recouped by collection efforts, such as the liquidation
of assets. In this equation, the cumulative net recovery rate[Footnote
11] for a cohort of loans was estimated as a function only of the age
of the loans in that cohort. In particular, this equation did not
include any economic variables, so forecasted recovery rates were
estimated to resemble historical recovery rates even though economic
conditions in the future might be quite different from the past.
According to documentation provided by SBA of the work done to develop
this equation, adding economic variables would not have increased the
precision of the recovery rate estimates.[Footnote 12]
The Model's Estimated Default and Recovery Rates Were in Line with
Historical Experience:
Our evaluation of the model's estimated default and recovery rates
found that these rates were in line with historical experience of the
7(a) program. There are some limitations to evaluating expected future
loan performance compared with historical data because over time the
economy changes and underwriting criteria and other factors that affect
loan performance may also change. Therefore, one would not expect the
estimated loan performance to exactly mirror historical experience.
However, these types of comparisons are useful to evaluate the model's
estimated default and recovery cash flows. Because recently issued
loans do not have significant experience and historical data can be
summarized in several ways, we evaluated the new model's estimated
default and recovery rates compared with historical data in two ways to
determine whether the estimates were in line with historical
experience.
In August 2001, we reported that from fiscal year 1992 through fiscal
year 2000, SBA overestimated the cost of the 7(a) program by about $1
billion, primarily because it overestimated defaults by approximately
$2 billion. Over this same period, SBA's estimated recoveries closely
matched actual loan performance. SBA's prior method to estimate costs
was based on averages of historical loan performance. As previously
discussed, SBA's current model estimated defaults significantly
differently than the prior method in that it considered economic
variables and loan specific information. Meanwhile, at the time of our
review, the model continued to estimate recoveries based on historical
patterns.
While it was currently not possible to determine the accuracy of the
model's estimated default rate, as shown in the following two figures,
the rate appeared to more closely match recent historical experience
than SBA's previous method. Figure 2 shows how the model's estimated
default rate compared with the estimated default rates calculated with
SBA's previous method and with the average default experience of loans
issued between 1992 and 2001.[Footnote 13] We could have included more
or fewer years of loans in our analysis, but we believe data since 1992
are sufficient to evaluate the model's estimated default rate compared
with historical experience because it included several years of loans
that have been through their peak default period, which for 7(a) loans
is generally between years 2 and 5.
Figure 2: Estimated Default Rates Compared with Average Default
Experience from 1992 through 2001:
[See PDF for image]
[End of figure]
As previously mentioned, since historical data may be summarized
differently, figure 3 shows how the new model's estimated default rate
compared with the estimated default rate calculated with SBA's previous
method and to actual default experience during fiscal year 2001 for the
loans issued since 1986.[Footnote 14] This comparison allowed us to
evaluate the
estimated default rate over a longer period of time since data from
older loans that have been outstanding for a longer period of time was
included.[Footnote 15]
Figure 3: Estimated Default Rates Compared with Fiscal Year 2001 Actual
Default Experience:
[See PDF for image]
[End of figure]
SBA's Model Could Be Enhanced by Adding Information on Borrowers,
Correcting Errors, and Updating Some Data:
SBA could enhance the reliability of its model's estimates by adding
information on both the businesses and the owners to the econometric
equations and reestimating the equations and by correcting errors in
the model. The econometric equations SBA used at the time of our review
to predict default and prepayments included some variables describing
the businesses and loans and two economic indicators, GDP and
unemployment rates. But they did not include some variables other
analysts and financial institutions often use that are associated with
businesses and business owners, such as credit scores. In addition,
during our review, we found some errors that resulted in
underestimating the cost of the 7(a) program that was included in the
fiscal year 2004 President's Budget. Correcting these errors would have
increased the estimated cost of the program by about $6.5 million.
Including Additional Information on Businesses and Business Owners
Could Enhance the Model's Reliability:
The quantitative relationships between the default and prepayment rates
and the current independent variables would probably change if new
information were included. In our review of the literature and
discussions with large banks, additional information was mentioned as
having an influence on defaults and prepayments. The information cited
was more detail on the loans, the business, and on business owners,
including credit scores.[Footnote 16]
Our review of the academic literature and discussions with some
commercial lenders indicated that private lenders often include
variables SBA did not consider in forecasting the financial performance
of small businesses.[Footnote 17] At the time of our review, the
current SBA model included loan variables (age and term) and some
business variables (new business indicators, form of ownership, and
loan amount, among others) but was missing detailed information on
businesses that can help predict financial viability. These variables
include earnings, capital, payment records, and available collateral,
all of which have been shown to affect creditworthiness and likelihood
of default. Profit levels, for example, help predict a business's
ability to generate cash internally to cover loan payments. Records of
debt payments help determine whether a business can cover its
obligations, while available collateral tells a lender whether a
business has the resources to cover outstanding debts during a
financial crisis. Adding and periodically updating this information
could enhance the predictive ability of SBA's econometric model by
providing more accurate estimates of potential defaults and
prepayments.
In addition, analysts and banks have found that variables describing
business owners can aid in evaluating credit risk, and many large banks
have started to underwrite and monitor small businesses using credit
scores. Information from business owners' credit records, such as
income, personal debt, employment tenure, homeownership status, and
previous personal defaults or delinquencies, can help predict
delinquencies and defaults in the businesses themselves. Although at
the time of our review SBA's current model did not include variables
that measure these characteristics, the agency was developing a new
loan monitoring system that SBA officials told us was intended to track
this type of information. This is an important issue since, if banks
use credit scores and the SBA does not, the SBA may be left with
riskier loans. SBA could then determine whether such variables also
reflect risks in SBA loans and could be used to help evaluate the costs
of SBA loan guarantees.
SBA's 2004 Subsidy Rate Estimate Included Errors:
During our review of the model used to generate the cost estimate of
the 7(a) subsidy that was included in the fiscal year 2004 budget, we
found errors that resulted in underestimates of program costs of about
$6.5 million. Based on the estimated subsidy rate and the projected
loan volume included in the fiscal year 2004 President's Budget, the
estimated cost of the program was about $94.9 million. If the errors we
found had been detected and corrected by SBA before the budget was
submitted, the estimated cost of the program with the same projected
loan volume would have increased to about $101.4 million.
These errors related to SBA's method of estimating recoveries, annual
guarantee fee cash flows, and projections of borrower interest rates.
First, the recovery estimates were based on the assumption that loans
would be issued during fiscal year 2003 instead of during fiscal year
2004, although default and prepayment estimates were based on the later
year. As a result, the model estimated that recovery cash flows would
occur 1 year early, affecting the net present value[Footnote 18] of the
cash flows and the subsidy rate. Second, formulas SBA used to summarize
the output of the cash flow segment of the model indicated that the
same annual guarantee fees collected during the first quarter of fiscal
year 2004 would be collected from about years 5-27, even though the
fees would decline as loan balances were paid off. SBA officials
indicated that these two errors would be corrected before the
submission of the 2005 budget. Third, in estimating the cost of loans
issued in the future, SBA assumed the loans would have characteristics
similar to those of loans issued during fiscal year 2001. However, SBA
did not adjust the borrower interest rates to levels that would be more
appropriate for loans to be issued during fiscal year 2004. SBA
officials indicated that this adjustment was not necessary because it
would not significantly affect the cost of the program. However, SBA
had made this adjustment when it calculated the subsidy cost for loans
to be issued during fiscal year 2003. When we corrected the previously
described errors, the estimated cost of the program for fiscal year
2004 increased by $6.5 million. We also found an error related to
estimating prepayment penalties. SBA officials stated that they were
aware of this error but believed that fixing it would be complicated
and that these cash flows would be immaterial to the cost of the
program. In the officials' view, fixing the error would not be cost
beneficial.
Cohort Data Could Be Updated:
In addition, the model could also be further enhanced if SBA were to
update the model to include new information as it becomes available.
For example, SBA used the 2001 cohort of loans to generate estimates of
the 2003 and 2004 subsidy. But, they were not sure if they were going
to use the 2002 cohort of loans for the 2005 estimate because they said
that updating the cohort is complicated as a result of changes in
program policies or in the composition of the 7(a) loan portfolio.
However, the model would likely produce more reliable estimates if the
most recent loan data were being used to generate the forecast rather
than continuing to use an older cohort of loans.
SBA Collaborated with OFHEO and OMB to Develop the Model:
SBA contracted with OFHEO economists, with expertise in econometric
modeling of mortgage defaults and prepayments, to develop its subsidy
model, which included determining the variables to be included in the
econometric equations. SBA consulted with OMB officials, who are
required by FCRA to approve agency subsidy estimates. SBA also hired a
private consulting firm to conduct a limited review of the model as
part of its ongoing review process to minimize errors in estimating the
subsidy.
SBA Entered into an Agreement with OFHEO to Develop the Subsidy Model:
In February 2002, SBA entered into an agreement with OFHEO to assist in
developing the subsidy model. According to SBA staff, they selected
OFHEO because it had staff with expertise and experience in econometric
modeling and was less expensive than a private contractor.[Footnote 19]
According to SBA staff, the OFHEO economists followed a four-step
process to develop the model. The first step was refining and building
the data set that would be used to generate the estimates. The data set
OFHEO used was constructed from the SBA databases that were used to
track loan payment history and personal financial information on
borrowers. The second step was the design and estimation of the
default, prepayment, and recovery equations, including the selection of
variables for these equations. The third step of the process was the
construction of the cash flow module, and, the fourth step was the
construction and testing of the model that OFHEO would deliver for use
by SBA.
OMB Officials Approved SBA's Model:
OMB officials also played a key role in the development of the model
because, under FCRA, OMB has final responsibility for approving
estimation methodologies and determining subsidy estimates. SBA
officials said they consulted with OMB during the model's development
until OMB approved it in the fall of 2002. OMB officials told us that
they considered the model to be an improvement over the previous method
that SBA used to calculate the program subsidy rate because it used
better data and the econometric equations allowed for more accurate
estimates of future cash flows. In addition, SBA could now use the
model to consider both programmatic and economic variables in
estimating the subsidy rate. For example, they said SBA could model how
such variables as lender type affected the subsidy rate.[Footnote 20]
In reviewing the model, OMB officials told us that they focused on the
methodology of the model, the cash flow projections, appropriate use of
variables in the econometric equations, and the validity of the data
used to make the calculations. They approved the model in November
2002.
SBA Hired a Private Consulting Firm to Review the Model:
SBA hired a private consulting firm to conduct an independent limited
review of the model for September 2002 to October 2002, as part of its
ongoing process to identify errors before OMB approved the model. The
consulting firm assessed the model conceptually and evaluated its
underlying computer programming--specifically, the key data inputs that
were the primary source of the model's cash flows and the model's
programming specifications (to ensure they were correctly coded and
that the code functioned properly). The firm also assessed the model's
compliance with the relevant statutes and regulations and conducted
scenario testing to evaluate how it performed under different economic
assumptions. The consulting firm concluded that although the model
performed reasonably well in estimating the subsidy cost, SBA had made
errors in estimating loan guaranty and servicing fees, the calculation
of recoveries, and prepayment penalties. SBA made changes to the model
to address the identified discrepancies for fees and recoveries, the
net effect of which was, to increase the subsidy rate estimate by about
36 percentage points. The consulting firm also determined that the
model lacked adequate documentation and they were, therefore, unable to
review the econometric component of the model. However, OFHEO
subsequently provided SBA with a report documenting the model's
development to a limited extent.
Lack of Adequate Model Documentation Hampered Independent Reviews of
SBA's Model:
In developing its new econometric model, SBA did not prepare adequate
supporting documentation to enable independent reviewers to understand
and evaluate the process that was used. For example, the independent
contractor SBA hired to review the 7(a) credit subsidy model was
hampered by the lack of adequate documentation and, as a result, this
team's review of the model's theoretical basis and its working features
was severely limited. While SBA later developed some general
documentation of its model development process, this documentation did
not contain, among other things, an adequate discussion of alternative
variables, or combinations of variables, that it considered, tested,
and rejected, and the reasons for rejecting them. SBA officials told us
that they did not prepare this type of documentation because they
believed that there was no specific requirement to do so. Current
guidance is either silent or unclear about supporting documentation
needed to explain the development of econometric models used to
generate credit subsidy estimates for the budget and financial
statements. Nevertheless, we believe that maintaining adequate
documentation on how such models were developed is a sound internal
control practice that would provide SBA and other agencies the
opportunity to demonstrate and explain the rationale and basis for key
aspects of their models that provide important cost information for
budgets, financial statements, and congressional decision makers.
Moreover, as a practical matter, this documentation would help
facilitate SBA's and other agencies' annual financial statement audits.
SBA's 7(a) Credit Subsidy Model Documentation Was Inadequate for
Outside Reviewers:
BearingPoint, the independent contractor hired to perform an initial
review of the SBA 7(a) credit subsidy model prior to its finalization,
was hampered by the lack of adequate documentation. In response to our
inquiry, the contractor stated that the team did not validate the model
which, from an audit perspective, would have encompassed a more robust
effort. In its final report to SBA, the contractor reported that SBA
lacked sufficient supporting documentation for a "thorough review of
its [the model's] theoretical basis (including alternative modeling
methodologies explored), its working features, or the update and
maintenance procedures necessary to use the model on an ongoing basis.
This lack of adequate documentation severely limited our ability to
assess certain critical parts of the model in detail, including its
econometric components." Further, the contractor recommended that "SBA
develop a robust set of documentation to support this model" including
"the modeling methodology, alternate methodologies considered, data
inputs and outputs, and model maintenance and update requirements.":
In its January 30, 2004, audit report, Cotton and Company, the
independent public accounting firm, identified in its internal control
report 9 specific
deficiencies in the model's documentation.[Footnote 21] These
deficiencies included, for example, a lack of technical references for
the statistical method used for the performance of the model, the
absence of mathematical specifications, the fact that important
variables were not clearly identified, and that units of measure for
key variables were not specified. In addition, the audit report stated
that the documentation that was provided was "self-contradictory" about
the quality of the default and prepayment model and lacked a discussion
of the assumptions and limitations of SBA's modeling approach. In
responding to the independent public accountant's internal control
report, SBA's Chief Financial Officer generally agreed with the
report's findings, including the deficiencies in SBA's model
documentation, and stated that the internal control report presented
"fundamentals of good financial management and SBA is committed to
accomplishing as many of these items as possible in the coming year.":
In response to BearingPoint's recommendation, SBA's OFHEO contractor
prepared some documentation for the model, but this documentation was
not sufficient to allow us and SBA's financial statement auditor to
gain an adequate understanding of certain key parts of the model
development process. For example, the documentation that SBA provided
included a broad overview of how the model works, a list of the
variables that the final econometric equations included, the estimated
coefficients of the equations, and figures showing how well the
equations fit the data during the historical period. For some
variables, SBA's documentation indicated how the variables were
expected to influence default or prepayment probabilities, but did not
provide any reasons, conceptual justification, or supporting empirical
analysis. Some of these statements seemed intuitive, such as when the
output of the economy increases, as measured by the percent change in
real GDP, it is expected that default rates will drop. However, other
statements were not intuitive. For example, SBA's documentation
indicated that larger loans were expected to default at elevated levels
and did not include any support for this assertion.
Additionally, the model documentation did not explain in sufficient
detail why SBA excluded some variables. Rather, the model documentation
included a table of 29 variables that were tested and rejected and
stated that the information presented was "a list of most variables
tested." The documentation also provided a general overview about why
these 29 variables were excluded. SBA's documentation stated that
"variables were removed for a variety of reasons. Some of the reasons
include--insignificant, highly correlated with other variables, low
economic importance (significant but impact on probabilities was
negligible), inconsistent results (variable was not robust to different
specifications), and incoherent results (results could not be
reconciled with any economic logic)." While the documentation that SBA
provided to us contained acceptable reasons that economists could cite
in rejecting variables, the documentation's lack of specificity did not
allow us to determine which variables were rejected for which reasons.
Further, we were unable to determine whether these were the only
criteria or whether they were consistently applied throughout the model
development process.
SBA and the OFHEO contractor told us that, during the model development
process, approximately 800 pages of raw testing information were
generated and retained in an electronic file. They further stated that
these 800 pages were not organized in any fashion and that there was no
summary document or road map with greater detail than the model
documentation provided us that would describe the variable-testing
process or the results of that process in an understandable fashion. In
addition, SBA and the contractor told us that the variables reflected
in the 800 pages were not recorded in English words, but rather in
mnemonics, and that there was no crosswalk or key still in existence to
decode the mnemonics. Based on these representations by SBA and its
contractor, we initially concluded that this information would be of
questionable or no usefulness in assessing SBA's development of the
assumptions and selection of variables used in the modeling process.
SBA eventually provided us access to the 800 pages of material that
contained some information on variables that were considered and
rejected. This document was a partial compilation of analyses conducted
during the model development process with no explanation or discussion
of what was learned from each analysis conducted. Thus, on its own,
this document provided little additional information regarding the
process that SBA's contractor followed in developing the econometric
equations used in the subsidy model. Further, the document was written
in mnemonics and was not organized in any logical manner. In addition,
SBA officials could not identify any specific parts of this
documentation that related to alternative variables that were
considered and rejected during the model development process.
Documenting the basis for selecting and rejecting variables from an
econometric model used to develop credit subsidy estimates is an
important internal control that would also help to provide financial
statement auditors reasonable assurance that a bias was not introduced
into the credit subsidy estimates by systematically excluding variables
to influence the subsidy rate in a particular direction. Statement on
Auditing Standards Number 57, Auditing Accounting Estimates (SAS No.
57), states that "even when management's estimation process involves
competent personnel using relevant and reliable data, there is
potential for bias in the subjective factors." When evaluating the
reasonableness of an estimate, the auditor should concentrate on, among
other things, "key factors and assumptions that are subjective and
susceptible to misstatement and bias." Because of the nature of
econometric models and the effect that variables used have on future
loan default and prepayment projections, auditors need to understand
both what was included and excluded from the model to assess the
reasonableness of the credit subsidy estimate from a financial
accounting perspective.
As our work demonstrated, changing the variables that were included in
the model changed the subsidy rate. Because of the lack of adequate
documentation on SBA's 7(a) model development process, we were unable
to determine whether a bias in selecting variables existed in the
model. Further, SBA's lack of adequate documentation on the 7(a) model
development process could have impeded our ability to reach a
conclusion on SBA's loan accounts in connection with the audit of the
consolidated financial statements of the federal government.
Specific Guidance on Credit Subsidy Model Development Documentation Is
Limited:
Currently, there is limited specific guidance on the nature and extent
of documentation that agencies must prepare related to the development
of models to generate credit subsidy estimates. OMB Circular A-11,
Preparation, Submission, and Execution of the Budget, provides guidance
on how agencies should prepare credit subsidy estimates. Circular A-11
does not include any guidance to the agencies for documenting their
model development process including selection and rejection of
variables for use in the models that generate federal credit subsidy
estimates. However, Federal Financial Accounting and Auditing Technical
Release 6, Preparing Estimates for Direct Loan and Loan Guarantee
Subsidies under the Federal Credit Reform Act Amendments to Technical
Release 3: Preparing and Auditing Direct Loan and Loan Guarantee
Subsidies under the Federal:
Credit Reform Act,[Footnote 22] provides some implementation guidance
about the nature and extent of documentation agencies should have for
their models. Technical Release 6 states that agencies should document
the cash flow model(s) used and the rationale for selecting the
specific methodologies. Agencies should also document the sources of
information, the logic flow, and the mechanics of the model(s)
including the formulas and other mathematical functions. In addition,
because the model is the basis for budget and financial statement
credit subsidy estimates, this documentation also facilitates an OMB
budget analyst's review, if the analyst is not involved in the
development process, the external financial statement audit, and other
independent reviews. Technical Release 6 also states that agency
documentation for subsidy estimates and reestimates should be complete
and stand on its own, enabling an independent person to perform the
same steps and replicate the same results with little or no outside
explanation or assistance. In addition, if the documentation were from
a source that would normally be destroyed, then copies should be
maintained in the file for the purposes of reconstructing the estimate.
Technical Release 6 does not specifically address expected
documentation of an agency's model development process, including a
detailed discussion of alternative variables that are considered, the
reasons for their rejection, and specific examples based on results of
earlier regressions. Nevertheless, in our view, the documentation
principles in this Technical Release represent sound internal control
practice that could also be applied to an agency's development of a
model used to generate budget and financial statement credit subsidy
estimates. Such documentation would introduce transparency into an
agency's budget process and enable agencies' models and the resulting
estimates to withstand scrutiny and inquiry from independent reviewers.
For example, such documentation would allow validation of an agency's
model by independent reviewers, and provide reasonable assurance that
the agency selected and rejected assumptions and variables for the
model on a sound basis. Further, this documentation would help
demonstrate to congressional stakeholders sound decision making and
stewardship over millions of dollars in appropriated funds.
SBA Had a Process to Help Ensure Data Quality and the Data Used in the
Model and SBA's Loan Level Databases Were Consistent:
Calculating a reliable credit subsidy estimate requires that the key
cash flow data, such as defaults or recoveries and the timing of these
events be reliable, or the credit subsidy estimate could be affected.
Internal control standards call for agencies to have a process to help
ensure the completeness, accuracy, and validity of all transactions
processed. SBA's monthly reconciliation process, combined with lender
incentives and loan sales, helped ensure the quality of the underlying
data used in its credit subsidy estimation process. Although at the
time of our review, some errors in its data existed in SBA's databases,
the nature and magnitude of these errors was unlikely to significantly
alter the subsidy rate. Further, we tested the data used by SBA's new
econometric model and found them to be consistent with the data in
SBA's loan systems at the time of our review.
SBA Had a Process to Identify and Correct Data Errors:
The primary method that SBA used to help ensure the integrity of its
loan data is its Form 1502 reconciliation process. Reconciliations are
an important internal control established to ensure that all data
inputs are received and are valid and all outputs from a particular
system are correct. This process, which has been in effect since
October 1997, utilized an SBA contractor to conduct monthly matches of
borrower data submitted by 7(a) program lenders on SBA's Form 1502 to
the information in the agency's Portfolio Management Query Display
System to help ensure the completeness and accuracy of the agency's
data. The information on the Form 1502 included a wide variety of data
for an individual loan, some of which was used in the credit subsidy
estimation process, and included, among other things, loan
identification number; loan status such as current, past due, or in
liquidation; loan interest rate; the portion of the loan guaranteed by
SBA; and the ending balance of the loan's guaranteed portion. Errors
identified by this match were loaded each month into SBA's Portfolio
Management Guaranty Information System, and it was accessed by the
various district office staff to work with lenders to correct the
erroneous data.
Although we did not independently test the data match conducted by
SBA's contractor or the field office staff's correction of identified
errors, we reviewed summary reports of the errors in the Guaranty Loan
Reporting System for each district office over a 4 month period during
fiscal year 2003 and found that most of these reported errors were
resolved during the month the errors were identified. During the months
we reviewed, the percentage of errors resolved ranged from a low of
about 65 percent to a
high of nearly 89 percent.[Footnote 23] Although one month we reviewed
had only a 65 percent resolution rate, leaving 4,860 errors uncorrected
at the end of the month, as explained in the following paragraph, not
all of these errors would affect the subsidy estimate and this number
is relatively small compared to the large volume of loan transaction
level data used in the credit subsidy estimation process. Our review of
the underlying data used in the model showed that about 5.7 million
data records were used to record the quarterly loan performance of
392,315 loans from 1988-2001.
In order to assess whether the remaining errors in SBA's data base
would likely have a significant affect on the credit subsidy estimation
process, we reviewed the 38 different error codes that are reported
monthly by the Guaranty Loan Reporting System and found that less than
half of these error codes were related to data used by the econometric
model and, as a result, could have affected the credit subsidy
estimate. For example, the Guaranty Loan Reporting System identified
errors for lender contact name and phone number--data that were not
used by the new econometric model and would not affect the subsidy
estimate. Other error codes relating to the guaranteed portion
principal balance or whether a loan was in liquidation status could
affect the credit subsidy estimate if the number of errors and their
dollar volume were significant.
We reviewed a 6-month summary error report from the Guaranty Loan
Reporting System for activity between February and July 2003 and found
that, for those error codes that could affect the credit subsidy
estimate, only two of these codes had error rates that exceeded 1
percent of the transactions. One of these codes indicated that the loan
status was not correct because the loan was in liquidation and had an
average error rate of about 1.4 percent for the 6-month period we
reviewed. The other error code indicated that the bank did not report
any information for a particular loan and had an average error rate of
about 2.4 percent for the same time period. The remaining 11 error
codes that could have affected the credit subsidy estimate had rates of
less than 1 percent. We assessed the error rates on this report in
aggregate to determine if these could affect the credit subsidy
estimate and found that the average aggregate error rate was about 6.5
percent during this period. However, given that most of these errors
were corrected in the month the error was identified, it was unlikely
that the remaining uncorrected errors would affect the credit subsidy
estimate at the time of our review.
Lender Incentives and Loan Sales Help Ensure Data Integrity:
In addition to the monthly loan data reconciliation process, lender
incentives also helped ensure the integrity of the underlying data used
in the credit subsidy estimates. In accordance with current SBA policy,
the agency can reduce or completely deny a lender's claim payment if
the defaulted loan data are not correct. According to SBA officials,
this policy gives the 7(a) program lenders an incentive to correct data
errors because it helps ensure they will be paid the full guarantee
amount if the borrower subsequently defaults on the loan. SBA provided
us with repair and denial data for fiscal years 1999 through the first
three quarters of fiscal year 2003 showing that the agency exercised
these options 2,177 times during this time, totaling at least $69.9
million.[Footnote 24]
Further, an ancillary benefit of SBA's loan sales program was to help
ensure data integrity. Prior to a sale, SBA district office staff, as
well as contractors, reviewed loan files as part of the "due diligence"
reviews to provide accurate information about the loans available for
sale to potential investors so that they may make informed bids. SBA
officials told us that prior to selling a loan, discrepancies between
the lenders' data and SBA had to be resolved.
Data Used by the Econometric Model Were Consistent with SBA Databases:
In order to assess the consistency between the data used in SBA's
econometric approach and the data in SBA's loan system, we selected and
tested a stratified random sample of 400 items to test key data that
could affect the credit subsidy estimate and found no errors.[Footnote
25] Specifically, we randomly selected 100 default and recovery
transactions and compared the amounts and transaction dates between the
loan system data and loan-level data used for the credit subsidy
estimate. In addition, we randomly selected 100 loans identified by the
model to be prepaid and reviewed the loan histories in SBA's database
and determined that all of these loans were paid off prior to their
scheduled termination date. Further, we tested 100 additional loans and
compared their status such as current, paid off, or default to ensure
their status in the model was correct and found no errors.
We also assessed the magnitude of 7(a) loans that were excluded from
the model in order to determine whether excluding these potentially
valid loans would likely affect the credit subsidy estimate. Our
earlier work on SBA's previous 7(a) credit subsidy model that primarily
used historical averages of defaults and recoveries found that
excluding loans from certain years that had higher default rates would
lower the overall average default rate. Excluding large numbers of
loans from this model would likely have a similar effect on the
estimated subsidy rate. To assess the magnitude of excluded loans, we
reviewed the computer coding for the econometric model and found that
SBA excluded loans when critical data for the model were missing such
as the initial disbursement date, the loan amount, or demographic
information on the borrowers. For most of the years between 1988 and
2001, the number of loans excluded because they lacked these essential
data ranged from 1 percent to 2 percent and overall, we concluded that
the degree of excluded loans was acceptable and would not significantly
affect the credit subsidy estimation calculation, at the time of our
review.
Conclusions:
Overall, we found that from an economics perspective, SBA's econometric
equations for its 7(a) credit subsidy model were reasonable. However,
from an audit perspective, SBA's lack of adequate documentation of the
model development process precluded us from (1) independently
evaluating the model's development; (2) determining whether SBA used a
sound and consistently applied method to select and reject variables to
be included in the model; and (3) determining whether a bias in
selecting variables existed in the model.
Based on our review, SBA's econometric equations for estimating
defaults, prepayments, and recoveries, which were used to derive the
estimate of its fiscal year 2004 subsidy costs, were reasonable. This
model's methodology has the potential to produce more reliable
estimates than the previous method of using historical averaging to
project the estimated program cash flows because this model relies on
economic reasoning in addition to historical program data. However, the
precision of any econometric model is limited because any estimate
produced by such a model should be considered one point in a range
within which the actual subsidy cost will likely fall. Because the
budget process requires agencies to select a specific estimate rather
than project a range, there will likely be some variance between the
forecasted and actual subsidy amounts. Using additional data that SBA
anticipates gathering in its new loan monitoring system, such as
borrower-specific data, could further enhance the reliability of SBA's
estimates of the subsidy cost. Therefore, further enhancements could
produce more reliable results.
Although the errors we identified in the model did not materially
affect the subsidy cost estimate, they did indicate that the process
SBA used to validate the model could be improved. Therefore, it is
important to invest the resources needed to periodically reevaluate the
underlying assumptions of any model to ensure that they are correct and
comprehensive, and that any errors or erroneous assumptions are
corrected so that the model continues to yield reasonable results.
While we found SBA's equations to be reasonable from an economics
perspective, the lack of adequate documentation of the model's
development process hampered three independent reviews of the 7(a)
model. Notwithstanding the current lack of clear OMB Circular A-11
guidance, SBA could benefit from applying the documentation principles
embodied in Technical Release 6 to the development of the 7(a)
econometric model and other credit subsidy estimation models it has
recently developed or is currently developing. Without adequate
documentation, SBA will be unable to transparently demonstrate the
rationale and basis for key aspects of models that provide important
cost information for budgets, financial statements, and congressional
decision makers. Although OMB provides guidance on how agencies should
prepare credit subsidy estimates in Circular A-11, it does not include
any guidance to the agencies for documenting their model development
process including the selection and rejection of variables for use in
the models that generate federal credit subsidy estimates. A lack of
improved OMB guidance for model documentation will continue to hamper
adequate external oversight and validation of models used to generate
credit subsidy estimates.
Recommendations for Executive Action:
We are making three recommendations to SBA and one to OMB. To further
enhance the reliability of SBA's subsidy estimates, we recommend that
the SBA Administrator take the following two actions:
* determine how best to include in future subsidy models borrower-
specific information, such as credit scores and loan-to-value ratios,
to be collected in the new loan monitoring system; and:
* ensure that the model remains reasonable by establishing a process
for periodically evaluating the model to correct any errors and
revising it to reflect changes in the 7(a) business loan program or
other factors that could affect the subsidy estimate.
To demonstrate and explain the rationale and basis for the 7(a)
econometric model and all other models developed, we recommend that the
SBA Administrator take the following action:
* prepare and retain adequate documentation of the model development
process including a detailed discussion of the alternative variables or
combinations of variables that were considered, tested, and rejected,
as well as the reasons for rejecting them.
To facilitate (1) validation of models used to generate credit subsidy
estimates, (2) external oversight, and (3) financial statement audits,
we recommend that the Director, OMB, take the following action:
* revise OMB Circular A-11 to require that agencies document the
development of their credit subsidy models, including the process
followed for selecting modeling methodologies over alternatives, and
variables tested and rejected, along with the basis for excluding them.
Agency Comments and Our Evaluation:
We provided an initial draft and a revised draft, based on our review
of additional model documentation, to both SBA and OMB for review and
comment. While our initial draft was at the agencies for comment, we
continued to pursue additional documentation that SBA had to further
explain its 7(a) model development process, including what variables
were selected, rejected, and why. When we eventually obtained access to
the 800 pages of SBA material, we determined that it was not organized
and included no road map to describe the variable testing process or
its results. We concluded that this information was of questionable or
no usefulness to our assessment of SBA's modeling process. We addressed
the weaknesses in SBA's documentation in the revised draft report and
provided it to SBA and OMB for comment. In commenting on the initial
draft, SBA's Chief Financial Officer (CFO) generally agreed with our
findings and the first two recommendations related to actions to
further enhance the reliability of the model's subsidy estimates. OMB
did not provide any comments on the initial draft report. We received
comments on the revised draft from SBA's CFO who generally disagreed
with our findings and recommendations related to the lack of adequate
documentation supporting the model's development process. We also
received comments on the revised draft from the OMB Assistant Director
for Budget and the Controller who disagreed with our recommendation
that OMB revise Circular A-11. Their written comments are reprinted in
appendixes III and IV, respectively, and are summarized below. Both
agencies provided technical comments that we have incorporated into the
report as appropriate.
In commenting on our final draft report, SBA stated that it had
provided us with extensive documentation, briefings, and explanations
about how the model was developed. We met with SBA officials and their
contractor who constructed the model and discussed their methodology,
but we were unable to corroborate this information with the
documentation they subsequently provided. SBA's comment letter stated
that it provided us with 800 pages of material that contained some
information on variables that were considered and rejected. During our
subsequent review of this material, we found that this documentation
was a partial compilation of analyses conducted during the model
development process with no explanation or discussion of what was
learned from each analysis conducted. After reviewing all of this
documentation, as discussed in the report, we concluded that it
provided little additional information to enable us to understand and
corroborate the process and criteria that SBA used to select and reject
variables for its 7(a) model.
Our conclusions regarding the lack of adequate documentation for the
model's development process were consistent with those of both the
independent contractor SBA hired to review the model in 2002 prior to
its implementation and the independent public accounting firm that
audited SBA's fiscal year 2003 financial statements. As part of its
January 30, 2004, audit report, the independent public accounting firm
identified in its internal control report 9 specific deficiencies in
the model's documentation. These deficiencies included, for example, a
lack of technical references for the statistical method used for the
performance of the model, the absence of mathematical specifications,
that important variables were not clearly identified, and that units of
measure for key variables were not specified. In addition, the audit
report stated that the documentation that was provided was "self-
contradictory" about the quality of the default and prepayment model
and lacked a discussion of the assumptions and limitations of SBA's
modeling approach. While SBA's CFO agreed with the independent
accounting firm's findings regarding the lack of adequate documentation
for the credit subsidy model, he disagreed with similar weaknesses
identified in our report.
SBA disagreed that its lack of adequate documentation on the 7(a) model
development process could impede our ability to reach a conclusion
about SBA's loan accounts in connection with the audit of the
consolidated financial statements of the federal government. Instead,
SBA believed mandating additional documentation would establish a new
and unnecessary requirement. Our comment was in regard to our
responsibility as the auditor of the consolidated financial statements
of the federal government and does not establish a new or unnecessary
requirement for SBA. For the consolidated financial statement audit, we
evaluate the reasonableness of credit program estimates based on audit
guidance in SAS No. 57.[Footnote 26] In auditing estimates, SAS No. 57
states that an auditor should consider, among other things, the process
used by management to develop the estimate, including determining
whether or not (1) relevant factors were used, (2) reasonable
assumptions were developed, and (3) biases influenced the factors or
assumptions. SBA's lack of adequate documentation of the 7(a) model
development process impaired our ability to make such an assessment.
OMB disagreed with the recommendation that Circular A-11 should be
revised and believed that the report did not demonstrate that revisions
were needed. OMB officials commented that they worked closely with SBA
during the model development process and believed that the
documentation SBA provided to OMB was adequate for them to determine
that the subsidy estimates and reestimates were reasonable. OMB also
did not concur with our statement that a lack of improved OMB guidance
hampered adequate external oversight. Unlike OMB, in this case, we and
other external reviewers did not have the opportunity to work with SBA
during the model development process and, as a result, relied on oral
explanations and documentation provided by SBA staff and its contractor
who developed the model. Further, we attempted to corroborate SBA's
statements with the documentation that SBA provided. However, as we
reported, three independent external reviews of SBA's 7(a) model were
hampered by a lack of adequate documentation of SBA's model development
process. We reaffirm our conclusion that adequate documentation is
needed for the SBA 7(a) model's development and that independent
external review and oversight will continue to be hampered without a
requirement to provide adequate documentation about how econometric
models are developed.
OMB stated that Ernst and Young was able to independently validate
SBA's 7(a) model with the available documentation. According to OMB,
this firm stated that the 7(a) model assumptions and methodology
appeared to be reasonable and accurate. We obtained and reviewed the
reports OMB cited and found that the firm was not hired to validate or
review the same segments of the model that we reviewed. This series of
reports was related to the cash flow module of the 7(a) model, as well
as the model used to calculate reestimates, but did not review the
econometric equations or the model's development process. In its
report, the firm explicitly stated that it was not reviewing the same
parts of the model that we reviewed. We confirmed this information in
conversations with the accounting firm's engagement partner and
concluded that this firm's work was not relevant to the findings and
conclusions presented in our report.
OMB also commented that SAS No. 57 states that internal controls over
accounting estimates may or may not be documented. While SAS No. 57
does state that the process for preparing accounting estimates may not
be documented, it also states that auditors should assess whether there
are additional key factors or alternative assumptions that need to be
included in the estimate and assess the factors that management used in
developing the assumptions. Further, SAS No. 57 states that auditors
should concentrate on key factors and assumptions that are subjective
and susceptible to misstatement and bias. We believe this includes the
selection and rejection of variables that can be included in the model.
Without adequate documentation on the credit subsidy model development
process, it is difficult for auditors to fulfill their responsibilities
to assess these areas.
OMB also commented that SBA fulfilled the management responsibilities
described in SAS No. 57 regarding internal controls for accounting
estimates. We disagree with this statement and point out that SAS No.
57 provides guidance for auditing accounting estimates as part of
conducting financial statement audits rather than directing agency
management's actions. Management's responsibility for internal
controls are contained in our "Standards for Internal Control in the
Federal Government," which states, among other things, that "internal
control and all transactions and other significant events need to be
clearly documented, and the documentation should be readily available
for examination."[Footnote 27] Further, as previously stated, Cotton
and Company also identified the lack of adequate model documentation as
an internal control weakness. Moreover, SBA's CFO generally agreed with
the independent public accountant's report's findings, including the
deficiencies in SBA's model documentation, and stated that the internal
control report presented "fundamentals of good financial management and
SBA is committed to accomplishing as many of these items as possible in
the coming year.":
OMB also stated that requiring agencies to prepare additional
documentation of the variables tested and rejected would be unduly
burdensome. We disagree with this statement and note that this
documentation would only need to be prepared when a model is developed
or when significant updates are implemented. Further, this requirement
would be consistent with other segments of OMB Circular A-11 that
require agencies to provide supporting documentation for their budget
submissions. However, as we mentioned in the report, there is currently
no explicit guidance for agencies to document the development of the
models that are used to generate credit subsidy estimates.
OMB also commented that we received sufficient information to test
alternative variables to measure the reasonableness of the final SBA
credit subsidy model. We note that our work demonstrated that using
additional variables that were also reasonable changed the subsidy
estimate. We believe that this work highlights the need for agencies to
document their basis for rejecting variables or combinations of
variables from their final credit subsidy models. By documenting this
work, agencies will be able to demonstrate to independent reviewers
that a bias from variable selection does not exist in the final model.
Both agencies provided technical comments that we incorporated into the
report as appropriate. The written comments of both agencies are
reprinted in appendixes III and IV.
We are sending copies of this report to the Chair of the Senate
Committee on Small Business and Entrepreneurship, other appropriate
congressional committees, the Administrator of the Small Business
Administration, and the Director of the Office of Management and
Budget. We also will make copies available to others upon request. In
addition, the report will be available at no charge on the GAO Web site
at [Hyperlink, http://www.gao.gov].
If you have any questions about this report, please contact me at (202)
512-8678 or [Hyperlink, dagostinod@gao.gov] or Katie Harris, Assistant
Director, at (202) 512-8415 or [Hyperlink, harrism@gao.gov]. Key
contributors to this report are listed in appendix V.
Signed by:
Davi M. D'Agostino Director, Financial Markets and Community
Investment:
[End of section]
Appendixes:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
As agreed with your staff, we (1) assessed the reasonableness of the
model's econometric equations and evaluated the model's estimated
default, prepayment, and recovery rates based on the 7(a) program's
recent historical loan experience; (2) identified additional steps the
SBA could take to further enhance the reliability of its subsidy
estimate produced by the model; (3) reviewed SBA's process for
developing the subsidy model; (4) evaluated the model's supporting
documentation, including its discussion of what variables were tested
and rejected; and (5) determined what steps SBA has taken to ensure the
integrity of the data used in the model and determined whether these
data are consistent with information in its databases. We did not
validate SBA's model.
Assessing the Reasonableness of the Model's Econometric Equations and
Evaluating the Model's Estimated Default, Prepayment, and Recovery
Rates:
To analyze the model, we obtained from SBA copies of the model as
approved by the Office of Management and Budget (OMB), along with the
loan-level data that were used to develop the subsidy estimates. We
analyzed the econometric equations to determine whether they were
reasonable based on the variables they included, the statistical
techniques used, and the results obtained. For example, we determined
whether the econometric equations included appropriate variables and
whether the variables used in the equations were statistically
significant. To evaluate the model's estimated default and recovery
rates, we compared these rates with recent historical loan experience
of the 7(a) program provided by SBA. Using SBA's data, we also
calculated what SBA would have estimated for default and recovery rates
based on the estimation methodology it used prior to its fiscal year
2003 budget submission. (See app. II for a detailed discussion of our
analysis of the reasonableness of the model's econometric equations.):
Identifying Additional Steps SBA Could take to Further Enhance the
Reliability of the Model:
To identify additional steps SBA could take to enhance the reliability
of its model, we considered additional types of data that SBA might
collect and consider including in its econometric equations. As part of
this analysis, we reviewed the academic literature on default modeling
and interviewed officials with several banks engaged in similar
efforts.
Reviewing SBA's Process of Developing the Subsidy Model:
To determine SBA's process for developing the model, we met with SBA
officials in the Chief Financial Office who were responsible for
estimating the 7(a) program subsidy costs. We also met with OMB
officials who were responsible for approving the model. Finally, we
also reviewed available documentation on the model's development
provided by SBA and the report by the private consultant who reviewed
the model.
Evaluating the Model's Supporting Documentation, Including Its
Discussion of What Variables Were Tested and Rejected:
To evaluate the model's supporting documentation, including its
discussion of what variables were tested and rejected, we obtained and
analyzed available relevant documents and met with SBA officials and
their contractor who developed the model. We compared the information
presented in SBA's model documentation with existing credit subsidy
guidance including OMB Circular A-11 and Federal Financial Accounting
and Auditing Technical Release 6: Preparing Estimates for Direct Loan
and Loan Guarantee Subsidies under the Federal Credit Reform Act
Amendments to Technical Release 3: Preparing and Auditing Direct Loan
and Loan Guarantee Subsidies under the Federal Credit Reform Act. We
also assessed the impact the lack of documentation would have on SBA's
financial statement audit by comparing the documentation with Statement
on Auditing Standards Number 57, Auditing Accounting Estimates. SBA and
its contractor told us that 800 pages of raw testing information
contained in an electronic file was not organized in any fashion, and
that there was no summary document or road map that had greater detail
than the model documentation provided us that described the variable-
testing process or the results of that process in an understandable
fashion. In addition, SBA and the contractor told us that the variables
reflected in the 800 pages were not recorded in English words, but
rather in mnemonics, and that there was no crosswalk or key still in
existence to decode the mnemonics. Thus, no documentation existed that
would link the variable names used in the programming to a table of
variable descriptions. We obtained and reviewed a copy of this
documentation and confirmed the representations of SBA and its
contractor.
Determining What Steps SBA Took to Ensure the Integrity of the Data
Used in the Model and Whether These Data Were Consistent with
Information in Its Databases:
To determine what steps SBA took to ensure the integrity of the data
used by the model, we met with SBA officials to gain a general
understanding of the agency's data integrity efforts. We also assessed
the number of errors that were resolved by the district offices each
month by analyzing 4 months of fiscal year 2003 field office activity
from the Form 1502 Guaranty Loan Reporting System. We further assessed
whether the remaining errors at the end of the month would likely
affect the credit subsidy estimate by analyzing the types of errors
tracked by the system and determining which errors affected data used
by the new model. We also assessed the magnitude of these errors by
analyzing 6 months of fiscal year 2003 activity in the Guaranty Loan
Reporting System. To determine whether the data in the new model was
consistent with data in SBA's loan-level databases, we selected and
tested a stratified random sample of 400 key data elements that could
affect the credit subsidy estimate.[Footnote 28] Specifically, we
randomly selected 100 default and 100 recovery transactions and
compared the amounts and transaction dates between the loan system data
and loan-level data used for the credit subsidy estimate; 100 loans
identified by the model to be prepaid and reviewed the loan histories
in SBA's database to determine whether all of these loans were paid off
prior to their scheduled termination date; 100 additional loans and
compared their status such as current, paid off, or default to
determine if their status in the model agreed with SBA's loan-level
databases.
[End of section]
Appendix II: Analysis of Default, Prepayment, and Recoveries
Econometric Equations:
This appendix provides more detail on the three econometric equations
that the Small Business Administration (SBA) used to estimate the
subsidy rate for its 7(a) loan guarantee program and the expanded
equations that we developed. These equations are used to forecast
defaults, prepayments, and recoveries. The first section of this
appendix describes the variables that SBA used in the default and
prepayment equations and presents SBA's estimated coefficients. The
second section explains how we created the variable that we used to
represent the borrower's industry and presents the estimated
coefficients from our expanded default and prepayment equations. The
third section describes the equation that SBA used to forecast
recoveries and presents the estimated coefficients from that equation.
SBA's Default and Prepayment Equations:
In its new model for estimating the subsidy rate for the 7(a) loan
program, SBA uses multinomial logistic regression to estimate the
likelihood of defaults and prepayments as functions of a variety of
explanatory variables. Because multinomial regression is a simultaneous
estimation process, the default and prepayment equations are
identically specified (that is, the same explanatory variables are used
in each equation). SBA conducts its analysis at the level of the
individual loan, using loans that were disbursed from 1988 through
2001. For each loan, SBA's data set contains an observation for each
quarter that the loan is active. For example, if a loan prepays at the
end of the third year (counting the disbursement year as the first
year), then it is active during 12 quarters and, therefore, there are
12 observations for that loan in the data set.
For each observation, the dependent variable measures whether in that
quarter the borrower defaults on the loan, prepays the loan, or keeps
it active. As a result, the coefficients in the default or prepayment
equation are estimates of the association of each explanatory variable
with the likelihood of the loan defaulting or prepaying in that
quarter.
There are several categories of explanatory variables included in the
default and prepayment equations. The first group consists of a set of
dummy variables that indicate the age of the loan. These variables thus
serve to reflect the fact that prepayment and default behavior change
as a loan seasons. Specifically, there is a dummy variable for each of
the first ten quarters of the life of a loan. From the eleventh quarter
to the thirty-fourth quarter, there is a dummy variable for each two
consecutive quarters. Finally, if a loan remains active past an age of
thirty-four quarters, there is one more dummy variable.
The second set of explanatory variables concern loan characteristics. A
set of dummy variables indicates the contractual term of the loan at
origination. The categories are less than 5 years, 5 to up to 10 years,
10 years to up to 15 years, and 15 years or greater. Less than five
years serves as the omitted category in the regression. Loan amount is
another characteristic and is measured in millions of dollars. SBA also
includes a dummy variable that shows whether a loan was delivered
through the SBA Express Program. Also known as Subprogram 1027, this
program allows lenders to originate a loan using their own loan
documents instead of SBA documents and processing, but the loan
guarantee is only up to 50 percent. By comparison, the typical SBA
guarantee is almost 80 percent. Finally, there is a set of dummy
variables for type of lender: Regular, Preferred, and Certified. In the
regression, the regular type serves as the omitted category.
The next set of explanatory variables provides information on the
borrower. A set of dummy variables identifies ownership structure. The
categories are sole proprietorship, corporation, or partnership. Sole
proprietorship is the omitted category in the regression. An additional
dummy variable indicates whether the borrower is a new business.
Finally, there is a set of dummy variables that indicate the U.S.
Census Bureau region where the borrower is located.
The final set of explanatory variables contains two measures of
economic conditions. The first is the state unemployment rate where the
borrower is based. The source for these data is the U.S. Bureau of
Labor Statistics. The second is the quarterly percentage change in
gross domestic product. SBA obtained these data from the U.S. Bureau of
Economic Analysis.[Footnote 29] Table 1 summarizes the explanatory
variables.
Table 1: Variable Names and Descriptions:
Variable name; Age dummy variables: i1;
Variable description: 1 if loan is 1 quarter old, else 0.
Variable name; Age dummy variables: i2;
Variable description: 1 if loan is 2 quarters old, else 0.
Variable name; Age dummy variables: i3;
Variable description: 1 if loan is 3 quarters old, else 0.
Variable name; Age dummy variables: i4;
Variable description: 1 if loan is 4 quarters old, else 0.
Variable name; Age dummy variables: i5;
Variable description: 1 if loan is 5 quarters old, else 0.
Variable name; Age dummy variables: i6;
Variable description: 1 if loan is 6 quarters old, else 0.
Variable name; Age dummy variables: i7;
Variable description: 1 if loan is 7 quarters old, else 0.
Variable name; Age dummy variables: i8;
Variable description: 1 if loan is 8 quarters old, else 0.
Variable name; Age dummy variables: i9;
Variable description: 1 if loan is 9 quarters old, else 0.
Variable name; Age dummy variables: i10;
Variable description: 1 if loan is 10 quarters old, else 0.
Variable name; Age dummy variables: i1112;
Variable description: 1 if loan is 11 or 12 quarters old, else 0.
Variable name; Age dummy variables: i1314;
Variable description: 1 if loan is 13 or 14 quarters old, else 0.
Variable name; Age dummy variables: i1516;
Variable description: 1 if loan is 15 or 16 quarters old, else 0.
Variable name; Age dummy variables: i1718;
Variable description: 1 if loan is 17 or 18 quarters old, else 0.
Variable name; Age dummy variables: i1920;
Variable description: 1 if loan is 19 or 20 quarters old, else 0.
Variable name; Age dummy variables: i2122;
Variable description: 1 if loan is 21 or 22 quarters old, else 0.
Variable name; Age dummy variables: i2324;
Variable description: 1 if loan is 23 or 24 quarters old, else 0.
Variable name; Age dummy variables: i2526;
Variable description: 1 if loan is 25 or 26 quarters old, else 0.
Variable name; Age dummy variables: i2728;
Variable description: 1 if loan is 27 or 28 quarters old, else 0.
Variable name; Age dummy variables: i2930;
Variable description: 1 if loan is 29 or 30 quarters old, else 0.
Variable name; Age dummy variables: i3132;
Variable description: 1 if loan is 31 or 32 quarters old, else 0.
Variable name; Age dummy variables: i3334;
Variable description: 1 if loan is 33 or 34 quarters old, else 0.
Variable name; Age dummy variables: i35p;
Variable description: 1 if loan is older than 34 quarters, else 0.
Variable name; Loan characteristics: t5_10;
Variable description: 1 if term of loan is at least 5 years but less
than 10, else 0.
Variable name; Loan characteristics: t10_15;
Variable description: 1 if term of loan is at least 10 years but less
than 15, else 0.
Variable name; Loan characteristics: t15p;
Variable description: 1 if term of loan is 15 years or more, else 0.
Variable name; Loan characteristics: sub1027;
Variable description: 1 if loan delivered through SBA Express Program,
else 0.
Variable name; Loan characteristics: loan_amt;
Variable description: Gross guaranteed disbursed amount in millions.
Variable name; Loan characteristics: Lender_PLP;
Variable description: 1 if lender is part of the Preferred Lender
Program, else 0.
Variable name; Loan characteristics: Lender_CLP;
Variable description: 1 if lender is part of the Certified Lender
Program, else 0.
Variable name; Borrower characteristics: Corporation;
Variable description: 1 if borrower is incorporated, else 0.
Variable name; Borrower characteristics: Partnership;
Variable description: 1 if borrower is a partnership, else 0.
Variable name; Borrower characteristics: NewBusiness;
Variable description: 1 if borrower is a new business, else 0.
Variable name; Borrower characteristics: Northeast;
Variable description: 1 if located in U.S. Census Bureau's Northeast
Region, else 0.
Variable name; Borrower characteristics: Midwest;
Variable description: 1 if located in U.S. Census Bureau's Midwest
Region, else 0.
Variable name; Borrower characteristics: South;
Variable description: 1 if located in U.S. Census Bureau's South
Region, else 0.
Variable name; Economic conditions: Urate;
Variable description: Unemployment rate in the state where firm is
located.
Variable name; Economic conditions: pc_gdp96;
Variable description: Quarterly percent change in constant dollar GDP.
Source: GAO.
[End of table]
The coefficients in the SBA equations indicate that the probability of
both defaults and prepayments generally increase and then decline as a
loan seasons. Defaults peak during the eighth quarter while prepayments
peak around quarters 27 and 28. Longer-term loans are less likely to
default or prepay. By comparison, larger loans are more likely to
default or prepay. Good economic conditions, as reflected by the
coefficients on unemployment and the percentage change in gross
domestic product, reduce the chances of default and increase the
likelihood of prepayment. The positive coefficients on the variable for
new business indicate that such firms are more likely to default and
prepay. Corporations and partnerships are less likely to default and
more likely to prepay than sole proprietors. Finally, loans granted
under Subprogram 1027 are less likely to default and more likely to
prepay. Table 2 presents the coefficients in SBA's default and
prepayment equations as well as some summary statistics.
Table 2: Multinomial Logistic Regression Coefficient Estimates[A]:
Variables: Constant;
Predicting to defaults: Base model: -9.7650;
Predicting to prepayments: Base model: -5.2762.
Variables: i1;
Predicting to defaults: Base model: 2.1151;
Predicting to prepayments: Base model: 1.1203.
Variables: i2;
Predicting to defaults: Base model: 3.1174;
Predicting to prepayments: Base model: 1.6016.
Variables: i3;
Predicting to defaults: Base model: 3.8158;
Predicting to prepayments: Base model: 1.9374.
Variables: i4;
Predicting to defaults: Base model: 4.2247;
Predicting to prepayments: Base model: 2.1063.
Variables: i5;
Predicting to defaults: Base model: 4.5187;
Predicting to prepayments: Base model: 2.2865.
Variables: i6;
Predicting to defaults: Base model: 4.6659;
Predicting to prepayments: Base model: 2.4113.
Variables: i7;
Predicting to defaults: Base model: 4.7487;
Predicting to prepayments: Base model: 2.5805.
Variables: i8;
Predicting to defaults: Base model: 4.8211;
Predicting to prepayments: Base model: 2.7080.
Variables: i9;
Predicting to defaults: Base model: 4.8068;
Predicting to prepayments: Base model: 2.8163.
Variables: i10;
Predicting to defaults: Base model: 4.8121;
Predicting to prepayments: Base model: 2.9133.
Variables: i1112;
Predicting to defaults: Base model: 4.8033;
Predicting to prepayments: Base model: 3.0540.
Variables: i1314;
Predicting to defaults: Base model: 4.7772;
Predicting to prepayments: Base model: 3.1439.
Variables: i1516;
Predicting to defaults: Base model: 4.7101;
Predicting to prepayments: Base model: 3.3111.
Variables: i1718;
Predicting to defaults: Base model: 4.6214;
Predicting to prepayments: Base model: 3.4554.
Variables: i1920;
Predicting to defaults: Base model: 4.6136;
Predicting to prepayments: Base model: 3.6945.
Variables: i2122;
Predicting to defaults: Base model: 4.5156;
Predicting to prepayments: Base model: 3.5201.
Variables: i2324;
Predicting to defaults: Base model: 4.4297;
Predicting to prepayments: Base model: 3.6685.
Variables: i2526;
Predicting to defaults: Base model: 4.2945;
Predicting to prepayments: Base model: 3.8222.
Variables: i2728;
Predicting to defaults: Base model: 4.3414;
Predicting to prepayments: Base model: 4.0106.
Variables: i2930;
Predicting to defaults: Base model: 4.2515;
Predicting to prepayments: Base model: 3.6142.
Variables: i3132;
Predicting to defaults: Base model: 4.2036;
Predicting to prepayments: Base model: 3.7143.
Variables: i3334;
Predicting to defaults: Base model: 4.1378;
Predicting to prepayments: Base model: 3.7914.
Variables: i35p;
Predicting to defaults: Base model: 4.1027;
Predicting to prepayments: Base model: 3.9950.
Variables: t5_10;
Predicting to defaults: Base model: -0.0462[A];
Predicting to prepayments: Base model: -0.6568.
Variables: t10_15;
Predicting to defaults: Base model: -0.7596;
Predicting to prepayments: Base model: -1.1013.
Variables: t15p;
Predicting to defaults: Base model: -0.7395;
Predicting to prepayments: Base model: -1.1014.
Variables: sub1027;
Predicting to defaults: Base model: -0.5800;
Predicting to prepayments: Base model: 0.0812.
Variables: loan_amt;
Predicting to defaults: Base model: 0.2578;
Predicting to prepayments: Base model: 0.1189.
Variables: corporation;
Predicting to defaults: Base model: -0.0434;
Predicting to prepayments: Base model: 0.0989.
Variables: partnership;
Predicting to defaults: Base model: -0.1982;
Predicting to prepayments: Base model: 0.0211[A].
Variables: northeast;
Predicting to defaults: Base model: 0.3612;
Predicting to prepayments: Base model: -0.2054.
Variables: midwest;
Predicting to defaults: Base model: 0.2184;
Predicting to prepayments: Base model: -0.1869.
Variables: south;
Predicting to defaults: Base model: 0.4142;
Predicting to prepayments: Base model: -0.0928.
Variables: Lender_PLP;
Predicting to defaults: Base model: -0.1761;
Predicting to prepayments: Base model: 0.0824.
Variables: Lender_CLP;
Predicting to defaults: Base model: -0.1688;
Predicting to prepayments: Base model: -0.0014[B].
Variables: NewBusiness;
Predicting to defaults: Base model: 0.2773;
Predicting to prepayments: Base model: 0.0678.
Variables: urate;
Predicting to defaults: Base model: 0.1043;
Predicting to prepayments: Base model: -0.0957.
Variables: Pc_gdp96;
Predicting to defaults: Base model: -0.1261;
Predicting to prepayments: Base model: 0.0661.
Summary statistics for multinomial logistic regression models: N of
Observations;
Predicting to prepayments: Base model: 5,736,628.
Summary statistics for multinomial logistic regression models:
Variables: Likelihood Ratio Chi Sq;
Predicting to prepayments: Base model: 120,478.
Summary statistics for multinomial logistic regression models:
Variables: Degrees of Freedom;
Predicting to prepayments: Base model: 76.
Summary statistics for multinomial logistic regression models:
Variables: Significance levels;
Predicting to prepayments: Base model:
