Telecommunications
GSA Has Accumulated Adequate Funding for Transition to New Contracts but Needs Cost Estimation Policy Gao ID: GAO-07-268 February 23, 2007The General Services Administration (GSA) and its customer agencies are preparing to transition new governmentwide telecommunications contracts known as the Networx program. GSA estimated the costs for which it is responsible to be $151.5 million. This report addresses (1) the soundness of the analysis GSA used to derive the estimate of funding that would be required for the transition and (2) whether GSA will have accumulated adequate funding to pay for transition costs. In performing this work, GAO reviewed cost estimation best practices, analyzed relevant GSA documents, and performed an uncertainty analysis on GSA's estimate.
GSA did not use sound analysis when estimating the amount of funding needed to meet its transition-related commitments. Specifically, its analysis was not sufficiently accurate, comprehensive, documented, or validated. A primary weakness is that the estimate is largely based an assumption--known as the transition traffic factor--that 76 percent of the services provided under the current contracts would be moved to a different provider under the Networx contracts. However, according to program officials, this assumption is intentionally conservative and represents a worst-case scenario that is unlikely to occur. Additionally, GSA may have double-counted a cost and did not update its analysis to reflect a nearly 2-year delay. Finally, GSA did not document significant assumptions and data sources used in its analysis, or validate it. These weaknesses can be attributed in part to the lack of a cost estimation policy that reflects best practices. While GSA's intentionally conservative approach minimizes the risk that it would have inadequate funds to pay for committed transition costs, it increases the risk that GSA will retain excess funds that could be used for other purposes. GSA has accumulated adequate funding to support its anticipated transition costs. As of fiscal year-end 2006, GSA had approximately $142 million in a transition reserve. GAO analysis of the estimate indicates it is unlikely that GSA will need more than it has already accumulated to fund the transition. Specifically, the $142 million already retained will be adequate to cover anticipated costs 96 percent of the time. The recent merger of two GSA funds gives the agency additional flexibility that reduces its need to accumulate the entire $151.5 million it estimated would be needed. With Networx contracts scheduled to be awarded starting in March 2007, GSA will soon have the information necessary to reassess the main assumption underlying its estimate--the transition traffic factor--and address the weaknesses GAO identified. Once this has been accomplished, GSA can reevaluate the funding needed to meet anticipated commitments.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-07-268, Telecommunications: GSA Has Accumulated Adequate Funding for Transition to New Contracts but Needs Cost Estimation Policy
This is the accessible text file for GAO report number GAO-07-268
entitled 'Telecommunications: GSA Has Accumulated Adequate Funding for
Transition to New Contracts but Needs Cost Estimation Policy' which was
released on March 26, 2007.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Ranking Member, Committee on Oversight and Government
Reform, House of Representatives:
United States Government Accountability Office:
GAO:
February 2007:
Telecommunications:
GSA Has Accumulated Adequate Funding for Transition to New Contracts
but Needs Cost Estimation Policy:
GAO-07-268:
GAO Highlights:
Highlights of GAO-07-268, a report to the Ranking Member, Committee on
Oversight and Government Reform, House of Representatives
Why GAO Did This Study:
The General Services Administration (GSA) and its customer agencies are
preparing to transition new governmentwide telecommunications contracts
known as the Networx program. GSA estimated the costs for which it is
responsible to be $151.5 million. This report addresses (1) the
soundness of the analysis GSA used to derive the estimate of funding
that would be required for the transition and (2) whether GSA will have
accumulated adequate funding to pay for transition costs. In performing
this work, GAO reviewed cost estimation best practices, analyzed
relevant GSA documents, and performed an uncertainty analysis on GSA‘s
estimate.
What GAO Found:
GSA did not use sound analysis when estimating the amount of funding
needed to meet its transition-related commitments. Specifically, its
analysis was not sufficiently accurate, comprehensive, documented, or
validated. A primary weakness is that the estimate is largely based an
assumption”known as the transition traffic factor”that 76 percent of
the services provided under the current contracts would be moved to a
different provider under the Networx contracts. However, according to
program officials, this assumption is intentionally conservative and
represents a worst-case scenario that is unlikely to occur.
Additionally, GSA may have double counted a cost and did not update its
analysis to reflect a nearly 2 year delay. Finally, GSA did not
document significant assumptions and data sources used in its analysis,
or validate it. These weaknesses can be attributed in part to the lack
of a cost estimation policy that reflects best practices. While GSA‘s
intentionally conservative approach minimizes the risk that it would
have inadequate funds to pay for committed transition costs, it
increases the risk that GSA will retain excess funds that could be used
for other purposes.
GSA has accumulated adequate funding to support its anticipated
transition costs. As of fiscal year-end 2006, GSA had approximately
$142 million in a transition reserve. GAO analysis of the estimate
indicates it is unlikely that GSA will need more than it has already
accumulated to fund the transition. Specifically, the $142 million
already retained will be adequate to cover anticipated costs 96 percent
of the time. The recent merger of two GSA funds gives the agency
additional flexibility that reduces its need to accumulate the entire
$151.5 million it estimated would be needed (see table). With Networx
contracts scheduled to be awarded starting in March 2007, GSA will soon
have the information necessary to reassess the main assumption
underlying its estimate”the transition traffic factor”and address the
weaknesses GAO identified. Once this has been accomplished, GSA can
reevaluate the funding needed to meet anticipated commitments.
Table: GSA's Transition Cost Estimate:
Cost element: GSA contractor support;
Estimated cost: $35.0.
Cost element: Certain agency transition costs;
Estimated cost: 116.5.
Cost element: Total;
Estimated cost: $151.5.
Source: GSA.
[End of table]
What GAO Recommends:
GAO recommends that the GSA Administrator establish a cost estimation
policy that reflects best practices. In addition, GAO recommends that
the Administrator revise the transition cost estimate using best
practices after the award of the Networx contracts and reassess the
funding needed to meet its commitments. GSA concurred with these
recommendations but questioned the way GAO characterized the soundness
of GSA‘s analysis and whether this transition was comparable with the
previous one. GAO clarified its characterization of GSA‘s analysis.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-268].
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Linda Koontz at (202) 512-
6240 or koontzl@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
GSA's Analysis Was Not Sound:
GSA Has Accumulated Adequate Funding for the Transition:
Conclusions:
Recommendations:
Agency Comments and Our Evaluation:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Uncertainty Analysis for GSA's Transition Estimate:
Appendix III: Comments from the GSA Administrator:
Appendix IV: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: GSA Transition Cost Estimate:
Table 2: Cumulative Effect of Quantifiable Weaknesses:
Table 3: Risk Adjusted Factors:
Figures:
Figure 1: Estimated Cost of Transition Using Various Percents of
Transition Traffic:
Figure 2: Confidence That Certain Amounts of Funding Will Account for
Actual Costs:
Abbreviations:
GSA: General Services Administration:
IMC: Interagency Management Council:
IT: Information Technology:
United States Government Accountability Office:
Washington, DC 20548:
February 23, 2007:
The Honorable Tom Davis:
Ranking Member:
Committee on Oversight and Government Reform:
House of Representatives:
Dear Mr. Davis:
As innovations in telecommunications services continue to transform the
way the federal government conducts business, the General Services
Administration's (GSA) governmentwide telecommunications acquisition
programs offer federal agencies the opportunity to apply innovative
services and solutions to their operations. With the current set of
governmentwide telecommunications contracts approaching expiration, GSA
and its customer agencies will have to transition the services acquired
under these contracts to their replacements, known collectively as
Networx.
GSA will incur program management costs associated with planning and
executing this transition. It has also made a commitment to absorb
certain agency transition costs. To ensure it would have the funds
necessary to pay for these costs, GSA estimated that it would need to
set aside approximately $151.5 million.
This report responds to your request that we determine (1) the
soundness of the analysis GSA used to derive the estimate of funding
that would be required for the transition and (2) whether GSA will have
accumulated adequate funding to pay for its transition management
costs. To accomplish the first objective, we conducted a search of over
250 documents from both government and industry[Footnote 1] for
examples of best practices in the field of cost estimation and
identified common characteristics among them. We determined that high-
quality, reliable estimates should be accurate, comprehensive, well-
documented, and validated. Using these characteristics, we analyzed
transition estimate documentation developed by GSA, documentation
provided by GSA on the previous transition, the Networx Request for
Proposals, and other relevant documents. To accomplish the second
objective, we analyzed financial and operational documents from related
GSA programs. In addition, we interviewed GSA program officials about
both objectives and conducted an analysis of GSA's estimate to examine
the effects of varying the main cost driver in the estimate. We
conducted our work at GSA's Washington, D.C., area headquarters between
June 2006 and January 2007 in accordance with generally accepted
government auditing standards. A detailed discussion of our objectives,
scope, and methodology can be found in appendix I.
Results in Brief:
GSA did not use sound analysis when estimating the amount of funding
needed to meet its transition-related commitments. Specifically, its
analysis was not sufficiently accurate, comprehensive, documented, or
validated. A primary weakness is that the estimate is largely based on
the assumption that 76 percent of the services provided under the
current contracts will transition to a different provider under the
Networx contracts. However, according to program officials, this
assumption is intentionally conservative and represents a worst-case
scenario that is unlikely to occur. Additionally, GSA may have double-
counted a cost, did not update its analysis to reflect a nearly 2-year
delay, and did not adequately document or validate its analysis. The
weaknesses we identified can be attributed in part to the lack of a
policy requiring cost estimates to be developed using best practices.
Without such a policy, GSA's future cost estimates could exhibit
similar weaknesses, increasing the risk that it will retain excess
funds that could be reallocated for other purposes.
GSA has accumulated adequate funding to support its anticipated
transition costs. As of fiscal year-end 2006, GSA had approximately
$142 million in a transition reserve to be used to pay for its costs
associated with the transition. Our analysis of the estimate indicates
that it is unlikely that GSA will need to accumulate the entire $151.5
million it estimated and that the funds it has already accumulated
should be sufficient to fund the transition. Specifically, the $142
million already retained will be adequate to cover anticipated costs 96
percent of the time. Further, the recent merger of two GSA funds
increases the agency's flexibility and could provide additional money,
if needed. With Networx contracts scheduled to be awarded starting in
March 2007, GSA will soon have the information necessary to reassess
the main cost driver underlying its estimate and address the weaknesses
we identified. Once it has a current, accurate estimate, GSA can
reevaluate the funding needed to meet anticipated commitments.
To ensure that future cost estimates by GSA are sound and can be used
as a reliable basis for decisions, we recommend that the GSA
Administrator establish an agencywide policy requiring that cost
estimates be developed using best practices. In addition, we recommend
that the Administrator revise the transition cost estimate using best
practices after the award of contracts under the Networx program and,
if feasible, reallocate any excess funds for other purposes or return
them to the Treasury.
In written comments on a draft of this report, the GSA Administrator
concurred with our recommendations and emphasized the importance of
supporting a successful governmentwide telecommunications transition.
Her comments also state that GSA's transition cost estimate was within
2 percent of our independent analysis using the same assumptions.
However, while we identified only $3 million in quantifiable errors
(roughly 2 percent of GSA's total estimate), we also determined that if
the extent of transitioning services is significantly lower than GSA's
intentionally conservative assumption, actual costs could be more than
$110 million less than GSA's estimate.
In addition, the Administrator presented two main objections to our
draft. First, she questioned whether our reported findings were
balanced given the facts and results presented. We clarified our report
to ensure that our findings better reflect the information we discuss.
Second, she stated that we incorrectly suggested comparability between
the pending Networx transition and the prior transition. We maintain
that the two transitions are comparable, particularly because GSA's
analysis is based, in part, on the results of the previous transition.
Appendix III provides the full text of GSA's comments. GSA also
provided technical comments that have been incorporated in our report
as appropriate.
Background:
As part of its mission of providing federal agencies with acquisition
services and solutions at best value, GSA's technology programs offer
agencies options to acquire needed telecommunications services. An
option chosen by more than 135 agencies is the FTS2001 program, which
consists of two large governmentwide telecommunications contracts--one
awarded to Sprint[Footnote 2] in December 1998 and the other to
MCI[Footnote 3] in January 1999--and FTS2001 crossover
contracts.[Footnote 4]
GSA is planning to replace the FTS2001 contracts, FTS2001 crossover
contracts, and separate wireless contracts with a new set of contracts.
Collectively known as the Networx program, these new contracts are to
provide governmentwide telecommunications services through two
indefinite-delivery/indefinite-quantity acquisitions--Networx Universal
and Networx Enterprise. The Universal acquisition is expected to
satisfy the requirements for a full range of national and international
network services and, according to GSA, to ensure the continuity of
broad-ranging services with global geographic coverage rendered under
expiring contracts. The Enterprise acquisition is expected to offer
agencies leading-edge services and solutions with less extensive
geographic and service requirements than Universal. The services
required in these contracts focus on Internet-based offerings and
related security and management services.
GSA expects the transition to begin when Networx Universal awards are
made in March 2007 and to continue until fiscal year 2010. Because the
FTS2001 contracts with Sprint Nextel Corporation and Verizon Business
expired in December 2006 and January 2007, respectively, GSA and the
incumbent vendors negotiated separate sole-source contracts that
essentially extend the terms of the FTS2001 contracts for 42
months.[Footnote 5] These sole-source contracts were awarded to ensure
uninterrupted service and allow agencies adequate time to complete the
transition to Networx.
GSA is working with representatives of federal agencies to prepare for
the upcoming transition to Networx, both directly and through the
Interagency Management Council (IMC), a group of senior federal
information resource officials who advise GSA on issues related to
telecommunications contracts. The IMC has worked with GSA to document
lessons learned from the transition to FTS2001 that began in the late
1990s, GSA's most recent governmentwide telecommunications transition.
One important lesson learned was that GSA and agency plans for funding
transition expenses should be determined early to allow agencies to
gauge the impact of transition expenses on their budgets. Specifically,
the lessons-learned document recommended that guidelines be established
to allow agencies to complete the financial planning required to ensure
that the resources needed for transition were available. This led the
IMC and GSA to develop a Taxonomy and Allocation of Transition Costs
document that identified which Networx transition costs would be borne
by GSA and which would be borne by transitioning agencies.
This taxonomy document indicated that GSA will incur or reimburse
agencies, including:
* GSA contractor support costs to, for example, aid in planning for the
transition and oversight of Networx contractors. GSA officials also
indicated that contractor support will be used to develop a methodology
for tracking transition progress and the establishment of a transition
coordination center; and:
* certain costs incurred by agencies during transition.
In 2004, following the development of the taxonomy document, GSA
generated an estimate of its costs for the transition. GSA's
methodology for its estimate was to:
* develop assumptions for the estimate,
* define and develop a baseline for the estimate based on experiences
and lessons learned from the previous transition,
* determine the network growth as well as the total business volume
projections for fiscal year 2006 based on historical traffic and cost
trends,
* define the transition cost elements,
* define and develop a formula for calculating an estimate for each
cost element, and:
* design and develop the estimated transition cost model for
sensitivity analysis.
Using this methodology, GSA estimated that it would need a total of
$151.5 million for a 30-month transition, most of which would be used
to reimburse agencies' transition costs. Table 1 below details the
transition costs GSA expects to pay.
Table 1: GSA Transition Cost Estimate:
Cost element: GSA contractor support;
Estimated cost: $35.0.
Cost element: Certain agency transition costs;
Estimated cost: 116.5.
Cost element: Total;
Estimated cost: $151.5.
Source: GSA.
[End of table]
GSA planned to pay for its costs and the reimbursement of certain
agency costs using its Information Technology (IT) Fund.[Footnote 6]
The IT Fund was a full-cost recovery revolving fund[Footnote 7] whereby
GSA fully recovered all costs of its technology programs and its
operations via estimated fee rates. The fees, which are charged to
agencies for the use of GSA contracts cover the direct costs of its
operations--such as the development and management of contract
vehicles--and indirect costs associated with its headquarters, such as
support for the Offices of the Chief Information Officer and the Chief
Financial Officer. The IT Fund allowed GSA to stabilize rates for its
services when expenses varied and was used to provide funding for the
previous transition to FTS2001. The IT Fund also contained a working
capital reserve that was used to offset losses due to fluctuations in
business volumes and other unexpected contingencies. At each fiscal
year-end, the uncommitted balance of funds remaining was to be
transferred to the general fund of the treasury as miscellaneous
receipts.
Recently, a new law changed the structure of the IT Fund and GSA's
organization. On October 6, 2006, the General Services Administration
Modernization Act[Footnote 8] combined the IT Fund with another GSA
revolving fund--the General Supply Fund--to create the Acquisition
Services Fund. This legislation, which also merged GSA's technology and
supply programs,[Footnote 9] made all capital assets and balances
remaining in the IT and General Supply Funds available for the purposes
of the Acquisition Services Fund. According to GSA, the Federal
Acquisition Service will increase agency savings, enhance GSA's
capability to meet customer requirements for excellence, and improve
internal efficiencies.
GSA's Analysis Was Not Sound:
The analysis GSA used to derive its estimate of $151.5 million was not
sound because it was not sufficiently accurate, comprehensive,
documented, or validated. The analysis was not sufficiently accurate
because it is largely based on the assumption that agencies will
transition 76 percent of the services acquired under the current
FTS2001 contracts to a different provider under Networx--an
intentionally conservative scenario that GSA program officials believe
is unlikely to occur. Further, the analysis has not been updated after
a nearly 2-year delay in the contract award. While GSA appears to have
included all pertinent costs, it may have double-counted a cost,
calling into question the comprehensiveness of its analysis. In
addition, GSA did not document significant assumptions and data.
Finally, GSA's analysis was not validated by an independent cost
estimate nor was an uncertainty analysis performed that would allow GSA
to quantify the level of confidence it has in its estimate. These
weaknesses can be attributed in part to the lack of a cost estimation
policy that would help to ensure that such estimates are developed
using best practices. While an intentionally conservative approach
minimizes the risk that GSA would have inadequate funds to pay for
committed transition costs, it increases the risk that GSA will retain
excess funds that could be used for other purposes.
GSA's Analysis Was Not Sufficiently Accurate:
Estimates are accurate when they are not overly conservative, based on
an assessment of the most likely costs, and adjusted properly for
inflation. Best practices further dictate that as schedules change,
cost estimates should be revised to provide management with insight
into the current program status, effective control of the program, and
the ability to balance resources and the budget.
The analysis GSA used to derive the estimate, however, was not
sufficiently accurate. First, the estimate's main cost driver is based
on the assumption that agencies will transition 76 percent of the
services acquired under the current FTS2001 contracts to a different
provider under Networx--60 percent due to agencies being forced to
change providers and 16 percent due to voluntary changes. However,
according to program officials, the 76 percent "transition traffic
factor" is intentionally conservative and represents a worst-case
scenario that is unlikely to occur.
The 76 percent transition traffic factor is also overly conservative
when compared with the previous transition. Then, approximately 60
percent of services were shifted from an incumbent to a different
provider under FTS2001, the bulk being forced to change providers when
an incumbent providing more than half of the services (AT&T) was not
awarded an FTS2001 contract. This forced shift happened in part because
GSA limited the FTS2001 awards to only two vendors. In contrast, GSA
has placed no such limit on the number of Networx vendors, and
therefore, all incumbent FTS2001 vendors could potentially be awarded a
Networx contract. Further, because the most-used FTS2001 incumbent
(Verizon Business) provides approximately 50 percent of the services
under the current FTS2001 contracts, the assumption of a 60 percent
forced shift would only be realized if no awards were made to Verizon
Business and at least one other incumbent. Finally, GSA's assumption of
a 16 percent voluntary shift is significantly higher than the 3 percent
that voluntarily changed providers during the previous transition.
Program officials could not identify any basis for the assumption that
voluntary changes will reach 16 percent. However, the officials stated
that the percentage of voluntary changes could be higher because
agencies will likely have more options when selecting vendors and
because of improved guidance on federal requirements regarding the fair
opportunity process, which is intended to give each awardee an equal
opportunity to compete for agencies' telecommunications services
requirements based on agency-established selection criteria.
Second, the analysis has not been updated to reflect schedule changes.
When the estimate was developed in 2004, GSA expected to award the
first Networx Universal contracts in mid-2005. Since that time, delays
have pushed the time frame back almost 2 years. Universal awards are
now expected in March 2007. GSA's analysis has not been adjusted to
reflect additional inflation during this period, which could increase
the estimates total by as much as $9 million.
Finally, GSA has not revised its analysis to reflect an assessment of
most likely costs using currently available information. The analysis
assumes that reimbursable agency transition costs would be greater than
during the previous transition due to growth in the volume of services
ordered. While GSA estimated that service levels would grow 60 percent
over the previous transition by the time of contract award, as of
August 2006, service levels had actually grown by 55.9 percent and are
now expected to remain stable. The overestimation of service growth
added approximately $1.7 million to the estimate.
GSA's Analysis May Not Be Comprehensive:
Estimates are comprehensive when their level of detail ensures that all
pertinent costs are included and no costs are double-counted. It is
important to ensure the completeness, consistency, and realism of the
information contained in the cost estimate.
GSA appears to have included all pertinent costs in its analysis;
however, according to officials, a specific agency cost valued at $4.3
million may have been double-counted. For several of the transition
costs estimated, GSA based its calculations on the actual charges
incurred during the previous transition. However, officials indicated
that a specific agency cost that is estimated separately in its current
estimate may have already been included in a more general category of
its previous costs. Despite this uncertainty, GSA calculated the
separate total for this specific agency cost based on current service
levels and added it to its estimate. As a result, it is likely this
specific agency cost is double-counted and therefore overstated.
GSA's Analysis Was Not Adequately Documented:
Cost estimates are well-documented when they can be easily repeated or
updated and can be traced to original sources through auditing.
Rigorous documentation increases the credibility of an estimate and
helps support an organization's decision-making process. The
documentation should explicitly identify the primary methods,
calculations, results, rationales or assumptions, and sources of the
data used to generate each cost element.
GSA provided us documentation of its methodology, the calculations it
used to derive each cost element, results, and many of the previous
transition costs.[Footnote 10] However, it did not document significant
assumptions. Specifically, GSA did not document the rationale behind
its 76 percent transition traffic factor or why it used a 30-month time
period for the transition--two key assumptions of its analysis.
GSA also did not provide documentation of certain data sources.
Specifically, program officials could not provide supporting data used
to estimate an agency transition cost valued at $4.7 million. In
addition, GSA could not document the data sources used to estimate
costs for contractor support in planning and implementing the
transition. While many costs in its estimate are based on the charges
incurred during the previous transition, GSA officials stated that it
was not appropriate to use previous costs as a basis for the contractor
cost element. These officials explained that unlike the previous
transition, GSA would not provide agencies with on-site contractor
support. Officials made this decision, in part, because the 2½ years of
transition planning that has occurred to date is expected to result in
better preparation by agencies and the ability for them to facilitate
their transitions without direct assistance from GSA or its
contractors. Instead of basing their projection of contractor costs on
prior charges, program officials told us that GSA management decided
that contractor support costs should not exceed $35 million. Program
officials could not provide any data or analysis to support this
decision.
GSA's Analysis Was Not Validated:
Estimates are adequately validated when they have been cross-checked
with an independent cost estimate, and when a level of uncertainty
associated with the estimate is identified. An independent cost
estimate provides the estimator with an unbiased test of the
reasonableness of the estimate and reduces the cost risk associated
with the project by demonstrating that alternate methods generate
similar results. In performing an uncertainty analysis, an entity
examines the effects of varying multiple elements and, as a result, is
able to express a level of confidence in its estimate.
GSA did not validate its analysis against an independent cost estimate
or perform an uncertainty analysis. GSA program officials could not
provide a rationale of why these activities were not performed.
Quantifiable Weaknesses Are Small, but Effect of Estimate's
Intentionally Conservative Assumption Could Be Significant:
The cumulative effect of the quantifiable weaknesses we identified in
GSA's analysis is relatively small, resulting in an underestimation of
$3 million, or roughly 2 percent of the total $151.5 million estimate
(as shown in table 2). The underestimation of costs, related to
inflation during the extended delay in making award, was offset by the
overestimation of service growth and the possible double-counting of a
specific agency transition cost.
Table 2: Cumulative Effect of Quantifiable Weaknesses:
Dollars in millions.
Issue identified: The estimate has not been adjusted to reflect
schedule changes;
Aspect of estimate affected: Calculation of inflation;
Approximate effect: +$9.0.
Issue identified: The estimate does not reflect an assessment of most
likely costs due to currently available information;
Aspect of estimate affected: Actual growth in services;
Approximate effect: -1.7.
Issue identified: The estimate may have double-counted a cost;
Aspect of estimate affected: Agency transition costs;
Approximate effect: -4.3.
Issue identified: Cumulative effect;
Aspect of estimate affected: [Empty];
Approximate effect: +$3.0.
Source: GAO analysis of GSA information.
[End of table]
In contrast, if the actual level of services transitioning to a
different vendor is lower than the 76 percent transition traffic factor
assumed by GSA, the effect will be greater. Because this factor is the
primary cost driver in the estimate, significant changes in
transitioning traffic result in similarly significant changes in total
costs. This effect can be illustrated using GSA's transition cost
estimate model (see fig. 1). For example, if a transition traffic
factor of 66 percent is used (GSA's assumption of 16 percent for a
voluntary transition plus a forced shift of 50 percent that would occur
if Verizon Business is not awarded a Networx contract), the total
estimated cost of the transition falls to about $136 million.
Similarly, if all incumbents are awarded Networx contracts and only
GSA's assumption of a 16 percent voluntary transition occurs, the total
estimated cost of the transition is reduced to approximately $40
million. GSA officials explained that they used a risk-averse
transition traffic factor to minimize the possibility of
underestimating costs.
Figure 1: Estimated Cost of Transition Using Various Percents of
Transition Traffic:
[See PDF for image]
Source: GSA cost estimate model.
[End of figure]
GSA Does Not Have a Policy to Ensure That Cost Estimates Are Sound:
The weaknesses in GSA's analysis can be attributed in part to the lack
of a policy requiring cost estimates to be developed using best
practices. Officials from the Offices of the Chief Acquisition Officer,
the Chief Information Officer, and the program office's Controller
confirmed that GSA does not have centralized policy or guidance on cost
estimation. Instead, the officials that prepared the estimate stated
that they based their work loosely on best practices learned as a
result of past experiences and were comfortable with the estimate.
Officials believe there is no reason to revise their estimate to
address the issues we raised in this report because their purpose in
producing it was to minimize the risk of underestimating costs. While
GSA's approach minimized the risk of having inadequate funds to fulfill
its commitments, it increased the risk that GSA will retain excess
funds that could be used for other purposes.
GSA Has Accumulated Adequate Funding for the Transition:
GSA has accumulated adequate funding to support its anticipated
commitments related to the Networx transition. As of fiscal year-end
2006, GSA accumulated approximately $142 million in a reserve dedicated
to costs associated with the transition from FTS2001 to Networx. The
uncertainty analysis we performed on GSA's cost estimate indicates that
the funding GSA has already accumulated will most likely be adequate to
pay for expected transition costs. By varying the transition traffic
factor, our analysis indicates that the $142 million already retained
should be adequate to cover expected expenses 96 percent of the time.
Figure 2 illustrates the probability that a particular level of funding
will be adequate to account for the total actual costs incurred. For
example, the $151.5 million estimate represents a confidence level of
100 percent: there is almost no chance that costs will exceed the
estimate. The full detail of our methodology is detailed in appendix
II.
Figure 2: Confidence That Certain Amounts of Funding Will Account for
Actual Costs:
[See PDF for image]
Source: GAO analysis of GSA data.
[End of figure]
The merger of the IT and General Supply Funds provides GSA with
additional flexibility, further reducing the need to retain the entire
amount of the estimate. As discussed, legislation[Footnote 11] combined
the two funds into an Acquisition Services Fund, making their capital
assets and balances available for the purposes of the new Federal
Acquisition Service. The legislation also allows the Federal
Acquisition Service to establish a reserve to retain surplus revenues
from GSA's technology and service programs specifically for the purpose
of offsetting losses and other unexpected contingencies. Further, at
the end of each year, the statute requires GSA to return to the
Treasury any funds not expended or held in a working capital reserve.
During fiscal year 2006, GSA's technology programs experienced losses,
but its supply programs reported overall positive earnings of
approximately $126 million. If operations continue in this fashion,
excess revenue will be available in the combined fund to offset losses
or account for contingencies, such as the Networx transition, be
retained within the Acquisition Services Fund, or be returned to the
Treasury.
With Networx contracts scheduled to be awarded starting in March 2007,
GSA will soon be in a position to reassess its main assumption, the
transition traffic factor, and the resulting level of funding needed to
meet anticipated commitments. Unless GSA revises its estimate, it risks
unnecessarily retaining funds that could be reallocated to other agency
priorities or returned to the Treasury.
Conclusions:
While GSA achieved its goal of minimizing the risk that it would have
inadequate funds to pay its transition commitments, the analysis used
to develop the transition cost estimate was not sound because it was
not sufficiently accurate, comprehensive, documented, or validated. The
weaknesses can be attributed in part to a lack of a policy at GSA to
ensure that such estimates are developed using best practices and make
the best use of agency resources. While the quantifiable effect of the
weaknesses in accuracy and comprehensiveness is small, the effect of
potential inaccuracies resulting from the intentional use of an overly
conservative assumption about a main cost driver could be more
significant. Without the use of a cost estimation policy that reflects
best practices, GSA could continue to produce similarly unsound
estimates, increasing the risk that it will unnecessarily retain funds
that could be reallocated for other purposes.
Despite the weaknesses in its analysis, GSA has accumulated adequate
funding to support its anticipated commitments related to the Networx
transition. Our analysis indicates that it is highly unlikely that GSA
will need more than the $142 million it has already accumulated. In
addition, the merger of two revolving funds gives it increased
flexibility in meeting costs. Once Networx contracts are awarded
beginning in March 2007, GSA will be able to forecast the number of
forced transitions more accurately, and, if necessary, reduce the
amount of funding it plans to accumulate in the future or free already
accumulated funds for other purposes. This reassessment will also be an
opportunity for GSA to address the other weaknesses in its analysis
that resulted from its deviation from best practices.
Recommendations:
To improve GSA's program management, we are making two recommendations.
First, to ensure that future cost estimates are sound and can be used
as a reliable basis for decisions, we recommend that the GSA
Administrator establish a policy for cost estimation efforts at GSA.
Specifically, this policy should reflect best practices by requiring
that estimates are:
* accurate (not overly conservative, based on an assessment of the most
likely costs, and adjusted properly for inflation);
* comprehensive (their level of detail ensures that all pertinent costs
are included and no costs are double-counted);
* well-documented (can be easily repeated or updated and can be traced
to original sources through auditing); and:
* validated (they have been cross-checked with an independent cost
estimate and a level of uncertainty associated with the estimate has
been identified.)
Second, to ensure the most efficient use of federal funds, we recommend
that the Administrator revise the transition cost estimate following
the award of contracts under the Networx program. Specifically, this
revision should reflect best practices, include a more precise
transition traffic factor, and address the overestimation of service
growth, the possible double-counting of a nonrecurring charge, and the
effects of inflation during the extended delay in making awards. If the
results of this new estimate indicate that the full $151.5 million is
not needed to reasonably support the transition effort, GSA should
reallocate any excess funds for other purposes allowable within the
Acquisition Services Fund or return them to the Treasury.
Agency Comments and Our Evaluation:
In written comments on a draft of this report, the GSA Administrator
concurred with our recommendations and emphasized the importance of
supporting a successful governmentwide telecommunications transition.
To address our recommendations, she stated that GSA will issue improved
policy guidance on cost estimating and review and adjust the cost
estimate as additional information becomes available.
The Administrator also commented that GSA's transition cost estimate
was within 2 percent of our analysis using the same assumptions.
However, while we identified only $3 million in quantifiable errors
(roughly 2 percent of GSA's total estimate), our report also states
that if the extent of transitioning services is significantly lower
than GSA's intentionally conservative assumption, the actual costs of
the transition could be considerably less than GSA's estimate.
Specifically, if all incumbents are awarded Networx contracts and only
GSA's assumption of a 16 percent voluntary transition occurs, the total
cost of the transition could be reduced to approximately $40 million--
over $110 million less than GSA's estimate.
In addition, the Administrator stated that GSA does not concur with the
entirety of the draft report. She presented two main objections. First,
she questioned whether our reported findings were balanced given the
facts and results presented. We clarified our report to ensure that our
findings better reflect the information we discussed. Second, she
raised a concern that we incorrectly suggested comparability between
the pending Networx transition and the prior transition. We maintain
that the two transitions are comparable, particularly because GSA's
analysis is based, in part, on the results of the previous transition.
Specifically, GSA's analysis relied on experiences and lessons learned
from the previous transition to establish costs for the current
estimate.
Appendix III provides the full text of GSA's comments. GSA also
provided technical comments that have been incorporated in this report,
as appropriate.
As agreed with your office, unless you publicly announce the contents
of this report earlier, we plan no further distribution until 30 days
from the report date. At that time, we will send copies of this report
to the GSA Administrator and interested congressional committees. We
will also make copies available to others upon request. In addition,
the report will be available at no charge on the GAO Web site at
http://www.gao.gov.
Should you or your offices have any questions about matters discussed
in this report, please contact me at (202) 512-6240 or by e-mail at
koontzl@gao.gov. Contact points for our Offices of Congressional
Relations and Public Affairs may be found on the last page of this
report. GAO staff who made major contributions to this report are
listed in appendix IV.
Sincerely,
Signed by:
Linda D. Koontz;
Director, Information Management Issues:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
Our objectives were to determine (1) the soundness of the analysis the
General Services Administration (GSA) used to derive the estimate of
funding that would be required for the transition and (2) whether GSA
will have accumulated adequate funding to pay for its transition
management costs.
To determine the soundness of GSA's analysis, we conducted an intensive
search of over 250 source documents of both government and industry
literature for examples of best practices in the field of cost
estimation. This included literature on cost estimation from the
Society of Cost Estimating and Analysis, the Department of the U.S.
Army, the Office of the Secretary of Defense, the National Aeronautics
and Space Administration, and the Department of Energy. This indicated
that high quality, reliable cost estimates are:
* accurate (not overly conservative, based on an assessment of the most
likely costs, and adjusted properly for inflation);
* comprehensive (their level of detail ensures that all pertinent costs
are included and no costs are double-counted);
* well-documented (can be easily repeated or updated and can be traced
to original sources through auditing); and:
* validated (they have been cross-checked with an independent cost
estimate and a level of uncertainty associated with the estimate has
been identified).
To determine the extent to which GSA followed these practices, we
analyzed documentation supporting the transition estimate,
documentation provided by GSA on the previous transition estimate, and
the Networx Request for Proposals. We also interviewed GSA Networx
program managers and attended a GSA sponsored transition conference and
meetings of the Interagency Management Council Transition Working
Group.[Footnote 12]
To address whether GSA has accumulated adequate funding to pay for its
transition management costs, we obtained and analyzed Cost and Capital
Requirements Plans for the Information Technology Fund submitted to the
Office of Management and Budget and legislation for GSA's Information
Technology, General Supply, and Acquisition Services Funds. In
addition, to determine the extent of funding held in GSA's related
accounts, we analyzed financial statements for GSA's technology and
service programs. To verify the reliability of these records, we
obtained and analyzed the results of the most recent GSA financial
audits and audit reports from GSA's Inspector General, and we
interviewed GSA's independent financial auditor regarding the quality
control procedures in place. The independent auditor did not
specifically review the dollar amounts in the Information Technology or
General Supply Funds for accuracy but did test the controls in place
for compliance with laws and regulations. This auditor stated that
there were no reportable findings associated with either fund, and it
was reasonable to assume that these accounts were fairly stated. As a
result, we determined the data were sufficiently reliable for the
purposes of this report. We also interviewed officials from the Office
of Management and Budget and officials from GSA's technology and
service programs, Office of the Chief Acquisition Officer, Office of
the Chief Financial Officer, and Office of the Chief Information
Officer.
To assess the adequacy of the level of funding already accumulated by
GSA, we performed an uncertainty analysis for GSA's estimate using a
Monte Carlo simulation.[Footnote 13] A Monte Carlo simulation provides
a perspective on the potential variability of the cost estimate should
the facts, circumstances, or assumptions change. We chose to vary only
the transition traffic factor because it is the main driver of the
costs in GSA's estimate. To carry out this simulation, we identified a
minimum, maximum, and median value for the transition traffic factor
based on information received from GSA.
We conducted our work between May 2006 and January 2007 in accordance
with generally accepted government auditing standards.
[End of section]
Appendix II: Uncertainty Analysis for GSA's Transition Estimate:
An uncertainty analysis provides decision makers with a perspective on
the potential variability of the estimate should the facts,
circumstances, and assumptions change. By examining the effects of
varying the estimates elements, a degree of uncertainty about the
estimate can be expressed, possibly as an estimated range or qualified
by some factor of confidence. For example, an estimate that produces a
100 percent confidence level indicates that, based on the methodology
used to create that estimate, there is almost no chance that costs will
exceed the estimate.
We performed our uncertainty analysis on GSA's estimate using a Monte
Carlo simulation.[Footnote 14] We chose to vary only the transition
traffic factor because it is the main driver of the costs in GSA's
estimate. To carry out this analysis, we identified a minimum, maximum,
and median value for the transition traffic factor, based on
information received from GSA.
The transition traffic factor represents the possible percentage of
services under FTS2001 that may transition to a different provider
under the Networx contracts. For this factor, we chose a minimum of 3
percent, which represents the voluntary shift to a different vendor
that occurred during GSA's previous transition to FTS2001. For the
maximum, we used 76 percent, as this value was chosen by GSA officials
as a worst-case scenario in an effort to mitigate the risks of
underestimating costs. The median of these two numbers is 39.5 percent.
Table 3 shows the variations of each factor used in our uncertainty
analysis.
Table 3: Risk Adjusted Factors:
Transition traffic factor;
GSA estimate: 76%;
Minimum: 3%;
Median: 39.5%;
Maximum: 76%.
Source: GAO analysis of GSA data.
[End of table]
[End of section]
Appendix III: Comments from the GSA Administrator:
GSA Administrator:
U.S. General Services Administration:
1800 F Street, NW:
Washington, DC 20405-002:
Telephone: (202)501-0800:
Fax: (202)-219-1243:
www.gsa.gov:
February 12, 2007:
The Honorable David M. Walker:
Comptroller General of the United States:
Government Accountability Office:
441 G Street NW:
Washington, DC 20548:
Dear Mr. Walker: -:
Thank you for the opportunity to comment on the Government
Accountability Office (GAO) draft report, "GSA Has Accumulated Adequate
Funding for Transition to New Contracts but Needs Cost Estimation
Policy" (GAO-07-268). I am pleased with GAO's finding that the General
Services Administration (GSA) has assured the availability of adequate
funding to support a successful Networx program transition. As you
know, Networx is a very important Governmentwide telecommunications
acquisition program that will allow GSA to continue to offer our
Federal agency customers innovative services and solutions to support
and improve the efficiency and effectiveness of their operations.
Networx, with its substantial number of new services and features, will
also allow agencies to operate with greater security while supporting
the President's Management Agenda and such Presidential mandates as
Internet Protocol Version 6 (IPv6).
The availability of funds to support the Networx transition is
enormously important to GSA and our customer agencies. As noted in your
report, our paramount consideration is the ability for GSA to guarantee
that transition funding will be available to all agencies, consistent
with the provisions of the GSA/Interagency Management Council Taxonomy
Document, under all possible circumstances. I appreciate your
independent review and GAO's observation that our transition cost
estimate, while conservative by design, was within two percent of GAO's
estimation using the same assumptions. This confirms that GSA met the
important goal of ensuring adequate funding for the Networx transition.
I concur with both recommendations in the report. As you noted, one of
the weaknesses inherent in our initial estimate is the uncertainty
associated with many variables that will impact the cost of the
transition to Networx. As GSA oversees and manages contract
implementation activities, the transition cost estimate will be
reviewed and adjustments will be made as additional information becomes
available. GSA will also issue improved policy guidance on cost
estimating.
Despite the overall positive findings in your draft report, there are
several observations in it that I believe either inaccurately represent
the facts or indicate an incomplete understanding of conditions
associated with the Networx transition. For example, in some instances
the wording used to characterize the reported findings is not balanced
with the facts presented and exaggerates their significance, given the
results of your own independent analysis. In others, assumptions are
made that! believe incorrectly suggest comparability between pending
Networx transition and prior contract transition efforts. For these
reasons, !cannot concur with the entirety of the draft report. My
specific concerns are detailed in the enclosure.
As always. I welcome an open dialogue with you and your staff on
oversight activities. Your efforts are valuable in helping us improve
the quality of the programs through which we deliver so many important
services to our customers.
Cordially,
Signed by:
Lurita Doan:
Administrator:
Enclosure:
cc: Linda D. Koontz, Director:
Information Management Issues:
Government Accountability Office:
441 G Street NW:
Washington, DC 20548:
[End of section]
Appendix IV: GAO Contact and Staff Acknowledgments:
GAO Contact:
Linda D. Koontz (202) 512-6240 or koontzl@gao.gov:
Staff Acknowledgments:
In addition to the contact named above, James R. Sweetman, Jr.,
Assistant Director; Jamey A. Collins; Neil J. Doherty; Jennifer K.
Echard; Wilfred B. Holloway; Ethan J. Iczkovitz; Frank Maguire; Karen
A. Richey; Glenn D. Slocum; and Amos A. Tevelow made key contributions
to this report.
FOOTNOTES
[1] These documents included published literature on cost estimation
from the Society of Cost Estimating and Analysis, the Department of the
U.S. Army, the Office of the Secretary of Defense, the National
Aeronautics and Space Administration, and the Department of Energy.
[2] Sprint Corporation merged with Nextel Communications, Inc., to form
Sprint Nextel Corporation in August 2005.
[3] MCI merged with Verizon to form Verizon Business in January 2006.
[4] In August 2001, GSA allowed contractors that had been awarded local
telecommunications contracts in selected metropolitan areas, through
GSA's Metropolitan Area Acquisition program, to offer long-distance
services on the FTS2001 contracts. This process is termed "crossover."
[5] GSA negotiated sole-source contracts for a 24-month base period
with three 6-month optional periods, for a total of 42 months.
[6] The IT Fund was authorized by the Paperwork Reduction
Reauthorization Act of 1986, Public Law 99-500 and 99-691; 40 U.S.C.
322.
[7] A revolving fund is established by Congress to finance a cycle of
businesslike operations through amounts received by the fund. A
revolving fund charges for the sale of products or services and uses
the proceeds to finance its spending, usually on a self-sustaining
basis. Instead of recording the collections in receipt accounts, the
budget records the collections and the outlays of revolving funds in
the same account. A revolving fund is a form of permanent
appropriation.
[8] General Services Administration Modernization Act, Pub. L. No. 109-
313, 120 Stat. 1734 (2006).
[9] GSA's supply programs provide agencies a source for commercial
products and services such as office supplies, vehicle purchasing,
travel, and furniture.
[10] For example, GSA provided data on contractor support costs
incurred during the previous transition.
[11] Public Law No. 109-313.
[12] This included documentation available on a Transition Manager Web
site established by GSA, such as presentations, meeting minutes, and
FTS2001 lessons learned.
[13] A Monte Carlo simulation allows the model's parameters to vary
according to their associated probability distribution. The result is a
set of estimated probabilities of achieving alternative outcomes, given
the uncertainty in the underlying parameters.
[14] See appendix I, footnote 2, for a description of Monte Carlo
simulation.
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site (www.gao.gov). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site.
To have GAO e-mail you a list of newly posted products every afternoon,
go to www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125
Washington, D.C. 20548:
Public Affairs:
Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548:
