Customs Service Modernization
Third Expenditure Plan Meets Legislative Conditions, but Cost Estimating Improvements Needed Gao ID: GAO-02-908 August 9, 2002The U.S. Customs Service has begun a multiyear, multibillion-dollar project: the Automated Commercial Environment (ACE), a new import processing system that is planned to support effective and efficient movement of goods into the United States. By congressional mandate, Customs' expenditure plans for ACE must meet certain conditions, including being reviewed by GAO. This report addresses whether Customs' latest plan satisfies these conditions and provides observations about the plan and Customs' efforts to implement GAO's open recommendations for improving ACE management. Customs' May 2002 ACE expenditure plan is the third in a series of legislatively required plans. This plan provides for the design, development, and deployment of the second release of the first of four planned ACE increments. The plan also meets the legislative conditions governing investment in ACE that Congress imposed on Customs. Since 1999, GAO has reported on Customs' management of ACE and made a series of recommendations to correct deficiencies. Customs currently has efforts under way to respond to all of GAO's recommendations. One of these deficiencies that affects the third expenditure plan is Customs' lack of effective cost estimating capabilities. Specifically, the cost estimate in the third expenditure plan is questionable because of limitations in how it was derived and inconsistencies between this estimate and an independent estimate developed by a Customs contractor. Customs is currently determining the cause of the variance in the estimates.
GAO-02-908, Customs Service Modernization: Third Expenditure Plan Meets Legislative Conditions, but Cost Estimating Improvements Needed
This is the accessible text file for GAO report number GAO-02-908
entitled 'Customs Service Modernization: Third Expenditure Plan Meets
Legislative Conditions, but Cost Estimating Improvements Needed' which
was released on August 09, 2002.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products‘ accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
Report to Congressional Committees:
August 2002:
Customs Service Modernization:
Third Expenditure Plan Meets Legislative Conditions, but Cost
Estimating Improvements Needed:
GAO-02-908:
Contents:
Letter:
Agency Comments:
Appendixes:
Appendix I: Custom‘s Third Automated Commercial Evironment (ACE):
Appendix II: Comments from the U.S. Customs Service:
Abbreviations:
ACE: Automated Commercial Environment:
ACS: Automated Commercial System:
CMM: Capability Maturity Model:
CMO: Customs Modernization Office:
COTS: commercial, off the shelf:
eCP: e-Customs Partnership:
FMCSA: Federal Motor Carrier Safety Administration:
GAO: General Accounting Office:
IBM: International Business Machines:
ITDS: International Trade Data System:
OMB: Office of Management and Budget:
SA-CMM: Software Acquisition Capability Maturity Model:
SEI: Software Engineering Institute:
Highlights:
August 2002:
DRAFT United States General Accounting Office DRAFT:
CUSTOMS SERVICE MODERNIZATION:
Third Expenditure Plan Meets Legislative Conditions, but Cost
Estimating Improvements Needed:
Why GAO Did This Study:
The U.S. Customs Service has begun a multiyear, multibillion-dollar
project: the Automated Commercial Environment (ACE), a new import
processing system that is planned to support effective and efficient
movement of goods into the United States. By congressional mandate,
Customs‘ expenditure plans for ACE must meet certain conditions,
including being reviewed by GAO. This study addresses whether Customs‘
latest plan satisfies these conditions and provides observations about
the plan and Customs‘ efforts to implement GAO‘s open recommendations
for improving ACE management.
Highlights of GAO-02-908, a report to the Subcommittee on Treasury and
General Government, Senate Committee on Appropriations, and the
Subcommittee on Treasury, Postal Service, and General Government, House
Committee on Appropriations.
What GAO Found:
Customs‘ May 2002 ACE expenditure plan is the third in a series of
legislatively required plans. This plan provides for the design,
development, and deployment of the second release of the first of four
planned ACE increments. The plan also meets the legislative conditions
governing investment in ACE that Congress imposed on Customs.
Since 1999, GAO has reported on Customs‘ management of ACE and made a
series of recommendations to correct deficiencies. Customs currently
has efforts under way to respond to all of GAO‘s recommendations. One
of these deficiencies that affects the third expenditure plan is
Customs‘ lack of effective cost estimating capabilities. Specifically,
the cost estimate in the third expenditure plan is questionable because
of limitations in how it was derived and inconsistencies between this
estimate and an independent estimate developed by a Customs contractor.
Customs is currently determining the cause of the variance in the
estimates.
The following table compares Customs‘ expenditure plan cost estimate
with the Software Engineering Institute‘s seven questions for
determining the reliability of an estimate:
Table:
Question: Are the objectives of the estimate clear and correct?; Yes:
No:
[Empty].
Question: Has the task been appropriately sized?; Yes: [Empty]; No:
Question: Are the estimated cost and schedule consistent with
demonstrated accomplishments on other projects?; Yes: [Empty]; No:
Question: Have the factors that affect the estimate been identified and
explained?; Yes: [Empty]; No:
Question: Have steps been taken to ensure the integrity of the
estimating process?; Yes: [Empty]; No:
Question: Is the organization‘s historical evidence capable of
supporting a reliable estimate?; Yes: [Empty]; No:
Question: Has the situation changed since the estimate was prepared?;
Yes: [Empty]; No:
[End of table]
Source: GAO analysis.
What GAO Recommends:
Because GAO has an open recommendation to Customs for correcting its
cost estimating deficiencies, which Customs reports will be fully
implemented in December 2002, GAO is not making another recommendation
at this time. Customs concurred with GAO‘s report and described actions
that it is taking to improve its expenditure plan cost estimating.
This is a test for developing Highlights for a GAO report. The full
report, including GAO‘s objectives, scope, methodology, and analysis is
available at www.gao.gov/cgi-bin/getrpt?GAO-02-908. For additional
information about the report, contact Randolph C. Hite (202-512-3439).
To provide comments on this test highlights, contact Keith Fultz (202-
512-3200) or E-mail HighlightsTest@gao.gov.
Letter:
August 9, 2002:
The Honorable Byron L. Dorgan
Chairman
The Honorable Ben Nighthorse Campbell
Ranking Minority Member
Subcommittee on Treasury and General Government
Committee on Appropriations
United States Senate:
The Honorable Ernest J. Istook, Jr.
Chairman
The Honorable Steny H. Hoyer
Ranking Minority Member
Subcommittee on Treasury, Postal Service,
and General Government
Committee on Appropriations
House of Representatives:
In May 2002, the U.S. Customs Service submitted to Congress its third
expenditure plan seeking release of $190.2 million for its Automated
Commercial Environment (ACE) project, pursuant to Customs‘ fiscal year
2002 appropriation.[Footnote 1] ACE is to be Customs‘ new import
processing system and the first project under the Customs Modernization
Program. As required by the act, we reviewed the expenditure plan. Our
objectives were to (1) determine whether the third ACE expenditure plan
satisfies the legislative conditions, (2) determine whether the plan is
consistent with our open ACE recommendations, and (3) provide
observations about the plan and Customs‘ management of ACE.
On June 28, 2002, we briefed your offices on the results of this
review. This report officially transmits the results of our work. The
full briefing, including our scope and methodology, is reprinted as
appendix I.
Concerning our first objective, Customs‘ expenditure plan satisfies the
legislative conditions specified in the appropriations act. That is,
the plan provides for (1) meeting the capital planning and investment
control review requirements of the Office of Management and Budget
(OMB); (2) complying with Customs‘ enterprise architecture;[Footnote 2]
and (3) complying with federal acquisition rules, requirements,
guidelines, and systems acquisition management practices. Further, the
plan was reviewed and approved by the Joint Capital Investment Review
Board[Footnote 3] and OMB.
Customs is also making progress in addressing our open recommendations,
which are as follows:
1. Justify and make ACE investment decisions incrementally.
2. Before building each ACE release, certify to Customs‘ appropriations
subcommittees that Customs‘ enterprise architecture has been
sufficiently extended and updated.
3. Develop and implement a Customs Modernization Office (CMO) human
capital management strategy.
4. Develop and implement a rigorous and analytically verifiable cost
estimating program that embodies the tenets of effective estimating as
defined in the institutional and project-specific estimating guidance
of the Software Engineering Institute (SEI).
5. Limit future expenditure plan requests for management reserve funds
to 10 percent of the total funds requested for the ACE program, or
adequately justify any management reserve requests in excess of 10
percent.
6. Implement certain process controls embodied primarily in the second
level of SEI‘s Software Acquisition Capability Maturity Model (SA-
CMM),[Footnote 4] and by September 30, 2002, assess and report to
Customs‘ appropriations subcommittees on the CMO‘s software acquisition
process maturity.
7. Address the risks associated with the accelerated ACE acquisition
strategy and report to Customs‘ appropriations subcommittees on the
strategy and plans for mitigating the risks associated with this
strategy.
Finally, we determined that the cost estimate in the third expenditure
plan is questionable because Customs is still in the process of
addressing our recommendation (see recommendation 4) for having an
effective cost estimating program. While the full impact of the
weaknesses in the process used to derive the estimate was
indeterminable, at least one limitation (an inflated labor rate) led to
an overestimate of about $36 million for designing, developing, and
deploying ACE increment 1 release 2. Because the expenditure plan
includes a management reserve of 10 percent of the funding requirement,
this means that the management reserve is also overstated by about $3.6
million.
According to an ACE prime contractor executive, the overstatement is
offset by a corresponding understatement in the estimated number of
labor hours, which the official attributed to Customs‘ not having a
complete set of requirements at the time that the plan‘s estimate was
derived. However, Customs also arranged for an independent cost
estimate by another contractor, which led to an estimate that was $77.9
million lower than the expenditure plan cost estimate. Customs and
support contractor officials stated that this independent estimate is
incomplete and reflects a misunderstanding by the cost estimate
contractor of the scope of work to be performed. Customs is currently
determining the cause of the variance.
Because we have an open recommendation to Customs for correcting its
cost estimating deficiencies, which Customs reports will be fully
implemented in December 2002, we are not making a recommendation at
this time.
Agency Comments:
In written comments on a draft of this report, the Acting Director,
Office of Planning, U.S. Customs Service, concurred with the report and
described actions that are being taken to improve expenditure plan cost
estimating. These comments are reprinted in appendix II.
We are sending copies of this report to the Chairmen and Ranking
Minority Members of other Senate and House committees and subcommittees
that have authorization and oversight responsibilities for the Customs
Service. We are also sending copies to the Secretary of the Treasury,
the Commissioner of the Customs Service, and the Director of OMB. We
also will make copies available to others upon request. In addition,
the report will be available at no charge on the GAO Web site at http:/
/www.gao.gov.
Should you or your staff have any questions on matters discussed in
this report, please contact me at (202) 512-3439. I can also be reached
by E-mail at HiteR@gao.gov. Key contributors to this report were Mark
Bird, Barbara Collier, Tamra Goldstein, Randolph Tekeley, Scott Pettis,
and Aaron Thorne.
Randolph C. Hite
Director, Information Technology Architecture
and Systems Issues:
Signed by Randolph C. Hite:
[End of section]
Appendix I: Custom‘s Third Automated Commercial Environment (ACE)
Expenditure Plan:
[See PDF for image]
[End of figure]
[End of section]
Appendix II: Comments from the U.S. Customs Service:
U.S. Customs Service:
Memorandum:
DATE: July 31, 2002:
MEMORANDUM FOR RANDOLPH C. HITE:
U.S. GENERAL ACCOUNTING OFFICE:
FROM: Acting Director Office of Planning:
SUBJECT: Draft Audit Report on the United States Customs Service‘s
Third
Automated Commercial Environment (ACE) Expenditure Plan:
Thank you for providing us with a copy of your draft report entitled
’Customs Service Modernization: Third Expenditure Plan Meets
Legislative Conditions, but Cost Estimating Improvements Needed“ and
the opportunity to discuss the issues in this report.
We are pleased that the General Accounting Office (GAO) recognizes the
progress we are making to satisfy the recommendations from their
previous reviews. We will continue to address all open recommendations
aggressively.
GAO gives particular attention to the cost estimation methodology
supporting the third expenditure plan, noting that we are still in the
process of addressing their prior recommendation on establishing an
effective cost estimating program. We continue to pursue efforts to
improve our cost estimates supporting expenditure plans. Our fourth
expenditure plan will be supported by a detailed internal review of the
eCP inputs and independent expenditure plan cost estimates. Each of
these efforts will be based on the Systems Engineering Institute‘s
(SEI) guidance for validating the reliability of an estimate. We are
confident that these efforts will provide our stakeholders with greater
confidence in the cost estimates supporting expenditure plan 4.
Further, Customs is proceeding with acquiring the services of an
independent contractor to develop and maintain a Modernization Life
Cycle Cost Model which will then support all planning activities. This
model will also be based on the SEI estimating guidance.
If you have any questions regarding these comments, please contact Ms.
Michele Donahue at (202) 927-0957.
Christine E. Gaugler:
Signed by Christine E. Gaugler:
[End of Section]
FOOTNOTES
[1] Pub. L. No. 107-67, 115 Stat. 514, 520 (2001).
[2] An enterprise architecture is an institutional blueprint for
guiding and constraining investments in business process change and
systems.
[3] To expedite reviews of ACE expenditure plans, the Customs and
Treasury Investment Review Boards were consolidated to form the Joint
Capital Investment Review Board, which first met in November 2001.
[4] Capability Maturity ModelSM (CMM) is a service mark of Carnegie
Mellon University, and CMM is registered in the U.S. Patent and
Trademark Office. The SA-CMM identifies key process areas that are
necessary to effectively manage software-intensive system
acquisitions. Level 2 is the second level of the SA-CMM‘s five-level
scale; achieving this level means that an organization has the software
acquisition rigor and discipline to repeat project successes.
GAO‘s Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO‘s commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO‘s Web site (www.gao.gov) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as ’Today‘s Reports,“ on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select ’Subscribe to daily E-mail alert for newly
released products“ under the GAO Reports heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone: Voice: (202) 512-6000
TDD: (202) 512-2537
Fax: (202) 512-6061
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800
U.S. General Accounting Office, 441 G Street NW, Room 7149
Washington, D.C. 20548:
