Information Technology
Customs Has Made Progress on Automated Commercial Environment System, but It Faces Long-Standing Management Challenges and New Risks Gao ID: GAO-06-580 May 31, 2006The Department of Homeland Security (DHS) is conducting a multiyear, multibillion-dollar acquisition of a new trade processing system, planned to support the movement of legitimate imports and exports and strengthen border security. By congressional mandate, plans for expenditure of appropriated funds on this system, the Automated Commercial Environment (ACE), must meet certain conditions, including GAO review. This study addresses whether the fiscal year 2006 plan satisfies these conditions; it also describes the status of DHS's efforts to implement prior GAO recommendations for improving ACE management, and provides observations about the plan and DHS's management of the program.
The fiscal year 2006 ACE expenditure plan, including related program documentation and program officials' statements, either satisfied or partially satisfied the legislative conditions imposed by the Congress; however, more can be done to better address several aspects of these conditions. In addition, DHS has addressed some recommendations that GAO has previously made, but progress has been slow in addressing several recommendations aimed at strengthening ACE management. For example, DHS has more to do to implement the recommendation that it establish an ACE accountability framework that, among other things, ensures that expenditure plans report progress against commitments made in prior plans. Implementing a performance and accountability framework is important for ensuring that promised capabilities and benefits are delivered on time and within budget. In addition, describing progress against past commitments is essential to permit meaningful congressional oversight. Among GAO's observations about the ACE program and its management are several related to the need to effectively set and use performance goals and measures. Although the program set performance goals, these targets were not always realistic. For example, in fiscal year 2005, the program set a target that 11 percent of all Customs and Border Protection (CBP) employees would use ACE. However, this target does not reflect the fact that many CBP employees will never need to use the system. Additionally, the program has established 6 program goals, 11 business results, 23 benefits, and 17 performance measures, but the relationships among these are not fully defined or adequately aligned with each other. For example, not every goal has defined benefits, and not every benefit has an associated performance measure. Without realistic ACE performance measures and targets that are aligned with the overall program goals and desired results, DHS will be challenged in its efforts to establish an accountability framework for ACE that will help to ensure that the program delivers its expected benefits. In addition, DHS plans to develop several increments, referred to as "releases," concurrently; in the past, such concurrency has led to cost overruns and schedule delays because releases contended for the same resources, and resources that were to be used on later releases were diverted to earlier ones. However, because of DHS's belief that such concurrent development will allow it to deliver ACE functionality sooner, it is reintroducing the same problems that resulted in past shortfalls.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-06-580, Information Technology: Customs Has Made Progress on Automated Commercial Environment System, but It Faces Long-Standing Management Challenges and New Risks
This is the accessible text file for GAO report number GAO-06-580
entitled 'Information Technology: Customs Has Made Progress on
Automated Commercial Environmental System, but It Faces Long-Standing
Management Challenges and New Risks' which was released on May 31,
2006.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Committees:
United States Government Accountability Office:
GAO:
May 2006:
Information Technology:
Customs Has Made Progress on Automated Commercial Environment System,
but It Faces Long-Standing Management Challenges and New Risks:
GAO-06-580:
Contents:
GAO Highlights:
Highlights of GAO-06-580, a report to Subcommittees on Homeland
Security, Senate and House Committees on Appropriations.
Why GAO Did This Study:
The Department of Homeland Security (DHS) is conducting a multiyear,
multibillion-dollar acquisition of a new trade processing system,
planned to support the movement of legitimate imports and exports and
strengthen border security. By congressional mandate, plans for
expenditure of appropriated funds on this system, the Automated
Commercial Environment (ACE), must meet certain conditions, including
GAO review. This study addresses whether the fiscal year 2006 plan
satisfies these conditions; it also describes the status of DHS‘s
efforts to implement prior GAO recommendations for improving ACE
management, and provides observations about the plan and DHS‘s
management of the program.
What GAO Found:
The fiscal year 2006 ACE expenditure plan, including related program
documentation and program officials‘ statements, either satisfied or
partially satisfied the legislative conditions imposed by the Congress;
however, more can be done to better address several aspects of these
conditions. In addition, DHS has addressed some recommendations that
GAO has previously made, but progress has been slow in addressing
several recommendations aimed at strengthening ACE management. For
example, DHS has more to do to implement the recommendation that it
establish an ACE accountability framework that, among other things,
ensures that expenditure plans report progress against commitments made
in prior plans. Implementing a performance and accountability framework
is important for ensuring that promised capabilities and benefits are
delivered on time and within budget. In addition, describing progress
against past commitments is essential to permit meaningful
congressional oversight.
Among GAO‘s observations about the ACE program and its management are
several related to the need to effectively set and use performance
goals and measures. Although the program set performance goals, these
targets were not always realistic. For example, in fiscal year 2005,
the program set a target that 11 percent of all Customs and Border
Protection (CBP) employees would use ACE. However, this target does not
reflect the fact that many CBP employees will never need to use the
system. Additionally, the program has established 6 program goals, 11
business results, 23 benefits, and 17 performance measures, but the
relationships among these are not fully defined or adequately aligned
with each other. For example, not every goal has defined benefits, and
not every benefit has an associated performance measure. Without
realistic ACE performance measures and targets that are aligned with
the overall program goals and desired results, DHS will be challenged
in its efforts to establish an accountability framework for ACE that
will help to ensure that the program delivers its expected benefits.
In addition, DHS plans to develop several increments, referred to as
’releases,“ concurrently; in the past, such concurrency has led to cost
overruns and schedule delays because releases contended for the same
resources, and resources that were to be used on later releases were
diverted to earlier ones. However, because of DHS‘s belief that such
concurrent development will allow it to deliver ACE functionality
sooner, it is reintroducing the same problems that resulted in past
shortfalls.
What GAO Recommends:
To help ensure the success of ACE, GAO recommends, among other things,
that DHS develop realistic ACE performance measures and targets; align
these with ACE program goals, benefits, and desired business outcomes;
and minimize concurrent development of ACE releases. DHS agreed with
GAO‘s recommendations and described actions that it has under way and
planned to address them.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-580].
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Randolph C. Hite at (202)
512-3429 or hiter@gao.gov.
[End of Section]
Letter:
Compliance with Legislative Conditions:
Status of Our Open Recommendations:
Observations about ACE Management:
Conclusions:
Recommendations for Executive Action:
Agency Comments:
Appendix I: Briefing to Subcommittees on Homeland Security, House and
Senate Committees on Appropriations:
Appendix II: Comments from the Department of Homeland Security:
Appendix III: Contact and Staff Acknowledgments:
Abbreviations:
ACE: Automated Commercial Environment:
CBP: Customs and Border Protection:
CBPMO: Customs and Border Protection Modernization Office:
CIO: chief information officer:
CMU: Carnegie Mellon University:
DHS: Department of Homeland Security:
EA: enterprise architecture:
EVM: earned value management:
IT: information technology:
IV&V: independent verification and validation:
OMB: Office of Management and Budget:
ORR: operational readiness review:
SA-CMM®: Software Acquisition Capability Maturity Model:
SEI: Software Engineering Institute:
US-VISIT: United States Visitor and Immigrant Status Indicator
Technology:
United States Government Accountability Office:
Washington, DC 20548:
May 31, 2006:
The Honorable Judd Gregg:
Chairman:
The Honorable Robert C. Byrd:
Ranking Minority Member:
Subcommittee on Homeland Security:
Committee on Appropriations:
United States Senate:
The Honorable Harold Rogers:
Chairman:
The Honorable Martin Olav Sabo:
Ranking Minority Member:
Subcommittee on Homeland Security:
Committee on Appropriations:
House of Representatives:
In February 2006, U.S. Customs and Border Protection (CBP), within the
Department of Homeland Security (DHS), submitted to the Congress its
fiscal year 2006 expenditure plan for the Automated Commercial
Environment (ACE) program. ACE is to be CBP's new import and export
processing system. The program's goals include:
* supporting border security by enhancing analysis and information
sharing with other government agencies and:
* streamlining time-consuming and labor-intensive tasks for CBP
personnel and the trade community through the provision of a single Web-
based interface.
DHS currently plans to acquire and deploy ACE in 11 increments,
referred to as "releases," in approximately 8.5 years. The first three
releases are deployed and operating, and the fourth release is being
deployed. Other releases are in various states of definition and
development. For the ACE life-cycle, the risk-adjusted cost estimate is
about $2.8 billion; through fiscal year 2005, ACE-appropriated funding
has been about $1.6 billion.
The Department of Homeland Security Appropriations Act of 2006
[Footnote 1] appropriates about $320 million for ACE. The act states
that DHS may not obligate any funds for ACE until it submits for
approval to the House and Senate Committees on Appropriations a plan
for expenditure that:
1. meets the capital planning and investment control review
requirements established by the Office of Management and Budget (OMB),
including Circular A-11, part 7;[Footnote 2]
2. complies with DHS's enterprise architecture;
3. complies with the acquisition rules, requirements, guidelines, and
systems-acquisition management practices of the federal government;
4. includes a certification by the DHS Chief Information Officer that
an independent verification and validation agent is currently under
contract;
5. is reviewed and approved by the DHS Investment Review Board
(IRB),[Footnote 3] the Secretary of Homeland Security, and OMB; and:
6. is reviewed by GAO.
As required by the act, we reviewed the ACE fiscal year 2006
expenditure plan. Our objectives were to (1) determine whether the
expenditure plan satisfies certain legislative conditions, (2)
determine the status of our open ACE recommendations, and (3) provide
any other observations about the expenditure plan and DHS's management
of the ACE program.
On March 10, 2006, we briefed your offices on the results of this
review. This report transmits the results of our work. The full
briefing, including our scope and methodology, can be found in appendix
I.
Compliance with Legislative Conditions:
The legislative conditions that the Congress placed on the use of
fiscal year 2006 ACE appropriated funds have been either partially or
fully satisfied by the latest expenditure plan and related program
documentation and activities. However, more can be done to better
address several aspects of these conditions. For example:
One legislative condition states that the plan should meet OMB's
capital planning and investment control review requirements, which
include addressing security and privacy issues. However, a privacy
impact assessment[Footnote 4] for ACE has been in draft for several
months and is not yet approved. Another capital planning and investment
control review requirement is that performance goals and measures be
provided in the business case for ACE. Although CBP describes selected
performance goals and measures, the goals (i.e., targets) are not
always realistic (we provide further discussion of this issue later in
this report).
According to another legislative condition, the expenditure plan must
comply with DHS's enterprise architecture. However, DHS does not have a
documented methodology for evaluating programs for compliance with its
enterprise architecture, other than relying on the professional
expertise of its staff.
According to a third legislative condition, the DHS Chief Information
Officer is to certify that an independent verification and validation
(IV&V) agent is under contract. Although DHS satisfied this condition,
the scope of the IV&V contractor's activities is not consistent with
the operative industry standard, which states that IV&V should extend
to key system products and development processes.[Footnote 5]
Status of Our Open Recommendations:
CBP has addressed some recommendations, while progress has been slow on
others. Each recommendation, along with the status of actions to
address it, is summarized below.
Ensure that future expenditure plans are based on cost estimates that
are reconciled with independent cost estimates.
Complete.[Footnote 6] In October 2005, CBP, with contractor support,
compared the program plan cost estimate with the independent cost
estimate. According to the analysis performed, the two estimates are
consistent.
Develop and implement a rigorous and analytically verifiable cost
estimating program that embodies the tenets of effective estimating, as
defined in the institutional and project-specific estimating models
developed by the Software Engineering Institute (SEI).[Footnote 7]
In progress. CBP has taken steps such as (1) hiring a contractor to
develop cost estimates (including contract task order cost estimates)
that are independent of CBP's estimates, and (2) tasking a support
contractor with evaluating both the independent and CBP estimates
against the criteria defined by SEI. According to the results of the
support contractor's evaluation, the independent estimates satisfied
the SEI criteria; CBP's estimates largely satisfied the criteria.
However, according to the support contractor, CBP's cost estimating had
limitations. First, the CBP estimate did not adequately consider past
projects in its cost and schedule estimates. In addition, the CBP
estimate was an aggregation of estimates developed separately for three
ACE components, each according to a different cost estimating
methodology; the support contractor advised against this approach,
recommending that component estimates be based on the same methodology.
Immediately develop and implement a human capital management strategy
that provides both near-and long-term solutions to program office human
capital capacity limitations, and report quarterly to the
Appropriations Committees on the progress of efforts to do so.
In progress. CBP has expanded its contractor and government workforce
dedicated to the ACE program by merging staff assigned to trade-related
legacy systems with the ACE program staff. In addition, it is beginning
to use subject matter experts from existing field operations advisory
boards to help program officials define requirements for future
releases. However, it does not have a documented human capital strategy
covering its ACE program.
Have future ACE expenditure plans specifically address any proposals or
plans, whether tentative or approved, for extending and using ACE
infrastructure to support other homeland security applications,
including any impact on ACE of such proposals and plans.
In progress. The expenditure plan describes steps both planned and
under way to ensure that ACE infrastructure supports both ACE and other
homeland security applications. For example, it states that both ACE
and the United States Visitor and Immigrant Status Indicator
Technology[Footnote 8] (US-VISIT) program should conform to the DHS
enterprise architecture, which is to define standard shared services
that the two systems can request. Such a services oriented architecture
is intended to promote reuse, as well as reducing overlap and
duplication.
Define measures, and collect and use associated metrics, for
determining whether prior and future program management improvements
are successful.
In progress. CBP continues to make changes that are intended to improve
overall program management, but it has not consistently defined
measures to determine whether the changes are successful. For example,
CBP has reorganized its Office of Information Technology; this
reorganization is intended to improve program management by providing
(1) enhanced government oversight of ACE development, (2) better
definition of requirements for future ACE releases, and (3) faster and
cheaper delivery of ACE capabilities. However, program officials told
us that they have not established measures or targets for determining
whether the reorganization is providing these benefits.
Define and implement an ACE accountability framework that fulfills
several conditions:
1. The framework should cover all program commitment areas, including
key expected or estimated system (a) capabilities, use, and quality;
(b) benefits and mission value; (c) costs; and (d) milestones and
schedules.
In progress. CBP has prepared an initial version of an accountability
framework that it intends to improve as it proceeds. The framework is
built around measuring progress against costs, milestones, schedules,
and risks for select releases; however, the benefit measurement has not
been well defined, and the performance targets are not always
realistic.
2. The framework should ensure currency, relevance, and completeness of
all program commitments made to the Congress in expenditure plans.
In progress. The fiscal year 2006 expenditure plan includes inaccurate,
dated, and incomplete information and omits other relevant information.
For example, the plan did not include information regarding CBP's
decision to eliminate the dependencies among the screening and
targeting releases and the cargo releases, and to take advantage of the
capabilities of its existing Automated Targeting System.[Footnote 9]
3. The framework should ensure reliable data that are relevant to
measuring progress against commitments.
In progress. The data that CBP uses to measure progress against
commitments are not consistently reliable. For example, data in the
defect tracking system show that defects in Release 4 (which is now
operational) have not been closed; however, program officials told us
that many of these defects have been resolved.
4. The framework should ensure that future expenditure plans report
progress against commitments contained in prior expenditure plans.
In progress. The current expenditure plan does not adequately describe
progress against commitments made in previous plans. For example, the
plan provides a summary of the funding requested in each of the
previous six expenditure plans, but it does not provide information on
whether these funding amounts were actually expended or obligated as
planned.
5. The framework should ensure that criteria for exiting key readiness
milestones adequately consider indicators of system maturity, such as
the severity of open defects.
In progress. ACE milestone exit criteria provide for addressing the
risk associated with severe defects that are unresolved. Using these
criteria, CBP passed several release milestones with severe defects
still open. However, CBP officials were unable to provide us with any
documentation on how they assessed the inherent risks of passing these
milestones with open severe defects.
6. The framework should ensure clear and unambiguous delineation of the
respective roles and responsibilities of the government and the prime
contractor.
Complete. The current ACE program plan describes general roles and
responsibilities for the government and the prime contractor. More
detailed roles have been documented in a roles and responsibilities
matrix that assigns primary responsibility for each activity.
Report quarterly to the House and Senate Appropriations Committees on
efforts to address our open recommendations.
In progress. CBP submitted quarterly reports to both Committees on its
efforts to address our open recommendations; however, progress in
addressing our recommendations was not always reported accurately.
Observations about ACE Management:
We have several observations about the development of ACE releases, as
well as several more concerning the performance of ACE releases that
are deployed and operating.
ACE development: Steps have been taken to address a past pattern of ACE
release shortfalls, but new release management weaknesses are emerging.
As we have previously observed, CBP established a pattern of borrowing
resources from future releases to address problems with the quality of
earlier releases; this led to schedule delays and cost overruns. This
pattern has continued with the most recently deployed cargo release,
which developed problems that caused delays with a subsequent screening
and targeting release.[Footnote 10] CBP took steps to mitigate this
problem by eliminating the dependencies between the cargo releases and
the screening and targeting releases.
However, CBP's planned schedule for developing additional releases
includes a significant level of concurrence, because of CBP's interest
in delivering ACE functionality sooner. Such concurrence between ACE
release activities has led to cost overruns and schedule delays in the
past. Thus, the revised ACE plans and actions are potentially
reintroducing the same problems that produced past shortfalls.
We made several specific observations related to these weaknesses,
including the following:
On two recent releases, key milestones were passed despite unresolved
severe defects. Officials told us that the risk of proceeding did not
outweigh the need to get the releases to users, and thereby gaining
user acceptance and feedback. However, the risks were not documented
and formally managed.
Concurrence in developing early ACE releases caused schedule slips and
cost overruns. Despite these experiences, CBP has established a risky
plan that involves considerable overlap across the development
schedules for three future releases.
Although the use of earned value management (EVM) is an OMB
requirement, it was not being used to manage the development of two
recent releases.[Footnote 11] For example, CBP discontinued use of EVM
on one release because this method was not familiar to staff who were
transferred to work on the program.
ACE operations: Operational performance has been mixed, and mission
impact is unclear.
ACE releases one through four are in operation. To date, these
releases' operational performance has been uneven. For example, ACE has
largely been meeting its goals for being available and responsive in
processing virtually all daily transactions, and has decreased truck
processing times at some ports. However, ACE is not being used by as
many CBP and trade personnel as was expected, and truck processing
times at other ports have increased. Moreover, overall user
satisfaction has been low.
In addition, ACE goals, expected mission benefits, and performance
measures are not fully defined and adequately aligned with each other.
For example, not every goal has defined benefits, every benefit is
defined only in terms of efficiency gains, not every benefit has an
associated business result, and not every benefit and business result
has associated performance measures.
Further, where performance measures have been defined, the associated
targets are not always realistic. For example, the performance target
in fiscal year 2005 for ACE usage was that 11 percent of all CBP
employees would use ACE. However, that many CBP employees will never
need to use the system. This performance target does not reflect that.
Because performance measures are not always realistic or aligned with
program goals and benefits, it is unclear whether ACE has realized--or
will realize--the mission value that it was intended to bring to CBP's
and other agencies' trade-and border security-related operations.
Conclusions:
The legislative conditions that the Congress placed on the use of
fiscal year 2006 ACE appropriated funds have been either partially or
fully satisfied by the latest expenditure plan and related program
documentation and activities. Nevertheless, more can be done to better
address several aspects of these conditions, such as ensuring that the
program's privacy impact assessment is approved, measuring ACE
performance and results, ensuring architectural alignment, and
employing effective IV&V practices. Given that the legislative
conditions are collectively intended to promote accountability and
increase the chances of program success, it is important that each
receives DHS's full attention.
Also important to ACE's success is the swift and complete
implementation of the recommendations that we have previously made to
complement the legislative conditions and improve program management,
performance, and accountability. In this regard, some recommendations
have been addressed, while progress has been slow on others, such as:
* accurately reporting to the Appropriations Committees on CBP's
progress in implementing our prior recommendations;
* developing and implementing a strategic approach to meeting the
program's human capital needs;
* using criteria for exiting key milestones that adequately consider
indicators of system maturity, such as severity of open defects and the
associated risks; and:
* developing and implementing a performance and accountability
framework for ensuring that promised capabilities and benefits are
delivered on time and within budget.
To its credit, CBP has taken several steps to stem the pattern of cost,
schedule, and performance shortfalls that it experienced on early ACE
releases. However, future releases are unlikely to realize the impact
of these steps because revised ACE plans and actions are reintroducing
the same pattern that led to early release shortfalls. This pattern
includes not formally and transparently considering, and proactively
addressing, the risks associated with passing key release milestones
with known severe defects; building considerable overlap and
concurrence in the development schedules of releases that will contend
for the same resources; and not performing EVM on all releases. If this
pattern continues, the prospects for a successful program will be
diminished.
Although availability and responsiveness targets are largely being met
and long-standing help desk limitations are being addressed, the
prospects for a successful program nevertheless remain unclear. The
true measure of ACE's success is arguably the mission value that it
brings to CBP's and other agencies' trade-and border security-related
operations and users. Such value depends both on the operational
performance of ACE and on CBP's ability to demonstrate that this
performance is achieving program goals, delivering expected benefits,
and producing desired business results. At this juncture, however,
neither the system's performance nor its value is clear because of
several factors: the operational performance of deployed releases has
been mixed; users' satisfaction has been low; the relationships among
goals, benefits, and desired business outcomes are not evident; and the
range of measures needed to create a complete and realistic picture of
ACE's performance is missing.
In summary, a number of ACE activities have been and are being done
well; these have contributed to the program's progress to date and will
go a long way in determining the program's ultimate success. However,
it will be important for CBP to effectively address long-standing ACE
management challenges along with emerging problems. Until it does so,
ACE will remain a risky program.
Recommendations for Executive Action:
To assist CBP in managing ACE--and increasing the chances that it will
deliver required capabilities on time and within budget, demonstrating
promised mission benefits and results--we recommend that the Secretary
of Homeland Security direct the appropriate departmental officials to
fully address those legislative conditions associated with having an
approved privacy impact assessment and ensuring architectural
alignment.
We also recommend that the Secretary, through CBP's Acting
Commissioner, direct the Assistant Commissioner for Information and
Technology to:
* fully address those legislative conditions associated with measuring
ACE performance and results and employing effective IV&V practices;
* accurately report to the appropriations committees on CBP's progress
in implementing our prior recommendations;
* include in the June 30, 2006, quarterly update report to the
appropriations committees a strategy for managing ACE human capital
needs and the ACE framework for managing performance and ensuring
accountability;
* document key milestone decisions in a way that reflects the risks
associated with proceeding with unresolved severe defects and provides
for mitigating these risks;
* minimize the degree of overlap and concurrence across ongoing and
future ACE releases, and capture and mitigate the associated risks of
any residual concurrence;
* use EVM in the development of all existing and future releases;
* develop the range of realistic ACE performance measures and targets
needed to support an outcome-based, results-oriented accountability
framework, including user satisfaction with ACE; and:
* explicitly align ACE program goals, benefits, desired business
outcomes, and performance measures.
Agency Comments:
In written comments on a draft of this report signed by the Director,
Departmental GAO/OIG Liaison, DHS agreed with our findings concerning
progress in addressing our prior recommendations, and it agreed with
the recommendations in this report. DHS also described actions that it
has under way and planned to address the recommendations. The
department's comments are reprinted in appendix II.
We are sending copies of this report to the Chairmen and Ranking
Minority Members of other Senate and House committees and subcommittees
that have authorization and oversight responsibilities for homeland
security. We are also sending copies to the DHS Secretary, the CBP
Commissioner and, upon their request, to other interested parties. In
addition, the report will be available at no charge on the GAO Web site
at [Hyperlink, http://www.gao.gov].
Should you or your offices have any questions on matters discussed in
this report, please contact me at (202) 512-3459 or at hiter@gao.gov.
Contact points for our Offices of Congressional Relations and Public
Affairs may be found on the last page of this report. Other contacts
and key contributors to this report are listed in appendix III.
Signed by:
Randolph C. Hite:
Director, Information Technology Architecture and Systems Issues:
[End of section]
Appendix I: Briefing to Subcommittees on Homeland Security, House and
Senate Committees on Appropriations:
Information Technology: Customs Has Made Progress on Automated
Commercial Environment System, but it Faces Long-Standing Management
Challenges and New Risks:
Briefing to the Staffs of the Subcommittees on Homeland Security,
Senate and House Committees on Appropriations:
March 10, 2006:
Briefing Overview:
Introduction:
Objectives:
Results in Brief:
Background:
Results:
* Legislative Conditions:
* Status of Recommendations:
* Observations:
Conclusions:
Recommendations:
Agency Comments:
Attachment 1: Scope and Methodology:
Introduction:
The Department of Homeland Security's (DHS) Customs and Border
Protection (CBP)[Footnote 12] is about 5 years into its trade
processing modernization program, known as the Automated Commercial
Environment (ACE). The goals of ACE are as follows:
Support border security by enhancing analysis and information sharing
with other government agencies to deal with increasing new security
threats to our nation. Provide CBP personnel with the technology and
information needed to decide, before a shipment reaches the border,
what should be targeted[Footnote 13] because it is a security threat,
and what should be expedited because it complies with U.S. laws.
Provide an integrated, fully automated information system to enable the
efficient collection, processing, and analysis of commercial import and
export data. Streamline time-consuming and labor-intensive tasks for
CBP personnel and the trade community, through a single, Web-based
interface, reducing costs for the government and the trade community.
Enable users to process, view, and manage their accounts nationally,
and obtain historical information on cargo, conveyances, and crew,
based on screening and targeting rules.
Enable CBP to comply with legislative mandates to improve efficiency/
effectiveness and provide better customer service to U.S. citizens.
The Department of Homeland Security Appropriations Act, 2006,[Footnote
14] appropriates about $320 million for ACE and states that DHS may not
obligate any funds for ACE until it submits for approval to the House
and Senate Committees on Appropriations a plan for expenditure that:
1. meets the capital planning and investment control review
requirements established by the Office of Management and Budget (OMB),
including Circular A-11, part 7;[Footnote 15]:
2. complies with DHS's enterprise architecture;
3. complies with the acquisition rules, requirements, guidelines, and
systems acquisition management practices of the federal government;
4. includes a certification by the Chief Information Officer of DHS
that an independent verification and validation agent is currently
under contract;
5. is reviewed and approved by the DHS Investment Review Board
(IRB),[Footnote 16] the Secretary of Homeland Security, and OMB; and:
6. is reviewed by GAO.
On February 2, 2006, DHS submitted its fiscal year 2006 expenditure
plan for $316.8 million to the House and Senate Appropriations
Subcommittees on Homeland Security.
DHS currently plans to acquire and deploy ACE in 11 increments,
referred to as releases. The first three releases are fully deployed
and operating, and the fourth release is being deployed. Other releases
are in various stages of definition and development.
Objectives:
As agreed, our objectives were to:
* determine whether the ACE fiscal year 2006 expenditure plan satisfies
the legislative conditions,
* determine the status of our open recommendations on ACE, and:
* provide any other observations about the expenditure plan and DHS's
management of the ACE program.
We conducted our work at CBP headquarters and contractor facilities in
the Washington, D.C., metropolitan area, as well as at the port in
Blaine, Washington, from July 2005 through March 2006, in accordance
with generally accepted government auditing standards. Details of our
scope and methodology are provided in attachment 1.
Results in Brief:
Objective 1: Satisfaction of legislative conditions:
The legislative conditions that the Congress placed on the use of
fiscal year 2006 ACE appropriated funds have been either partially or
fully satisfied by the latest expenditure plan and related program
documentation and activities. Nevertheless, more can be done to better
address several aspects of these conditions, such as having an approved
privacy impact assessment, measuring ACE performance and results,
ensuring architectural alignment, and employing effective independent
verification and validation (IV&V) practices. The following table
summarizes the status of each of the legislative conditions.
Legislative condition; Status.
1. Meets the capital planning and investment control review
requirements established by OMB, including OMB Circular A-11, part 7;
Status: Partially satisfied[A].
2. Complies with DHS's enterprise architecture;
Status: Partially satisfied.
3. Complies with the acquisition rules, requirements, guidelines, and
systems acquisition management practices of the federal government;
Status: Satisfied[B].
4. Includes a certification by the Chief Information Officer of DHS
that an independent verification and validation agent is currently
under contract;
Status: Satisfied.
5. Is reviewed and approved by the DHS Investment Review Board,
Secretary of Homeland Security, and OMB;
Status: Satisfied.
6. Is reviewed by GAO;
Status: Satisfied.
Source: GAO.
[A]Partially satisfied means that the plan, in combination with
supporting documentation, either satisfied; or provided for satisfying
many, but not all, key aspects of the condition that we reviewed.
[B]Satisfied means that the plan, in combination with supporting
documentation, either satisfied or provided for satisfying every aspect
of the condition that we reviewed.
[End of Table]
Objective 2: Status of actions to implement our open recommendations:
Some recommendations have been addressed, while progress has been slow
on others, such as:
* accurately reporting to the Appropriations Committees on CBP's
progress in implementing our prior recommendations;
* developing and implementing a strategic approach to meeting the
program's human capital needs;
* using criteria for exiting key milestones that adequately consider
indicators of system maturity, such as severity of open defects and the
associated risks; and:
* developing and implementing a performance and accountability
framework for ensuring that promised capabilities and benefits are
delivered on time and within budget.
The following table summarizes the status of each of the open
recommendations.
GAO recommendation: 1. Ensure that future expenditure plans are based
on cost estimates that are reconciled with independent cost estimates:
Number of months open: 7 months[[A]];
Status: Complete [[B] , C].
GAO recommendation: 2. Develop and implement a rigorous and
analytically verifiable cost estimating program:
Number of months open: 46 months;
Status: in progress[[D]].
GAO recommendation: 3. Immediately develop and implement a human
capital management strategy that provides both near-and long-term
solutions; develop and implement missing human capital practices:
Number of months open: 46 months;
Status: In progress.
GAO recommendation: 4. Have future ACE expenditure plans specifically
address any proposals or plans for extending and using ACE
infrastructure to support other homeland security applications:
Number of months open: 36 months
Status: In progress.
GAO recommendation: 5. Define measures, and collect and use associated
metrics, for determining whether prior and future program management
improvements are successful:
Number of months open: 22 months
Status: In progress.
GAO recommendation: 6. Define and implement an ACE accountability
framework that ensures:
GAO recommendation: 6a. coverage of all program commitment areas,
including key expected or estimated system (1) capabilities, use, and
quality; (2) benefits and mission value; (3) costs; and (4) milestones
and schedules:
Number of months open: 12 months
Status: In progress.
GAO recommendation: 6b. currency, relevance, and completeness of all
such commitments made to the Congress in expenditure plans:
Number of months open: 12 months
Status: In progress.
GAO recommendation: 6c. reliable data relevant to measuring progress
against committments;
Number of months open: 12 months;
Status: In progress.
GAO recommendation: 6d. reporting in future expenditure plans progress
against commitments contained in prior expenditure plans:
Number of months open: 12 months;
Status: In progress.
GAO recommendation: 6e. use of criteria for exiting key readiness
milestones that adequately consider indicators of system maturity, such
as severity of open defects:
Number of months open: 12 months;
Status: In progress.
GAO recommendation: 6f. clear and unambiguous delineation of the
respective roles and responsibilities of the government and the prime
contractor:
Number of months open: 12 months;
Status: Complete.
GAO recommendation: 7. Report quarterly to the House and Senate
Appropriations Committees on efforts to address open GAO
recommendations:
Number of months open: 22 months;
Status: In progress.
Source: GAO.
[A] Recommendation was also completed with respect to the fiscal year
2005 expenditure plan
[B] With respect to the fiscal year 2006 expenditure plan
[C] Complete means that actions have been taken to fully implement the
recommendation
[D] In progress means that actions are under way to implement the
recommendation
[End of table]
Objective 3: Observations:
Steps have been taken to address past pattern of ACE release
shortfalls, but new release management weaknesses are emerging.
* Release 4 pilot revealed performance problems that caused the pilot
period to be extended and the pilot scope to be reduced.
* Release 4 operational readiness review was passed despite unresolved
severe defects, and Release 4 is now being deployed.
* Release 4 quality problems and enhancement needs have led to changes
in how ACE release requirements are defined.
* Release 4 problems delayed Screening 1 and led to a revised strategy
for delivering all screening and targeting releases.
* Screening 1 key milestones were passed despite unresolved severe
defects.
* Past pattern of cost, schedule, and performance shortfalls with
Releases 2, 3, and 4 makes new strategy of concurrently developing
Releases 5, 6, and 7 risky.
* Earned value management (EVM), a technique for measuring progress
toward meeting deliverables, is not being used to manage Screening 1
and Release 5.
ACE's operational performance has been mixed, and mission impact is
unclear.
* Availability and responsiveness performance targets are largely being
met.
* Processing times for trucks crossing the border at key ports vary.
* Long-standing help desk limitations are being addressed.
* Usage by CBP and the trade is lower than expected.
* User satisfaction was reported as low.
* Performance targets are not always realistic.
* Goals, expected mission benefits, and performance measures are not
adequately aligned.
To assist CBP in managing ACE and increasing the chances that it will
deliver required capabilities on time and within budget and demonstrate
promised mission benefits and results, we are recommending that DHS:
* fully address legislative conditions associated with having an
approved privacy impact assessment, measuring ACE performance and
results, ensuring architectural alignment, and employing effective IV&V
practices;
* accurately report to the Appropriations Committees on CBP's progress
in implementing our prior recommendations;
* include in the June 30, 2006, quarterly update report to the
Appropriations Committees a strategy for managing ACE human capital
needs and the ACE framework for managing performance and ensuring
accountability;
* document key milestone decisions in a way that reflects the risks
associated with proceeding with unresolved severe defects and provides
for mitigating these risks;
* minimize the degree of overlap and concurrency across ongoing and
future ACE releases, and capture and mitigate the associated risks of
any residual concurrency;
* use EVM in the development of all existing and future releases;
* develop the range of realistic ACE performance measures and targets
needed to support an outcome-based, results-oriented accountability
framework, including user satisfaction with ACE; and:
* explicitly align ACE program goals, benefits, desired business
outcomes, and performance measures.
In their oral comments on a draft of this briefing, DHS and CBP
officials, including the Executive Director of Cargo Management
Systems, CBP, generally agreed with our findings, conclusions, and
recommendations and stated that the presentation was fair and balanced.
They also provided clarifying information that we incorporated as
appropriate in this briefing.
Background:
ACE-Related Business Functions:
ACE is to support eight major CBP business areas.
1. Release[Footnote 17] Processing: Processing of cargo for import or
export; tracking of conveyances, cargo, and crew; and processing of in-
bond, warehouse, Foreign Trade Zone, and special import and export
entries.
2. Entry[Footnote 18] Processing: Liquidation and closeout of entries
and entry summaries related to imports, and processing of protests and
decisions.
3. Finance: Recording of revenue, performance of fund accounting, and
maintenance of the general ledger.
4. Account Relationships: Maintenance of trade accounts, their bonds
and CBP-issued licenses, and their activity.
5. Legal and Policy: Management of import and export legal, regulatory,
policies and procedures, and rulings issues.
6. Enforcement: Enforcement of laws, regulations, policies and
procedures, and rulings governing the import and export of cargo,
conveyances, and crew.
7. Business Intelligence: Gathering and reporting data, such as
references for import and export transactions, for use in making
admissibility and release decisions.
8. Risk: Decision making about admissibility and compliance of cargo
using risk-based mitigation, screening,[Footnote 19] and
targeting.[Footnote 20]
ACE Technical Architecture:
The ACE technical architecture is to consist of layers or tiers of
computer technology:
* The Client Tier includes user workstations and external system
interfaces.
* The Presentation Tier provides the mechanisms for the user
workstations and external systems to access ACE.
* The Integration Services Tier provides the middleware for integrating
and routing information between ACE software applications and legacy
systems.
* The Applications Tier includes the ACE software applications
comprising commercial products (e.g., SAP[Footnote 21]) and custom-
developed software that provide the functionality supporting CBP
business processes.
* The Data Tier provides the data management and warehousing services
for ACE, including database backup, restore, recovery, and space
management.
Security and data privacy are to be embedded in all five layers.
Figure1: Simplified View of ACE Technical Architecture:
[See PDF for image]
Source: GAO analysis based on CBP data.
[End of figure]
Acquisition Strategy:
CBP is the component agency within DHS that is responsible for ACE.
Currently, CBP is headed by an Acting Commissioner. Within CBP, the ACE
program is located within the Office of Information Technology, which
is headed by the Assistant Commissioner for Information and Technology.
ACE is being acquired and implemented through a series of incremental
releases. On April 27, 2001, CBP awarded a contract to IBM Global
Services to develop and implement ACE. IBM and its subcontractors are
collectively called the ACE Support Team (formerly called e-Customs
Partnership).
CBP currently plans to acquire the 11 ACE releases in about 8.5 years.
Screening 2 is to be acquired in two "drops," or subreleases; Release 5
is to be acquired in two drops; Release 6 is to be acquired in three
drops; and Release 7 is to be acquired in two drops.
Summary of ACE Releases:
Figure 2: Planned Schedule for ACE:
[See PDF for Image]
[End of Figure]
The following presents the functionality provided by the 11 ACE
releases, their status, and associated plans.
Release 1 (ACE Foundation): Provide information technology (IT)
infrastructure-computer hardware and system software-to support
subsequent system releases. This release was deployed in October 2003
and is operating.
Release 2 (Account Creation): Give initial group of CBP national
account managers[Footnote 22] and importers access to account
information, such as trade activity. This release was deployed in
October 2003 and is operating.
Release 3 (Periodic Payment): Provide additional account managers and
importers, as well as brokers and carriers,[Footnote 23] access to
account information; provide initial financial transaction processing
and CBP revenue collection capability, allowing importers and their
brokers to make monthly payments of duties and fees. This release was
deployed in July 2004 and is operating.
Release 4 (e-Manifest: Trucks): Provide electronic truck
manifest[Footnote 24] processing and interfacing to legacy enforcement
systems and databases. As discussed later, this release is operating at
39 truck border crossings as of March 8, 2006. Additional enhancement
releases for Release 4 have been deployed since May 2005.
Screening 1 (Screening Foundation): Establish the foundation for
screening cargo and conveyances by centralizing criteria and results
into a single standard database; allow users to define and maintain
data sources and business rules. This release is scheduled for
deployment beginning in March 2006.
Screening 2 (Targeting Foundation): Establish the foundation for
advanced targeting capabilities by enabling CBP's National Targeting
Center to search multiple databases for relevant facts and actionable
intelligence. This release is scheduled for deployment in two drops:
* Screening 2 Targeting Platform (TP): Provide a platform to collect
and search relevant data and other information from multiple databases.
This drop is scheduled for deployment beginning in June 2006.
* Screening 2 Targeting Foundation (TF): Build on the targeting
platform to add new data sources, enhance screening business rules, and
provide reporting capabilities. This drop is scheduled for deployment
beginning in October 2006; however, CBP deployed a prototype to the
National Targeting Center as part of an effort to gather detailed
requirements.
Release 5 (Entry Summary, Accounts, and Revenue): Leverage SAP
technologies to enhance and expand accounts management, financial
management, and entry summary functionality. This release is being
developed in two drops:
* Master Data and Enhanced Accounts (Drop A1): Use SAP to deliver
enhanced account creation and maintenance functionality and expand the
types of accounts managed in ACE. This drop is scheduled for deployment
beginning in May 2007.
* Entry Summary and Revenue (Drop A2): Expand ACE to encompass entry
summary, interfaces with participating government agencies, calculation
of duties and fees, reconciliation processing, and refunds. This drop
is scheduled for deployment beginning in July 2008.
Screening 3 (Advanced Targeting Capabilities): Provide enhanced
screening for reconciliation, intermodal manifest, Food and Drug
Administration data, and in-bond, warehouse, and Foreign Trade Zone
authorized movements; integrate additional data sources into targeting
capability; and provide risk management capability. This release is
scheduled for deployment beginning in February 2007.
Screening 4 (Full Screening and Targeting): Provide full screening and
targeting functionality supporting all modes of transportation and all
transactions within the cargo management life cycle, including enhanced
screening and targeting capability with additional technologies. This
release is scheduled for deployment beginning in December 2008.
Release 6 (e-Manifest: All Modes and Cargo Security): Provide
electronic manifest capability for rail, air, and sea shipments;
provide a multimodal manifest;[Footnote 25] enable full tracking of
cargo, conveyances, individuals, and equipment; and enhance enforcement
processes for rail, air, and sea. This release is planned for
development in three drops:
E-Manifest: Rail and Sea (Drop M1): Extend the electronic manifest
functionality to rail and sea shipments; convert rail, sea, and truck
electronic manifests into the multimodal manifest. Drop M1 is scheduled
for deployment beginning in July 2008.
E-Manifest: Air (Drop M2): Provide the electronic manifest capability
to air shipments, and bring all modes of transportation into the
multimodal manifest. Drop M2 is scheduled for deployment beginning in
October 2007.
E-Manifest: Enhanced Tracking (Drop M3): Provide the capability to
track cargo, conveyances, individuals, and equipment, providing more
timely and accurate shipment status information. Drop M3 is scheduled
for deployment beginning in June 2009.
Release 7 (Exports and Cargo Control): Implement the remaining accounts
management, revenue, manifest, and release and export functionality.
This release is planned for development in two drops:
* ESAR: Drawback, Protest, and IASS (Drop A3): Provide the import
activity summary statement (IASS),[Footnote 26] drawback functionality,
and enhanced protest; provide on-line processing for trade account
applications. Drop A3 is scheduled for deployment beginning in December
2009.
* E-Manifest: Final Exports and Manifest (Drop M4): Extend the
electronic manifest for mail, pipeline, and hand carry; provide for
electronic export processing. Drop M4 is scheduled for deployment
beginning in December 2009.
Deployment Status:
As of March 8, 2006, ACE, Releases 1 through 4 capabilities have been
deployed to 39 of the 99 truck ports (see table).
Table 3:
[See PDF for Image]
[End of Table]
ACE Satisfaction of Modernization Act Requirements:
ACE is intended to support CBP satisfaction of the provisions of Title
VI of the North American Free Trade Agreement, commonly known as the
Modernization Act. Subtitle B of the Modernization Act contains the
various automation provisions that were intended to enable the
government to modernize international trade processes and permit CBP to
adopt an informed compliance approach with industry. The following
table illustrates how each ACE release is to fulfill the requirements
of Subtitle B.
Figure 3: ACE Satisfaction of Modernization Act Requirements:
[See PDF for Image]
Source: CBP.
[End of Figure]
Contract Tasks:
Thus far, CBP has executed 21 contract task orders-12 have been
completed and 9 are active.
Table 4: Status and Description; of Completed Task Orders.
[See PDF for Image]
[End of table]
Nine contract task orders are active.
Table 5: Status and Description of Active Task Orders.
[See PDF For Image]
[End of table]
Chronology of Seven ACE Expenditure Plans:
Since March 2001, seven ACE expenditure plans have been
submitted.[Footnote 27] Collectively, the seven plans have identified a
total of $1,698.1 million in funding.
On March 26, 2001, CBP submitted to its appropriations committees the
first expenditure plan seeking $45 million for the modernization
contract to sustain Customs and Border Protection Modernization Office
(CBPMO) operations, including contractor support. The appropriations
committees subsequently approved the use of $45 million, bringing the
total ACE funding to $50 million.
On February 1, 2002, the second expenditure plan sought $206.9 million
to sustain CBPMO operations; define, design, develop, and deploy
Increment 1, Release 1 (now Releases 1 and 2); and identify
requirements for Increment 2 (now part of Releases 5, 6, and 7 and
Screenings 1 and 2). The appropriations committees subsequently
approved the use of $188.6 million, bringing total ACE funding to
$238.6 million.
On May 24, 2002, the third expenditure plan sought $190.2 million to
define, design, develop, and implement Increment 1, Release 2 (now
Releases 3 and 4). The appropriations committees subsequently approved
the use of $190.2 million, bringing the total ACE funding to $428.8
million.
On November 22, 2002, the fourth expenditure plan sought $314 million
to operate and maintain Increment 1 (now Releases 1, 2, 3, and 4); to
design and develop Increment 2, Release 1 (now part of Releases 5, 6,
and 7 and Screening 1); and to define requirements and plan Increment 3
(now part of Releases 5, 6, and 7 and Screenings 2, 3, and 4). The
appropriations committees subsequently approved the use of $314
million, bringing total ACE funding to $742.8 million.
On January 21, 2004, the fifth expenditure plan sought $318.7 million
to implement ACE infrastructure; to support, operate, and maintain ACE;
and to define and design Release 6 (now part of Releases 5, 6, and 7)
and Selectivity 2 (now Screening 2 and 3). The appropriations
committees subsequently approved the use of $316.8 million, bringing
total ACE funding to $1,059.6 million.
On November 8, 2004, the sixth expenditure plan sought $321.7 million
for design and development of Release 5 and Screening 2, definition of
Screening 3, ACE program management, architecture and engineering, and
operations and maintenance. The appropriations committees subsequently
approved the use of $321.7 million, bringing total ACE funding to
$1,381.3 million.
* On February 02, 2006, CBP submitted its seventh expenditure plan,
seeking $316.8 million for detailed design and development of Release
5, development of Release 6, deployment of Screening 2, development and
deployment of Screening 3, program management and operations, and ACE
operations, maintenance, and infrastructure implementation.
Summary of Funding:
Table 6: Summary of the ACE Fiscal Year 2006 Expenditure Plan:
[See PDF for Image]
Source: CBP.
[A] Millions of dollars.
[End of table]
The plan does not include management reserve funding. As of December
15, 2005, the program had about $33.8 million in unused management
reserve funding from prior years.
ACE Testing and Related Milestones.
Development of each ACE release includes system integration and system
acceptance testing, followed by a pilot period that includes user
acceptance testing. Generally, the purpose of these tests is to ensure
that the system meets defined system requirements or satisfies user
needs. The associated readiness reviews are to ensure that the system
is ready to proceed to the next stage of testing or operation.
Tests and their related milestones are described in the following
table.
[See PDF for Image]
Source: eCP.
[A] Generally, the identified milestone review comes at the conclusion
of the related test.
[End of table]
Defect Categories.
Defects identified during testing and operation of the system are
classified into one of four severity categories, as described below.
[See PDF for Image]
[End of Table]
Previous GAO Observations:
Since 2001, we have reviewed CBP's six prior expenditure plans and made
a number of observations that relate to cost overruns, schedule delays,
quality limitations, and program management shortcomings. Among other
things, we observed the following:
* Release 1 and 2 testing revealed a sufficient volume and significance
of system defects to result in schedule delays.[Footnote 28]:
* Following cost overruns and schedule delays with Release 1, steps
were taken to avoid future problems, but the means for measuring the
success of the actions was not in place.:
* Delays in Release 2 began a pattern of increased reliance on
concurrent activities, which in turn caused future release delays and
cost overruns.[Footnote 29]:
* Release 3 testing and pilot activities were delayed and produced
system defect trends that raised questions about decisions to pass key
milestones and about the state of system maturity.[2]
* Release 4 test phases were delayed and overlapped, and tests revealed
a higher than expected volume and significance of defects, which again
raised questions about decisions to pass key milestones and about the
state of system maturity.':
* Progress toward activating ACE importer accounts had not met
expectations.[Footnote 30]:
Objective 1: Legislative Conditions Capital Planning Requirements:
DHS and OMB satisfied or partially satisfied each of its legislative
conditions; GAO satisfied its legislative condition.
Condition 1. The plan is to meet the capital planning and investment
control review requirements established by OMB, including Circular A-
11, part 7, which establishes policy for planning, budgeting,
acquisition, and management of federal capital assets. The plan, in
conjunction with related program documentation and program officials'
statements, partially satisfied the condition.
The table that follows provides examples of the results of our
analysis.
Table: Legislative Conditions:
[See PDF for Image]
[End of Table]
Objective 1: Legislative Conditions Enterprise Architecture Compliance:
Condition 2. The plan is to comply with DHS's enterprise architecture
(EA). The plan, including related program documentation and program
officials' statements, partially satisfied this condition.
The DHS Enterprise Architecture Board, supported by the Enterprise
Architecture Center of Excellence, is responsible for ensuring that
projects demonstrate adequate technical and strategic compliance with
the department's EA.
In May 2005, CBP requested that the DHS Enterprise Architecture Board
evaluate its analysis of ACE's alignment with the department's EA.
Using the ACE fiscal year 2006 business case, the ACE program plan, and
other documentation, the Center of Excellence evaluated alignment with
version 2.0 of the DHS EA business model, data architecture, technical
reference model, and transition strategy. In July 2005, the center
approved CBP's analysis and recommended that the request for program
alignment be approved. The Enterprise Architecture Board subsequently
concurred with the center's recommendation.
DHS required CBP to provide documentation to support ACE's alignment
with aspects of the EA such as the business architecture, the data
model, the transition strategy, and the technical reference model.
However, DHS officials told us that they do not have a documented
methodology for evaluating programs for compliance with the DHS EA,
other than relying on the professional expertise of the members of the
Center of Excellence. Moreover, no analysis or documentation was
produced by the evaluators that could be used to verify the degree of
alignment.
Condition 3. The plan is to comply with the acquisition rules,
requirements, guidelines, and systems acquisition management practices
of the federal government. The plan, in conjunction with related
program documentation, satisfied this condition.
The Software Acquisition Capability Maturity Model (SA-CMMO), developed
by Carnegie Mellon University's Software Engineering Institute (SEI),
is consistent with the acquisition guidelines and systems acquisition
management practices of the federal government, and it provides a
management framework that defines acquisition practices for such
process areas as acquisition planning, solicitation, requirements
development and management, project management, contract tracking and
oversight, and evaluation.
In November 2003, SEI assessed ACE acquisition management processes and
practices against the SA-CMM and assigned a level 2 rating, indicating
that CBP had instituted basic acquisition management processes and
controls in the following areas: acquisition planning, solicitation,
requirements development and management, project management, contract
tracking and oversight, and evaluation.
Independent Verification and Validation:
Condition 4. The DHS Chief Information Officer (CIO) is to certify that
an independent verification and validation (IV&V) agent is under
contract. DHS satisfied this condition.
On January 26, 2006, the DHS CIO certified that an IV&V agent is under
contract for the ACE program. However, the CIO also reported that the
contractor's approach needs to be improved. For example, the CIO said
that the contractor needs to address ACE satisfaction of quality
standards and user needs.
Further, the scope of the contractor's activities is not consistent
with the operative industry standard, which states that IV&V should
extend to key system products and development processes.[Footnote 31]
CBP's IV&V contract, awarded in December 2004, recognizes the
importance of this scope of activities by requiring the contractor to
implement a program consistent with this standard.
In fiscal year 2005, CBP expended approximately $310,000 on IV&V.
However, these resources have been used to assess certain program
management processes, such as ACE help desk activities and progress in
hiring Office of Information Technology staff. They have not been used
to, for example, examine the development of ACE requirements or the
quality of ACE software releases.
According to CBP officials, the scope of the IV&V contractor's
activities has not included ACE product quality because they believe
that such verification and validation activities would duplicate the
program's own quality assurance and testing activities. While we agree
that the IV&V agent should not duplicate work that is already performed
by an entity that is independent of the program's cost and schedule
processes, both the DHS CIO and the IV&V agent's statement of work have
directed that product quality be addressed. For fiscal year 2006, CBP
plans to spend $856,000 on IV&V.
Review by DHS and OMB:
Condition 5. The plan is to be reviewed and approved by the DHS
Investment Review Board (IRB), the Secretary of Homeland Security, and
OMB. DHS and OMB satisfied this condition.
On November 21, 2005, the DHS IRB reviewed the ACE program. The Under
Secretary for Management approved the expenditure plan on behalf of the
Secretary of Homeland Security on February 2, 2006.
OMB approved the plan on December 30, 2005.
Review by GAO:
Condition 6. GAO is to review the plan. We satisfied this condition.
Our review was completed on March 10, 2006.
Open Recommendations:
CBP has implemented one of our seven open recommendations, and
implementation of the remaining six is in progress. The status of each
of these recommendations is summarized below.
Table: Open Recommendations:
[See PDF for Image]
[A] Recommendation was also completed with respect to the fiscal year
2005 expenditure plan
[B] With respect to the fiscal year 2006 expenditure plan
[C] Complete means that actions have been taken to fully implement the
recommendation
[D] In progress means that actions are under way to implement the
recommendation
[End of Table]
Independent Cost Estimates:
Open recommendation 1: Ensure that future expenditure plans are based
on cost estimates that are reconciled with independent cost estimates.
Status: Complete[Footnote 32]:
The cost estimate in the fiscal year 2006 expenditure plan is based on
the estimates in the current ACE program plan. Cl3P, with contractor
support, compared the program plan cost estimate with the independent
cost estimate. According to the analysis performed, the two estimates
are consistent. The analysis was completed in October 2005, about 3
months before the fiscal year 2006 expenditure plan was submitted to
the Appropriations Committees.
Effective Cost Estimating:
Open recommendation 2: Develop and implement a rigorous and
analytically verifiable cost estimating program that embodies the
tenets of effective estimating as defined in SEI's institutional and
project-specific estimating models.[Footnote 33]:
Status: In progress:
CBP has defined and documented processes for estimating program costs
for the expenditure plan (including management reserve costs). It has
also hired a contractor to develop cost estimates, including contract
task order cost estimates, that are independent of CBP's estimates.
Further, to ensure sufficiency and completeness of the estimates, CBP
tasked a support contractor with evaluating both the independent and
the CBP estimates against the criteria defined by SEI.
According to the results of the support contractor's evaluation, the
independent estimates satisfied all seven of the SEI criteria, and
CBP's estimates satisfied six of the criteria and partially satisfied
the remaining one. For the partially satisfied criterion, the
evaluation found that the CBP estimate did not adequately consider past
projects in its cost and schedule estimates.
In addition, the support contractor found that CBP's estimate was the
aggregation of three ACE component estimates, each of which was
developed by the group responsible for a given component using
different cost estimating methodologies. As a result, the support
contractor recommended that CBP ensure that component estimates be
based on the same methodology.
Human Capital Strategy:
Open recommendation 3: Immediately develop and implement a human
capital management strategy that provides both near-and long-term
solutions to program office human capital capacity limitations, and
report quarterly to the Appropriations Committees on the progress of
efforts to do so.
Status: In progress:
CBP does not have a documented human capital strategy covering its ACE
program. As we have previously reported, such a strategy should provide
for defining the positions needed (including core competencies) to
perform core program functions; assessing and inventorying current
workforce skills and abilities; assessing any gaps between needed and
existing workforce levels and capabilities; and filling identified gaps
via such means as new staff, training existing staff, and augmenting
staff with contractor support.
In lieu of a documented human capital strategy, CBP officials told us
that they have taken various steps to bolster their ACE workforce as
part of a less formal strategy. For example:
* CBP expanded its contractor and government workforce dedicated to the
ACE program by merging staff assigned to trade-related legacy systems
with the ACE program staff-creating a Cargo Management Systems Program
Office. According to the officials, this merger has enabled them to
leverage the knowledge of staff who have been working with cargo
systems for the past 10 to 20 years, and thus increased the level of
expertise available to ACE.
* CBP recently trained staff working on a major release on earned value
management.
* CBP began using subject matter experts from existing field operations
advisory boards to help program officials define requirements for
future releases.
Further, CBP has reported quarterly to the appropriations committees on
its human capital goals and objectives for ACE.
Nevertheless, CBP officials acknowledged that they have not developed
and followed a formal strategy that systematically compares competency-
based staffing needs to on-board capabilities and defines and
implements long-term and short-term plans for closing any shortfalls
and associated strategies. Further, they have not reported to the
appropriations committees on these shortfalls. They stated that they
intend to develop a formal human capital strategy, and in doing so,
will reflect the activities that they have already taken.
Extending ACE Infrastructure:
Open Recommendation 4: Have future ACE expenditure plans specifically
address any proposals or plans, whether tentative or approved, for
extending and using ACE infrastructure to support other homeland
security applications, including any impact on ACE of such proposals
and plans.
Status: In progress:
The expenditure plan describes steps under way and planned to leverage
ACE's relationship with other homeland security applications. According
to the plan,
ACE and US-VISIT[Footnote 34] conform to the DHS enterprise
architecture, which is to define standard shared services that the two
systems can request. Such a services oriented architecture is intended
to promote reuse and reduce overlap and duplication. According to CBP
officials, they are currently exploring shared services areas.
* ACE and US-VISIT have begun to use a common infrastructure to deploy
and operate their systems. For example, in locations where ACE and US-
VISIT have been fully deployed, such as at the port of Blaine,
Washington, the two systems operate on the same network and the same
workstations.
However, the plan does not discuss the impact to ACE (e.g., cost and
schedule) with regard to these efforts and plans.
Besides what is described in the plan, ACE officials told us that they
meet once a month with US-VISIT officials to share lessons learned on
program management topics, such as risk management, change management,
and configuration management.
Measuring Success:
Open recommendation 5: Define measures, and collect and use associated
metrics, for determining whether prior and future program management
improvements are successful.
Status: In progress:
CBP continues to make changes that are intended to improve overall
program management.
* CBP has merged aspects of its Office of Information Technology that
managed trade-related legacy systems with its former ACE program
office, thereby creating the Cargo Management Systems Program Office.
This reorganization, according to CBP, is to result in (1) enhanced
government oversight of ACE development, (2) better definition of
requirements for future releases, and (3) faster and cheaper delivery
of ACE capabilities. However, program officials told us that they have
not established measures or targets for determining whether the
reorganization is providing these benefits.
CBP eliminated the dependencies between the screening and targeting
releases and the cargo releases by leveraging its existing Automated
Targeting System in delivering ACE screening and targeting
capabilities. (This topic is discussed in further detail in the
observations section.) It has also established measures for determining
the impact of this change, including saving the program $10 million in
life-cycle costs and allowing the screening and targeting releases to
be fully deployed 1 year ahead of schedule.
Accountability Framework:
Open recommendation 6: Define and implement an ACE accountability
framework that fulfills several conditions:
a. Covers all program commitment areas, including key expected or
estimated system (a) capabilities, use, and quality, (b) benefits and
mission value; (c) costs, and (d) milestones and schedules.
Status: In progress:
CBP has prepared an initial version of an accountability framework that
program officials said they intend to improve, but have nevertheless
begun using. This framework is built around measuring progress against
costs, milestones and schedules, and risks for select releases. It is
also intended to permit measurement at different levels of aggregation,
and for whatever incremental periods of time desired (e.g., monthly).
However, as we discuss later, the benefit commitments have not been
well defined and the performance targets are not always realistic.
b. Ensures currency, relevance, and completeness of all program
commitments made to the Congress in expenditure plans.
Status: In progress:
The fiscal year 2006 expenditure plan continues to include inaccurate,
dated, and incomplete information and to omit other relevant
information.
The plan did not include information regarding CBP's decision to
eliminate the dependencies among the screening and targeting releases
and the cargo releases and to leverage its existing Automated Targeting
System capabilities, nor did the plan reflect the $10 million cost
reduction and the 1 year schedule acceleration that this change is
intended to produce.
The plan includes milestone completion dates for Releases 5, 6, and 7
and Screening 1 that are not accurate. For example, the expenditure
plan shows that the preliminary design review for Release 5, drops A1
and A2, was scheduled to occur in August 2005; however, it did not
occur until November 2005. Similarly, although the plan states that the
critical design review for Release 6, drop M1, was scheduled to occur
in November 2005, it is currently scheduled to take place August 2006.
According to CBP officials, they did not update the plan because they
wanted to provide it to the Appropriations Committees as soon as
possible. They also stated that they use the congressional quarterly
reports to provide the Appropriations Committees with more current,
relevant, and complete information about ACE. However, these quarterly
reports are generally submitted 3 to 4 months after the end of each
quarter.
c. Ensures reliable data relevant to measuring progress against
commitments.
Status: In progress:
The data that CBP uses to measure progress against commitments are not
consistently reliable. For example:
* Data in the defect tracking system show that Release 4 is operating
with long-standing defects and that new defects have not been closed.
For example, as of January 18, 2006, the system showed that a number of
defects were open:
Number: 12:
Severity: 2:
Number: 4:
Severity: 3:
Number: 3:
Severity: 4:
Program officials told us that many of these defects are in fact
resolved. However, the defect tracking system does not accurately
reflect this status for two reasons:
* staff were using multiple systems to track defects and these systems
were not always reconciled; and:
* newer staff were inexperienced in using the defect tracking system.
According to program officials, they are manually reconciling the data
between the multiple systems, and plan to implement a new system that
will track all defects in one central system.
d. Ensure future expenditure plans report progress against commitments
contained in prior expenditure plans.
Status: In progress:
The fiscal year 2006 expenditure plan does not adequately describe
progress against commitments made in previous plans. For example:
* The plan provides a summary of the funding requested in each of the
previous six expenditure plans. However, it does not provide
information on whether these funding amounts were actually expended or
obligated as planned.
* The plan includes a new schedule for ACE releases, but it does not
report progress relative to the schedule presented in the fiscal year
2005 plan.
* The plan does not explain the extent to which Release 5 design and
development planned in the fiscal year 2005 plan was actually
accomplished.
e. Ensure criteria for exiting key readiness milestones adequately
consider indicators of system maturity, such as severity of open
defects.
Status: In progress:
According to CBP officials, ACE milestone exit criteria provide for
considering the risk associated with having unresolved severe defects.
Specifically, the criteria state that critical and severe defects must
be resolved or there must be a plan in place to resolve them. Using
these criteria, CBP passed several release milestones with severe
defects still open:
* Release 4 operational readiness review was passed with 9 severe
defects open,
* Screening 1 test readiness review was passed with 3 severe defects
open, and:
* Screening 1 production readiness review was passed with 2 severe
defects open.
In making these decisions, CBP officials told us that they considered
the associated risk and concluded that the risk was acceptable. In
particular, they stated that it was more important to get the releases
in the hands of users and thereby gain user acceptance and receive user
feedback sooner than it was to first resolve the defects.
However, CBP officials were unable to provide us with any documentation
on how the inherent risks of passing these milestones with open severe
defects were assessed, and the ACE risk inventory does not include any
risks associated with these decisions.
f. Ensures clear and unambiguous delineation of the respective roles
and responsibilities of the government and the prime contractor.
Status: Complete.
The current version of the ACE program plan describes general roles and
responsibilities for the government and the prime contractor. More
detailed roles and responsibilities of CBP, the prime contractor, and
the support contractors have been documented in a roles and
responsibilities matrix that assigns primary responsibility for each
activity, such as testing, training, and maintenance. In addition, the
task orders describe the specific responsibilities and expectations of
the contractors.
Implementation Reporting:
Open recommendation 7: Report quarterly to the House and Senate
Appropriations Committees on efforts to address open GAO
recommendations.
Status: In progress:
CBP submitted reports to both Committees on its efforts to address open
GAO recommendations for the quarters ending March 31, 2005; June 30,
2005; and September 30, 2005. CBP also plans to submit a report for the
quarter ending December 31, 2005.
However, progress in addressing our recommendations was not always
reported accurately. For example, CBP reported that it will review the
expenditure plan throughout the approval process to ensure that it
incorporates the most current program commitments. However, it did not.
Additionally, in the September 2005 report, CBP stated that the fiscal
year 2006 expenditure plan would have a section that describes progress
made against program commitments made in all prior expenditure plans.
However, the expenditure plan does not include this information.
Objective 3: Development Observations:
Steps have been taken to address past pattern of ACE release
shortfalls, but new release management weaknesses are emerging.
As we have observed in our previous reviews, CBP established a pattern
of addressing quality problems with earlier releases by borrowing
resources from future releases, which led to schedule delays and cost
overruns. This pattern has continued with Release 4, which developed
problems that caused delays with Screening 1. CBP has taken steps to
mitigate this problem by eliminating the dependencies between the cargo
releases and the screening and targeting releases.
However, CBP's planned schedule for developing Releases 5, 6, and 7
includes a significant level of concurrency, because of CBP's interest
in delivering ACE functionality sooner. As we have reported in the
past, such concurrency between release activities has led to cost
overruns and schedule delays. Thus, the revised ACE plans and actions
are potentially reintroducing the same problems that resulted in
shortfalls in the past.
Release 4 Pilot Problems:
Release 4 pilot revealed performance problems that caused the pilot
period to be extended and the pilot scope to be reduced.
The Release 4 pilot was intended to ensure that the release met its
requirements before it was deployed to all truck ports. Examples of
pilot activities include training of CBP and trade personnel in how to
use the release and conducting user acceptance testing. As planned, the
pilot was to be conducted at two truck ports-Blaine, Washington, and
Buffalo, New York-and the testing was to occur during a 10 week period.
The pilot was to conclude with the operational readiness review on
February 23, 2005.
However, the pilot only occurred at Blaine, the pilot period covered 17
weeks, and the operational readiness review did not occur until April
14, 2005. The following are significant events that occurred during the
pilot.
* During the initial days of the pilot, Release 4 slowed truck
processing. To address this, users identified needed system
enhancements, primarily intended to reduce the number of steps required
to process trucks. As a result, the pilot was suspended 11 days after
it began so that the enhancements could be developed and implemented.
The pilot resumed about 5 weeks after it was suspended. However, new
problems were encountered, such as slow system response times and
screen freezes. Additionally, the release was not properly displaying
alerts for potential criminals or terrorists. As a result, the pilot
was again suspended about 6 weeks after it was resumed.
The pilot resumed about 3 weeks after the second suspension. On April
14, 2005, about 7 weeks later than planned, Release 4 passed
operational readiness review.
Because of the ongoing problems with Release 4 in Blaine, CBP cancelled
its plans to include Buffalo in the pilot.
A comparison of the planned versus actual pilot schedules is summarized
in the following figure.
Figure: Planned versus Actual Time Frame for Release 4 Pilot Programs:
[See PDF for image]
[End of figure]
Release 4 Milestone Reviews and Quality Problems:
Release 4 operational readiness review was passed despite unresolved
severe defects, and Release 4 is now being deployed.
CBP's criteria for passing key milestones-such as the Release 4
operational readiness review (ORR)-stipulate that all critical and
severe defects must either be resolved or there must be plans in place
to resolve the defect. As noted earlier, we have recommended that any
decisions to pass key milestones adequately consider indicators of
system maturity, such as open severe defects.
At the time of the Release 4 operational readiness review, CBP reported
that 9 severe defects remained open, and that it had plans in place to
resolve each of these defects. Of these 9, one was subsequently
cancelled, 4 were closed within approximately 6 weeks of ORR, 3 were
closed several months after ORR, and 1 remains open about 10 months
later. According to program officials, the remaining open defect has
been lowered from severe to moderate status, but the change has yet to
be reflected in the defect tracking system.
CBP officials told us that they considered the risk associated with
passing ORR with these 9 severe defects, and concluded that the risk
was acceptable. In particular, they indicated that it was important to
get Release 4 in the hands of users and thereby gain user acceptance
and receive user feedback sooner.
However, it is important to note that deploying a system with known
operational problems, while likely to encourage user feedback, may
actually limit user acceptance, particularly given the number of
Release 4 enhancements needed. (The next section discusses these
enhancements.)
Further, CBP officials were unable to provide us with any documentation
on how the inherent risks of passing this milestone with open severe
defects were assessed, and the ACE risk inventory does not include any
risks associated with this decision.
Since ORR, Release 4 has been deployed to 38 truck ports on the
northern and southern U.S. borders; by December 2006, CBP plans to
deploy Release 4 to the remaining 60 truck ports.
4 Definition Limitations and Enhancements:
Release 4 quality problems and enhancement needs have led to changes in
how ACE release requirements are defined.
Inadequate requirements definition was a major reason for the problems
and delays encountered during the Release 4 pilot in Blaine.
Specifically, program officials told us that the requirements
definition process did not fully identify the key functionality
contained in the legacy system that ACE needed to provide. Also, the
process did not adequately consider the capabilities that ACE would
need to provide in an actual operational environment. For example,
Release 4 requirements did not reflect the large volumes of
transactions common in an operational setting, such as documenting
notifications of potential security violations.
According to program officials, the requirements definition process was
insufficient in part because personnel involved in defining
requirements did not have sufficient experience with the legacy systems
that ACE was to replace and/or interface with. Moreover, they said that
although Release 4 met the requirements that were defined for it in
2001, since then CBP's mission and operations have changed, creating
the need to introduce additional system requirements.
To address these limitations, the number and scope of release
enhancements (subreleases) have been larger than anticipated. To date,
CBP has implemented two enhancement releases that add, for example,
performance enhancements to transaction processing and improvements to
the usability of reports. Although CBP has yet to determine the total
number of enhancement subreleases that will be developed for Release 4,
program officials stated that more are needed.
To improve requirements definition for future releases, CBP has changed
its requirements definition process. For instance, program officials
said that they are:
* leveraging existing field operations advisory boards to augment
program staff defining the requirements for future releases,
* using contractors to analyze legacy system functionality to ensure
that ACE requirements reflect it, and:
* regularly involving key representatives from the trade community to
more fully define ACE requirements and help ensure that ACE meets
users' needs.
Impacts on Screening 1:
Release 4 problems delayed Screening 1 and led to a revised strategy
for delivering all screening and targeting releases.
The Release 4 problems caused delays in developing and testing
Screening 1 because resources that were to be used on Screening 1 were
diverted to Release 4. For example:
* The test environment's availability to support Screening 1 was
delayed about 11 weeks because it was still supporting Release 4.
* The staff targeted for Screening 1 development and test activities
were being held on Release 4 longer than planned.
As a result, the combined critical design review/test readiness review
for Screening 1 was delayed by 60 days. As we have previously reported,
this diversion of resources from future releases to address problems on
prior releases has been a pattern on ACE for many years, owing largely
to the concurrency in the development of releases and the releases'
dependency on the same resources.
To correct this pattern, as well as to leverage existing targeting
functionality, CBP decided to take two steps:
* to "decouple" all screening and targeting releases from ACE's cargo
releases and:
* to use its existing system, the Automated Targeting System (and
associated resources), to deliver needed screening and targeting
capabilities.
In addition to addressing the pattern of delays caused by the releases
competing for the same resources, CBP estimates that these changes will
save the program $10 million and allow the screening and targeting
releases to be fully deployed 1 year ahead of schedule.
Screening 1 Milestone Reviews and Quality Problems
Screening 1 key milestones were passed despite unresolved severe
defects.
As previously noted, CBP's criteria for passing key milestones, such as
the test readiness review and production readiness review for Screening
1, stipulate that all critical and severe defects must either be
resolved or have work-off plans in place. Also, we have recommended
that any decisions to pass key milestones adequately consider
indicators of system maturity, such as open severe defects.
A number of severe defects were open at the time of these milestones:
* At the test readiness review on November 14, 2005, CBP reported that
three severe defects were open.
* At the production readiness review on December 22, 2005, CBP reported
that two severe defects were open.
According to CBP officials, these key milestones were passed because
work-off plans were in place for resolving each severe defect. Thus, in
their view the risk of proceeding did not outweigh the need to get
Screening 1 to the users and thereby gain user acceptance and receive
user feedback sooner.
However, deploying a system with known operational problems, while
likely to encourage user feedback, may actually limit user acceptance.
Further, CBP officials were unable to provide us with any documentation
on how the inherent risks of passing these milestones with open severe
defects were assessed, and the ACE risk inventory does not include any
associated risks.
Screening 1 is scheduled for deployment beginning on March 23, 2006.
Concurrent Development Risks:
Past pattern of development problems with Releases 2, 3, and 4 makes
new strategy of concurrently developing Releases 5, 6, and 7 risky.
As we have previously reported,[Footnote 35] concurrent development of
system components introduces risks that can adversely impact program
cost and schedules. According to ACE contractors, these risks can
include limited understanding of requirements before design and
development activities begin, uncertainty regarding the timely
availability of commercial hardware and software products, and
increased near-term funding requirements. Other risks include
contention for limited resources (such as key personnel, as well as
development and testing equipment and facilities) and dependencies
among releases not being met.
Concurrency in developing early ACE releases caused schedule slips and
cost overruns. As we previously reported, overlapping the development
of Releases 2 and 3 caused delays in Release 3 and resulted in Releases
1 through 4 costing more than planned.
Factors contributing to the schedule slips and cost overruns include
the following:
* Additional resources were needed to eliminate Release 1 and 2
defects.
* Unavailable testing and development environments extended Release 3
and 4 schedules.
* Increased scope of Releases 3 and 4 to include Release 2 deferred
requirements.
Despite these experiences, CBP established a new ACE program plan in
July 2005 that involves considerable overlap across the development
schedules for Releases 5 and 6 and for Releases 6 and 7. According to
CBP, the additional risk introduced by this concurrency is outweighed
by the potential for delivering ACE functionality sooner.
However, performance to date in meeting the highly concurrent
milestones in the July 2005 program plan shows that delays are already
occurring that are introducing even more schedule overlap and thus
program risk. For example, both Releases 5 and 6 have experienced
significant design-related delays, as shown by the table below. These
delays have in turn increased the amount of development overlap across
Releases 5, 6 and 7, as shown by the next slide, which compares the
July 2005 and January 2006 schedules for these releases.
Table: Delays in Meeting Release 5 and 6 Design-Related Milestones:
[See PDF for Image]
[End of table]
Figure: Original versus Revised Schedules for Developing Releases 5, 6,
and 7:
[See PDF for Image]
Source: GAO analysis of CBP data.
[End of Figure]
The risk associated with this concurrency in development schedules is
exacerbated by several factors.
* Releases/drops have extensive data and resource (e.g., testing
environments) dependencies. For example, Release 6, drop M1, which is
to provide electronic manifest capabilities to rail and sea ports, is
dependent on the data that will be provided by Release 5, drop A1,
which is to add trade account types and corresponding data. Further,
CBP officials have said that Release 5, drop A2, and Release 6, drop
M1, must be tested together in the same testing environment. Therefore,
if either of the drops is delayed, the other will be delayed as well.
* Just as delays on drops have already increased contention for
resources, further delays could introduce additional contention.
* Release 5, 6, and 7 include the vast majority of ACE's functionality
(87 percent) and are to produce the more significant mission benefits.
Earned Value Management Not Being Used:
Earned value management is not being used to manage Screening 1 and
Release 5.
CBP has not used earned value management (EVM) to manage Release 5 and
Screening 1. EVM is a management tool for measuring program progress by
comparing, during a given period of time, the value of work
accomplished with the amount of work expected to be accomplished; this
comparison permits performance to be evaluated based on calculated
variances from the planned cost and schedule. EVM is both an industry
accepted practice and an OMB requirement.
For Screening 1, CBP discontinued use of EVM in June 2005, when it made
the decision to decouple the screening and targeting releases from the
cargo releases. At that time, it decided to take advantage of the
functionality of its legacy targeting system, Automated Targeting
System (ATS), as well as the expertise of ATS staff. According to CBP
officials, when ATS staff began working on Screening 1, they were
unfamiliar with EVM and therefore did not use it. CBP officials stated
that in lieu of EVM, they monitored the actual costs of work performed.
In addition, CBP has not used EVM for Release 5, which has been under
development for 22 months and for which $29.5 million has reportedly
been expended. According to CBP officials, use of EVM was not possible
because the revision of the Release 5 scope and strategy delayed
definition of requirements and prevented cost and schedule baselines
from being established. Therefore, Release 5 work had to be conducted
under intermittent authorizations to proceed, which did not have
measurable baselines.
To respond to these EVM limitations, according to program officials,
Screening 1 staff have now been trained on EVM. They also agreed that
prior Release 5 work should have been managed by measurable baselines.
They said that they plan to use EVM for future Release 5 work once
fiscal year 2006 funds become available. In addition, they said that
they intend to establish baselines for any work performed under
authorizations to proceed, and appropriate performance metrics will be
applied.
Objective 3: Operation Observations:
ACE's operational performance has been mixed, and mission impact is
unclear.
As described earlier, ACE Releases 1 to 4 are in operation. To date,
these releases' operational performance has been uneven. For example,
ACE has largely been meeting its goals for being available and
responsive in processing virtually all daily transactions, and it has
decreased truck processing times at some ports. However, ACE is not
being used by as many CBP and trade personnel as was expected, and
truck processing times at other ports have increased. Moreover, overall
user satisfaction has been low.
In addition, ACE goals, expected mission benefits, and performance
measures are not fully defined and adequately aligned. Where
performance measures have been defined, the associated targets are not
always realistic. As a result, it is unclear whether ACE has realized
or will realize the mission value that it was intended to bring to
CBP's and other agencies' trade-and border security-related operations.
Availability Targets Largely Met:
Availability and responsiveness targets are largely being met.
CBP has defined ACE availability in terms of the percentage of
transactions that are to be executed successfully each day. According
to a service level agreement between the ACE Support Team and CBP, 99.9
percent of all ACE transactions on any given day are to be successful.
The ACE Support Team reports that ACE met this requirement on all but
22 days between January 1, 2005, and January 27, 2006.
For each of the 22 days that the system did not meet the agreement, the
ACE Support Team identified and corrected the root cause. For example,
outages were caused by:
* a server accidentally being shut down by data center personnel,
* a software error that disabled a transaction function, and:
* a network switch that malfunctioned.
To address these causes, the ACE Support Team instituted new data
center procedures, deleted ACE code that caused the transaction error,
and established methods for identifying network problems sooner.
Another service level agreement between the ACE Support Team and CBP
requires the system to execute all transactions within 6 seconds 95
percent of the time. The ACE Support Team reports that ACE met this
requirement on all but 16 days between January 1, 2005, and January 27,
2006, and no incidents have been reported since August 5, 2005.
For each of the 16 days that the system did not meet the agreement, the
ACE Support Team identified the root cause. For instance:
* eight incidents were due to a problem with a program that measures
network performance, which has since been addressed through changes to
operational procedures, and:
* one incident was caused by an improperly configured server that has
since been reconfigured.
Truck Processing Times Vary:
Processing times for trucks crossing the border at key ports vary.
CBP has identified more efficient truck processing at the ports as an
expected ACE benefit. To ascertain whether the benefit is being
realized, it also defined truck processing time as the performance
measure to be used, and it set a performance target of reducing
processing times at the ports by 6 percent in fiscal year 2005.
However, at the two ports for which CBP established baselines to
measure truck processing against performance targets in fiscal year
2005, CBP reports that truck processing times have actually increased.
For example:
* At Pembina, North Dakota, by the end of fiscal year 2005, processing
time had increased by about 14 percent. CBP officials attributed this
increase to a lack of user proficiency and confidence with ACE.
* At Nogales, Arizona, by the end of fiscal year 2005, processing time
had increased by about 70 percent. CBP officials attributed this
increase to ACE-related changes to booth operations, such as the new
requirement to enter empty trucks in the system.
According to CBP officials, widespread truck processing efficiency
gains will not be realized until a majority of truck manifests are
submitted electronically, which is not expected to occur until use of
electronic manifests becomes a requirement in early fall 2006.
Nevertheless, since fiscal year 2005, CBP reports that some ports have
experienced processing improvements. For example:
* At the Ambassador Bridge in Detroit, Michigan, processing time
decreased by approximately 27 percent between October 3 and December
15, 2005. CBP officials attributed this decrease to improvements in the
quality of training and a new user interface toolbar feature.
Help Desk Improvements:
Long-standing help desk limitations are being addressed.
According to an independent technology research firm,[Footnote 36]
effective help desk services include providing users timely updates on
the status of their requests; conducting user satisfaction surveys; and
establishing, collecting, and reporting operational metrics, such as
the number of requests resolved during the initial call to the help
desk.
The current ACE help desk does not perform all the practices associated
with an effective help desk. To its credit, the existing help desk
does, for example, monitor and measure such activities as the number of
requests that are resolved during the initial call to the help desk,
the number of calls that are not received because the user hangs up,
and the average time it takes to answer a help desk call. However, it
does not:
* collect data and inform users on the status of their help desk
requests or:
* survey users on their satisfaction with help desk services.
CBP has long recognized the limitations of the ACE help desk. In
January 2003, it decided to implement a new system that would provide
greater help desk capabilities. However, the first phase of the new
system was not implemented until about 3 years later (February 2006).
According to CBP officials, the delay was due to competing demands for
limited resources.
* The first phase is to enable users to check the status of their
existing requests and to resolve certain problems without calling the
help desk. In addition, the new system is to automatically e-mail users
with a notification of resolution, which is to provide a link to a
customer satisfaction survey.
* The second phase is to include more advanced functionality such as
allowing users to open and update their own help requests. CBP is
working to develop a schedule for the second phase.
In addition to the new help desk service, CBP has established an ACE
Portal Support Center to provide additional support to CBP and the
trade community on nontechnical issues: for example, submitting and
processing electronic truck manifests, setting up and using accounts,
generating reports, and resetting passwords.
Usage Lower Than Expected:
Usage by CBP and the trade is lower than expected.
CBP and trade usage of deployed ACE releases has been lower than
expected. Specifically:
* The goal for fiscal year 2005 was for 11 percent of CBP employees to
use ACE. However, as of the end of the fiscal year, 8 percent were
using it. According to CBP officials, they are rethinking this goal to
recognize that not all CBP employees have a need to use ACE.
* The goal for fiscal year 2005 was for 20 percent of all monthly
payments of fees and duties to be collected using ACE. However, as of
the end of fiscal year 2005, ACE collected about 11 percent of the
total fees and duties. To increase ACE use in paying fees and duties,
CBP has:
- eliminated its requirement for importers paying their way to be
members of the Customs-Trade Partnership against Terrorism (this
organization is focused on developing, enhancing, and maintaining
effective security practices throughout the trade industry) and:
- eliminated some of the paperwork and other requirements that have
since been deemed unnecessary.
The following graph depicts the expected and actual percentage of ACE-
collected fees and duties for fiscal year 2005.
Figure: Fiscal Year 2005 Expected versus Actual Percentage of Duties
and Fees Collected Using ACE:
[See PDF for Image]
Source: GAO analysis of ACE data.
[End of Figure]
The goal for fiscal year 2005 was for 5 percent of all
manifests[Footnote 37] to be filed electronically using ACE. However,
the actual percentage filed in this manner was less than 1 percent. CBP
officials attributed this low percentage to the trade community's
reluctance to invest resources to change, and the fact that electronic
manifests must be submitted in advance of truck's arrival. They also
said that even though the goal for fiscal year 2006 is 20 percent,
significant increase in electronic manifests is unlikely to occur until
it is mandatory for the trade to use this method, which is not expected
to occur until early fall 2006.
The following table shows CBP's progress towards reaching its fiscal
year 2005 and 2006 electronic manifest goals.
Table: CBP's Limited Progress Towards Reaching Fiscal Year 2005 and
2006.
[See PDF for Image]
Source: GAO analysis of CBP data.
[End of table]
User Satisfaction Low:
User satisfaction was reported as low.
In February and March of 2005, a CBP user satisfaction survey was
conducted that covered, among other things, CBP IT systems, including
ACE. Of the 187 respondents, 39 percent indicated that they are
dissatisfied or very dissatisfied with ACE. The reason most often given
for this response was that the system was not easy to use. For example,
according to the survey responses, ACE required officers to navigate
through several screens in order to process each truck.
Other ACE user concerns identified include the following:
* Passwords did not allow users to access the system.
* Response times were slow.
* Initial training of users was not adequate.
In response to the ACE survey results, CBP officials are developing a
prioritized list of recommendations for improving user satisfaction, as
well as a strategy for surveying CBP and trade users at several ports
at which Release 4 has been deployed. The goal is to gain further
insights into users' satisfaction and to identify potential areas for
improvement. According to CBP, they plan to start surveying the ports
in the spring of 2006.
Performance Targets Not Realistic:
Performance targets are not always realistic.
Meaningful measurement of operational performance requires, among other
things, realistic performance targets against which to gauge results.
However, defined ACE performance targets are not always realistic. For
example:
* The performance target in fiscal year 2005 for ACE usage was that 11
percent of all CBP employees would use ACE. However, many CBP employees
will never need to use the system. Thus, the defined performance target
does not reflect this. CBP officials stated that they plan to redefine
this measure to focus on CBP employees who have a reason to use ACE.
* The performance target for fiscal year 2005 for truck processing was
a 6 percent decrease in truck processing times at each port. However,
each port varies in terms of truck volumes, operational hours, cargo
and antiterrorism activities, and port policies. A single performance
goal for every port does not recognize these differences.
* According to CBP officials, the fiscal year 2005 and 2006 targets for
processing electronic manifests (5 and 20 percent, respectively) were
arbitrarily set. They stated that until electronic filing of manifests
is required which is expected to take place in early fall 2006-it is
unlikely that there will be any significant increase in the rate of
electronic submissions.
Benefits and Measures Not Aligned:
Goals, expected mission benefits, and performance measures are not
fully defined and adequately aligned.
The Clinger-Cohen Act and associated OMB guidance[Footnote 38] require
the use of effective IT management practices, including measuring the
contributions of IT investments to achieving agency mission outcomes.
To this end, OMB requires[Footnote 39] that agencies should, among
other things,
* establish program performance goals and expected benefits;
* develop outcome-based performance measures to assess actual program
performance (i.e., achievements) against expected benefits; and:
* ensure that goals, expected benefits, and performance measures are
properly aligned.
CBP has defined ACE goals, expected benefits, desired business results,
and performance measures. The six ACE goals cited earlier are
summarized below.
* Support border security through enhanced analysis and interagency
information sharing.
* Provide information to enable decisions before a shipment reaches the
border as to what to target and what to expedite.
* Enable efficient collection, processing, and analysis of commercial
import and export data.
* Streamline time-consuming, labor-intensive tasks for CBP and the
trade.
* Enable national account management and informed, rule-based screening
and targeting.
* Comply with legislative mandates to improve efficiency/effectiveness
and provide better customer service.
CBP has also identified 23 ACE benefits, all of which relate to gains
in efficiency. Examples of these benefits are as follows:
* importer account profile efficiency gains,
* forms processing efficiency gains,
* driver verification efficiency gains, and:
* cargo release efficiency gains.
CBP has also identified 11 ACE desired business results. Examples are
as follows:
* improved accuracy and timeliness of information to support threat
assessment decisions,
* detected unfair/illegal trade activities,
* increased compliance rates, and:
* improved responsiveness and adaptability to policy, statutory, and
regulatory changes and trade volume increases.
Page 130 GAO-06-580 Customs Modernization:
In addition, CBP has defined 17 performance measures. Examples are as
follows:
* percentage of internal CBP population using ACE functionality to
manage trade information,
* percentage of total duties and fees paid through periodic monthly
statements, and:
* percentage reduction of truck processing time.
However, the relationships among these program goals, benefits,
business results, and performance measures have not been clearly
established and are not always apparent. Further, not every goal has
defined benefits, every benefit is defined only in terms of efficiency
gains, not every benefit has an associated business result, and not
every benefit and business result have associated performance measures.
For example:
* The two ACE goals focused on homeland security do not align with any
stated ACE benefit.
* The ACE benefit of greater efficiency in processing forms does not
clearly align to any performance measures.
* The desired business result related to improved threat assessment
decisions does not align to any ACE benefit or performance measure.
* There are 23 ACE benefits but only 17 measures for gauging
performance relative to these benefits.
The following table illustrates the lack of clearly defined
relationships among ACE benefits, desired business results, and
performance measures for Releases 2, 3, and 4.
Table: Operation Observations.
[See PDF for Image]
[End of table]
Conclusions:
The legislative conditions that the Congress placed on the use of
fiscal year 2006 ACE appropriated funds have either been partially or
fully satisfied by the latest expenditure plan and related program
documentation and activities. Nevertheless, more can be done to better
address several aspects of these conditions, such as having an approved
privacy impact assessment, measuring ACE performance and results,
ensuring architectural alignment, and employing effective IV&V
practices. Given that the legislative conditions are collectively
intended to promote accountability and increase the chances of program
success, it is important that each receives DHS's full attention.
Also important to ACE's success is the swift and complete
implementation of the recommendations that we have previously made to
complement the legislative conditions and improve program management,
performance, and accountability. In this regard, some recommendations
have been addressed, while progress has been slow on others, such as:
* accurately reporting to the Appropriations Committees on CBP's
progress in implementing our prior recommendations;
* developing and implementing a strategic approach to meeting the
program's human capital needs;
* using criteria for exiting key milestones that adequately consider
indicators of system maturity, such as severity of open defects and the
associated risks; and:
* developing and implementing a performance and accountability
framework for ensuring that promised capabilities and benefits are
delivered on time and within budget.
To its credit, CBP has taken several steps to stem the pattern of cost,
schedule, and performance shortfalls that it experienced on early ACE
releases. However, future releases are unlikely to realize the impact
of these steps because revised ACE plans and actions are reintroducing
the same pattern that led to early release shortfalls. This pattern
includes not formally and transparently considering and proactively
addressing the risks associated with passing key release milestones
with known severe defects, building considerable overlap and
concurrency in the development schedules of releases that will contend
for the same resources, and not performing EVM on all releases. If this
pattern continues, the prospects for a successful program will be
diminished.
Although availability and responsiveness targets are largely being met
and long-standing help desk limitations are being addressed, the
prospects for a successful program nevertheless remain unclear. The
true measure of ACE's success is arguably the mission value that it
brings to CBP's and other agencies' trade-and border security-related
operations and users. Such value depends both on the operational
performance of ACE and on CBP's ability to demonstrate that this
performance is achieving program goals, delivering expected benefits,
and producing desired business results. At this juncture, however,
neither the system's performance nor its value is clear because of
several factors: the operational performance of deployed releases has
been mixed; users' satisfaction has been low; the relationships among
goals, benefits, and desired business outcomes are not evident; and the
range of measures needed to create a complete and realistic picture of
ACE's performance is missing.
In summary, a number of ACE activities have been and are being done
well; these have contributed to the program's progress to date and will
go a long way in determining the program's ultimate success. However,
CBP needs to effectively address long-standing ACE management
challenges along with emerging problems. Until it does so, ACE will
remain a risky program.
Recommendations:
To assist CBP in managing ACE and increasing the chances that it will
deliver required capabilities on time and within budget and demonstrate
promised mission benefits and results, we recommend that the DHS
Secretary direct the appropriate departmental officials to fully
address those legislative conditions associated with having an approved
privacy impact assessment and ensuring architectural alignment.
We also recommend that the DHS Secretary, through CBP's Acting
Commissioner, direct the Assistant Commissioner for Information and
Technology to:
* fully address those legislative conditions associated with measuring
ACE performance and results and employing effective IV&V practices;
* accurately report to the Appropriations Committees on CBP's progress
in implementing our prior recommendations;
* include in the June 30, 2006, quarterly update report to the
Appropriations Committees a strategy for managing ACE human capital
needs and the ACE framework for managing performance and ensuring
accountability;
* document key milestone decisions in a way that reflects the risks
associated with proceeding with unresolved severe defects and provides
for mitigating these risks;
* minimize the degree of overlap and concurrency across ongoing and
future ACE releases, and capture and mitigate the associated risks of
any residual concurrency;
* use EVM in the development of all existing and future releases;
* develop the range of realistic ACE performance measures and targets
needed to support an outcome-based, results-oriented accountability
framework, including user satisfaction with ACE; and:
* explicitly align ACE program goals, benefits, desired business
outcomes, and performance measures.
Agency Comments:
In their oral comments on a draft of this briefing, DHS and CBP
officials, including the Executive Director of Cargo Management
Systems, CBP, generally agreed with our findings, conclusions, and
recommendations and stated that the presentation was fair and balanced.
They also provided clarifying information that we incorporated as
appropriate in this briefing.
Attachment 1: Scope and Methodology:
Scope and Methodology:
To accomplish our objectives, we analyzed the ACE fiscal year 2006
expenditure plan and supporting documentation, comparing them to
relevant federal requirements and guidance, applicable best practices,
and our prior recommendations. We also interviewed DHS and CBP
officials, ACE program contractors, and officials at the port of
Blaine, Washington. In particular, we reviewed:
* DHS and CBP investment management practices, using OMB A-11, part 7;
* DHS and CBP certification activities for ensuring ACE compliance with
the DHS enterprise architecture;
* DHS and CBP acquisition management efforts, using SEI's SA-CMM;
* CBP cost estimating program and cost estimates, using SEI's
institutional and project-specific estimating guidelines;
* independent verification and validation (IV&V) activities using the
Institute of Electrical and Electronics Engineers Standard for Software
Verification and Validation;
* CBP actions to coordinate ACE with US-VISIT program documentation;
* CBP's reorganization documentation, including the new organizational
charts and roles and responsibilities matrix;
* ACE's accountability framework;
* ACE's performance using service level agreements;
* ACE's quality, using the ACE Support Team defect data and testing
results for Release 4 and Screening 1;
* cost and schedule data and program commitments from program
management documentation;
* CBP's progress toward increasing usage of ACE, against established
targets;
* level of user satisfaction, against survey scores, and:
* reliability of performance measures, by mapping the measures to
benefits.
For DHS-, CBP-, and contractor-provided data that our reporting
commitments did not permit us to substantiate, we have made appropriate
attribution indicating the data's source.
We conducted our work at CBP headquarters and contractor facilities in
the Washington, D.C., metropolitan area and at the port of Blaine,
Washington, from July 2005 through March 2006, in accordance with
generally accepted government auditing standards.
[End of Slide Presentation]
[End of section]
Appendix II: Comments from the Department of Homeland Security:
Homeland Security:
May 9, 2006:
Mr. Randolph C. Hite:
Director:
Information Technology Architecture and System Issues:
U.S. Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Mr. Hite:
Thank you for the opportunity to review and comment on the Government
Accountability Office's (GAO's) draft report GAO-06-580 entitled
INFORMATION TECHNOLOGY. Customs Has Made Progress on Automated
Commercial Environment System, but it Faces Long-Standing Management
Challenges and New Risks.
The U.S. Customs and Border Protection (CBP) agrees with the status of
open recommendations and new recommendations for CBP executive action.
The report indicates that an earlier recommendation regarding the roles
and responsibilities of the government and prime contractor has been
satisfied. The report also indicates that a previous recommendation on
reconciling Modernization expenditure plans with independent cost
estimates has been successfully satisfied for the second consecutive
year. CBP will continue using independent cost estimates to develop
future expenditure plans.
CBP is committed to fully addressing the remaining open recommendations
regarding: (1) cost estimating; (2) human capital management; (3) use
of the Automated Commercial Environment (ACE) for other Department of
Homeland Security (DHS) applications; (4) program management metrics
and measurements; and (5) implementation of an accountability
framework, including (a) coverage of all program commitment areas, (b)
currency and completeness of commitments made in expenditure plans, (c)
reliable data, (d) reporting on progress against commitments in future
expenditure plans, (e) establishing milestone exit criteria that
account for system maturity; and (6) quarterly reporting to Congress.
CBP notes that recommendations 1 and 3, aspects of recommendation 5,
and recommendation 6 are related to normal programmatic processes, and
as such, will likely remain open for the life of the program. CBP would
suggest that these recommendations not continue to be held as open
findings, but that they be continuously monitored through status
updates in quarterly reports to Congress on ACE.
CBP continues to follow established plans for developing and deploying
ACE capabilities that are enhancing cargo security while also
facilitating the flow of legitimate trade across our borders. Solid
program management and reporting systems have enabled the ACE program
to operate within 10 percent of the program baseline and within
approved program funding. CBP acknowledges the nine new recommendations
by the GAO in the draft report and intends to address them as part of
continuing efforts to enhance processes for balancing quality, cost,
and schedule, and maintain visibility of, and accountability for,
program commitments. The following is a summary of the agency's
progress toward, and plans for addressing, each of the nine new
recommendations detailed in the draft report.
Recommendation 1: Fully address those legislative conditions associated
with having an approved privacy impact assessment and ensuring
architectural alignment.
Response: CBP is committed to fully satisfying all legislative
conditions for the approval of Modernization expenditure plans. CBP is
taking steps to address the two subordinate elements of the
recommendation, as follows:
* The ACE Privacy Impact Assessment (PIA) has been updated based on
input from the DHS Privacy Office and the Federal Motor Carrier Safety
Administration (FMCSA). CBP provided a revised draft of the PIA to the
DHS Privacy Office on May 1, 2006 and will continue to work with that
staff to bring the matter to closure.
* CBP will work with DHS to ensure that an appropriate methodology is
used to evaluate the compliance of ACE with the DHS Enterprise
Architecture (EA) as part of the FY 2007 Modernization Expenditure Plan
development process.
Recommendation 2: Fully address those legislative conditions associated
with measuring ACE performance and results and employing effective
Independent Verification and Validation (IV&V) practices.
Response: Consistent with its commitment to fully satisfy all
legislative conditions for the approval of Modernization expenditure
plans, CBP is addressing the two subordinate elements of the
recommendation as follows:
* CBP will align and fully define program goals, benefits, business
results, and performance measures. CBP's approach for this effort,
which will be completed by July 1, 2006, and certified as complete by
the CBP Commissioner, is outlined below in the responses to
recommendations 8 and 9.
* Beginning October 2005, CBP took steps to align Modernization program
IV&V efforts more closely with the Institute of Electrical and
Electronic Engineers (IEEE) 1012-2004 Standard for Software
Verification and Validation. To further align with this standard, CBP
will complete and implement Version 2.0 of the IV&V Implementation and
Management Plan by June 1, 2006. Implementation of version 2.0 of the
subject plan will ensure IV&V efforts are aligned with the IEEE 1012-
2004 standard and address satisfaction of quality:
standards for all ACE products, as well as user needs, as defined
through requirements, use cases, and design documents.
Recommendation 3: Accurately report to the Appropriations Committees on
CBP's progress in implementing our prior recommendations.
Response: CBP will continue to report on progress toward addressing
open GAO recommendations through quarterly status updates to the
Appropriations and Authorization Committees. CBP intends to meet with
GAO representatives to ensure that quarterly status reports accurately
reflect a common understanding of the intent of each new and previous
recommendation, as well as critical success factors for fully
satisfying each recommendation. CBP will provide ACE third and fourth
quarter reports, including the status of prior recommendations, to the
Appropriations and Authorization Committees by September 1, 2006, and
December 1, 2006, respectively.
Recommendation 4: Include in the June 30, 2006, quarterly update report
to the Appropriations Committees a strategy for managing ACE human
capital needs and the ACE framework for managing performance and
ensuring accountability.
Response: Based on the Office of Information Technology (OIT) Strategic
Human Capital Management Plan, which is now under development, the ACE
program office will develop a complementary plan that maps OIT human
capital strategies to the ACE program, and as such, provides a strategy
for managing ACE human capital needs. By July 1, 2006, OIT will achieve
certification by the Acting CBP Commissioner that the OIT Strategic
Human Capital Management Plan is complete and mapped to the ACE
program.
The December 31, 2005, quarterly report on ACE, as well as the March
31, 2006, quarterly report, now under review, includes a copy of the
accountability framework that is being used as the basis for tracking
progress against program commitments. CBP will include, as part of the
third quarter report on ACE, a copy of the aforementioned
accountability framework and an outline of the key elements of the plan
for managing ACE human capital needs. CBP representatives intend to
meet with their GAO counterparts to ensure that the agency fully
understands the intent of the foregoing recommendation and the actions
that will be required to satisfy this intent. CBP plans to provide the
ACE third quarter report, including the strategy for managing ACE human
capital needs and the ACE framework for managing performance and
accountability, to the Appropriations and Authorization Committees by
September 1, 2006.
Recommendation 5: Document key milestone decisions in a way that
reflect the risks associated with proceeding with unresolved severe
defects and provide for mitigating these risks.
Response: The ACE program office has strengthened the Software
Development Lifecycle (SDLC) gate review process by ensuring that
specific risk assessment and acceptance at each review is a requirement
for proceeding to the next stage of the SDLC. In addition, the ACE Risk
and Issue:
Management Process has been updated to account for the need to identify
risks associated with proceeding beyond SDLC gate reviews. Under the
updated process, the designated ACE risk manager is working with
project teams to identify appropriate risks, mitigation plans, and
impact assessments prior to gate reviews so that gate review decisions
will be based on documentation that includes risks and their associated
impact. Documented risks are entered into ACE program office risk
management software to ensure that CBP has visibility of these risks
and can take action to mitigate them as appropriate. In addition, CBP
is working with the DHS Chief Information Officer to certify that each
release is ready to proceed beyond the Critical Design Review and
Production Readiness Review. The ACE program office will implement the
strengthened SDLC gate review process at the next scheduled SDLC gate
review.
Recommendation 6: Minimize the degree of overlap and concurrency across
ongoing and future ACE releases, and capture and mitigate the
associated risks of any residual concurrency.
Response: As noted in the draft GAO report, CBP has taken steps to
"decouple" Screening and Targeting (S&T) releases from ACE secure cargo
management releases, and will augment the Automated Targeting System
with new S&T capabilities. This approach will reduce system development
interdependencies between ACE S&T and secure cargo management
capabilities. CBP has also taken specific action in three areas to
reduce potential contention for common resources across ACE releases.
First, CBP has conducted extensive planning to ensure that development
milestones eliminate contention for computer hardware environments
needed for development, integration, testing, and training activities.
Second, CBP is centrally managing underlying ACE shared software
services to maximize efficient use of resources, enhance responsiveness
to workload peaks, and provide consistent technical management
approaches across releases. Third, CBP has divided ACE releases into
smaller groups of capabilities or "drops," which, in turn, will be
packaged into "deliveries" of ACE that can include capabilities from
more than one release. Managing ACE deliveries allows hardware
environments, system testing, integration with legacy systems,
training, and deployment activities, as well as required staffing, to
be managed across releases, thereby improving planning and reducing
resource contention.
CBP has in place a solid program for managing remaining concurrent
project and program activities and associated risks. Key elements of
this program management foundation include the following:
* Requirements Traceability Matrix (RTM): The RTM provides a complete
view of all ACE release and task order requirements, as well as the
interrelationships between these requirements, thus minimizing the
potential for duplicative efforts.
* Integrated Master Schedule (IMS) for ACE secure cargo management
capabilities: The IMS provides (1) the visibility of all project
release/delivery schedules and project interdependencies; (2) a
centralized repository for information on all gate reviews, milestones,
and project schedules; (3) comprehensive schedule information on the
use and availability of computer hardware "environments" that helps
reduce delays by ensuring environments are available when needed for
development efforts; (4) a calculated critical path for the entire
program that shows which activities are critical to staying on
schedule; (5) a view of all measurable project activity at the project
work package level; and (6) a view of what deliverables and/or
milestones are on track for timely completion or are behind schedule.
When status updates to the IMS place key program milestones at risk,
the IMS automatically notifies the ACE program office scheduling team
via e-mail. This feature allows program office staff members to
actively monitor program milestones and computer hardware environment
dependencies. Collectively, IMS capabilities provide a valuable tool
for evaluating the progress of the program and determining where
corrective actions may be required.
* Bi-weekly integration meetings: OIT directors and senior managers of
the prime contractor meet bi-weekly to discuss program issues and
concerns.
* Monthly Program Management Reviews (PMRs): OIT conducts monthly PMRs
to review the ACE Accountability Framework report, which details the
status of all ACE task orders and releases relative to program
commitments.
* Active Risk Manager (ARM): ARM, a leading software tool used across
industry and government, provides visibility of program risks and
issues and attendant mitigation plans. OIT uses ARM to track and manage
all identified program risks for the ACE program. The process for
identifying risks is reviewed and revised, as appropriate, to further
strengthen the risk identification process and ensure that it will
successfully lead to the identification of risks associated with
concurrent activities across the program.
* Risk Issue Forum (RIF): Monthly RIF meetings are conducted to
evaluate the status and impacts of risks and issues; take steps to
mitigate risks and issues, as appropriate; and determine whether new
risks and issues should be tracked across the program.
* Cost and schedule risk analysis: The ACE program office identifies.
potential program risks as part of the cost and schedule analysis that
is completed for each ACE Program Plan and Modernization Expenditure
Plan update. A sensitivity analysis is conducted to quantify these
risks and ensure that sufficient management reserve will be available
should the risk materialize. The output of cost and schedule analysis
are risk-adjusted cost and schedule estimates that are used to ensure
that sufficient time and funding are available within the program
baseline to allow for successful development of ACE. Each risk
identified as part of the cost and schedule risk analysis is entered
into the aforementioned ARM tool and actively managed to diminish the
probability that a risk will materialize.
* Earned Value Management (EVM): As discussed below in the response to
recommendation 7, EVM is a key tool for managing the development of ACE
releases. EVM provides early warning signals of potential problems as
well as the basis for making course corrections across the program.
Recommendation 7: Use EVM in the development of all existing and future
releases.
Response CBP is currently using EVM for the development of e-Manifest:
Trucks (Release 4) production baseline enhancements and Targeting
Foundation (S2). CBP plans to use EVM in the development of all future
releases, and to effect the implementation of EVM within 45 days after
task order award to the prime contractor. CBP intends to implement use
of EVM for e-Manifest: All Modes and Cargo Security (Release 6), e-
Manifest: Rail and Sea (M1), by May 31, 2006. The agency projects that
EVM will be implemented for Entry Summary, Accounts, and Revenue (ESAR)
(Release 5) Master Data and Enhanced Accounts (Al) and Entry Summary
and Revenue (A2) by June 30, 2006. Contract negotiations for Advanced
Targeting (S3) began in April 2006. CBP projects that EVM will be
implemented for ESAR Master Data and Enhanced Accounts (A 1) and Entry
Summary and Revenue (A2) by June 30, 2006.
Recommendation 8: Develop the range of realistic ACE performance
measures and targets needed to support an outcome-based, results-
oriented accountability framework, including user satisfaction with
ACE.
Response: CBP is committed to aligning and fully defining program
goals, benefits, business results, and performance measures. Toward
this end, CBP will update the performance measurement framework for
ACE, which is the basis for the overall ACE performance measurement
program. The updated framework will demonstrate the relationship
between CBP objectives and Desired Business Results (DBRs). Through
validation of DBRs and supporting performance measures, CBP will
institute a more realistic, appropriate, and comprehensive performance
measurement framework that includes assessments of user satisfaction,
CBP operational efficiency, and trade facilitation benefits. Based on
these performance measures, CBP will institutionalize data collection
on a monthly basis and report results against performance measures
through management reports such as the ACE Accountability Framework.
CBP expects to develop ACE performance measures by July 1, 2006.
Recommendation 9: Explicitly align ACE program goals, benefits, and
desired business outcomes, and performance measures.
Response: As discussed above in response to recommendation 8, CBP will
align ACE program goals, benefits, DBRs, and performance measures by
(1) adapting the Federal Enterprise Architecture and CBP Performance
Reference Model framework for ACE performance measures, including DBRs;
and (2) revising specific performance objectives to ensure they are
realistic and aligned with DBRs. The foregoing planned efforts are
expected to further clarify the mission impact of ACE and provide a
solid foundation for evaluating progress against commitments. CBP
expects to align ACE program goals, benefits, and desired business
outcomes, and performance measures by July 1, 2006.
Mindful of the imperative to detect terrorist efforts to exploit our
Nation's supply chain, while also facilitating legitimate trade, CBP is
focused on ensuring that ACE will meet high standards for usability and
operational effectiveness within cost and on schedule. CBP looks
forward to working with GAO to address previous and new GAO
recommendations, and will continue efforts to make transparent the
agency's progress toward satisfying these recommendations.
Thank you again for the opportunity to comment on this draft report and
we look forward to working with you on future homeland security issues.
Sincerely,
Signed by:
Steven J. Pecinovsky:
Director:
Departmental GAO/OIG Liaison Office:
[End of section]
Appendix III: Contact and Staff Acknowledgments:
GAO Contact:
Randolph C. Hite, (202) 512-3459:
Staff Acknowledgments:
In addition to the person named above, Justin Booth, Barbara Collier,
William Cook, Neil Doherty, Michael Marshlick, Shannin O'Neill, Tomas
Ramirez, and Jennifer Vitalbo made key contributions to this report.
FOOTNOTES
[1] Pub. L. 109-90 (Oct. 18, 2005).
[2] OMB Circular A-11 establishes policy for planning, budgeting,
acquisition, and management of federal capital assets.
[3] The purpose of the Investment Review Board is to integrate capital
planning and investment control, along with the budgeting, acquisition,
and management of investments. It is also to ensure that spending on
investments directly supports and furthers the mission, and that this
spending provides optimal benefits and capabilities to stakeholders and
customers.
[4] The purpose of a privacy impact assessment is to ensure that there
is no collection, storage, access, use, or dissemination of
identifiable personal or business information that is not both needed
and permitted.
[5] Institute of Electrical and Electronics Engineers Computer Society,
Standard for Software Verification and Validation 1012-1998 (June 8,
2005).
[6] CBP's implementation of this recommendation is complete with
respect to the fiscal year 2006 expenditure plan.
[7] SEI's institutional and project-specific estimating guidelines are
defined respectively in Robert E. Park, Checklists and Criteria for
Evaluating the Cost and Schedule Estimating Capabilities of Software
Organizations, CMU/SEI-95-SR-005, and A Manager's Checklist for
Validating Software Cost and Schedule Estimates, CMU/SEI-95-SR-004
(Pittsburgh, Pa.: Carnegie Mellon University Software Engineering
Institute, 1995).
[8] US-VISIT is a governmentwide program to collect, maintain, and
share information on foreign nationals for enhancing national security
and facilitating legitimate trade and travel, while adhering to U.S.
privacy laws and policies.
[9] The Automated Targeting System is a DHS system that targets
containers for inspection based on a perceived level of risk.
[10] Screening is the method of determining high-risk people or
shipments before their arrival at a port. Targeting is the risk-based
determination of whether a shipment should undergo additional
documentary review or physical inspection.
[11] Earned value management is a tool for measuring program progress
by comparing the value of work accomplished during a given period with
that of the work expected in that period; this comparison permits
performance to be evaluated based on calculated variances from the
planned cost and schedule.
[12] CBP was formed from the former U.S. Customs Service and other
entities with border protection responsibility.
[13] Targeting capabilities are currently being provided by DHS's
Automated Targeting System. This system targets containers for
inspection based on a perceived level of risk. ACE is intended to
leverage these capabilities.
[14] Pub. L. 109-90 (Oct. 18, 2005).
[15] OMB Circular A-11 establishes policy for planning, budgeting,
acquisition, and management of federal capital assets.
[16]The purpose of the Investment Review Board is to integrate capital
planning and investment control, budgeting, acquisition, and management
of investments. It is also to ensure that spending on investments
directly supports and furthers the mission and that this spending
provides optimal benefits and capabilities to stakeholders and
customers.
[17]A release is the act of CBP permitting imported merchandise to
enter the United States.
[18] An entry is the documentation required to be submitted to CBP in
order for it to permit imported merchandise to enter the United States.
[19] Screening is the method of determining high-risk people or
shipments before their arrival at a port.
[20] Targeting is the risk-based determination of whether a shipment
should undergo additional documentary review or physical inspection.
[21] SAP is a commercial enterprise resource planning software product
that has multiple modules, each performing separate but integrated
business functions. ACE will use SAP to support many of its business
processes and functions. CBP's Modernization Office is also using SAP
as part of a joint project with its Office of Finance to support
financial management, procurement, property management, cost
accounting, and general ledger processes.
[22] CBP national account managers work with the largest importers.
[23] Brokers obtain licenses from CBP to conduct business on behalf of
the importers by filling out paperwork and obtaining a bond; carriers
are individuals or organizations engaged in transporting goods for
hire.
[24] Manifests are lists of passengers or invoices of cargo for a
vehicle, such as a truck, ship, or plane.
[25] The multimodal manifest involves the processing and tracking of
cargo as it transfers between different modes of transportation, such
as cargo that arrives by ship, is transferred to a truck, and then is
loaded onto an airplane.
[26] An import activity summary statement is a summary of an importer's
shipment activities over a specific period of time that is transmitted
electronically to CBP on a periodic basis by importers and brokers.
[27] In March 2001, appropriations committees approved the use of $5
million in stopgap funding to fund program management office
operations.
[28] GAO, Information Technology. Early Releases of Customs Trade
System Operating, but Pattern of Cost and Schedule Problems Needs to Be
Addressed, GAO-04-719 (Washington, D.C.: May 14, 2004).
[29] GAO, Information Technology. Customs Automated Commercial
Environment Progressing, but Need for Management Improvements
Continues, GAO-05-267 (Washington, D.C.: Mar. 14, 2005).
[30] GAO-05-267.
[31] IEEE Computer Society, Standard for Software Verification and
Validation 1012-1998 (June 8, 2005).
[32]With respect to the fiscal year 2006 expenditure plan.
[33] For these models, see Robert E. Park, Checklists and Criteria for
Evaluating the Cost and Schedule Estimating Capabilities of Software
Organizations (Pittsburgh, Pennsylvania: SEI, Carnegie Mellon
University, 1995); A Manager's Checklist for Validating Software Cost
and Schedule Estimates (Pittsburgh, Pennsylvania: SEI, Carnegie Mellon
University, 1995).
[34] United States Visitor and Immigrant Status Indicator Technology
(US- VISIT) is a governmentwide program to collect, maintain, and share
information on foreign nationals for enhancing national security and
facilitating legitimate trade and travel, while adhering to U.S.
privacy laws and policies.
[35] GAO-04-719.
[36]Chip Gliedman, Thirty-One Best Practices for the Service Desk
(Forrester Research, Inc., 2005).
[37] Electronic manifests provide truck information such as driver/
passenger data, vehicle data, and shipment details to CBP officers.
[38] Clinger-Cohen Act of 1996, 40 U.S.C. 1101-11703; and OMB Circular
A-130, Management of Federal Information Resources (Nov. 30, 2000).
[39] OMB Circular No. A-11, Part 7 (revised June 2005).
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
