Information Technology
Improvements for Acquisition of Customs Trade Processing System Continue, but Further Efforts Needed to Avoid More Cost and Schedule Shortfalls Gao ID: GAO-08-46 October 25, 2007The Department of Homeland Security (DHS) established the Automated Commercial Environment (ACE) program to replace and supplement existing cargo processing technology. According to the fiscal year 2007 DHS appropriations act, DHS is to develop and submit an expenditure plan for ACE that satisfies certain conditions, including being reviewed by GAO. GAO reviewed the plan to (1) determine whether the expenditure plan satisfies the legislative conditions, (2) determine the status of 15 open GAO recommendations, and (3) provide observations about the expenditure plan and DHS's management of the program. To address the mandate, GAO assessed plans and related documentation against federal guidelines and industry standards and interviewed the appropriate DHS officials.
The ACE expenditure plan satisfies many--but not all--of the legislative conditions specified in the fiscal year 2007 DHS appropriations act. Specifically, the plan (with related program documentation and officials' statements) complies with acquisition rules, requirements, guidelines, and management practices of the federal government; includes a DHS certification that an independent verification and validation agent is under contract; was reviewed and approved by DHS and the Office of Management and Budget (OMB); and was reviewed by GAO. In addition, it partially satisfies conditions for meeting the capital planning and investment control review requirements established by OMB in Circular A-11 (part 7), including information security, and for complying with the DHS enterprise architecture. DHS has implemented eight open GAO recommendations made during the past 4 years, including those related to performance measures and targets, independent verification and validation, cost estimation, and program reporting. Seven other recommendations made during this time are in the process of being implemented. With respect to these seven, DHS has taken steps to satisfy each, such as establishing an accountability framework, reducing overlap and concurrence among ACE releases, and completing a privacy impact assessment, and actions are under way or planned to more fully address them. GAO is making three new observations about the expenditure plan and the management of ACE. First, the program is taking needed steps to redefine requirements for several ACE releases because of limitations in the completeness of original requirements, but this redefinition is likely to introduce significant program schedule delays and cost increases. Second, the changes to ACE requirements have led to replacement of a key commercial product, but the new product carries the risk of negatively impacting user productivity. Third, the automated database used for managing ACE risks is incomplete and does not contain information needed to adequately inform program decisions. All told, DHS has continued to make progress on ACE, and the program is better positioned today for delivering promised capabilities and benefits than it has been in the past. Nevertheless, key program management practices relating to, for example, human capital management, requirements management, and risk management remain a challenge, and other management areas, such as information security and architecture alignment, continue to require attention. As a result, GAO sees major program schedule delays and cost overruns on the horizon. To improve ACE management and minimize exposure to risk, it is important for DHS to remain vigilant in its efforts to satisfy ACE legislative requirements, fully implement prior GAO recommendations, and keep Congress fully informed about the program's status, plans, and risk.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-08-46, Information Technology: Improvements for Acquisition of Customs Trade Processing System Continue, but Further Efforts Needed to Avoid More Cost and Schedule Shortfalls
This is the accessible text file for GAO report number GAO-08-46
entitled 'Information Technology: Improvements for Acquisition of
Customs Trade Processing System Continue, but Further Efforts Needed to
Avoid More Cost and Schedule Shortfalls' which was released on October
25, 2007.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Committees:
October 2007:
Information Technology:
Improvements for Acquisition of Customs Trade Processing System
Continue, but Further Efforts Needed to Avoid More Cost and Schedule
Shortfalls:
GAO-08-46:
GAO Highlights:
Highlights of GAO-08-46, a report to congressional committees.
Why GAO Did This Study:
The Department of Homeland Security (DHS) established the Automated
Commercial Environment (ACE) program to replace and supplement existing
cargo processing technology. According to the fiscal year 2007 DHS
appropriations act, DHS is to develop and submit an expenditure plan
for ACE that satisfies certain conditions, including being reviewed by
GAO.
GAO reviewed the plan to (1) determine whether the expenditure plan
satisfies the legislative conditions, (2) determine the status of 15
open GAO recommendations, and (3) provide observations about the
expenditure plan and DHS‘s management of the program. To address the
mandate, GAO assessed plans and related documentation against federal
guidelines and industry standards and interviewed the appropriate DHS
officials.
What GAO Found:
The ACE expenditure plan satisfies many”but not all”of the legislative
conditions specified in the fiscal year 2007 DHS appropriations act.
Specifically, the plan (with related program documentation and
officials‘ statements) complies with acquisition rules, requirements,
guidelines, and management practices of the federal government;
includes a DHS certification that an independent verification and
validation agent is under contract; was reviewed and approved by DHS
and the Office of Management and Budget (OMB); and was reviewed by GAO.
In addition, it partially satisfies conditions for meeting the capital
planning and investment control review requirements established by OMB
in Circular A-11 (part 7), including information security, and for
complying with the DHS enterprise architecture.
DHS has implemented eight open GAO recommendations made during the past
4 years, including those related to performance measures and targets,
independent verification and validation, cost estimation, and program
reporting. Seven other recommendations made during this time are in the
process of being implemented. With respect to these seven, DHS has
taken steps to satisfy each, such as establishing an accountability
framework, reducing overlap and concurrence among ACE releases, and
completing a privacy impact assessment, and actions are under way or
planned to more fully address them.
GAO is making three new observations about the expenditure plan and the
management of ACE. First, the program is taking needed steps to
redefine requirements for several ACE releases because of limitations
in the completeness of original requirements, but this redefinition is
likely to introduce significant program schedule delays and cost
increases. Second, the changes to ACE requirements have led to
replacement of a key commercial product, but the new product carries
the risk of negatively impacting user productivity. Third, the
automated database used for managing ACE risks is incomplete and does
not contain information needed to adequately inform program decisions.
All told, DHS has continued to make progress on ACE, and the program is
better positioned today for delivering promised capabilities and
benefits than it has been in the past. Nevertheless, key program
management practices relating to, for example, human capital
management, requirements management, and risk management remain a
challenge, and other management areas, such as information security and
architecture alignment, continue to require attention. As a result, GAO
sees major program schedule delays and cost overruns on the horizon. To
improve ACE management and minimize exposure to risk, it is important
for DHS to remain vigilant in its efforts to satisfy ACE legislative
requirements, fully implement prior GAO recommendations, and keep
Congress fully informed about the program‘s status, plans, and risk.
What GAO Recommends:
GAO is making recommendations to further strengthen ACE management and
accountability by disclosing program information in quarterly reports
to Congress related to unmet legislative conditions, open GAO
recommendations, program changes, and risk management. DHS agreed with
GAO‘s findings and recommendations and described actions that it has
under way and planned to address them. Most of the described actions
are consistent with GAO‘s recommendations.
To view the full product, including the scope and methodology, click on
[hyperlink, http://www.GAO-08-46]. For more information, contact
Randolph C. Hite at (202) 512-3459 or hiter@gao.gov.
[End of section]
Contents:
Letter:
Compliance with Legislative Conditions:
Status of GAO Recommendations:
Observations on the Expenditure Plan and Management of ACE:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendixes:
Appendix I: Briefing to Subcommittees on Homeland Security, House and
Senate Committees on Appropriations:
Appendix II: Comments from the Department of Homeland Security:
Appendix III: Contact and Staff Acknowledgments:
ACE: Automated Commercial Environment:
CBP: U.S. Customs and Border Protection:
CIO: chief information officer:
COTS: commercial off-the-shelf:
DHS: Department of Homeland Security:
GBB: global business blueprint:
ITS: internet transaction server:
OMB: Office of Management and Budget:
PIA: privacy impact assessment:
October 25, 2007:
The Honorable Robert C. Byrd:
Chairman:
The Honorable Thad Cochran:
Ranking Minority Member:
Subcommittee on Homeland Security:
Committee on Appropriations:
United States Senate:
The Honorable David E. Price:
Chairman:
The Honorable Harold Rogers:
Ranking Minority Member:
Subcommittee on Homeland Security:
Committee on Appropriations:
House of Representatives:
The U.S. Customs and Border Protection (CBP), as part of the Department
of Homeland Security (DHS), submitted to Congress in February 2007 its
fiscal year 2007 expenditure plan for the Automated Commercial
Environment (ACE) program pursuant to the Department of Homeland
Security Appropriations Act, 2007.[Footnote 1]
Begun in 2001, ACE is to replace and supplement existing cargo
processing technology. The system is being developed and deployed in a
series of increments to about 300 ports of entry and is expected to be
fully deployed by the end of 2011. The goals of ACE include (1)
supporting border security by enhancing analysis and information
sharing with other government agencies and providing CBP with the means
to decide before a shipment reaches the border what should be targeted
because it is a security threat and what should be expedited because it
complies with U.S. laws and (2) streamlining time-consuming and labor-
intensive tasks for CBP personnel and the trade community through a
national trade account and a single Web-based interface.
As required by the appropriations act, we reviewed ACE's fiscal year
2007 expenditure plan. Our objectives were to (1) determine whether the
expenditure plan satisfies the legislative conditions, (2) determine
the status of 15 open GAO recommendations for ACE,[Footnote 2] and (3)
provide observations about the expenditure plan and DHS's management of
the program.
On July 26, 2007, we provided a briefing to the staffs of the
Subcommittees on Homeland Security, Senate and House Committees on
Appropriations on the results of our review. This report transmits
those results. The full briefing, including our scope and methodology,
is reprinted in appendix I.
Compliance with Legislative Conditions:
The ACE expenditure plan--including related program documentation and
program officials' statements--satisfies or partially satisfies the six
legislative conditions specified in the appropriations act.
Specifically, the plan satisfies the conditions that it (1) comply with
the acquisition rules, requirements, guidelines, and systems
acquisition management practices of the federal government;[Footnote 3]
(2) include a certification by the DHS Chief Information Officer that
an independent verification and validation agent is currently under
contract for the program; (3) be reviewed by the DHS Investment Review
Board, the Secretary of DHS, and Office of Management and Budget (OMB);
and (4) be reviewed by us. The plan and its supporting information
partially satisfies the conditions that it (5) meet the capital
planning and investment control review requirements established by OMB
(including Circular A-11, part 7) and that it (6) comply with the DHS
enterprise architecture. For example, although DHS determined that ACE
was aligned with the DHS enterprise architecture, the agency's analysis
did not address key aspects of an architectural alignment, such as
alignment with the architecture's data reference model.
Status of GAO Recommendations:
DHS has implemented some, but not all, of the recommendations
pertaining to ACE that we have made since 2003. These recommendations,
along with their status, are summarized here.
Eight recommendations have been implemented. For example, DHS has:
* developed a range of ACE performance measures and targets needed to
support an outcome-based, results-oriented accountability framework,
including user satisfaction with ACE;
* aligned ACE program goals, benefits, desired business outcomes, and
performance measures;
* addressed those legislative conditions associated with measuring ACE
performance and results and employing effective independent
verification and validation practices; and:
* ensured that future expenditure plans are based on cost estimates
that are reconciled with independent cost estimates.
The remaining seven recommendations are in the process of being
implemented, as described here.
* Recommendation: Define and implement an ACE accountability framework
that fulfills several conditions,[Footnote 4] to include ensuring the
currency, relevance, and completeness of commitments made to Congress
in expenditure plans and reporting in future expenditure plans the
progress against commitments that were contained in prior expenditure
plans.
In progress. The program has established an accountability framework
that is providing input for both the annual expenditure plan and
quarterly congressional reports. However, as with prior expenditure
plans, the fiscal year 2007 expenditure plan did not reflect the most
current program information. For example, information on milestones,
earned value management, and risks were about 4 months old when the
plan was submitted to the appropriations committees in February 2007,
and program commitments were no longer current. Further, while program
officials continue to use the quarterly ACE status reports to provide
the appropriations committees with more detailed information, these
reports are generally submitted to Congress 3 to 4 months after the end
of each quarter, thus limiting their currency and relevance as well.
CBP and DHS officials told us that the delays were due to the DHS
review and approval process and that they are exploring ways to
accelerate the process.
The 2007 expenditure plan did not adequately report on progress against
previous plan commitments. For example, the plan did not (1) report
progress against milestones in the fiscal year 2006 plan or explain why
these milestones were not achieved, (2) report actual obligation or
expenditure of funds relative to the planned uses of these funds in
prior expenditure plans, or (3) address progress against the milestone
dates for each stage of a release that was included in the prior year's
plan. According to program officials, the quarterly congressional
reports provide more current information on the program's progress
against prior commitments. However, these reports have also not fully
addressed the commitments made in prior expenditure plans. Program
officials stated that they intend to start providing this information
in the last quarterly report of each year.
* Recommendation: Define measures and collect and use associated
metrics for determining whether prior and future program management
improvements are successful.
Planned. The program office has made changes that are to improve
overall program management and, according to its December 2006
quarterly report to Congress, the program office plans to measure the
impact of future management improvements. Moreover, this report stated
that the program anticipates more changes, including creation of a
cargo requirement management board to decide the disposition of all
change requests to production systems; establishment of a new invoice
review policy; and colocation of personnel within a given business
area. However, program officials told us that they have yet to define
measures to determine the impact of such changes and thus are not yet
positioned to determine their success.
* Recommendation: Minimize the degree of overlap and concurrency across
ongoing and future ACE releases and capture and mitigate the associated
risks of any residual concurrence.
In progress. Since May 2006, the program office has reduced overlap and
concurrence of ACE releases and has taken actions to reduce potential
contention for limited resources by, for instance, decoupling (i.e.,
reducing dependencies among) certain program components; dividing
releases into smaller subreleases to provide more flexibility in
scheduling; improving planning for development, integration, testing,
training activities, and milestones to better schedule use of
development and test environments; and centralizing management of
shared software services. Further, the program office conducts regular
integration meetings with the teams supporting each release to discuss
concerns, decisions, and schedules associated with resource
availability and is using a software tool to track and mitigate release-
specific concurrency risks. However, this tool contains vague or
incomplete data relative to mitigating these risks. These data
limitations make it difficult to determine the status or the
effectiveness of the efforts to reduce the risks associated with
overlap and concurrence among releases.
* Recommendation: Direct the appropriate departmental officials to
fully address those legislative conditions associated with having an
approved privacy impact assessment (PIA)[Footnote 5] and ensuring
architectural alignment.
In progress. One legislative condition states that the plan should meet
OMB's capital planning and investment control review requirements,
which include addressing security and privacy issues. The program
office has developed a PIA for ACE release 4 (e-Manifest: Trucks),
which is currently operational. DHS approved this PIA on July 14, 2006.
This PIA addressed the major elements of DHS's guidance, and program
officials stated that they would update the assessment for each ACE
release. However, this PIA did not cover other recently completed
screening releases. Program officials told us that these screening
releases were considered to be part of the Automated Targeting System
and were covered by the Automated Targeting System's PIA. However, this
PIA does not specifically identify or address ACE screening releases.
With respect to architectural alignment, DHS determined in May 2007
that all required ACE products and technologies were aligned with the
DHS technical reference model and that ACE was thus aligned with the
DHS enterprise architecture. However, we have yet to receive sufficient
documentation describing the criteria and methodology used to make
these determinations or verifiable analysis supporting the
determinations. Moreover, the determinations were based on technical
alignment and did not address other relevant aspects of program
alignment to an enterprise architecture, such as data alignment.
* Recommendation: Develop and implement key human capital management
practices.
In progress. In June 2006, the Office of Information Technology
Strategic Human Capital Management Plan, which included an ACE-specific
appendix as a road map for effective management of human capital
resources, was approved. However, the plan does not address the basic
tenets of effective human capital management, such as defining the
positions needed (including core competencies) to perform core program
functions, assessing and inventorying current workforce skills and
abilities, assessing any gaps between needed and existing workforce
levels and capabilities, and filling identified gaps. Officials
acknowledged these limitations in the plans and stated that they are
developing an implementation plan to address these shortfalls.
* Recommendation: Include in the June 30, 2006, quarterly update report
to the appropriations committees a strategy for managing ACE human
capital needs and the ACE framework for managing performance and
ensuring accountability.
In progress. The June 30, 2006, quarterly report to the House and
Senate Appropriations Committees included the ACE program's strategy
for meeting its human capital needs and its accountability framework;
however, the human capital strategy does not meet the basic tenets of
strategic human capital management, as previously explained.
* Recommendation: Accurately report to the appropriations committees on
progress in implementing our prior recommendations.
In progress. Quarterly reports to the House and Senate Appropriations
Committees have contained information on the status of our open
recommendations since November 2002, but recent reports remain outdated
due to a 2-to 7-month time lapse between when the reports are produced
and when they are provided to the appropriations committees. DHS and
program officials stated that they are exploring ways to accelerate the
review process and thereby improve the timeliness and accuracy of their
reports to Congress.
Observations on the Expenditure Plan and Management of ACE:
We have three observations about ACE requirements, commercial product
selection, and risk management.
* Requirements: Redefinition of requirements for several ACE releases
is now under way to address limitations in completeness of originally
defined requirements, and this redefinition is likely to introduce
program schedule delays and cost increases.
In defining ACE requirements, the program office discovered that its
original approach did not adequately engage all key stakeholders, such
as software programmers and subject matter experts, for the legacy
system ACE is intended to replace. To address this, key stakeholders
are now collaborating and decomposing legacy code. However, this effort
is expected to significantly delay some system releases and drops.
Program officials have taken several actions that they say will
minimize the impact of the delays, such as prioritizing shared
functionality, dividing releases/drops into smaller increments, and
changing release deployment strategies. However, these changes have yet
to be approved, and the full extent of the cost and schedule
implications is not yet known. Moreover, neither the fiscal year 2007
expenditure plan nor the ACE quarterly reports have disclosed the
requirements redefinition and its impact, and neither has addressed any
changes to release deployment strategies.
* Commercial product selection: Significant changes to ACE requirements
have led to reevaluation and replacement of a key commercial off-the-
shelf (COTS) product previously selected and being prepared for use.
The program office conducted several analyses to determine which COTS
products would best meet ACE system requirements, including a general
review of various packages in 2002 to select a provider and a more
detailed analysis in 2004 to define and allocate ACE requirements and
select a specific product--the SAP Enterprise Portal. In December 2006,
however, the ACE Chief System Architect determined through additional
analysis that all planned SAP functionality could be provided by
Internet Transaction Server (ITS) technology and recommended that ITS
be adopted. The program office subsequently stopped work on SAP
Enterprise Portal design and configuration efforts and reported that
ITS would be used for release 5/drop A2 instead of the SAP Enterprise
Portal. This decision is expected to have some near-term schedule
impacts because much of the completed work for A2 had been based on the
planned use of SAP Enterprise Portal. Further, use of ITS raises the
risk of inadequate user response time, which would, in turn, negatively
impact user productivity and introduce a high probability of
significant cost and schedule impacts. Program officials reported that
actions are under way to mitigate the risk through performance modeling
and test planning. However, neither the fiscal year 2007 expenditure
plan nor the quarterly reports to Congress disclose this COTS product
change, its impact on release schedules and cost estimates, or the risk
to future system performance.
* Risk management: All program risks are not being effectively managed.
The program office has developed a process guide and implemented an
automated tool (database) for managing ACE risks in accordance with
relevant guidance and best practices. Although the database contains
fields to provide a description, level (high, medium, or low), and
mitigation strategy (including start and end dates, exit criteria, and
implementation status) for each risk, the completeness and quality of
this information varies. Because of such database limitations, we could
not determine the status of and mitigation progress on 17 risks.
Moreover, these database limitations were not reflected in the
documentation used at key program events, indicating that the program
does not have the risk-related information that it needs to inform its
program decisions and to reduce the chances of potential problems
becoming actual problems. Program officials stated that they are taking
steps to improve risk management, including establishing a group to
ensure the quality and completeness of the database, holding regular
group meetings with contract staff and team leads to discuss risks and
their impacts, and conducting risk management training. To date,
however, program risks have not been communicated to oversight
organizations through the 2007 expenditure plan or recent quarterly
reports to the House and Senate Appropriations Committees.
Conclusions:
Over the past 7 years, CBP and DHS have worked to fulfill legislatively
mandated annual expenditure plan requirements and to implement dozens
of our recommendations related to these plans and management of the
program. Among other things, these requirements and recommendations
have promoted effective program management and accountability for
performance and results. As a result of these years of effort, the ACE
program is better positioned today for delivering promised capabilities
and benefits than it has been in the past.
Nevertheless, key program management practices relating to, for
example, human capital management, requirements management, and risk
management continue to remain a challenge, and other management areas,
such as information security and architecture alignment, continue to
require attention. As a result, avoiding major program schedule delays
and cost overruns remains a challenge as more of each appears to be on
the horizon. To further improve ACE management and minimize its
exposure to risk, it is important for CBP and DHS to remain vigilant in
their efforts to satisfy ACE legislative requirements and to fully
implement our prior recommendations. Moreover, it is important that
they keep Congress fully informed on where the program stands and what
changes are planned to address emerging cost overruns and schedule
delays.
Recommendations for Executive Action:
To further strengthen ACE management and promote accountability for ACE
performance and results, we are recommending that the Secretary of
Homeland Security direct the CBP Commissioner to ensure that future
quarterly reports to the House and Senate Appropriations Committees
disclose:
* the risks and associated mitigation strategies of not having fully
satisfied the expenditure plan legislative conditions and not having
completed implementation of all our prior recommendations;
* the status and impacts on the program's estimated cost and schedule
and lessons learned from ongoing efforts to redefine requirements and
to implement a different COTS product than originally selected; and:
* the program's plans and actions for improving ACE risk management and
its current inventory of program risks, including their associated
mitigation strategies and the status of the strategies' implementation.
Agency Comments and Our Evaluation:
In written comments on a draft of this report signed by the Director,
Departmental GAO/OIG Liaison, and reprinted in appendix II, DHS agreed
with our findings and stated that it is committed to addressing them.
Further, the department agreed with our recommendations and described
actions that it said are under way or planned to address them. While
most of the department's stated actions are consistent with our
recommendations, in one case they may not be sufficient. Specifically,
the department stated that it would ensure that future expenditure
plans meet all OMB capital planning and investment control review
requirements by attaching the required OMB budget submission for ACE,
referred to as an Exhibit 300, to all future plans. However, we
reviewed the fiscal year 2007 ACE Exhibit 300 as part of our review of
the fiscal year 2007 ACE expenditure plan and found that it did not
meet the OMB requirements cited above. Therefore, unless DHS improves
the quality of future ACE Exhibit 300s by addressing the weaknesses we
have identified, this action alone may not fully address our
recommendation.
We are sending copies of this report to the Chairmen and Ranking
Minority Members of the other Senate and House committees and
subcommittees that have authorization and oversight responsibilities
for homeland security. We are also sending copies to the DHS Secretary,
the CBP Commissioner and, on their request, to other interested
parties. In addition, the report will be available at no charge on the
GAO Web site at [hyperlink, http://www.gao.gov].
Should you or your offices have any questions on matters discussed in
this report, please contact me at (202) 512-3459 or at h [Hyperlink,
hiter@gao.gov] iter@gao.gov. Contact points for our Offices of
Congressional Relations and Public Affairs may be found on the last
page of this report. Other contacts and key contributors to this report
are listed in appendix III.
Signed by:
Randolph C. Hite:
Director, Information Technology Architecture:
and Systems Issues:
[End of section]
Appendix I: Briefing to Subcommittees on Homeland Security, House and
Senate Committees on Appropriations:
Information Technology: Management Improvements for Acquisition of
Customs Trade Processing System Continue, but Further Efforts Needed to
Avoid More Cost and Schedule Shortfalls:
Briefing to the Staffs of the Subcommittees on Homeland Security,
Senate and House Committees on Appropriations:
July 26, 2007:
Briefing Overview:
Introduction:
Objectives:
Results in Brief:
Background:
Results:
* Legislative Conditions:
* Status of Recommendations:
* Observations:
Conclusions:
* Recommendations for Executive Action:
Agency Comments:
Attachment 1. Scope and Methodology:
Attachment 2. Related GAO Products:
Introduction:
The U.S. Customs and Border Protection[Footnote 6](CBP) is developing a
new import and export processing system, referred to as the Automated
Commercial Environment (ACE), to replace and supplement existing cargo
processing technology. Begun in 2001, this system is being developed
and deployed in a series of increments to about 300 ports of entry and
is expected to be fully deployed by the end of 2011.
The goals of ACE include:
* supporting border security by enhancing analysis and information
sharing with other government agencies and providing CBP with the means
to decide before a shipment reaches the border what should be targeted
because it is a security threat and what should be expedited because it
complies with U.S. laws; and:
* streamlining time-consuming and labor-intensive tasks for CBP
personnel and the trade community through a national trade account and
a single Web-based interface.
The Department of Homeland Security Appropriations Act, 2007,[Footnote
7] states that DHS may not obligate $216.8 million of the $316.8
million appropriated for ACE until the House and Senate Committees on
Appropriations receive a plan for expenditure that:
1. meets the capital planning and investment control review
requirements established by the Office of Management and Budget (OMB),
including Circular A-11, part 7;[Footnote 8]:
2. complies with DHS‘s information systems enterprise architecture;
3. complies with the acquisition rules, requirements, guidelines, and
systems acquisition management practices of the federal government;
4. includes a certification by the Chief Information Officer (CIO) of
DHS that an independent verification and validation agent (IV&V) is
currently under contract for the project;
5. is reviewed and approved by the DHS Investment Review Board
(IRB),[Footnote 9] the Secretary of Homeland Security, and OMB; and:
6. is reviewed by GAO.
On February 6, 2007, DHS submitted its fiscal year 2007 expenditure
plan for $316.8 million to the House and Senate Appropriations
Subcommittees on Homeland Security. In addition, CBP submits quarterly
reports to the House and Senate Appropriations Committees to keep them
apprised of ACE progress and issues.
Objectives:
As agreed, our objectives were to:
1. determine whether the ACE fiscal year 2007 expenditure plan
satisfies the legislative conditions,
2. determine the status of 15 open GAO recommendations for
ACE,[Footnote 10] and:
3. provide observations about the expenditure plan and DHS‘s management
of the ACE program.
We conducted our work at CBP headquarters and contractor facilities in
the Washington, D.C., metropolitan area from December 2006 through July
2007 in accordance with generally accepted government auditing
standards. Details of our scope and methodology are provided in
attachment 1. Related GAO products are in attachment 2.
Results in Brief: Objective 1:
Legislative Conditions:
Table: Summary of satisfaction of legislative conditions:
Legislative condition: Meets the capital planning and investment
control review requirements established by OMB, including OMB Circular
A-11, part 7;
Status [A] [B]: Partially satisfied.
Legislative condition: Complies with the DHS enterprise architecture;
Status [A] [B]: Partially satisfied.
Legislative condition: Complies with the acquisition rules,
requirements, guidelines, and systems acquisition management practices
of the federal government;
Status [A] [B]: Satisfied.
Legislative condition: Includes a certification by the DHS CIO that an
IV&V agent is currently under contract for the project;
Status [A] [B]: Satisfied.
Legislative condition: Is reviewed and approved by the DHS IRB,
Secretary of DHS, and OMB;
Status [A] [B]: Satisfied.
Legislative condition: Is reviewed by GAO;
Status [A] [B]: Satisfied.
Source: GAO.
[A] Partially satisfied means that the plan, in combination with
supporting documentation, either satisfied or provided for satisfying
many, but not all, key aspects of the condition that we reviewed.
[B] Satisfied means that the plan, in combination with supporting
documentation, either satisfied or provided for satisfying every aspect
of the condition that we reviewed.
[End of table]
Results in Brief: Objective 2:
Open Recommendations:
Table: Implementation of prior GAO recommendations GAO
Recommendations[A]:
GAO Recommendation: 1. Define and implement an ACE accountability
framework that ensures;
Status [B,C,D]: [Empty].
GAO Recommendation: 1. Define and implement an ACE accountability
framework that ensures: a. coverage of all program commitment areas,
including key expected or estimated system (1) capabilities, use, and
quality; (2) benefits and mission value; (3) costs; and (4) milestones
and schedules;
Status [B,C,D]: Complete.
GAO Recommendation: 1. Define and implement an ACE accountability
framework that ensures: b. currency, relevance, and completeness of
such commitments made to Congress in expenditure plans;
Status [B,C,D]: In progress.
GAO Recommendation: 1. Define and implement an ACE accountability
framework that ensures: c. reliable data relevant to measuring progress
against commitments;
Status [B,C,D]: Complete.
GAO Recommendation: 1. Define and implement an ACE accountability
framework that ensures: d. reporting in future expenditure plans
progress against commitments contained in prior expenditure plans;
Status [B,C,D]: In progress.
GAO Recommendation: 1. Define and implement an ACE accountability
framework that ensures: e. use of criteria for exiting key readiness
milestones that adequately consider indicators of system maturity, such
as severity of open defects, and document milestone decisions in a way
that reflects the risks associated with proceeding with unresolved
severe defects and provides for mitigating these risks.[E];
Status [B,C,D]: Complete.
GAO Recommendation: 2. Develop the range of realistic performance
measures and targets needed to support an outcome-based, results-
oriented accountability framework, including user satisfaction;
Status [B,C,D]: Complete.
GAO Recommendation: 3. Explicitly align program goals, benefits,
desired business outcomes, and performance measures;
Status [B,C,D]: Complete.
GAO Recommendation: 4. Define measures and collect and use associated
metrics for determining whether prior and future program management
improvements are successful;
Status [B,C,D]: Planned.
GAO Recommendation: 5. Fully address those legislative conditions
associated with measuring performance and results and employing
effective IV&V practices;
Status [B,C,D]: Complete.
GAO Recommendation: 6. Ensure that future expenditure plans are based
on cost estimates that are reconciled with independent cost estimates;
Status [B,C,D]: Complete.
GAO Recommendation: 7. Develop and implement a rigorous and
analytically verifiable cost estimating program that embodies the
tenets of effective estimating;
Status [B,C,D]: Complete.
GAO Recommendation: 8. Use earned value management (EVM)[F]in
developing all existing and future releases;
Status [B,C,D]: Complete.
GAO Recommendation: 9. Have future expenditure plans specifically
address any proposals or plans for extending and using ACE
infrastructure to support other homeland security applications;
Status [B,C,D]: Complete.
GAO Recommendation: 10. Minimize the degree of overlap and concurrence
across ongoing and future releases, and capture and mitigate the
associated risks of any residual concurrence;
Status [B,C,D]: In progress.
GAO Recommendation: 11. Fully address those legislative conditions
associated with having an approved privacy impact assessment (PIA) and
ensuring architectural alignment;
Status [B,C,D]: In progress.
GAO Recommendation: 12. Develop and implement missing human capital
management practices;
Status [B,C,D]: In Progress.
GAO Recommendation: 13. Include in the June 30, 2006, quarterly update
report to the appropriations committees a strategy for managing human
capital needs and the framework for managing performance and ensuring
accountability;
Status [B,C,D]: In progress.
GAO Recommendation: 14. Report to House and Senate Appropriations
Committees on a quarterly basis on efforts to address open GAO
recommendations;
Status [B,C,D]: Complete.
GAO Recommendation: 15. Accurately report to the appropriations
committees on CBP's progress in implementing our prior recommendations;
Status [B,C,D]: In Progress.
Source: GAO.
[A] With respect to the fiscal year 2007 expenditure plan.
[B] Complete means that actions have been taken to fully implement the
recommendation.
[C] In progress means that actions are under way to implement the
recommendation.
[D] Planned means actions are planned to implement the recommendation.
[E] This is a combination of two related recommendations.
[F] EVM is a management tool for measuring progress and is both an
industry accepted practice and an OMB requirement.
[End of table]
Results in Brief: Objective 3:
Observations:
Summary of observations:
* Limitations in the completeness of the original requirements for
several ACE releases have resulted in changes to how requirements are
being defined, as well as changes to the requirements themselves. These
changes are likely to produce significant schedule delays and cost
growth, and neither the expenditure plan nor the quarterly reports to
Congress have disclosed these risks.
* Requirements changes have led to reevaluation and replacement of a
key commercial software product that was previously selected based on
incomplete requirements. This change carries schedule and cost risks
that neither the expenditure plan nor the quarterly reports have
disclosed.
* Management of ACE risks has not been effective, but improvements are
planned to better anticipate and avoid problems that cause schedule
delays and cost growth. ACE risks are not disclosed in either the
expenditure plan or in the quarterly reports.
Results in Brief:
Recommendations and Agency Comments:
To reduce ACE exposure to future schedule delays and cost growth and to
promote greater accountability, we are making recommendations to DHS
aimed at disclosing the nature of and progress in addressing the risks
associated with not having fully satisfied expenditure plan legislative
conditions, not having completed implementation of all prior GAO
recommendations, and having to redefine ACE requirements and reselect a
key commercial software product.
In oral comments on a draft of this briefing, DHS and CBP officials
agreed with our conclusions and recommendations, and provided
clarifying information and technical comments that we incorporated in
the briefing, as appropriate.
Background:
Program Overview:
CBP is about 6 years into its trade processing modernization program,
known as ACE. Among other things, ACE is to introduce reengineered
business processes and next generation cargo processing technology, and
it is to support CBP‘s mission of (1) protecting the American public
against terrorism and (2) enforcing the laws of the United States while
fostering our nation‘s economic security through lawful international
trade and travel.
ACE is also to support provisions of Title VI of the North American
Free Trade Agreement, commonly known as the Customs Modernization Act.
Subtitle B of the Act[Footnote 11] contains provisions that were
intended to enable the government to modernize international trade
processes and permit CBP to adopt an informed compliance approach with
industry using automated systems.
The goals of ACE are to:
* enhance analysis and information sharing with other government
agencies relative to new national security threats;
* provide CBP personnel with the technology and information needed to
decide, before a shipment reaches the border, what should be targeted
because it is a security threat and what should be expedited because it
complies with U.S. laws;
* enable the efficient collection, processing, and analysis of
commercial import and export data via an integrated, fully automated
information system;
* reduce costs for the government and the trade community by
streamlining time-consuming and labor-intensive tasks;
* enable government and trade communities to process, view, and manage
accounts nationally and obtain historical information on cargo,
conveyances, and crew, based on screening and targeting rules; and:
* enable government to comply with legislative mandates to improve
efficiency/effectiveness and provide better customer service to U.S.
citizens.
Background:
Program Organization:
Several CBP components support the execution of the ACE program.
* The Cargo Systems Program Office (referred to in this briefing as the
program office) is responsible for the implementation of ACE. This
office has primary responsibility for ACE program management; is
responsible for the day-to-day management of ACE modernization
activities; and develops the policies, standards, processes, and
metrics by which the program is managed and measured.
* The ACE program office is located within CBP‘s Office of Information
Technology (OIT), which is headed by the Assistant Commissioner for
Information and Technology.
* The ACE program office is supported by the Targeting and Analysis
Systems Program Office (a merger of ACE screening and targeting
resources and the Office of Information Technology‘s Interprocess
Solutions Branch), which is responsible for acquisition and development
of ACE screening and targeting functionality.
* The program office is supported by CBP‘s Office of Finance, Office of
Strategic Trade, Office of Regulations and Rulings, and more than 100
participating government agencies and offices.
Background:
Contractors:
CBP awarded a 15-year, indefinite delivery/indefinite quantity, prime
integration contract (5-year base with two 5-year options) to IBM
Global Services in April 2001 for development and implementation of
ACE. CBP exercised the first option in April of 2006.
* IBM and its subcontractors-collectively called the ACE Support
Team[Footnote 12]-provide ACE with program management support,
enterprise engineering, systems planning and development, and systems
business process reengineering; technology architectures; prototyping;
systems and application design; software development, testing, and
evaluation; deployment; and developmental operations support.
* IBM is also under contract for operations and maintenance of deployed
ACE releases, as well as for ongoing enhancements (new capabilities,
referred to as program baseline enhancements) implemented in sub-
releases that are scheduled in-between major ACE releases.
Background:
Acquisition Approach and Cost:
CBP also relies on support contractors to provide a range of management
support, such as program management, financial management, process
improvement, quality assurance, requirements and configuration
management, program communication, organizational learning, human
capital planning, specialized cost analysis and life cycle cost
estimating/modeling services, and procurement support.
In 2002, ACE was to be completed in four increments over 4 years at an
estimated cost of between $1.5 billion and $1.6 billion.
The ACE Program Plan (version 2.1, dated August 2006) states that ACE
is expected to be fully deployed and operational by the end of 2011 at
a cost of about $3.3 billion. However, program officials told us that
cost and schedule estimates are being revised, but have not yet been
approved and therefore were not provided for our review. In the fiscal
year 2007 ACE expenditure plan, CBP reports that it has spent about
$1.7 billion on the program.
ACE is now being acquired and implemented through a series of 10
increments (referred to as releases), which are further divided into
major system deliveries, known as ’drops.“ (See the following six
slides for a description of the increments and their status.)
Background:
Description of ACE Increments:
A description of the 10 ACE increments (releases and related drops)
follows.
Release 1 (ACE Foundation): Computer hardware and system software
(infrastructure) to support subsequent system releases.
Release 2 (Account Creation): Initial group of national account
managers1 and 41 importers access to account information, such as trade
activity.
Release 3 (Periodic Payment): Additional account managers and
importers, as well as brokers and carriers,2 with access to account
information; provides initial financial transaction processing and
revenue collection capability, allowing monthly payments of duties and
fees.
Release 4 (e-Manifest: Trucks): Electronic truck manifest3 processing
and interfacing to legacy enforcement systems and databases.
Background:
Description of ACE Increments:
Release 5 (Entry Summary, Accounts, and Revenue (ESAR)): SAP[Footnote
16] technologies to enhance and expend accounts management, financial
management, and entry summary functionality.
* Master Data and Enhanced Accounts (drop A1): SAP to deliver enhanced
account creation and maintenance functionality and expand the types of
accounts managed in ACE.
* Entry Summary and Revenue (drop A2): Entry summary, interfaces with
participating government agencies, calculation of duties and fees,
reconciliation processing, and refunds.
Release 6 (e-Manifest, All Modes, and Cargo Release): Electronic
manifest capability for rail, air, and sea shipments; provides a
multimodal manifest;[Footnote 17] enables full tracking of cargo,
conveyances, individuals, and equipment; and enhances enforcement
processes for rail, air, and sea.
* e-Manifest: Rail and Sea Manifest (drop M1): Electronic manifest
functionality for rail and sea shipments; rail, sea, and truck
electronic manifests into the multimodal manifest.
* e-Manifest: Air Manifest and Cargo Release (drop M2): Electronic
manifest to air shipment and brings all modes of transportation into
the multimodal manifest.
* e-Manifest: Exports and Mail Entry Writing System (drop M3): Tracking
of cargo, conveyances, individuals, and equipment for truck, sea, rail,
and air manifests.
Release 7 (Exports and Cargo Control): Remaining accounts management,
revenue, manifest, release, and export functionality.
* ESAR: Drawback, Protest, and Importer Activity Summary Statement
(IASS) (drop A3): Import activity summary statement,[Footnote 18]
drawback functionality, and enhanced protest; online processing for
trade account applications.
E-Manifest: Final Exports and Manifest (drop M4): Electronic manifest
for mail, pipeline, and hand carry; electronic export processing.
Screening S1 (Screening Foundation): Foundation for screening cargo and
conveyances by centralizing criteria and results into a single standard
database; allows user definition and maintenance of data sources and
business rules for air, rail, sea, and truck modes of transportation.
Screening S2 (Targeting Foundation): Platform and foundation for
advanced targeting capabilities by enabling CBP‘s National Targeting
Center to search multiple databases for relevant facts and actionable
intelligence and infer relationships between entities and data
elements; architecture for integrating new data sources (including
integrating external data sources and providing a single sign on
capability), implementing analytical tools, and deploying analytical
capabilities.
Screening S3 (Advanced Targeting Capabilities): Screening for
reconciliation, intermodal manifest, Food and Drug Administration data,
and in-bond, warehouse, and foreign trade zone authorized movements;
integrates additional data sources into targeting capability; and risk
management capability.
Background:
Status of ACE Increments:
According to CBP, as of July 2007,:
* Five increments are fully operational: Releases 1, 2, and 3, and
Screenings S1 and S2.
* One increment (Release 4) has been deployed at 94 of the 99 truck
land border ports.
* Three increments are at various stages of development and/or
deployment: Release 5/Drops A1 and A2, Release 6/Drop M1, and Screening
Background:
Status of ACE Increments:
Figure: ACE Schedule:
This figure is a bar chart showing the ACE schedule.
[See PDF for image] - graphic text:
Source: GAO analysis based on CBP data.
[End of table]
Background:
Status of ACE Use:
CBP reports increased use of operational ACE increments. For example:
* The number of external accounts[Footnote 19] grew from 41 in June
2003 to 736 in August 2005, and stood at about 4,100 in October 2006.
As of November 2006, about 4,500 corporate entities had been approved
to pay monthly duties and fees.
* Total revenue collections through ACE grew from $84,673 in June 2004
to $1.3 billion in July 2005, and as of October 2006 stood at about $8
billion.
* The number of e-Manifests that were filed using ACE grew from 20,847
in December 2006 to 45,548 in January 2007.
Background:
Funding for ACE Expenditure Plans:
Table: Funding for ACE Expenditure Plans:
Release: 1: ACE Foundation:-through Operational Readiness;
Budgeted amount: Prior years[A]: 114.8;
Budgeted amount: Fiscal year: [Empty];
Budgeted amount: Total: 114.8.
Release: 2: Account Creation-through Operational Readiness;
Budgeted amount: Prior years[A]: 114.8;
Budgeted amount: Fiscal year: [Empty];
Budgeted amount: Total: 114.8.
Release: 3: Periodic Payment:-through Operational Readiness;
Budgeted amount: Prior years[A]: 172.9;
Budgeted amount: Fiscal year: [Empty];
Budgeted amount: Total: 183.2.
Release: 4: e-Manifest Trucks-through Operational Readiness;
Budgeted amount: Prior years[A]: 10.3;
Budgeted amount: Fiscal year: [Empty];
Budgeted amount: Total: 183.2.
Release: 5(A1): ESAR: Master Data and Enhanced Amounts;
Budgeted amount: Prior years[A]: 172.3;
Budgeted amount: Fiscal year: [Empty];
Budgeted amount: Total: 233.6.
Release: 5(A2): ESAR: Entry Summary and Revenue;
Budgeted amount: Prior years[A]: 23.3;
Budgeted amount: Fiscal year: 38.0;
Budgeted amount: Total: 233.6.
Release: 6(M1): e-Manifest- Rail and Sea Manifest;
Budgeted amount: Prior years[A]: 46.3;
Budgeted amount: Fiscal year: 38.3;
Budgeted amount: Total: 95.0.
Release: 6(M2): e-Manifest- Air and Cargo Release;
Budgeted amount: Prior years[A]: 10.4;
Budgeted amount: Fiscal year: 38.3;
Budgeted amount: Total: 95.
Release: 6(M3): Exports and Mail Entry Writing System;
Budgeted amount: Prior years[A]: [Empty];
Budgeted amount: Fiscal year: 38.3;
Budgeted amount: Total: 95.
Release: 7(A3): ESAR: Drawback, Protest and IASS[B];
Budgeted amount: Prior years[A]: [Empty];
Budgeted amount: Fiscal year: [Empty];
Budgeted amount: Total: [Empty].
Release: 7(M4): e-Manifest Custodial Entities, Pipeline, and Batch
Processes[C];
Budgeted amount: Prior years[A]: [Empty];
Budgeted amount: Fiscal year: [Empty];
Budgeted amount: Total: [Empty].
Release: S1: Screening Foundation;
Budgeted amount: Prior years[A]: 48.7;
Budgeted amount: Fiscal year: [Empty];
Budgeted amount: Total: 48.7.
Release: S2: Targeting Foundation;
Budgeted amount: Prior years[A]: 39.3;
Budgeted amount: Fiscal year: [Empty];
Budgeted amount: Total: 39.3.
Release: S3: Advanced Targeting;
Budgeted amount: Prior years[A]: 21.8;
Budgeted amount: Fiscal year: [Empty];
Budgeted amount: Total: 21.8.
Activity: ACE Operations and Maintenance;
Budgeted amount: Prior years[A]: 143.1;
Budgeted amount: Fiscal year: 44.2;
Budgeted amount: Total: 187.3.
Activity: ACE Production Baselines;
Budgeted amount: Prior year[A]: [Empty];
Budgeted amount: Fiscal year: 22.6;
Budgeted amount: Total: 22.6.
Activity: ACE Foundation Architecture and Engineering;
Budgeted amount: Prior year[A]: 90.4;
Budgeted amount: Fiscal year: 35.7;
Budgeted amount: Total: 126.1.
Activity: ACE Implementation Infrastructure and Support;
Budgeted amount: Prior year[A]: 269.5;
Budgeted amount: Fiscal year: 62.9;
Budgeted amount: Total: 332.4.
Activity: ACE Foundation Program Management;
Budgeted amount: Prior year[A]: 137.5;
Budgeted amount: Fiscal year: 18.9;
Budgeted amount: Total: 156.4.
Activity: ACE Communications, Training, Outcome, and Deployment;
Budgeted amount: Prior year[A]: 31.7;
Budgeted amount: Fiscal year: 24.4;
Budgeted amount: Total: 56.1.
Activity: Other Tasks;
Budgeted amount: Prior year[A]: 20.6;
Budgeted amount: Fiscal year: [Empty];
Budgeted amount: Total: 20.6.
Activity: Cargo Systems Program Office;
Budgeted amount: Prior year[A]: 201.2;
Budgeted amount: Fiscal year: 25.3;
Budgeted amount: Total: 226.5.
Activity: International Trade Data System;
Budgeted amount: Prior year[A]: 59.4;
Budgeted amount: Fiscal year: 16;
Budgeted amount: Total: 75.4.
Activity: Management Reserve;
Budgeted amount: Prior year[A]: 92;
Budgeted amount: Fiscal year: 4.6;
Budgeted amount: Total: 96.6.
Activity: International Trade Data System Funding for Development[D];
Budgeted amount: Prior year[A]: [Empty];
Budgeted amount: Fiscal year: -14;
Budgeted amount: Total: -14.
Appropriated total;
Budgeted amount: Prior year[A]: 1707;
Budgeted amount: Fiscal year: 317;
Budgeted amount: Total: 2024.
Source: GAO analysis of CBP data.
[A] Prior funding consists of stopgap funding, approved by the
appropriations committees in March 2001, and the seven previous
expenditure plans.
[B,C] Funding for A3 and M4 has not yet been included in the
expenditure plans.
[D] This amount consists of prior year unobligated funds that ITDS
provided to ACE to address requirements for participating government
agencies.
[End of table]
Objective 1: Legislative Conditions:
Condition 1: Partially Satisfied:
The 6 legislative conditions have been either satisfied or partially
satisfied.
Table: Legislative condition:
Legislative condition: 1. Meets the capital planning and investment
control review requirements established by the Office of Management and
Budget (OMB), including OMB Circular A-11, part 7;
Status [A,B]: Partially satisfied.
Legislative condition: 2. Complies with DHS‘s enterprise architecture;
Status [A,B]: Partially satisfied.
Legislative condition: 3. Complies with the acquisition rules,
requirements, guidelines, and systems acquisition management practices
of the federal government;
Status [A,B]: Satisfied.
Legislative condition: 4. Includes a certification by the DHS CIO that
an IV&V agent is currently under contract for the project;
Status [A,B]: Satisfied.
Legislative condition: 5. Is reviewed and approved by the DHS IRB,
Secretary of Homeland Security, and OMB;
Status [A,B]: Satisfied.
Legislative condition: 6. Is reviewed by GAO;
Status [A,B]: Satisfied.
Source: GAO.
[A] Partially satisfied means that the plan, in combination with
supporting documentation, either satisfied or provided for satisfying
many, but not all, key aspects of the condition that we reviewed.
[B] Satisfied means that the plan, in combination with supporting
documentation, either satisfied or provided for satisfying every aspect
of the condition that we reviewed.
[End of table]
Condition 1: The plan, including related program documentation and
program officials‘ statements, partially satisfies the capital planning
and investment control review requirements established by OMB,
including Circular A-11, part 7.[Footnote 20]
The table that follows provides an overview of the results of our
analysis and selected examples in areas where A-11 requirements have or
have not been fully satisfied. Given that the A-11 requirements are
intended to minimize a program‘s exposure to risk, permit performance
measurement and oversight, and promote accountability, any areas in
which the program falls short of the requirements reduces the chances
of delivering cost effective capabilities and measurable results on
time and within budget.
Objective 1: Legislative Conditions:
Condition 1: Partially Satisfied:
Table: Condition 1: Partially Satisfied:
Table: Examples of A-11 Conditions Results of Our Analysis:
Provide a brief description of the investment and its status in the
capital planning and investment control review process, including major
assumptions made about the investment;
The expenditure plan and the ACE Exhibit 300 budget submission provide
brief descriptions of the ACE investment and its releases. The Exhibit
300 also describes the status of ACE relative to DHS‘s capital planning
and investment control process, and states that ACE is in the ’control“
stage of the process;
The Exhibit 300 and the ACE program plan also describe investment
assumptions, such as (a) an annual budget of $305.5 million will be
provided; (b) incremental deliveries within releases will mitigate
risks and reduce uncertainty of program estimates; (c) ITDS funding
will be timely, separate, and adequate; (d) stakeholder representation
and resources will be adequate.
Report performance goals and measures for existing investments and show
how the Federal Enterprise Architecture (FEA) Performance Reference
Model (PRM) applies to this investment;
The expenditure plan reports the program‘s performance goals, benefits,
objectives, performance measures, and their relationships for some, but
not all, increments, including actual performance for fiscal years 2005
through 2006. The plan includes the ACE accountability framework, which
is the program‘s overall tool for implementing CBP‘s PRM, which in turn
is based on the FEA PRM. CBP‘s PRM includes, for example, performance
measures for user satisfaction, efficiency, productivity, and data
reliability, among other elements, and the ACE accountability framework
incorporates a subset of these PRM measures, augmented by additional
measures focusing on the program‘s status;
ACE‘s Exhibit 300 also presents performance goals, targets, measures,
and results for 2003 through 2012. However, these are not fully
consistent with the expenditure plan. According to CBP, this is because
the ACE performance measures were aligned with DHS‘s goals, objectives,
strategies, and desired results as of July 2006, which is after the
fiscal year 2007 Exhibit 300 was submitted. In addition, the
performance measures continue to evolve as more understanding is gained
in the usefulness and meaning of measures. (See the open
recommendations section of this briefing for more information on
performance measures.)
Provide a summary of the investment‘s risk assessment, including how 19
OMB-identified risk elements are being addressed;
The expenditure plan presents some, but not all, of the identified
risks for the program‘s releases and activities. The Exhibit 300 also
presents program risks, assessments, mitigation strategies, and status,
and it organizes them by OMB risk categories. However, the risks in the
expenditure plan and the Exhibit 300 are not fully consistent with each
other or with the risks being tracked in the program‘s risk database.
Further, the most recent risk assessment is limited to security risks
for Release 4;
The ACE program has a documented risk management process that includes
identifying, classifying, reporting, and tracking risks. However, this
process has not been fully implemented. For example, some risks that
are identified in the risk database and the accountability framework
have missing or outdated status information and mitigation strategies.
As a result, the status of all risks is not clear. Program officials
stated that risk management is not yet mature, but that they are taking
steps to improve it. (See the open recommendation section of this
briefing for more information on risk management.)
Provide a summary of the investment‘s status in accomplishing baseline
cost and schedule goals through the use of an earned value management
(EVM) system or operational analysis, depending on the life-cycle
stage;
The expenditure plan reported some, but not all, EVM data for ACE
releases and activities. For example, the data includes planned,
estimated, and actual schedule milestones, but it does not include
schedule variances;
The EVM data in the expenditure plan was from the ACE accountability
framework that program officials rely on to manage the program. (See
the open recommendations section of this briefing for more information
on the accountability framework.) According to program officials, they
are using EVM to manage all ACE releases and screenings under contract
and EVM tracking begins with an Integrated Baseline Review, which forms
the basis for a realistic plan against which to objectively measure
work to be completed during the period of performance;
The Exhibit 300 also includes EVM data and describes contractor
requirements for EVM, including verification of EVM compliance with
industry standards. According to program officials, these contractual
requirements are still operative;
Nevertheless, the EVM program has still not been certified, as required
by OMB. According to program officials, this certification is scheduled
for the beginning of fiscal year 2008. As an interim measure, the prime
contractor performed an EVM compliance and internal surveillance audit
against relevant EVM criteria[Footnote 21] in March 2006 and found no
issues.
Provide a description of the investment's privacy and security issues.
Summarize the agency's ability to manage security at the system or
application level. Demonstrate compliance with the certification and
accreditation processes as well as the mitigation of IT security
weaknesses;
Privacy: The expenditure plan does not discuss privacy issues. However,
the Exhibit 300 states that ACE is subject to the privacy provisions of
the E-Government Act of 2002. It further states that a privacy impact
assessment was conducted in 2005 for Release 4 and that separate
assessments will be conducted for each subsequent release. This
assessment was approved by DHS on July 14, 2006, and our analysis shows
that it complied with relevant DHS guidance. This assessment does not
cover other recently completed screening releases S1 and S2. According
to program officials, S1 and S2 are considered to be part of the
Automated Targeting System (ATS), and are therefore covered by the ATS
PIA. However, our analysis of the ATS PIA showed that while it
addresses screening and targeting functions, it does not specifically
identify or address releases S1 and S2. (See the open recommendations
section of this briefing for further information on privacy issues);
Security: CBP has taken a number of actions related to ACE security
management. It conducted a security self-assessment for Release 4 in
September 2006 using the National Institute of Standards and Technology
(NIST) Security Self-Assessment Guide for Information Technology
Systems and did not report any major security vulnerabilities. In
addition, CBP officials stated that another security self-assessment
was conducted in April 2007. However, we have yet to receive this self-
assessment. CBP also reports that it has conducted two full contingency
plan tests at its disaster recovery facility;
Further, CBP accredited ACE Release 4 on November 22, 2004; Release
S1on July 18, 2006; and Release S2 on October 18, 2006. The program
office plans to conduct its next ACE certification and accreditation of
ACE in September 2007 for Release 5/Drop A1. However, since November
2004, the deployed ACE system has been subject to considerable change.
Specifically, about 7,100 trouble tickets, trouble reports, change
requests and install requests were generated”some of which have
resulted in system changes. Federal guidance recommends reaccreditation
whenever significant changes to the system or its operational
environment are likely to affect a system‘s security posture.[Footnote
22] However, CBP has not established explicit criteria for when to
reaccredit/recertify a system in the face of changes, or documented a
systematic procedure for determining the security risk from the
cumulative impact of changes made to the system. As a result, CBP has
not recertified or reaccredited the system;
In addition, the ACE security plan and security risk assessment have
not been updated to reflect key information, such as the results of the
April 2007 security self-assessment or a system access control risk
identified in May 2006 and included in the program‘s risk database.
Source: OMB and NIST criteria and GAO analysis of DHS documentation.
[End of table]
Objective 1: Legislative Conditions:
Condition 2: Partially Satisfied:
Condition 2: The plan, including related program documentation and
program officials‘ statements, partially satisfies the condition that
it comply with the DHS information systems enterprise architecture (EA)
as currently defined.
According to federal guidelines and best practices, investment
compliance with an EA is essential for ensuring that an organization‘s
investment in new and existing systems is defined, designed, and
implemented in a way that promotes integration and interoperability and
minimizes overlap and redundancy, thus optimizing enterprisewide
efficiency and effectiveness. A compliance determination is not a one-
time event that occurs when an investment begins, but is, rather, a
series of determinations that occurs throughout an investment‘s life
cycle as changes to key aspects of both the EA and the investment‘s
architecture are made (e.g., data, business, services, and technology).
The DHS Enterprise Architecture Board, supported by the Enterprise
Architecture Center of Excellence, is responsible for ensuring that
projects demonstrate adequate technical and strategic compliance with
the department‘s EA.
During 2006, the DHS Enterprise Architecture Board conducted three
reviews of ACE architectural alignment with the DHS EA Technical
Reference Model (TRM)[Footnote 23] as part of key milestone reviews for
three release/screening increments.
* Master Data and Enhanced Accounts (Drop A1), Critical Design Review,
June 2006;
* E-Manifest: Rail and Sea (Drop M1), Critical Design Review, September
2006, and:
* Targeting Foundation (S2), Production Readiness Review, November
2006.
On May 1, 2007, DHS EA officials reported that all required products
and technologies were aligned to the TRM, and thus that ACE was in
alignment with the DHS EA. In addition, the DHS CIO certified as part
of the above milestone reviews that each increment was in alignment
with the TRM. More specifically,
* In December 2006, the Center of Excellence determined that ACE was
conditionally compliant with the DHS EA, contingent on actions taken in
regard to four conditions. Three of the four conditions were resolved
by March 2007.
* On March 29, 2007, the EAB recommended approval of ACE‘s decision
request for program alignment, with one remaining condition”that by
June 30, 2007, the program submit technology insertions packages for
the products identified as needing alignment with the DHS TRM.
Notwithstanding these EA compliance determinations, DHS did not provide
us with sufficient documentation of its determinations to allow us to
understand the methodology and criteria, or to verify the analyses,
that were used to arrive at them. Moreover, documentation that was
provided showed that the determinations focused on ACE technical
alignment, and did not address, for example, alignment to the DHS EA
Data Reference Model.
Until DHS demonstrates, through verifiable documentation and
methodologically- based analysis, that ACE is aligned with all relevant
aspects of the DHS EA, including corporate data structures and
standards, the program will be at risk of being designed and
implemented in a way that does not support optimized departmental
operations, performance, and achievement of strategic goals and
outcomes, including those related to information sharing.
Objective 1: Legislative Conditions:
Condition 3: Satisfied:
Condition 3: The plan, including related program documentation and
program officials‘ statements, satisfies the condition that it comply
with the acquisition rules, requirements, guidelines, and systems
acquisition management practices of the federal government.[Footnote
24]
Federal acquisition rules, requirements, guidelines, and management
practices provide an acquisition management framework that is based on
the use of rigorous and disciplined processes for planning, managing,
and controlling the acquisition of IT resources.[Footnote 25] These
acquisition management processes are embodied in published best
practices models, such as the Capability Maturity Models® developed by
Carnegie Mellon University‘s Software Engineering Institute (SEI).
These models explicitly define, among other things, acquisition process
management controls that are recognized hallmarks of successful
organizations and that, if implemented effectively, can greatly
increase the chances of acquiring software-intensive systems that
provide promised capabilities on time and within budget.
In our prior reviews of the ACE program,[Footnote 26] we reported that
ACE had satisfied this condition based on SEI‘s November 2003
assessment of the program against the Software Acquisition Capability
Maturity Model (SA- CMM®).[Footnote 27] That assessment assigned the
program an SEI level 2[Footnote 28] rating, indicating that CBP had
instituted basic acquisition management processes and practices. Since
receiving its 2003 rating, the program office has not conducted another
acquisition capability assessment to ensure that it is continuing to
employ these basic acquisition controls, and does not plan to do so.
Furthermore, program officials told us that although the prime
contractor was contractually required to have an SEI CMM® Level
3[Footnote 29] capability, they are no longer required to do so because
funding contractor program management-related activities is now a lower
priority relative to other competing demands for ACE funding.
According to CBP officials, several steps have been taken to mitigate
any impact of no longer focusing on SEI CMM® compliance, such as:
* requiring the prime contractor and the program office‘s support
contractors to follow plans, processes, and procedures for such key
areas as EVM, configuration management, and problem reporting;
* having the program office‘s quality assurance staff, in collaboration
with the prime contractor‘s quality assurance function, monitor
adherence to key management controls through reviews of ACE processes
and products, including 48 such reviews in fiscal year 2006; and:
* having the program office‘s contracting function continuously monitor
adherence to contractual provisions;
* leveraging OIT assistance, through the Process Asset Group, to
conducts reviews of new and updated plans, policies, and
procedures;[Footnote 30] and:
* cataloguing job aids in a Process Asset Group library that specifies
the required activities for program management processes and
disseminating them to ACE staff.
Objective 1: Legislative Conditions:
Condition 4: Satisfied:
Condition 4: The plan satisfies the condition that it include
certification by the DHS CIO that an IV&V agent is currently under
contract.
On October 24, 2006, the DHS Deputy CIO certified in writing that an
IV&V agent is under contract for ACE and that the agent met applicable
requirements and standards. (See open recommendations section of this
briefing for more information on IV&V.)
Objective 1: Legislative Conditions:
Condition 5: Satisfied:
Condition 5. The plan, including related program documentation and
program officials‘ statements, satisfies the requirement that it be
reviewed and approved by the DHS IRB, the Secretary of Homeland
Security, and OMB.
* The DHS IRB reviewed the program and approved the expenditure plan on
December 12, 2006.
* The DHS Under Secretary for Management approved the expenditure plan
on behalf of the Secretary of Homeland Security on February 6, 2007.
* OMB approved the expenditure plan on January 22, 2007.
Objective 1: Legislative Conditions:
Condition 6: Satisfied:
Condition 6. The plan satisfies the requirement that it be reviewed by
GAO. Our review was completed on July 26, 2007.
Objective 2: Open Recommendations:
Accountability Framework:
Open Recommendation 1: Define and implement an ACE accountability
framework that fulfills these conditions:
a. Covers all program commitment areas, including key expected or
estimated system (1) capabilities, use, and quality; (2) benefits and
mission value; (3) costs; and (4) milestones and schedules.
Status: Complete:
Effective program management includes defining and measuring progress
against program commitments and being held accountable for results.
Such commitments generally cover expected or estimated (1) capabilities
and their associated use and quality, (2) benefits and mission value,
(3) costs, and (4) milestones and schedules.
Since 2003, we have reported that such commitments for ACE have not
always been defined, although improvements have been made. Most
recently, we reported[Footnote 31] that the program office had
developed an initial version of an accountability framework for
measuring several program commitments, such as capabilities and
milestones, but that other commitments, such as benefits, had not been
as well defined.
During the last year, the program office has continued to improve its
coverage of program commitments, as evidenced by the content of key
program documents” ACE accountability framework, ACE Program Plan
(August 2006), the CBP/ACE performance reference model, periodic
Program Management Review (PMR) reports, and the fiscal year 2007
expenditure plan. For example,
* The accountability framework now captures and integrates data on ACE
functional and performance capabilities, benefits, and mission value,
estimated and actual costs, milestones and schedule, accomplishments,
and earned value management status. Further, the framework provides
visibility into each ACE release at several levels of detail, and it is
being used by the ACE Executive Director and others as the means for
monitoring program progress, issues, and decisions.
* The PMR reports have included the accountability framework data.
* The fiscal year 2007 ACE expenditure plan included an example of the
accountability framework commitments from the September 2006 PMR report.
b. Ensures currency, relevance, and completeness of such commitments
made to Congress in expenditure plans.
Status: In progress:
We have previously reported that commitments made in expenditure plans
relative to program capabilities, benefits, costs, and schedules need
to be current, relevant, and complete. To the extent that they are not,
the currency and relevance of the plan and its utility to Congress as
an accountability mechanism are limited.
ACE expenditure plans have not always included such information. To
address this limitation, we reported last year[Footnote 32] that CBP
was relying on quarterly reports to Congress to provide the
appropriations committees with more current, relevant, and complete
information about the program than could be provided in the expenditure
plan. However, we also noted that the quarterly reports were generally
submitted to Congress 3 to 4 months after the end of each quarter, thus
limiting their currency and relevance.
The fiscal year 2007 expenditure plan continues to include information
about program commitments that is not current. Specifically,
* The latest expenditure plan was provided to the appropriations
committees on February 6, 2007. However, information from the ACE
accountability framework that was included in the plan, such as
milestones, EVM values, and risks, were as of October 2006, making it
about 4 months old. For example, the expenditure plan listed the
milestone for the Production Readiness Review (PRR) for a key release,
Release 5/Drop A1, as March 1, 2007. However, this milestone had
already slipped by more than 4 months--to July 12, 2007--by the
December 2006 PMR (held on January 5, 2007).
According to program officials, they continue to use the quarterly ACE
status reports to provide the appropriations committees with more
current information.[Footnote 33] However, recent quarterly reports
have not been current. For example,
* The December 2006 quarterly report was not provided until February
26, 2007, and it contained the same outdated PRR milestone as the
expenditure plan did for the release previously mentioned.
* Other quarterly reports have been submitted to the appropriations
committees as many as 7 months after the end of the quarter.
CBP and DHS officials told us that the delays were due to the DHS
review and approval process and that incorporating the most current
program information would delay the reports even longer. According to
the ACE and DHS officials, they are exploring ways to accelerate the
review process.
c. Ensures reliable data relevant to measuring progress against
commitments. Status: Complete:
The quality of the capabilities that a program is to deliver is a
relevant program commitment. One measure of the quality of system
capabilities is the trend in the number and severity of unresolved
system defects or problems. Reliable data about these defects are
needed so that system maturity can be understood and informed
investment decisions can be made.
We previously reported [Footnote 34] that ACE defect data were not
always consistent because the two tools that the program used to track
defects were not integrated and reconciled. As a result, the true
status of ACE defects, and thus an important measure of system quality,
was not known.
Since then, the program office has integrated the two tools using a
cross- referencing function and has instituted manual processes for
reconciling the data in each tool. The December 2006 quarterly
congressional report stated that these efforts have improved the
ability to record, assess, and report on system quality and performance.
d. Ensures reporting in future expenditure plans progress against
commitments contained in prior expenditure plans.
Status: In progress:
The value and utility of an expenditure plan for congressional
oversight and agency accountability depends in large part on whether
the plan reports on progress against commitments--system capabilities,
benefits, costs, and schedules--made in prior plans.
Historically, the ACE expenditure plans have not reported adequately on
progress against previous plan commitments. Most recently, we
reported[Footnote 35] that the fiscal year 2006 expenditure plan
contained several such reporting gaps, such as whether funding amounts
were actually obligated and expended as planned and whether releases
met schedules. In particular, we reported that the plan did not address
whether the design and development for Release 5 was actually
accomplished as planned.
The fiscal year 2007 expenditure plan still does not adequately
describe the program‘s progress against the commitments that were made
in the fiscal year 2006 plan. For example,
* The plan does not report progress against milestones in the fiscal
year 2006 plan or explain why these milestones were not achieved.
* The plan does not report actual obligation or expenditure of funds
relative to the planned uses of these funds in prior expenditure plans.
* The plan did not address progress against the milestone dates for
each stage of a release that was included in the prior year‘s plan. As
a case in point, the fiscal year 2006 expenditure plan described
specific functionality planned for Release 5/Drop A1 (Master Data and
Enhanced Accounts), such as online registration for trade
representatives. However, the fiscal year 2007 expenditure plan does
not include information on whether these functions were delivered,
stating only that Release 5 functionality would be deployed in two
phases.
According to program officials, the quarterly congressional reports
provide more current information on the program‘s progress against
prior commitments. However, we found that these reports have also not
fully addressed the commitments made in prior expenditure plans. For
example, recent quarterly reports describe progress against
developmental milestones for each release, but do not report whether
key functionality associated with each release was delivered. In
particular, the report for the first quarter of fiscal year 2007
identified planned future capabilities and associated milestones for
Release 5/Drop A1 but did not address the status of the key functions
associated with this release.
Program officials stated that they intend to start providing this
information in the last quarterly report of each year.
e. Ensures use of criteria for exiting key readiness milestones that
adequately consider indicators of system maturity, such as severity of
open defects, and document the milestone decisions in a way that
reflects the associated risks and plans for mitigating them.[Footnote
36]
Status: Complete:
As noted earlier, one measure of the quality of system capabilities
being delivered is the number and severity of unresolved system defects
or problems. As such, information on such defects is an important
consideration when program decisions, such as key milestone decisions,
are made.
We previously reported[Footnote 37] that several key milestones were
passed that had severe open defects and that program officials were
unable to provide documentation regarding how the risks associated with
these defects were assessed. We also reported that these risks were not
being tracked in the program‘s risk database.
Since then, the system life cycle gate review process for Office of
Information and Technology projects, including ACE, has been amended to
include milestone readiness decisions based on an assessment and
acceptance of risks (including the risks related to unresolved
defects). Further, the ACE Risk and Issue Management Process guidance
provides for the systematic identification, analysis, prioritization,
planning, execution, evaluation, and documentation of program risks and
issues and requires that any risks associated with going forward should
be identified as part of each life cycle gate review.
This process has been applied in the following two major gate reviews:
* Release A1 (ESAR) Critical Design Review (May 2006) and:
* Release M1 (E-Manifest: Rail and Sea Manifest) Critical Design Review
(August 2006).
In the case of Release 5/Drop A1, program officials reported that no
severe defects existed, but that all known risks associated with
proceeding past the milestone had been entered into the program‘s risk
database and documented in the certification package submitted to the
DHS CIO.
The quarterly reports to Congress similarly state that risks related to
gate review decisions and the associated impacts are entered into a
database to ensure visibility and mitigation and are included in the
package submitted to the DHS CIO for review and certification to pass a
given milestone. Going forward, officials stated that similar
assessments will be part of the following planned gate reviews for
Release 5/Drop A1:
* Operational Readiness Review (planned for August 23, 2007) and:
* Live operations (planned for August 25, 2007).
Open recommendation 2: Develop the range of realistic ACE performance
measures and targets needed to support an outcome-based, results-
oriented accountability framework, including user satisfaction with
ACE.
Status: Complete:
We have previously reported on both the absence of meaningful
performance measures to understand ACE progress, quality, and results,
and have raised concerns about the practicality and applicability of
some measures that have been defined. Most recently, we
reported[Footnote 38] that defined ACE performance targets are not
always realistic and that goals, expected mission benefits, and
performance measures are not fully defined and adequately aligned.
In response, the program office has established an initial set of
performance measures that are tied to program objectives and related
performance goals, and it has established processes for the collecting,
analyzing, and integrating performance data for each measure. Among
other things, these performance measures address user satisfaction,
efficiency, and productivity. Moreover, these measures have been made
an integral part of the ACE accountability framework and they were
approved by the CBP Commissioner and the DHS CIO on June 30 and July 6,
2006, respectively.
Objective 2: Open Recommendations:
Performance Measures:
According to program officials, these initial measures will continue to
be revised and augmented, and thus will evolve over time based on
lessons learned, changes to existing release capabilities, and
refinements of requirements for future releases. For example, the
program office reported in March 2007 that new measures are still being
developed, reviewed, and evaluated for Releases 2, 3, and 4 at the same
time that performance data is being collected and reported for existing
measures for these releases. According to program officials, the new
measures will be combined with existing performance measures.
In addition, CBP‘s December 2006 quarterly congressional report stated
that performance measures for future releases, such as Release 5, are
being scheduled for identification and development and that changes are
being made to existing measures when they are determined to be
inappropriate. For example, the program office learned that the
performance measure "percentage of truck manifests being filed
electronically" did not recognize that empty trucks are not required to
file manifests, and thus it revised the metric used to divide the
number of electronically filed truck manifests by the total number of
required truck manifests instead of the total number of trucks in order
to get a more meaningful reflection of ACE performance.
To assist in managing these performance measures, the program has
established a life cycle process for performance measures as well as a
database tool to record and manage modifications.
Objective 2: Open Recommendations:
Performance Measures Alignment:
Open recommendation 3: Explicitly align ACE program goals, benefits,
desired business outcomes, and performance measures.
Status: Complete:
As just discussed, the program office has developed an initial set of
ACE program measures that are tied to program goals, among other
things, and they intend to continue to update and evolve these as the
program moves forward. In addition, the program office has developed an
ACE Performance Reference Model that explicitly aligns the CBP
strategic goals, objectives, strategies, and desired results applicable
to the ACE program with specific performance measures. For example,
Table:
CBP Strategic Goal;
Preventing terrorism at ports of entry: Prevent terrorists and
terrorist weapons, including weapons of mass destruction and weapons of
mass effect, from entering the U. S..
CBP Objective;
Improve information and targeting.
CBP Strategy;
Use advanced passenger and cargo information (NTC, ATS-Air, ATS,
Screening and Targeting-ACE) to pre-screen, target, and identify
potential terrorists and terrorist shipments and any related activity.
Desired Result;
Increased use of targeting.
Performance Measure;
Number of security-focused selections generated by system (= or above
intensive threshold) by type.
Further, this model links the measures to specific ACE releases. For
example, the above performance measure applies to Screening Foundation
(S1) and Targeting Foundation (S2).
Objective 2: Open Recommendations:
Management Improvement Measures:
Open recommendation 4: Define measures and collect and use associated
metrics for determining whether prior and future program management
improvements are successful.
Status: Planned:
As we have previously reported,[Footnote 39] investments in program
management improvements should include defined measures of progress and
results. To date, the program office has implemented a number of such
improvements; however, it has not had measures or metrics to determine
the success of the improvements. Moreover, the December 2006 quarterly
congressional report stated that the program anticipates more changes,
including:
* creation of a cargo requirement management board to decide the
disposition of all change requests to production systems;
* establishment of a new invoice review policy; and:
* co-location of personnel within a given business area.
According to its December 2006 quarterly reports to Congress, the
program office plans to measure the impact of future management
improvements. However, program officials told us that they have yet to
define them and thus are not yet using such measures.
Objective 2: Open Recommendations:
Legislative Conditions:
Open recommendation 5: Fully address those legislative conditions
associated with measuring ACE performance and results and employing
effective IV&V practices.
Status: Complete:
Among the legislative conditions that ACE expenditure plans have been
required to meet are satisfaction of OMB guidance, including that
associated with measuring program performance and results and use of
effective IV&V practices. These conditions reflect good program
management practices and, if implemented properly, can reduce program
risks.
Performance and Results:
As previously discussed in this briefing, the program office has
developed a range of ACE performance measures and taken steps to align
them with program, CBP, and DHS strategic goals and outcomes. According
to program officials, they plan to continue to evolve and refine the
performance measures and include the measures in the ACE accountability
framework.
IV&V Practices:
In January 2006, the DHS CIO certified that an IV&V agent was under
contract for the ACE program, but noted several issues, including
these:
* mechanisms were needed to ensure that products were complete, of
sufficient quality, and met the needs of the user and:
* a more explicit technical approach, describing when and how certain
activities should (or should not be) performed, was needed.
Moreover, we subsequently reported[Footnote 40] that the scope of the
contractor‘s activities did not extend to both IV&V of key system
products and development processes.
On October 24, 2006, the Deputy CIO again certified that an IV&V agent
was under contract and stated that the previous issues had been
addressed. Since then, the program office has also developed an IV&V
Implementation Management Plan that addresses the concerns raised by
the DHS CIO and us. In particular, the plan:
* requires an IV&V program consistent with the industry standard;
* provides a set of objectives, guidelines, and expectations for IV&V
activities, including periodic independent reports on status,
observations, recommendations, and activities; and:
* addresses satisfaction of quality standards for ACE products and user
needs.
Program officials told us that IV&V has allowed early identification
and correction of program process and product weakness.
Objective 2: Open Recommendations:
Reconciliation of Cost Estimates:
Open recommendation 6: Ensure that future expenditure plans are based
on cost estimates that are reconciled with independent cost estimates.
Status: Complete:
It is important that expenditure plans be based on reliable estimates
of costs, to include reconciling differences between government and
independent cost estimates. We recently reported[Footnote 41] that the
cost estimate in the fiscal year 2006 expenditure plan was based on
government and independent cost estimates that had been compared and
found to be consistent.
For the fiscal year 2007 expenditure plan, our analysis showed that the
government and independent cost estimates differed by about 15 percent.
According to program officials, they reconciled these differences in
January 2007, and concluded that the results did not warrant changes to
the expenditure plan because the government estimates used in the
expenditure plan were more accurate.
According to program officials, two primary factors account for the
difference in estimates. Specifically,
* The estimates assumed different timelines for completing development
of all releases. The independent estimate assumed development
completion by fiscal year 2010, while the government estimate assumed
fiscal year 2011. According to program office officials, this accounts
for about five percent of the difference.
* The government estimate included a number of items that the
independent estimator did not. For example, the independent estimator
did not include training and outreach items, such as the cost of
conferences with trade associations, training materials, and associated
travel. As a result, the government estimated training and outreach
costs over the life of the program to be about $90 million, while the
independent estimate put these costs at about $39 million. The
independent estimate has since been amended to include the missing
items, and the reconciliation process is continuing.
Objective 2: Open Recommendations:
Rigorous Cost Estimation:
Open recommendation 7: Develop and implement a rigorous and
analytically verifiable cost estimating program that embodies the
tenets of effective estimating.[Footnote 42]
Status: Complete:
The reliability of cost estimates is largely a function of the quality
of the estimating process used to derive them. We previously
reported[Footnote 43] that the program did not have a well-defined cost
estimating process, but that it has since made progress in
strengthening its cost estimating program. Specifically, the program
office has:
* defined and documented processes for estimating program costs
(including management reserve costs) and:
* hired a contractor to develop costs estimates that were independent
of the government estimates.
In September 2006, an ACE support contractor reported that both the
government and independent cost estimation processes demonstrated
significant conformance to effective estimating practices and concluded
that the program is using a rigorous and verifiable cost estimating
approach.
Objective 2: Open Recommendations:
Earned Value Management:
Open recommendation 8: Use EVM in developing all existing and future
releases.
Status: Complete:
EVM is a program management tool to measure progress by comparing,
during a given period of time, the value of work accomplished with the
amount of work expected to be accomplished. This comparison permits
performance to be evaluated based on calculated variances from the
planned (baselined) cost and schedule. EVM is both an industry accepted
practice and an OMB requirement.
We recently reported[Footnote 44] that the program office was not using
EVM for all releases (e.g., Release 5) due to changes to release
baselines and the lack of familiarity on the part of program staff with
EVM practices.
Since then, the program office has established the performance
baselines needed to implement EVM for Release 5/Drops A1 and A2,
Screening 3, and Release 6/Drop M1.
For these releases, the program office reports that it has also applied
EVM standards in establishing baselines and included EVM data in the
accountability framework for management and decision-making purposes.
Program officials also reported that they plan to implement EVM on
future releases and task orders.
Objective 2: Open Recommendations:
ACE Infrastructure:
Open recommendation 9: Have future ACE expenditure plans specifically
address any proposals or plans, whether tentative or approved, for
extending and using ACE infrastructure to support other homeland
security applications, including any impact on ACE of such proposals
and plans.
Status: Complete:
Together, the fiscal year 2007 expenditure plan and the quarterly
reports to Congress address ACE‘s relationships with other trade
processing and DHS applications. For example, the plan discusses
efforts to work with participating government agencies in defining and
deploying the International Trade Data System (ITDS). The stated goal
is to deliver ACE/ITDS in an integrated and coordinated manner. In this
regard, the plan discusses workshops to gather requirements from
participating agencies for Release 6/Drop M2 (first held on July 19,
2006), a working group to address these agencies‘ HAZMAT issues
(established on July 17, 2006), and efforts to develop data element
inputs for ACE from other government agencies.
In addition, the quarterly reports to Congress cite coordination
efforts with related homeland security programs. For example, the
report for the first quarter of 2007 states that:
* Container Security Initiative will be supported by ACE Release 6 and
Screening 1-3 capabilities;
* Customs-Trade Partnership Against Terrorism is coordinating with ACE
Release 5/Drop A1 capabilities to provide both CBP and trade
representatives with the ability to view the status of CBP programs;
and:
* U.S.-Mexico Border Partnership Plan will coordinate with ACE to
implement any cargo screening standards derived from partnership plan
agreements.
In addition, CBP reports that ACE and other CBP applications[Footnote
45] share infrastructure (desktops, clients, and local area networks)
at the ports of entry. This infrastructure is purchased and maintained
by CBP‘s OIT Program Integration Division. Each application, including
ACE, is responsible for complying with OIT‘s infrastructure standards.
Objective 2: Open Recommendations:
Overlap and Concurrence:
Open recommendation 10: Minimize the degree of overlap and concurrency
across ongoing and future ACE releases and capture and mitigate the
associated risks of any residual concurrence.
Status: In progress:
Significant overlap and concurrency among major program activities,
such as releases, introduce considerable cost and schedule risks as
they can create contention for limited resources among the releases. We
have continued to report on extensive overlap and concurrence in ACE
releases and the cost overruns and schedule delays that have resulted.
Most recently, we reported[Footnote 46] that the ACE schedule continued
to provide for such overlap and concurrency and that the risks
associated with doing so were not being effectively addressed. Since
then, the program office has reduced this overlap and concurrence. For
example, it has:
* decoupled (i.e., reduced dependencies among) certain ACE program
components by separating Screening 1-3 from Releases 4-6 and:
* aligned the development and delivery of functionality for different
releases with the availability of required hardware environments.
Recent quarterly congressional reports stated that CBP has taken
actions to reduce potential contention for limited resources. Examples
include:
* dividing releases into smaller subreleases, called drops, to provide
more flexibility in scheduling;
* improving planning for development, integration, testing, and
training activities and milestones to better schedule use of
development and test environments; and:
* centralizing management of shared software services to address, among
other things, allocation of resources and responsiveness to workload
peaks and use of consistent technical management approaches across
releases. Moreover, the program‘s risk database identifies overlap and
concurrency-related risks. However, the mitigation strategies for these
risks contain vague or incomplete data. These data limitations make it
difficult to determine the status or the effectiveness of the efforts
to reduce the risks associated with overlap and concurrence among
releases. (See the observations section of this briefing for additional
information on program risk management.)
The program office is also conducting regular integration meetings with
the teams supporting each release to discuss concerns, decisions, and
schedules associated with resource availability and is using a software
tool to track and mitigate release- specific concurrency risks.
Notwithstanding these steps, the program continues to face challenges
in managing dependencies among releases, which any associated
concurrency in the program‘s development exacerbates. For example, the
schedule slips expected with Release 5/Drop A2 will affect Release
6/Drop M1, as resources have been shifted from M1 to address the A2
delay. In addition, further delays in A1 and A2 will impact the
implementation of M1.
Objective 2: Open Recommendations:
Legislative Conditions:
Open recommendation 11: Direct the appropriate departmental officials
to fully address those legislative conditions associated with having an
approved privacy impact assessment (PIA)[Footnote 47] and ensuring
architectural alignment.
Status: In progress:
The department approved the ACE PIA for Release 4 in July 2006,
however, this assessment does not cover other completed releases.
Further, DHS has determined that ACE is aligned with DHS architecture,
but we have yet to receive documentation to adequately understand and
verify the determination.
PIA (In progress):
In March 2006, DHS developed guidance on the development and content of
a PIA. As we have previously reported,[Footnote 48] the purpose of a
PIA is to ensure that there is no collection, storage, access, use, or
dissemination of identifiable personal or business information that is
not both needed and permitted. Development and use of a PIA are both a
requirement of OMB and the E-Government Act of 2002.[Footnote 49]
The program office has developed a PIA for Release 4 of the ACE e-
Manifest: Trucks and the International Trade Data System, and DHS
approved it on July 14, 2006. As noted earlier, our analysis shows that
this PIA addresses the major elements of DHS‘s guidance. Program
officials stated that they would update the assessment for each ACE
release. This assessment does not cover other recently completed
screening releases (i.e., S1 and S2). However, program officials told
us that S1 and S2 are considered to be part of the Automated Targeting
System (ATS), and are therefore covered by the ATS PIA. However, our
analysis of the ATS PIA showed that while it addresses screening and
targeting functions, it does not specifically identify or address
releases S1 and S2.
Architectural Alignment (in progress):
As discussed in the legislative conditions section of this briefing,
DHS has determined that ACE is aligned with the DHS EA.
For example,
* In December 2006, DHS‘s EA Center of Excellence determined that ACE
was conditionally compliant with the DHS architecture, but that the
program needed to take actions to address four conditions.
* On March 2007, the DHS EA Board reported that ACE had satisfied three
of the conditions and recommended approval of ACE‘s request for program
alignment with one remaining condition”alignment with the DHS Technical
Reference Model.
* In May 2007, DHS EA officials reported that all required products and
technologies were aligned with the Technical Reference Model and that
ACE was thus aligned with the DHS EA.
However, as was also discussed earlier in this briefing, we have yet to
receive sufficient documentation describing the criteria and
methodology used to make these determinations or verifiable analysis
supporting the determinations. Moreover, the determinations were based
on technical alignment and did not address other relevant aspects of
program alignment to an EA, such as data alignment. Thus, the program
is at risk of being designed and implemented in a way that is not
consistent with all relevant aspects of the DHS EA, such as data
structures and standards, and thus does not support optimized DHS-wide
operations, performance, and results.
Objective 2: Open Recommendations:
Human Capital Practices:
Open recommendation 12: Develop and implement key human capital
management practices.
Status: In progress:
As we have previously reported,[Footnote 50] effective strategic
management of program human capital includes, among other things,
defining the skill sets needed to perform program functions, assessing
and inventorying the skill sets of the program‘s current workforce and
assessing any associated skill gaps in meeting future needs, and
developing strategies to fill any identified gaps. Moreover, it
includes having a well-defined plan that provides for effective
implementation of these processes.
In June 2006, CBP executives approved the OIT Strategic Human Capital
Management Plan. The plan is intended to be a road map for supporting
CBP‘s workforce vision and ensuring effective management of human
capital resources across OIT to address enterprise-wide priorities. An
ACE-specific plan was included as an appendix to the CBP plan. This
appendix was intended to provide better near- and long-term human
capital management practices for the OIT offices involved in ACE
development.
Neither the OIT nor the ACE-specific plan addresses the basic tenets of
effective human capital management. For example, neither provides for:
* defining the positions needed (including core competencies) to
perform core program functions;
* assessing and inventorying current workforce skills and abilities;
* assessing any gaps between needed and existing workforce levels and
capabilities; and:
* filling identified gaps via such means as hiring new staff, training
existing staff, and augmenting staff with contractor support.
OIT officials acknowledged these limitations in the plans and stated
that they are developing an implementation plan to address these
shortfalls. The officials stated that the implementation plan will
include accountability, timeframes, and metrics to carry out the larger
OIT Strategic Human Capital Management Plan.
According to these officials, this implementation plan was to be
completed in January 2007, following presentation to the OIT Deputy
Director's Council and approval by the OIT Office Directors and
Assistant Commissioner. However, as of July 2007, it had not yet been
approved and thus was not available for our review.
In the interim, OIT and program officials reported taking various steps
to address ACE human capital needs. These include, for example,
* Using various methods to fill staff vacancies. As of May 2007, the
program reports that it has 62 full-time employees and one vacancy and
that additional staff are working on ACE either full- or part-time.
* Lobbying OPM for more flexibility in recruiting, to include higher
salaries and direct hiring authority.
• Reorganizing OIT into six program offices aligned to major mission
areas so that the number of government personnel responsible for ACE
development activities can be augmented by IT functional program
management expertise.
* Offering training to improve the skills of staff.
Nevertheless, these steps have not been guided by a well-defined plan
and thus represent activities that are not tied to strategic goals and
outcomes and thus cannot be measured against them. Without a plan, it
is unlikely that the program will be able to adequately ensure that it
has the right people at the right time to deliver ACE successfully.
Objective 2: Open Recommendations:
Quarterly Reports:
Open recommendation 13: Include in the June 30, 2006, quarterly update
report to the appropriations committees a strategy for managing ACE
human capital needs and the ACE framework for managing performance and
ensuring accountability.
Status: In progress:
The June 30, 2006 quarterly report to the House and Senate
Appropriations Committees included the program‘s strategy for meeting
its human capital needs and its accountability framework.
Human Capital Strategy (In Progress):
The June 30, 2006, quarterly report to Congress included a description
of the previously described ACE-specific appendix to the OIT Strategic
Human Capital Management Plan, including the plan‘s five goals and
strategies. Moreover, the report states that this ACE-specific plan is
aligned with OPM‘s Human Capital Assessment and Accountability
Framework.[Footnote 51] As previously stated, however, this ACE-
specific plan does not meet the basic tenets of strategic human capital
management, as defined in this framework and other relevant guidance.
Objective 2: Open Recommendations:
Quarterly Reports:
According to program officials, they are developing a new OIT human
capital strategy and implementation plan, but they have yet to provide
us with a date when it will be completed.
Accountability Framework (Complete):
The June 30, 2006, quarterly report included a description of the ACE
accountability framework. As previously discussed, this framework is
the means by which the program measures performance relative to
promised ACE capabilities, costs, schedules, earned value, risks, and
mission values and benefits. The report also included an appendix
illustrating the format and content of the accountability framework
tool. At that time, CBP reported that it would continue to work with
stakeholders to further enhance the format, readability, and utility of
the framework as a program management and reporting tool.
Since June 2006, the program office has refined and implemented the
accountability framework. The framework covers all program commitment
areas and key system aspects to support executive decision making and
provides external program stakeholders with information on the program.
Objective 2: Open Recommendations:
Quarterly Report on Open Recommendations:
Open recommendation 14: Report quarterly to the House and Senate
Appropriations Committees on efforts to address open GAO
recommendations.
Status: Complete:
CBP has submitted quarterly reports on ACE to both the House and the
Senate Appropriations Subcommittees since November 2002, including
reports for each quarter of fiscal year 2006 and the first and second
quarters of fiscal year 2007. These reports have addressed CBP‘s
efforts to address open GAO recommendations.
Objective 2: Open Recommendations:
Accurately Report on Open Recommendations:
Open recommendation 15: Accurately report to the appropriations
committees on CBP's progress in implementing our prior recommendations.
Status: In progress:
As previously stated, CBP has included information on progress in
meeting GAO‘s recommendations in its quarterly reports[Footnote 52] to
the appropriations committees since November 2002. However, some of
this information is dated and thus inaccurate due to the time lapse
between when the reports are produced and when they are provided to the
appropriations committees. Recently, this time lapse has been between
about two and seven months. According to program officials, they are
exploring ways to accelerate the review process and thereby improve the
timeliness and accuracy of their reports to Congress. DHS officials
also told us that they have ongoing efforts to improve the review
process; however, the process has not improved consistently or
significantly.
Objective 3: Observations:
Requirements Limitations Likely to Cause Major Delays:
Observation 1: Redefinition of requirements for several ACE releases is
now under way to address limitations in completeness of originally
defined requirements, and this redefinition is likely to introduce
program schedule delays and cost increases.
A key aspect of successful system acquisition programs is having well-
defined requirements. Among other things, requirements should be
complete, and to accomplish this, best practices advocate engaging all
key stakeholders in the requirements definition and management process.
In defining ACE requirements, the program office discovered that its
original requirements definition approach did not adequately engage all
key stakeholders, and to its credit, has since taken steps to address
this. Specifically,
* In the spring of 2004, the program office and the ACE support team
conducted more than 300 business process workshops with the ACE user
community to help define ACE requirements for the future releases. This
requirements definition approach was referred to as the Global Business
Blueprint (GBB) effort.
Objective 3: Observations:
Requirements Limitations Likely to Cause Major Delays:
* In June and July of 2005, a key group of ACE stakeholders that were
not originally involved in the GBB (application programmers familiar
with the legacy system that ACE is to replace) raised questions about
the completeness of the requirements. To address these questions, the
program office examined the GBB‘s completeness by first reverse-
engineering the legacy software for a small number of legacy system
programs and then comparing the reverse-engineered requirements to the
GBB-derived requirements. Based on this, which is referred to as legacy
code decomposition, the program office found that the GBB-derived
requirements were missing about 20 percent of needed ACE functionality.
* In August 2005, the program office decided that it needed to
decompose all of the legacy system code in order to completely capture
the requirements for all ACE releases. Work began with decomposition of
the legacy code related to ACE Release 5/Drop A2 (Entry Summary
Accounts and Revenue).
* In September 2006, the program office determined that the legacy code
decomposition approach alone was not sufficient to gain a full
understanding of the requirements given the size and complexity of A2.
As a result, the requirements definition process was expanded to engage
another key stakeholder group (business process subject matter
experts). Under the expanded approach, referred to as legacy code
decomposition and collaboration, legacy system software programmers and
subject matter experts were to examine the code line by line in
defining ACE requirements.
* In November 2006, the legacy system code decomposition and
collaboration effort for Release 5/Drop A2 fell behind schedule because
of lack of personnel with legacy system expertise and experience. The
schedule for A2 was tentatively revised, but at the time of our review,
the schedule was still under review, had not yet been approved, and
thus was not yet available for our review.
The program office has identified the A2 legacy code decomposition and
collaboration process as a high risk item that will significantly
impact the A2 schedule. For example, the decomposition and
collaboration process for part of the functionality on A2‘s critical
path”the Authorized Broker Interface Entry Summary”is not expected to
be completed before early 2008, at the earliest.
According to program officials, delays for other releases/drops, such
as Release 5/Drop A1 and Release 6/Drop M1, will not be as significant.
They also said that, while they have not yet estimated how long A2 will
be delayed and what the associated cost implications are, they do not
expect the cost increases to breach the current acquisition program
baseline of $3.3 billion, which translates into a cost increase of less
than $200 million. Moreover, they said that several actions have been
taken to minimize the impact of the delays. For example,
* A2 functionality necessary for M1 has been given priority in order to
support M1 deployment as originally planned and:
* A2 scope is being divided into increments to allow some functionality
to be delivered sooner and to minimize the impact on other drops.
However, these actions carry consequences, such as missed opportunities
to combine field training and an increase in the number of legacy
interfaces, thus increasing the potential for introducing software
problems.
In addition, program officials told us that they are considering
changes to the A2 and M1 deployment strategies to address stakeholder
concerns, and they said that these changes could also minimize the
magnitude of the A2 and M1 delays. Specifically, they said that they
had planned to deploy on a national basis, meaning that A2 and M1
functionality would be deployed to all ports at the same time and
concurrently adopted by all users nationwide. However, deployment may
change to a filer-by-filer basis, meaning that A2 and M1 functionality
would be deployed to all ports at the same time, but not all filers
would begin using it at the same time.
Objective 3: Observations:
Requirements Limitations Likely to Cause Delays:
According to program officials, they believe that the change in the
deployment strategy would both address stakeholder concerns and
minimize any A2 and M1 schedule delays caused by the redefinition of
requirements. However, they added that these changes have yet to be
approved, and the full extent of the cost and schedule implications are
not yet known. Moreover, neither the fiscal year 2007 expenditure plan
nor the ACE quarterly reports have disclosed the A2 and M1 requirements
redefinition and its impact, and neither has addressed any changes to
their deployment strategies.
Objective 3: Observations:
Key COTS Product Being Replaced:
Observation 2: Significant changes to ACE requirements have, in turn,
led to reevaluation and replacement of a key commercial off-the-shelf
product (COTS) previously selected and being prepared for use.
When acquiring commercial component-based systems, like ACE, best
practices advocate basing decisions on whether to employ a given COTS
product on thorough, rigorous, and continuous analysis of a number of
factors, including how well competing products satisfy defined system
requirements.
To the program office‘s credit, it reports having followed the CBP
system life cycle methodology to determine which COTS product would
best meet the requirements for Release 5/Drop A2. These analyses
include:
* In 2002, the program office reviewed, in general terms, various COTS
packages and determined that a solution using SAP (formerly Systems
Application and Products), a COTS provider, combined with other
commercial solutions and customized development, provided the best
combination of capability, performance, and cost for ACE.
* In 2004, a more detailed analysis was conducted as part of the
previously mentioned GBB process, which was intended to define and
allocate ACE requirements to future ACE releases and provide the basis
for, among other things, selecting a specific SAP product. At that
time, the SAP Enterprise Portal product was selected for Release 5/Drop
A2.
* In December 2006, the ACE Chief System Architect recommended that the
Internet Transaction Server (ITS) technology already used by CBP should
be adopted instead of the SAP Enterprise Portal, based on the
determination that all currently planned SAP functionality could be
presented using the ITS technology. This decision was based on improved
understanding of the requirements and previous analyses of ITS.
On the basis of these analyses and the Chief System Architect‘s
recommendation, the program office subsequently stopped work on SAP
Enterprise Portal design and configuration efforts and, in March 2007,
the program reported that the SAP Internet Transaction Server would be
used for Release 5/Drop A2 instead of the SAP Enterprise Portal. While
this decision was expected to have some near-term schedule impact
because much of the completed work for A2 had been based on the planned
use of SAP Enterprise Portal, the program office reports that these
impacts are offset by the cost advantages of other ACE releases already
using the Internet Transaction Server technology.
However, in March 2007, program officials identified a risk of
inadequate response time for the Internet Transaction Server”thus
negatively impacting user productivity”and that there was high
probability of significant cost and schedule impacts. Actions are
underway to mitigate the risk through performance modeling and test
planning.
Neither the fiscal year 2007 expenditure plan nor the quarterly reports
to Congress disclose this COTS product change, its impact on release
schedules and cost estimates, or risk to future system performance.
Objective 3: Observations:
Risks Not Being Effectively Managed:
Observation 3: All program risks are not being effectively managed.
Risk management is a continuous, forward-looking process that is
intended to either prevent program cost, schedule, and performance
problems from occurring or to minimize the impact if they occur.
According to relevant guidance and best practices, effective risk
management involves proactively identifying, assessing, and disclosing
risks; defining and implementing cost-effective strategies for
mitigating these risks; and measuring and disclosing progress in doing
so. To its credit, the program office has developed a process guide and
implemented an automated tool (database) for managing ACE risks in
accordance with relevant guidance and best practices. Among other
things, the database contains the description, level (high, medium, or
low), and mitigation strategy (including start and end dates, exit
criteria, and implementation status) for each risk.
However, the completeness and quality of the information on each of the
risks in the risk database[Footnote53] vary. For example,
* many risks were missing information on the status of efforts to
implement the mitigation strategy and:
* many risks (18) were missing criteria for completing mitigation
steps, clear descriptions of what the risk entailed, and start and end
dates for planned mitigation activities.
Because of such database limitations, we could not determine the status
of and mitigation progress on 17 risks. Moreover, these database
limitations were reflected in the documentation used at key program
events, such as PMRs. This means that the program does not have the
risk-related information that it needs to inform its program decisions
and to reduce the chances of potential problems becoming actual
problems.
Program officials, including the official responsible for risk
management, stated that risk management is immature and needs to be
strengthened. The reasons they gave for these risk management
weaknesses are due to:
* all staff not being trained on how to use the tool (last training was
provided in 2003, and the since then a number of people have joined the
program);
* the tool is unique to the ACE program and thus no CBP guidance exists
on its use); and:
* each ACE group addresses risk differently in its weekly meetings.
To improve ACE risk management, program officials told us that they
are:
* establishing a group to ensure the quality and completeness of the
database;
* holding regular group meetings with contract staff and team leads to
discuss risks and their impacts; and:
* conducting risk management training.
If implemented effectively, such steps should result in more meaningful
information about program risks that can be useful to DHS in managing
the program and to Congress in overseeing it. To date, however, ACE
program risks have not been communicated to oversight organizations
through the 2007 expenditure plan or recent quarterly reports to the
House and Senate Appropriations Committees.
Conclusions:
Over the past 7 years, CBP and DHS have worked to fulfill legislatively
mandated annual expenditure plan requirements and to implement dozens
of our recommendations related to these plans and management of the
program. Among other things, these requirements and recommendations
have promoted effective program management and accountability for
performance and results. As a result of these years of efforts, the ACE
program is better positioned today for delivering promised capabilities
and benefits than it has been in the past.
Nevertheless, key program management practices relating to, for
example, human capital management, requirements management, and risk
management continue to remain a challenge, and other management areas,
such as information security and architecture alignment, continue to
require attention. As a result, avoiding major program schedule delays
and cost overruns remains a challenge as more of each appear to be on
the horizon.
To further improve ACE management and minimize its exposure to risk, it
is important for CBP and DHS to remain vigilant in their efforts to
satisfy ACE legislative requirements and to fully implement our prior
recommendations. Moreover, it is important that they keep the Congress
fully informed on where the program stands and what changes are planned
to address emerging cost overruns and schedule delays.
Recommendations for Executive Action:
To further strengthen ACE management and promote accountability for ACE
performance and results, we are making the following recommendation to
the Secretary of Homeland Security to direct the CBP Commissioner to
ensure that future quarterly reports to the House and Senate
Appropriations Committees disclose:
(1)the risks and associated mitigation strategies of not having fully
satisfied the expenditure plan legislative conditions and not having
completed implementation of all our prior recommendations;
(2)the status and impacts on the program‘s estimated cost and schedule
and lessons learned from ongoing efforts to redefine requirements and
to implement a different COTS product than originally selected; and:
(3)the program‘s plans and actions for improving ACE risk management
and its current inventory of program risks, including their associated
mitigation strategies and the status of the strategies‘ implementation.
Agency Comments:
In oral comments on a draft of this briefing, DHS and CBP officials
agreed with our conclusions and recommendations, and provided
clarifying information and technical comments that we incorporated in
the briefing, as appropriate.
Attachment 1:
Scope and Methodology:
To accomplish our objectives, we analyzed the ACE fiscal year 2007
expenditure plan and supporting documentation, and compared them to
relevant federal requirements and guidance, applicable best practices,
and our prior recommendations. We also interviewed DHS and CBP
officials and ACE program contractors. In particular, we reviewed
* DHS and CBP investment management practices, using OMB A-11, part 7;
* DHS and CBP certification activities for ensuring ACE compliance with
the DHS enterprise architecture;
* DHS and CBP acquisition management efforts, using SEI‘s SA-CMM;
* CBP cost estimating program and cost estimates, using SEI‘s
institutional and project-specific estimating guidelines;[Footnote 54]
* independent verification and validation (IV&V) activities using the
Institute of Electrical and Electronics Engineers standard for software
verification and validation;[Footnote 55]
* CBP actions to coordinate ACE with related programs;
* CBP‘s reorganization documentation, including the organizational
charts and roles and responsibilities matrix;
* ACE‘s accountability framework; and:
* cost and schedule data and program commitments from program
management documentation.
For DHS-, CBP-, and contractor-provided data that we did not
substantiate, we have made appropriate attribution indicating the
data's source.
We conducted our work at CBP headquarters and contractor facilities in
the Washington, D.C., metropolitan area from December 2006 through July
2007 in accordance with generally accepted government auditing
standards.
Attachment 2:
Related GAO Products:
ACE Expenditure Plans:
Information Technology: Customs Has Made Progress on Automated
Commercial Environment System, but IT Faces Long-Standing Management
Challenges and New Risks. GAO-06-580. Washington, D.C.: May 31, 2006.
Information Technology: Customs Automated Commercial Environment
Program Progressing, but Need for Management Improvements Continues.
GAO-05-267. Washington, D.C.: March 14, 2005.
Information Technology: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule Problems Needs to Be
Addressed. GAO-04-719. Washington, D.C.: May 14, 2004.
Customs Service Modernization: Automated Commercial Environment
Progressing, but Further Acquisition Management Improvements Needed.
GAO-03-406. Washington, D.C.: February 28, 2003.
Customs Service Modernization: Third Expenditure Plan Meets Legislative
Conditions, but Cost Estimating Improvements Needed. GAO-02-908.
Washington, D.C.: August 9, 2002.
[End of section]
Appendix II: Comments from the Department of Homeland Security:
U.S. Department of Homeland Security:
Washington, DC 20528:
[hyperlink, http://www.dhs.gov]
October 10, 2007:
Mr. Randolph C. Hite:
Director, Information Technology Architecture and Systems Issues:
U.S. Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Mr. Hite:
Re: Draft Report GAO-08-46, Information Technology: Improvements for
Acquisition of Customs Trade Processing System Continue, but Further
Efforts Needed to Avoid More Cost and Schedule Shortfalls (GAO Job Code
310634)
The Department of Homeland Security (DHS) appreciates the opportunity
to review and comment on the draft report referenced above. Consistent
with the Fiscal Year 2007 DHS appropriations act, DHS is to develop and
submit an expenditure plan for the Automated Commercial Environment
(ACE) that satisfies certain conditions, including being reviewed by
the U.S. Government Accountability Office (GAO). The report correctly
notes that the expenditure plan satisfies many of the legislative
conditions specified in the act and that the Department continues to
make progress.
GAO recognizes that we have implemented eight GAO recommendations made
during the last four years and that seven other recommendations made
during that time are in the process of being implemented. Department
and U.S. Customs and Border Protection (CBP) officials are committed to
fully addressing these remaining open recommendations regarding the ACE
accountability framework; measures and associated program management
metrics for determining success of improvements; minimizing the degree
of overlap and concurrency across ACE releases and mitigating
associated risks; meeting the legislative conditions regarding a
privacy impact assessment and ensuring architectural alignment;
implementing key human capital management practices; managing human
capital needs; and reporting progress implementing GAO recommendations
to Congress quarterly.
GAO recommends three actions involving program information disclosure
to the House and Senate Appropriations Committees designed to further
strengthen ACE management and accountability for ACE performance and
results. U.S. Customs and Border
Protection officials agree with the recommendations. The new
recommendations and CBP's planned corrective actions follow.
Recommendation 1: [Future quarterly reports disclose] the risks and
associated mitigation strategies of not having fully satisfied the
expenditure plan legislative conditions and not having completed
implementation of all prior GAO recommendations.
Response: GAO concluded that CBP satisfied four of six legislative
conditions but only partially satisfied the following two conditions:
(1) Meet the capital planning and investment control review
requirements established by the Office of Management and Budget (OMB),
including OMB Circular A-11, Preparation, Submission, and Execution of
the Budget, Part 7 [Planning, Budgeting, Acquisition and Management of
Capital Assets]. The risk of not completely satisfying this requirement
as part of the expenditure plan is minimal as CBP does meet OMB's
requirements, but must better demonstrate compliance in future
expenditure plan submissions. Full alignment to OMB's investment
management requirements can be demonstrated via the Office of
Management and Budget Exhibit 300 form [ACE Exhibit 300] that will be
attached to all future ACE Expenditure Plan submissions.
(2) Comply with the DHS enterprise architecture. DHS officials
determined in May 2007 that all required ACE products and technologies
were aligned with the DHS technical reference model and that ACE was
thus aligned with the DHS enterprise architecture. CBP agrees with GAO
that a compliance determination is not a one- time event but a series
of determinations that occurs throughout an investment's life cycle.
CBP will continue to address DHS architectural requirements as DHS
evolves its alignment methodology and criteria to meet GAO
expectations.
Recommendation 2: [Future quarterly reports disclose] the status and
impacts on the program's estimated cost and schedule and lessons
learned from ongoing efforts to redefine requirements and to implement
a different commercial off-the-shelf product than originally selected.
Response: ACE program officials intend to disclose this information in
future quarterly reports.
Recommendation 3: Future quarterly reports disclose] the program's
plans and actions for improving ACE risk management and its current
inventory of program risks, including their associated mitigation
strategies and the status of the strategies' implementation.
Response: Officials responsible for ACE plan to strengthen risk
management by:
(1) Undertaking a new round of risk identification efforts to include
the appropriate contractor technical and management personnel as well
as government personnel.
The first session was held on September 14, 2007. After these
identification sessions, the list of risks will be analyzed and
prioritized. A response strategy will be developed for each risk.
(2) The risks will be categorized and linked to the 19 risk categories
in the OMB 300 document for information technology programs. This
effort will ensure that the critical areas of risk are covered.
(3) Reports from the risk management software will be made available to
the appropriate oversight personnel. The reports will include the
mitigation strategy and status of its implementation.
The foregoing actions should enable CBP to ensure that future
Congressional reports provide complete and current information
regarding program risks and attendant mitigation strategies, as well as
the status of efforts to mitigate these risks.
Sincerely,
Signed by:
Steven J. Pecinovsky:
Director:
Departmental GAO/OIG Liaison Office:
[End of section]
Appendix III: Contact and Staff Acknowledgments:
GAO Contact:
Randolph C. Hite, (202) 512-3459:
Staff Acknowledgments:
In addition to the person named above, Daniel Castro, Dawn Day, Neil
Doherty, Nancy Glover, Paula Moore, Jamelyn Payan, Nik Rapelje, and
Karen Talley made key contributions to this report.
[End of section]
Footnotes:
[1] Pub. L. No. 109-295 (Oct. 4, 2006).
[2] Two related open recommendations have been combined.
[3] We did not evaluate the compliance of the ACE program with respect
to the Federal Acquisition Regulation or OMB directives, but we did
evaluate its compliance with established best practices models that
incorporate IT investment requirements for federal agencies.
[4] This recommendation has multiple conditions, of which three are
completed and two are in progress. A sixth condition of this
recommendation--that the ACE accountability framework clearly and
unambiguously delineate roles and responsibilities of the government
and the prime contractor--was previously completed. See Information
Technology: Customs Has Made Progress on Automated Commercial
Environment System, but It Faces Long-Standing Management Challenges
and New Risks, GAO-06-580 (Washington, D.C.: May 31, 2006).
[5] 44 U.S.C. § 3501 note.
[6] The U.S. Customs and Border Protection (CBP), formerly the Bureau
of Customs and Border Protection, was formed in 2003 under the new
Department of Homeland Security from the former U.S. Customs Service
and other entities with border protection responsibilities.
[7] Pub. L. No. 109-295, (October 4, 2006).
[8] OMB Circular A-11 establishes policy for planning, budgeting,
acquisition, and management of federal capital assets.
[9] The purpose of the IRB is to integrate capital planning and
investment control, budgeting, acquisition, and management of
investments. It is also to ensure that spending on investments directly
supports and furthers the mission and that this spending provides
optimal benefits and capabilities to stakeholders and customers.
[10] Two related open recommendations have been combined.
[11] 19 U.S.C. Section 1411.
[12] This partnership was formerly known as the e-Customs partnership.
It includes Lockheed Martin, Bearing Point, Sandler and Travis,
Computer Sciences Corporation, and a number of small businesses.
[13] CBP national account managers work with the largest importers to
ensure their compliance with trade laws.
[14] Brokers obtain licenses from CBP to conduct business on behalf of
the importers by filling out paperwork and obtaining a bond; carriers
are individuals or organizations engaged in transporting goods for
hire.
[15] Manifests are lists of passengers or invoices of cargo for a
vehicle, such as a truck, ship, or plane.
[16] Systems Applications and Products (SAP) is a commercial enterprise
resource planning software that has multiple modules, each performing
separate but integrated business functions. ACE will use SAP to support
many of its business processes and functions. In addition, the CBP‘s
Modernization Office is using SAP as part of a joint project with its
Office of Finance to support financial management, procurement,
property management, cost accounting, and general ledger processes.
[17] The multimodal manifest involves the processing and tracking of
cargo as it transfers between different modes of transportation, for
example, cargo arrives by ship, is transferred to a truck, and then is
loaded onto an airplane.
[18] An import activity summary statement is a summary of an importer‘s
shipment activities over a specific period of time that is transmitted
electronically to CBP on a periodic basis by importers and brokers.
[19] These accounts include importers, brokers, carriers, authorized
service providers, and other members of the trade community who utilize
ACE.
[20] OMB Circular A-11, part 7 establishes policy for planning,
budgeting, acquisition, and management of federal capital assets.
[21] EVM is a management tool for measuring progress and is both an
industry accepted practice and an OMB requirement.
[22] Guide for the Security Certification and Accreditation of Federal
Information Systems, NIST Special Publication 800-37. NIST,
Gaithersburg, MD, May 2004.
[23] A technical reference model is list of approved IT industry
standards, hardware, and software products that ensures that IT
solutions developed by individual programs (such as ACE) are consistent
within DHS as a whole.
[24] For this condition, we did not evaluate the compliance of the ACE
program with respect to the Federal Acquisition Regulation, OMB
directives, or other governmentwide requirements.
[25] See, for example, the Clinger-Cohen Act of 1996 (P.L. No. 104-106,
40 U.S.C. §§11101 through §§11704) and OMB Circular A-130.
[26] GAO-06-580; GAO-05-267. As with our prior reviews of ACE
expenditure plans, the evaluation does not include compliance with
federal acquisition regulations or other federal rules and requirements
beyond those encompassed by SEI‘s Capability Maturity Models.
[27] The SA-CMM® is consistent with the acquisition guidelines and
systems acquisition management practices of the federal government, and
provides a management framework that defines acquisition practices for
such process areas as acquisition planning, solicitation, requirements
development and management, project management, contract tracking and
oversight, and evaluation.
[28] ACE‘s level 2 rating indicated that CBP had instituted basic
acquisition management processes and practices consistent with the
acquisition guidelines and management practices of the federal
government in the following areas: acquisition planning, solicitation,
requirements development and management, project management, contract
tracking and oversight, and evaluation.
[29] Level 3 capability is more advanced than level 2 and indicates
that acquisition management processes have been defined throughout the
organization.
[30] The program office refers to these plans, processes, and
procedures as assets.
[31] GAO-06-580.
[32] GAO-06-580.
[33] Quarterly reports should be received by Congress approximately 60
days after the end of the quarter.
[34] GAO-06-580.
[35] GAO-06-580.
[36] This is a combination of two prior recommendations. From GAO-05-
267, "Define and implement an ACE accountability framework that
—ensures use of criteria for exiting key readiness milestones that
adequately consider indicators of system maturity, such as severity of
open defects“ and from GAO-06-580, ’Document key milestone decisions in
a way that reflects the risks associated with proceeding with
unresolved severe defects and provides for mitigating these risks."
[37] GAO-06-580.
[38] GAO-06-580.
[39] GAO-04-719.
[40] GAO-06-580.
[41] GAO-06-580.
[42] See, for example, models developed by the Carnegie Mellon
University SEI, Checklists and Criteria for Evaluating the Cost and
Schedule Estimating Capabilities of Software Organizations, CMU/SEI-95-
SR-005 (Pittsburgh, Pa.: Carnegie Mellon University, 1995) and A
Manager's Checklist for Validating Software Cost and Schedule
Estimates, CMU/SEI-95-SR-004 (Pittsburgh, Pa.: Carnegie Mellon
University,1995).
[43] GAO-04-719, GAO-05-267, GAO-06-580.
[44] GAO-06-580.
[45] CBP applications include IT systems that provide tools and
information to help front-line officers ensure the security of our
nation. This includes applications for the United States Visitor and
Immigrant Status Indicator Technology (US-VISIT), Container Security
Initiative and Automated Targeting System.
[46] GAO-06-580.
[47] 44 U.S.C. § 3501 note.
[48] GAO-06-580.
[49] OMB, Guidance for Implementing the Privacy Provisions of the
EGovernment Act of 2002, OMB M-03-22 (Sept. 26, 2003).
[50] GAO-06-580.
[51] As revised by OPM in 2005, this framework reflects guidance from
the collaboration of OMB, OPM, and GAO.
[52] The reports are due to Congress approximately 60 days after the
completion of a fiscal quarter. ACE quarterly reports are due to DHS
approximately 30 days after the end of the quarter, and then DHS has 30
days to review the report and submit it to Congress.
[53] As of May 23, 2007 the risk database contained 46 risks.
[54] SEI's Institutional and project-specific estimating guidelines are
defined in Robert E. Park, Checklists and Criteria for Evaluating the
Cost and Schedule Estimating Capabilities of Software Organizations,
CMU/SEI-95-SR-005 (Pittsburgh, Pa.: Carnegie Mellon University Software
Engineering Institute, 1995) and A Manager's Checklist for Validating
Software Cost and Schedule Estimates, CMU/SEI-95-SR-004 (Pittsburgh,
Pa: 1995), respectively.
[55] Institute of Electrical and Electronics Engineers (IEEE) Computer
Society, Standard for Software Verification and Validation 1012-1998
(June 8, 2005).
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "Subscribe to Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Gloria Jarmon, Managing Director, JarmonG@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
