Information Technology
Immigration and Customs Enforcement Needs to Fully Address Significant Infrastructure Modernization Program Management Weaknesses Gao ID: GAO-07-565 April 27, 2007The Department of Homeland Security (DHS) fiscal year 2006 appropriations act provided $40.15 million for the Immigration and Customs Enforcement's (ICE) program to modernize its information technology (IT) infrastructure. As mandated by the appropriations act, the department is to develop and submit for approval an expenditure plan for the program, referred to as "Atlas," that satisfies certain legislative conditions, including a review by GAO. In performing its review of the Atlas plan, GAO (1) determined whether the plan satisfies certain legislative conditions and (2) provided other observations about the plan and management of the program. To do this, GAO analyzed the fiscal year 2006 Atlas expenditure plan and supporting documents against the legislative conditions, federal requirements, and related best practices. GAO also interviewed relevant DHS officials.
The fiscal year 2006 Atlas expenditure plan, in combination with related program documentation and program officials' statements, satisfies or partially satisfies the legislative conditions set forth by Congress. This satisfaction, however, is based on plans and commitments that provide for meeting these conditions rather than on completed actions to satisfy them. For example, to address the legislative condition related to capital planning and investment control review requirements, the program plans to, among other things, update its cost-benefit analysis in September 2007 to reflect emerging requirements and other program changes and to complete a privacy impact assessment by April 2007. In addition, the program is in the process of defining how it plans to use its independent verification and validation agent. GAO also observed that DHS has not implemented key system management practices. Specifically, (1) rigorous practices are not being fully adhered to in developing and managing system requirements, (2) key contract management and oversight controls have not been fully implemented, (3) planned risk management practices have yet to be implemented, and (4) performance management practices that are critical to measuring progress against program goals are still being implemented. Thus, much still needs to be accomplished to minimize the risks associated with the program's capacity to deliver promised IT infrastructure capabilities and benefits on time and within budget. Given that hundreds of millions of dollars are to be invested and the program is critical to supporting the ICE mission, it is essential that DHS follow through on its commitments to build the capability to effectively manage the program. Proceeding without it introduces unnecessary risks to the program and potentially jeopardizes its viability for future investment.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-07-565, Information Technology: Immigration and Customs Enforcement Needs to Fully Address Significant Infrastructure Modernization Program Management Weaknesses
This is the accessible text file for GAO report number GAO-07-565
entitled 'Information Technology: Immigration and Customs Enforcement
Needs to Fully Address Significant Infrastructure Modernization Program
Management Weaknesses' which was released on April 30, 2007.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Committees:
April 2007:
Information Technology:
Immigration and Customs Enforcement Needs to Fully Address Significant
Infrastructure Modernization Program Management Weaknesses:
GAO-07-565:
GAO Highlights:
Highlights of GAO-07-565, a report to congressional committees
Why GAO Did This Study:
The Department of Homeland Security (DHS) fiscal year 2006
appropriations act provided $40.15 million for the Immigration and
Customs Enforcement‘s (ICE) program to modernize its information
technology (IT) infrastructure. As mandated by the appropriations act,
the department is to develop and submit for approval an expenditure
plan for the program, referred to as ’Atlas,“ that satisfies certain
legislative conditions, including a review by GAO. In performing its
review of the Atlas plan, GAO (1) determined whether the plan satisfies
certain legislative conditions and (2) provided other observations
about the plan and management of the program. To do this, GAO analyzed
the fiscal year 2006 Atlas expenditure plan and supporting documents
against the legislative conditions, federal requirements, and related
best practices. GAO also interviewed relevant DHS officials.
What GAO Found:
The fiscal year 2006 Atlas expenditure plan, in combination with
related program documentation and program officials‘ statements,
satisfies or partially satisfies the legislative conditions set forth
by Congress (see table).
Table: DHS's Satisfaction of Legislative Conditions:
Legislative conditions: 1. Expenditure plan must meet the capital
planning and investment control review requirements established by the
Office of Management and Budget (OMB); Status: partially satisfies.
Legislative conditions: 2. DHS must ensure Atlas is compliant with
DHS's enterprise architecture; Status: partially satisfies.
Legislative conditions: 3. The plan must comply with the acquisition
rules, requirements, guidelines, and systems acquisition management
practices of the federal government; Status: partially satisfies.
Legislative conditions: 4. The plan must include a certification by
DHS's chief information officer that an independent verification and
validation agent is currently under contract for the project; Status:
partially satisfies.
Legislative conditions: 5. The plan must be reviewed and approved by
the DHS Investment Review Board, the Secretary of Homeland Security,
and OMB; Status: satisfies.
Source: GAO.
[End of table]
This satisfaction, however, is based on plans and commitments that
provide for meeting these conditions rather than on completed actions
to satisfy them. For example, to address the legislative condition
related to capital planning and investment control review requirements,
the program plans to, among other things, update its cost-benefit
analysis in September 2007 to reflect emerging requirements and other
program changes and to complete a privacy impact assessment by April
2007. In addition, the program is in the process of defining how it
plans to use its independent verification and validation agent.
GAO also observed that DHS has not implemented key system management
practices. Specifically,
* rigorous practices are not being fully adhered to in developing and
managing system requirements,
* key contract management and oversight controls have not been fully
implemented,
* planned risk management practices have yet to be implemented, and;
* performance management practices that are critical to measuring
progress against program goals are still being implemented.
Thus, much still needs to be accomplished to minimize the risks
associated with the program‘s capacity to deliver promised IT
infrastructure capabilities and benefits on time and within budget.
Given that hundreds of millions of dollars are to be invested and the
program is critical to supporting the ICE mission, it is essential that
DHS follow through on its commitments to build the capability to
effectively manage the program. Proceeding without it introduces
unnecessary risks to the program and potentially jeopardizes its
viability for future investment.
What GAO Recommends:
GAO is recommending that DHS minimize Atlas program risks by fully
adhering to requirements development and management practices,
implementing key contract management and oversight practices,
completing planned risk management activities, and implementing
critical performance management practices. DHS concurred with GAO‘s
recommendations and described actions planned and under way to
implement them.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-565].
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Randolph C. Hite at (202)
512-3439 or hiter@gao.gov.
[End of section]
Contents:
Letter:
Compliance with Legislative Conditions:
Observations on the Expenditure Plan and Management of Atlas:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendixes:
Appendix I: Briefing to the Staffs of the Subcommittees on Homeland
Security, Senate and House Committees on Appropriations:
Appendix II: Comments from the Department of Homeland Security:
Appendix III: GAO Contact and Staff Acknowledgments:
Abbreviations:
CCE: Common Computing Environment:
CIO: chief information officer:
CMMI: Capability Maturity Model Integrated:
COTR: contracting officer's technical representative:
DHS: Department of Homeland Security:
EA: enterprise architecture:
ICE: Immigration and Customs Enforcement:
IT: information technology:
IV&V: independent verification and validation:
OMB: Office of Management and Budget:
SACMM: Software Acquisition Capability Maturity Model:
SEI: Software Engineering Institute:
CCE: Common Computing Environment:
CIO: chief information officer:
CMMI: Capability Maturity Model Integrated:
COTR: contracting officer's technical representative:
DHS: Department of Homeland Security:
EA: enterprise architecture:
ICE: Immigration and Customs Enforcement:
IT: information technology:
IV&V: independent verification and validation:
OMB: Office of Management and Budget:
SACMM: Software Acquisition Capability Maturity Model:
April 27, 2007:
The Honorable Robert C. Byrd:
Chairman:
The Honorable Thad Cochran:
Ranking Member:
Subcommittee on Homeland Security:
Committee on Appropriations:
United States Senate:
The Honorable David E. Price:
Chairman:
The Honorable Harold Rogers:
Ranking Member:
Subcommittee on Homeland Security:
Committee on Appropriations:
House of Representatives:
The 2006 Department of Homeland Security (DHS) appropriations
act[Footnote 1] provided $40.15 million for Immigration and Customs
Enforcement's (ICE) program to modernize its information technology
(IT) infrastructure. The goals of the program--which consists of seven
related IT projects and is referred to by ICE as the "Atlas" program--
include improving information sharing and collaboration, strengthening
information security, and improving workforce productivity. The act
provides that none of the funds may be obligated until the department
develops a plan for how the funds are to be spent that satisfies
certain legislative conditions. The conditions included, among other
things, having GAO review the plan. On November 9, 2006, DHS submitted
its fiscal year 2006 expenditure plan to the Senate and House
Appropriations Subcommittees on Homeland Security. Pursuant to the act,
we reviewed the plan. Our objectives were to (1) determine whether the
plan satisfies legislative conditions specified in the act and (2)
provide other observations about the expenditure plan and the
department's management of the Atlas program.
On February 2, 2007, we provided our review results via a briefing to
the staffs of the Senate and House Appropriations Subcommittees on
Homeland Security. This report summarizes our findings, conclusions,
and recommendations from that briefing. The full briefing, including
our scope and methodology, is reprinted in appendix I.
Compliance with Legislative Conditions:
DHS's fiscal year 2006 Atlas expenditure plan, including related
program documentation and program officials' stated commitments,
satisfies or partially satisfies key aspects of (1) meeting the capital
planning and investment control review requirements of the Office of
Management and Budget (OMB);[Footnote 2] (2) complying with the DHS
enterprise architecture;[Footnote 3] (3) complying with acquisition
rules, requirements, guidelines, and systems acquisition management
practices of the federal government; (4) including certification by the
department's chief information officer (CIO) that an independent
verification and validation agent is currently under contract for the
project; and (5) having the plan reviewed and approved by the
department's Investment Review Board, the Secretary of Homeland
Security, and OMB.
Regarding the fourth legislative condition, the department certified
that an independent verification and validation agent is under contract
for Atlas, and the department is in the process of defining how it
plans to use its agent, but unresolved issues remain. For example, we
have previously reported[Footnote 4] that the independence of a
verification and validation agent is critical to providing management
with objective insight into program processes and associated work
products. We also reported[Footnote 5] that to be effective, the
verification and validation function should be performed by an entity
that is independent of the processes and products that are being
reviewed. When the agent engaged to provide the service is a
contractor, a practice to help ensure independence is to include in the
agent's contract a provision that prohibits the agent from soliciting,
proposing, or being awarded program work other than the independent
verification and validation services and products. However, the
contract that Atlas awarded to its agent did not include such a
provision.
Observations on the Expenditure Plan and Management of Atlas:
Our observations address program efforts regarding (1) requirements
development and management, (2) contract management and oversight, (3)
risk management, and (4) performance management. An overview of these
observations follows:
* Rigorous practices are not being fully adhered to in developing and
managing system requirements. On its three key projects,[Footnote 6]
the program has not fully implemented federal IT guidance, relevant
best practices, and agency policies and procedures (i.e., ICE System
Lifecycle Management Handbook) that provide for effective requirements
development and management activities. For example, on its Common
Computing Environment project, while the project established a policy
to guide project efforts in developing and managing requirements, it
did not among other things:
- document stakeholder comments as part of the requirements elicitation
process and review and obtain stakeholder approval on requirements
until approximately 1 year after deploying system capabilities;
- follow a disciplined change management process for several years
(between 2003 and 2006) after the project was initiated in March 2003
(during this time, requirements were introduced and implemented in an
ad hoc fashion); and:
- develop supporting analysis showing traceability among requirements,
systems design, and the contract.
* Key contract management and oversight practices are not fully
implemented. The program has not fully implemented practices that (1)
the Software Engineering Institute's CMMI®[Footnote 7] has identified
as essential to effectively managing and overseeing contracts that
support IT projects and (2) the program has largely established in its
existing policies and procedures. Specifically, the program has
implemented two key practices, but has not fully implemented four
others. For example, the program has assigned responsibility and
authority for performing contract management and oversight, and has
trained staff performing contract management and oversight. However, it
has not, among other things, fully documented each contract--including
clearly identifying the work to be performed and acceptance criteria--
and verified and accepted deliverables.
* Key risk management practices have yet to be implemented. We
previously reported[Footnote 8] that the program had developed a risk
management plan and process to guide program office staff in managing
risks throughout each project's life cycle and had recently begun to
implement the risk management process. Since then, the program has
taken additional steps to implement risk management, such as hiring a
risk management coordinator and conducting quarterly program management
meetings to review the program's progress in managing risks. However,
the program has not fully implemented key practices essential to
effectively managing project risks. For example, it does not have a
transparent, documented, and traceable process for inventorying and
resolving risks. Further, while the risk management plan defines a
process and associated practices for developing risk mitigation
strategies, the program has not fully implemented them. In addition,
although the risk management plan calls for developing a process for
elevating risks to management's attention, the program has not
developed and implemented such a process.
* Performance management practices are still being implemented and
available data show mixed results and may not be reliable. We
reported[Footnote 9] in July 2006 that Atlas had begun taking steps to
implement performance goals and measures. Since then, the program has
taken additional steps to define and implement them. For example, the
program identified 13 goals--for 5 of its 7 projects--that it has been
using to measure progress during fiscal year 2006. However, it has yet
to fully implement and institutionalize its performance goals and
measures. For example, the program has not yet developed performance
goals for 2 projects.
In addition, progress against performance goals, as reported by the
program, show mixed results. For example, of the 12 goals, the program
has met 3 of them. Further, underlying data used by the program to
measure performance may not be reliable in all cases. For example, on 1
goal, the program uses project cost and schedule data from the 2005
Atlas business case in measuring progress against the goal, even though
these data are out of date (e.g., they do not reflect project cost and
schedule slippages that have occurred since 2005).
Conclusions:
The fiscal year 2006 Atlas expenditure plan, in combination with
related program documentation and program officials' statements,
satisfies or partially satisfies the legislative conditions set forth
by Congress. However, this satisfaction is based on plans and
commitments that provide for meeting these conditions, rather than on
completed actions to satisfy the conditions. In addition, while steps
are being initiated that are intended to address significant program
management weaknesses, a number of improvements, including those
recommended in our past reports, have yet to be implemented.
Further, the program has not fully achieved many performance goals that
it set out to accomplish over the past year. The above factors continue
to put the program at risk and call for heightened and sustained
management attention to expeditiously address and resolve the issues.
Thus, there is much that still needs to be accomplished to minimize the
risks associated with the program's capacity to deliver promised IT
infrastructure capabilities and benefits on time and within budget.
This includes demonstrating better progress against established
performance goals in the coming year.
Given that hundreds of millions of dollars are to be invested and the
program is critical to supporting the ICE mission, it is essential that
DHS follow through on its commitments to build the capacity to
effectively manage the program. Proceeding without this capacity
introduces unnecessary risks to the program and potentially jeopardizes
its viability for future investment.
Recommendations for Executive Action:
To minimize risks to the Atlas program, we recommend that the Secretary
of Homeland Security direct the Assistant Secretary for Immigration and
Customs Enforcement to ensure that ICE follows through on its
commitments to implement effective management controls and capabilities
by implementing the following five recommendations:
* Employ practices essential to ensuring that the Atlas program's
independent verification and validation agent is and remains
independent, including incorporating requirements in future contracting
actions, such as the renegotiation or recompetition of the current
independent verification and validation contract, which will prohibit
the agent from soliciting, proposing, or being awarded program work
other than providing independent verification and validation services
and products.
* Fully adhere to requirements development and management practices,
including those specified in ICE's policies and procedures. This should
also include having the Atlas program manager develop a process
improvement plan for all of the projects that is consistent with the
ICE System Lifecycle Management Handbook and provide for making the
program manager and project managers responsible and accountable for
rigorously adhering to the requirements in the handbook.
* Fully implement key contract management and oversight practices,
including those specified in ICE's policies and procedures. This should
also include ensuring that the Atlas program manager, working with the
program's contracting officer's technical representative, follow
through on planned efforts to strengthen the program's compliance with
these practices by (1) revising the acquisition plan by May 2007; (2)
revising the ICE System Lifecycle Management Handbook by June 2007 to
incorporate key contract management activities such as contract
tracking and oversight; and (3) preparing statements of work (beginning
in February 2007) for future contracts to clearly define Atlas work
statements and acceptance criteria.
* Complete implementation of planned risk management activities. This
should include (1) updating the risk inventory to include risks for all
projects and risk areas and (2) fully implementing and
institutionalizing procedures for reporting to management on the
existence and status of risks and progress in implementing mitigation
strategies.
* Improve program performance management, including developing
performance goals for projects that do not have goals and reporting on
their progress in the fiscal year 2007 expenditure plan. Further, the
program should assess that the data being used to measure progress are
reliable, complete, and accurate.
Agency Comments and Our Evaluation:
In DHS's written comments on a draft of this report, signed by the
Director, Departmental GAO/Office of Inspector General Liaison Office,
the department concurred with all of our recommendations and described
actions planned and under way to implement them. The department also
orally provided technical comments updating Atlas obligation amounts
and the milestone for revising the acquisition plan. We incorporated
these comments in the report and briefing as appropriate. DHS's written
comments are reprinted in appendix II.
We are sending copies of this report to the Chairman and Ranking
Members of other Senate and House committees and subcommittees that
have authorization and oversight responsibilities for homeland
security. We are also sending copies to the Secretary of Homeland
Security, the Assistant Secretary for Immigration and Customs
Enforcement, and the Director of the Office of Management and Budget.
Copies of this report will also be available at no charge on our Web
site at [Hyperlink, http://www.gao.gov.].
Should you or your offices have any questions on matters discussed in
this report, please contact me at (202) 512-3439 or at hiter@gao.gov.
Contact points for our Offices of Congressional Relations and Public
Affairs may be found on the last page of this report. Key contributors
to this report are listed in appendix III.
Signed by:
Randolph C. Hite:
Director, Information Technology Architecture and Systems Issues:
[End of section]
Appendix I: Briefing to the Staffs of the Subcommittees on Homeland
Security, Senate and House Committees on Appropriations:
Information Technology: Immigration and Customs Enforcement Needs to
Fully Address Significant Infrastructure Modernization Program
Management Weaknesses:
Briefing to the staffs of the:
Subcommittees on Homeland Security:
Senate and House Committees on Appropriations:
February 2, 2007:
Introduction:
Objectives, Scope, and Methodology:
Results in Brief:
Background:
Results:
* Legislative Conditions:
* Observations:
Conclusions:
Recommendations for Executive Action:
Agency Comments:
Attachment I: Detailed Scope and Methodology:
Attachment II: GAO Analysis of Requirements Management:
Introduction:
The Department of Homeland Security's (DHS) Bureau of Immigration and
Customs Enforcement (ICE)[Footnote 10] is responsible for enforcing
immigration, border security, trade, and other laws by, among other
means, investigating and collecting intelligence on individuals and
groups that act to violate these laws. ICE is also responsible for
protecting federal facilities.
Atlas is the ICE program developed to modernize the bureau's
information technology (IT) infrastructure, which includes the hardware
(e.g., servers, routers, storage devices, communication lines) and
system software (e.g., database management and operating systems and
network management) that provide an environment for operating and
maintaining software applications.
According to ICE, the goals of the Atlas program include improving
information sharing, strengthening information security, and improving
workforce productivity.
The fiscal year 2006 Department of Homeland Security Appropriations
Act[Footnote 11] appropriated $40.15 million for Atlas[Footnote 12] and
stated that none of the funds may be obligated until the department
submits for approval to the Senate and House Committees on
Appropriations a plan for how the funds are to be spent that satisfies
the following legislative conditions:
meets the capital planning and investment control review requirements
established by the Office of Management and Budget (OMB), including OMB
Circular A-11, part 7[Footnote 13];
complies with DHS's information systems enterprise architecture;
complies with the acquisition rules, requirements, guidelines, and
systems acquisition management practices of the federal government;
includes certification by the DHS chief information officer (CIO) that
an independent verification and validation agent is currently under
contract for the project;
is reviewed and approved by DHS's Investment Review Board[Footnote 14],
the Secretary of Homeland Security, and OMB; and:
is reviewed by GAO.
DHS submitted its fiscal year 2006 expenditure plan for $39.75[Footnote
15] million on November 9, 2006, to the Senate and House Appropriations
Subcommittees on Homeland Security.
Objectives, Scope, and Methodology:
As agreed, our objectives were to:
determine whether the Atlas fiscal year 2006 expenditure plan satisfies
the legislative conditions and:
provide other observations about the expenditure plan and DHS's
management of the Atlas program.
To address our first objective, we analyzed the fiscal year 2006 Atlas
expenditure plan and supporting documents against legislative
conditions and relevant federal requirements and related best practices
to determine whether the conditions were being met.
To address our second objective, we evaluated supporting documentation
and interviewed relevant officials to determine progress in
establishing program management capabilities in areas such as
requirements development and management, contract management and
oversight, risk management, and performance management. In assessing
requirements management and contract management and oversight, we
focused on three projects-Common Computing Environment, Integration,
and ICE Mission Information-that collectively account for approximately
75 percent of the funds provided to the program and that the Atlas
program manager identified as being the most critical to the program's
success and ICE's mission. We conducted our work at DHS and ICE
headquarters in Washington, D.C., from July 2006 through January 2007
in accordance with generally accepted government auditing standards.
Details of our scope and methodology are provided in attachment 1.
Results in Brief:
Objective 1: Satisfaction of Legislative Conditions:
1;
Legislative conditions: Meets the capital planning and investment
control review requirements established by OMB,including OMB Circular A-
11, part 7;
Status: partially satisfies[Footnote 16].
2;
Legislative conditions: Complies with the DHS information systems
enterprise architecture;
Status: partially satisfies.
3;
Legislative conditions: Complies with the acquisition rules,
requirements, guidelines, and systems acquisition management practices
of the federal government;
Status: partially satisfies.
4;
Legislative conditions: Includes a certification by the chief
information officer of the Department of Homeland Security that an
independent verification and validation agent is currently under
contract for the project;
Status: partially satisfies.
5;
Legislative conditions: Is reviewed and approved by the DHS Investment
Review Board, Secretary of Homeland security, and OMB;
Status: satisfies[17].
6;
Legislative conditions: Is reviewed by GAO;
Status: satisfies.
Source: GAO.
[End of table]
Objective 2: Other Observations:
DHS has not implemented key system management practices essential to
establishing an effective program management control environment.
Specifically, in managing the Atlas program, the department has not:
fully adhered to rigorous practices called for in developing and
managing system requirements,
implemented key contract management and oversight controls,
completed implementation of planned risk management practices, or:
implemented performance management practices critical to measuring
progress against program goals.
These shortfalls are attributable in part to a number of factors,
including the Atlas program's expedited schedule to deploy modernized
IT infrastructure capabilities to users in the wake of the events of
September 11, 2001, and in the haste of doing so, not following these
practices or related ICE policies and procedures. Until these practices
are fully implemented, the program is at an increased risk of not being
able to deliver modern IT infrastructure capabilities on time and
within budget. Program officials acknowledge the shortfalls and have
efforts planned or under way to address some, but not all, of the
missing practices.
Accordingly, we are making recommendations to the Secretary of Homeland
Security to strengthen the Atlas program by implementing and
institutionalizing these key system management practices, thereby
helping to establish an effective program management control
environment. In commenting on a draft of this briefing, ICE officials,
including the deputy chief information officer who is also the Atlas
program manager, agreed with our results.
Background ICE:
ICE was formed as a component agency of DHS in 2003 when the law
enforcement functions of the Department of Justice's former Immigration
and Naturalization Service and Department of the Treasury's former
Customs Service and other agencies were merged into DHS.
The ICE mission is to ensure the security of the American people and
homeland by, among other means, investigating violators of and
enforcing the nation's immigration and customs laws; policing and
securing federal buildings and other facilities; and collecting,
analyzing, and disseminating intelligence to assist in these endeavors.
Headed by the Assistant Secretary of Immigration and Customs
Enforcement, ICE has approximately 15,000 employees in more than 400
offices in the United States and in other countries.
The Atlas program was started by the Immigration and Naturalization
Service in 2002. Responsibility for the program was transferred to ICE
in 2003. The figure on the following slide is a simplified version of
the current ICE and Atlas organizational placements within DHS.
Background: ICE and Atlas Organizational Placement:
[See PDF for image]
Source: GAO analysis of DHS data.
[End of figure]
Background: Program Impetus and Goals:
Impetus for Program:
According to ICE officials, Atlas was initiated to address information
sharing and security limitations within the former Immigration and
Naturalization Service caused by, for example,
obsolete hardware/software;
incompatible, noninteroperable information systems; and:
system security weaknesses.
These officials stated that these challenges were exacerbated by the
formation of ICE because the organizations merged into ICE had
different hardware/software environments (e.g., multiple e-mail
systems) and different missions and customer needs.
Program Goals:
The stated goals of Atlas are, among other things, to:
promote information sharing and collaboration,
strengthen information security, and:
enhance workforce productivity.
Background Atlas Projects:
Atlas Consists of Seven Projects:
Project: Common Computing Environment;
Description: * Deploy compatible desktops, servers, and standard office
automation applications;
* Deploy a common e-mail platform;
* Integrate ICE with a shared departmental active directory structure.
Project: Integration;
Description: * Migrate from the ICE network to the DHS OneNetwork
infrastructure;
* Provide streaming video;
* Refresh local area network cabling and electronics and replace legacy
firewalls;
* Refresh the wide area network and local area network communications
hardware at the overseas ICE attaché offices;
* Provide network support for all applications and provide converged
communications among all system users.
Project: ICE Mission Information;
Description: * Provide enterprise data warehousing capabilities and
integrate related ICE business processes;
* Organize information and support user access so ICE users can find
relevant, timely, and reliable information.
* Improve information searching and indexing capabilities and integrate
legacy applications with service-oriented techniques.
Project: Information Assurance;
Description: * Institute strong information, system/ application
platform, network, and computer security measures;
Project: Architecture Engineering;
Description: * Provide state-of-the-art engineering facilities and
tools to ensure new technologies align to the DHS enterprise
architecture (EA).
Project: Data Center Migration;
Description: * Migrate ICE hardware and applications from the
Department of Justice data centers to the common DHS data center;
* Ensure compliance with the DHS EA.
Project: Transformation Planning;
Description: * Oversee and coordinate the various Atlas projects;
* Ensure compliance with the DHS EA;
* Ensure adherence to project cost, schedule, and performance goals.
Source: GAO analysis of DHS data.
[End of table]
Figure:
[See PDF for image]
Source: GAO analysis based on DHS appropriation laws.
[A] Counterterrorism funding came from Pub. L. No. 107-117, Jan. 10,
2002; H.R. Rep. 107-350, Dec. 19, 2001.
[B] Pub. L. No. 108-90, Oct. 1, 2003.
[C] Pub. L. No. 108-334, Oct. 18, 2004.
[D] DHS's fiscal year 2006 appropriations (Pub. L. No. 109-90, Oct. 18,
2005) provided $40.15 million for Atlas. In December 2005, this amount
was reduced to $39.7 million via a 1 percent governmentwide rescission
in Pub. L. No. 109-148, Division B, Dec. 30, 2005. [E] pub. L. No. 109-
295, Oct. 4, 2006. The law mandates that DHS also submit an expenditure
plan prior to obligation of these funds.
[End of figure]
ICE reports that it has obligated $84.7 million of the $113.0 million
available for Atlas in fiscal years 2002, 2003, 2004, and 2005.
Table: Atlas funds obligated (by project) for fiscal years 2002-2005
(as of December 31, 2006):
Project (in order presented in the plan): Common Computing Environment;
Obligations (in millions): FY 2002: $0;
Obligations (in millions): FY 2003: $10.5;
Obligations (in millions): FY 2004: $3.2;
Obligations (in millions): FY 2005: $4.3;
Total: $18.0.
Project (in order presented in the plan): Integration;
Obligations (in millions): FY 2002: $6.1;
Obligations (in millions): FY 2003: $20.4;
Obligations (in millions): FY 2004: $2.5;
Obligations (in millions): FY 2005: $0.5;
Total: $29.5.
Project (in order presented in the plan): ICE Mission Information;
Obligations (in millions): FY 2002: $1.3;
Obligations (in millions): FY 2003: $4.6;
Obligations (in millions): FY 2004: $0.8;
Obligations (in millions): FY 2005: $4.0;
Total: $10.7.
Project (in order presented in the plan): Information Assurance;
Obligations (in millions): FY 2002: $10.2;
Obligations (in millions): FY 2003: $9.6;
Obligations (in millions): FY 2004: $0;
Obligations (in millions): FY 2005: $0.6;
Total: $20.4.
Project (in order presented in the plan): Architecture Engineering;
Obligations (in millions): FY 2002: $0;
Obligations (in millions): FY 2003: $0;
Obligations (in millions): FY 2004: $0;
Obligations (in millions): FY 2005: $0.7;
Total: $0.7.
Project (in order presented in the plan): Data Center Migration;
Obligations (in millions): FY 2002: $0;
Obligations (in millions): FY 2003: $0;
Obligations (in millions): FY 2004: $0;
Obligations (in millions): FY 2005: $0;
Total: $0.
Project (in order presented in the plan): Transformation planning;
Obligations (in millions): FY 2002: $0;
Obligations (in millions): FY 2003: $0.9;
Obligations (in millions): FY 2004: $3.3;
Obligations (in millions): FY 2005: $1.2;
Total: $5.4.
Total;
Obligations (in millions): FY 2002: $17.6;
Obligations (in millions): FY 2003: $46.0;
Obligations (in millions): FY 2004: $9.8;
Obligations (in millions): FY 2005: $11.3;
Total: $84.7.
Source: GAO analysis of DHS data.
[End of table]
Background: Planned Use of Fiscal Year 2006 Funding:
According to the fiscal year 2006 expenditure plan, the $39.75 million
is to be spent on the projects as follows.
Table: Distribution of funding for Atlas projects (in order presented
in the plan):
Project: Common Computing Environment;
Funding (in millions): $23.88;
Planned use:
* Complete acquisition of hardware refresh;
* Deploy active directory and e-mail exchange system.
Project: Integration;
Funding (in millions): $2.30;
Planned use:
* Replace outdated communications electronics at foreign attaches to
ensure compatibility with current ICE Net WAN and LAN infrastructure
standards;
* Complete acquisition of streaming video for field sites.
Project: ICE Mission Information;
Funding (in millions): $5.84;
Planned use:
* Complete acquisition of technology to efficiency integrate all
related ICE business processes;
* Complete 60 percent of acquisition of technology to support data
warehousing capabilities.
Project: Information Assurance;
Funding (in millions): $0.92;
Planned use:
* Acquire IT security specialists to design and develop single sign-on
and audit log capabilities.
Project: Architecture Engineering;
Funding (in millions): $0;
Planned use:
* No use of fiscal year 2006 funds identified.
Project: Data Center Migration;
Funding (in millions): $0.89;
Planned use:
* Plan for the migration of ICE hardware and applications from two
Department of Justice data centers to a common DHS data center.
Project: Transformation Planning;
Funding (in millions): $5.92;
Planned use:
* Provide $2.3 million to continue performing and planning program
management activities, including maintaining the Atlas business case;
* Provide $3.6 million in management reserve for unanticipated program
costs or to fund priority emerging IT modernization requirements.
Total: $39.75.
Source: GAO analysis of DHS data.
[End of table]
Objective 1 Results Legislative Conditions:
The Atlas expenditure plan satisfies or partially satisfies each of the
legislative conditions.
Condition 1 partially satisfied. The expenditure plan, including
related program documentation and statements from the program manager,
partially satisfies the capital planning and investment control review
requirements established by OMB, including Circular A-11, part 7, which
establishes policy for planning, budgeting, acquisition, and management
of federal capital assets.
Examples of our analysis are included in the following table. As the
table shows, not all OMB requirements have been satisfied, but oral
commitments have been made for doing so. Given that ICE has reportedly
already invested $75.4 million on Atlas projects and plans to invest
another $39.7 million this year, it is important for ICE to follow
through on these commitments.
Objective 1 Results Legislative Conditions:
GAO Analysis of Atlas Compliance with Legislative Conditions.
Examples of OMB Circular A- 11 requirements: Indicate whether the
investment was approved by an investment review committee;
Results of our analysis: The plan was approved on May 11, 2006, by
DHS‘s Deputy Secretary, who chairs the DHS Investment Review Board.
Examples of OMB Circular A- 11 requirements: Provide justification and
describe acquisition strategy;
Results of our analysis: A business case, including a cost-benefit
analysis, was issued in December 2005 to provide economic justification
for the program. It estimated the program‘s life cycle costs to be
approximately $1 billion through the year 2024. The program plans to
issue an updated cost-benefit analysis in September 2007 to reflect
emerging requirements and other program changes (e.g., cost and
schedule slippages). In addition, the expenditure plan and supporting
documents (e.g., fiscal year 2006 Atlas budget submission to OMB known
as an exhibit 300) provide aspects of a high-level acquisition
strategy, such as identifying the ICE contracts that are being used and
are to be used to acquire hardware and software products and program
support services. An acquisition plan was approved in February 2006
that includes a statement of need and the capabilities to be delivered
through these contracts. To help ensure compliance with contract
criteria and Atlas program objectives, a contracting officer‘s
technical representative (COTR) was hired in November 2006. The
program‘s efforts to ensure compliance with contract management and
oversight practices are more fully discussed in the Observations
section of this briefing.
Examples of OMB Circular A- 11 requirements: Summarize life cycle costs
and cost-benefit analysis, including the return on investment;
Results of our analysis: The December 2005 cost-benefit analysis
provides costs and benefits for the life cycle of Atlas, which is
through the year 2024. The analysis also included an estimated return
on investment for three alternative approaches. As we previously
reported[Footnote 18], the analysis was in large part consistent with
OMB and best practices guidance, but it did omit key practices such as
key mission requirements. According to the program manager, the
analysis is considered to be a ’living document“ and the program plans
to issue an update in September 2007 that, among other things,
addresses these requirements.
Examples of OMB Circular A- 11 requirements: Provide performance goals
and measures;
Results of our analysis: The expenditure plan and supporting
documentation (e.g., prior expenditure plans) identify 13 proposed
goals and measures, but two projects do not have measures. Proposed
measures have been drafted for these projects, but target dates for
their completion have not been identified by the program. In addition,
as part of the December 2005 Atlas business case, the program mapped
Atlas‘s goals to ICE‘s mission and goals. Our analysis of Atlas‘s
performance management is discussed in more detail in the Observations
section of this briefing.
Examples of OMB Circular A- 11 requirements: Address security and
privacy;
Results of our analysis: The plan and supporting documentation state
the importance of security and privacy and provide high-level
information on intended security measures, including one proposed
project”information assurance”that is intended to implement an ICE
security program. The plan allocates $0.9 million to this project in
addition to the $1.9 million provided in the fiscal year 2005 plan. In
addition, the program issued an Atlas system security plan, as well as
a security test and evaluation plan, in April 2006. The program also
recently issued an updated system security plan and security test and
evaluation plan in December 2006. ICE also developed a draft Atlas
privacy impact assessment in August 2005, submitted a revised
assessment in August 2006 to the DHS privacy office, and plans to
complete and submit the final Atlas privacy impact assessment by April
2007.
Examples of OMB Circular A- 11 requirements: Provide for managing risk;
Results of our analysis: The program issued a risk management plan in
January 2006. This plan provides guidance for identifying, analyzing,
and resolving program risks before they occur. In addition, a risk
management coordinator was hired in August 2006 to help identify and
monitor risks. According to the program manager, this official recently
vacated the position, and the program is currently in the process of
hiring a new coordinator. (At our Feb. 1, 2007, exit meeting with DHS
and ICE officials, the program manager told us a coordinator had been
hired on Jan. 22, 2007.) Further, while the program procured an
automated tool in May 2006 to help complete the inventory and manage
risks, it does not have, among other things, a complete inventory of
all program risks. Our complete findings related to risk management are
addressed in the Observations section of this briefing.
Source: GAO analysis of DHS data.
[End of table]
Condition 2 partially satisfied. The plan, including related program
documentation and DHS officials' statements, partially satisfies the
condition that the department ensures Atlas is compliant with DHS's
enterprise architecture (EA).
An EA provides a clear and comprehensive picture of an organization's
operations and its supporting systems and infrastructure. It is an
essential tool for effectively and efficiently engineering business
processes and for implementing and evolving supporting systems in a
manner that maximizes interoperability, minimizes overlap and
duplication, and optimizes performance. We have worked with Congress,
OMB, and the federal Chief Information Officers Council to highlight
the importance of architectures for both organizational transformation
and IT management. An important element of EA management is ensuring
that IT investments are compliant with the EA, including basing such
assessments on documented analysis.
The DHS Enterprise Architecture Board, supported by the Enterprise
Architecture Center of Excellence,[Footnote 19] is responsible for
ensuring that projects demonstrate adequate technical and strategic
compliance with the department's EA. To this end, the board conducts
compliance reviews at key decision points in an investment's life
cycle. Specifically, DHS guidance[Footnote 20] directs the board prior
to an investment's acquisition milestone (referred to by DHS as key
decision point 2) to assess the investment against the transition
strategy, data architecture, application component, and technology
architecture.
We previously reported[Footnote 21] in May 2005 that the Atlas program
manager requested that the center assess the program's compliance with
the EA. The center staff compared Atlas to version 2.0 of the DHS EA
and reported the results of its assessment to the board, stating that
Atlas was in compliance with the DHS EA. In July 2005, the board
approved this compliance determination.
Subsequently, we reported[Footnote 22] that ICE'S compliance
determination was not based on a documented analysis mapping Atlas's
infrastructure to the DHS EA. Specifically, the department did not have
a documented methodology for evaluating programs for compliance with
the DHS EA, other than relying on the expertise of the Center of
Excellence members.
Since the July 2005 compliance determination, DHS issued version 2006
of its EA.[footnote 23] In addition, Atlas has undertaken an effort to
define an architecture framework and technical environment for a data
warehouse aimed at providing reporting and analytical capabilities to
investigative and enforcement personnel. Despite these changes, no new
compliance reviews have been conducted. According to the Atlas program
manager, a compliance review was not required during this time as the
program had not yet reached a phase requiring it. The program manager
stated the department has scheduled an EA compliance review in March
2007.
In addition, program officials told us that they are taking other steps
aimed at ensuring compliance. These steps include:
participating in the DHS Enterprise Architecture Center of Excellence
and the Data Management Working Group; these groups are currently
tasked with developing the DHS EA and the DHS Data Reference Model,
participating in DHS EA initiatives, such as serving on the Service
Oriented Architecture Tactical Working Group,[Footnote 24] which is to
provide the framework for departmentwide service alignment and
interoperability efforts, and:
establishing an Atlas Interoperability Lab-under the Architecture
Engineering project-to among other things, help ensure the program's
projects are compliant with the EA.
These steps notwithstanding, until DHS demonstrates, through verifiable
documentation and a methodologically based analysis, that ICE is
aligned with the DHS enterprise architecture, the program will remain
at risk of being defined and implemented in a way that does not support
optimal departmentwide operations, performance, and achievement of
strategic goals and outcomes.
Condition 3 partially satisfied. The plan, including related program
documentation and statements from the Atlas program manager, partially
satisfies the condition to comply with the acquisition rules,
requirements, guidelines, and systems acquisition management practices
of the federal government. These practices provide a management
framework based on the use of rigorous and disciplined processes for
planning, managing, and controlling the acquisition of IT resources,
including:
acquisition planning, which ensures, among other things, that
reasonable plans, milestones, and schedules are developed and that all
aspects of the acquisition effort are included in these plans;
procurement, which involves making sure that (1) a request for
proposals delineating a project's requirements is prepared and (2)
consistent with relevant acquisition laws and regulations, a contractor
is selected that can cost-effectively satisfy these requirements;
requirements development and management, which includes establishing
and maintaining a common and unambiguous definition of requirements
among the acquisition team, the system users, and the development
contractor;
project management, which provides for management of the activities
within the project office and supporting contractors to ensure a
timely, efficient, and cost-effective acquisition;
contract tracking and oversight, which ensures that the development
contractor performs according to the terms of the contract; needed
contract changes are identified, negotiated, and incorporated into the
contract; and contractor performance issues are identified early, when
they are easier to address; and:
evaluation, which determines whether the acquired products and services
satisfy contract requirements before acceptance.
These acquisition management processes are also embodied in published
best practices models, such as the Software Engineering Institute's
Software Acquisition Capability Maturity Model[Footnote 25]
Examples of our analysis of ICE performance of these processes and
practices are shown in the following table. They show that not all
aspects of the processes and practices have been implemented, but that
oral commitments have been made for doing so.
Table: GAO Analysis of Atlas with Legislative Conditions.
Example of practices: Acquisition Planning; Ensures that reasonable
plans, milestones, and schedules are developed and that all aspects of
the acquisition effort are included in these plans;
Results of our analysis: The expenditure plan and supporting documents
(e.g., fiscal year 2006 Atlas budget submission to OMB known as an
exhibit 300) provide aspects of a high-level acquisition strategy, such
as identifying the ICE contracts that are being used and are to be used
to acquire products and program support services. An acquisition plan
was issued in February 2006, and it includes a statement of the
capabilities to be delivered through these contracts;
To manage and oversee the contracts, including helping ensure
compliance with contract performance criteria and Atlas program
objectives, the program hired a COTR in November 2006. In addition, the
business case issued in December 2005 provides details on alternatives,
cost, and schedule. The program plans to issue an updated business case
and cost-benefit analysis by September 2007 to reflect emerging
requirements and other program changes (e.g., project cost and schedule
slippages). In addition, Atlas is also updating and revising its
January 2006 acquisition program baseline, which is to include cost and
schedule baselines for its projects, with the goal of finalizing it by
March 2007;
According to the program manager, Atlas is currently following the
February 2005 ICE System Lifecycle Management Handbook.[Footnote 26]
The handbook addresses a number of key process areas such as project
management and requirements development and management; however, it
does not address certain key acquisition management activities such as
solicitation and contract tracking and oversight. According to the
Atlas program manager, he acknowledges this problem and plans to update
the handbook by June 2007 to include the missing acquisition management
activities. This area is more fully discussed in the Observations
section of this briefing.
Example of practices: Project Management; Provides for the management
of activities within the project office and supporting contractors to
ensure a timely, efficient, and cost-effective acquisition;
Results of our analysis: ICE has established a program office with
responsibility for managing the acquisition, deployment, operation, and
sustainment of Atlas. The Atlas program management office is to be
allocated funding of $2.3 million via the Transformation Planning
project in the expenditure plan. The current staffing of the program
office consists of a program manager, who is also the deputy chief
information officer; a deputy program manager; a manager for each of
the seven Atlas projects; and other personnel (e.g., a business analyst
or a communications specialist). According to the program manager, the
program office is now fully operational. In addition, a project plan
for each of the projects was approved by the program manager in the
latter part of 2006.
Source: GAO analysis of DHS data.
[End of table]
Condition 4 partially satisfied. DHS partially satisfies the
legislative condition requiring that the plan include a certification
by the department's CIO that an IV&V agent is currently under contract
for the project.
In an October 25, 2006, letter to ICE's CIO, DHS's Deputy CIO certified
that an IV&V agent is under contract for Atlas. However, there are
unresolved issues with the Atlas program's efforts in this area. First,
in the Deputy CIO's letter, he requested additional information and
made recommendations regarding the IV&V agent, asking for a response by
January 31, 2007. Specifically, this official:
requested a copy of the IV&V contract and the agent's work plan in
order to validate, among other things, whether planned IV&V activities
are in accordance with industry quality standards and whether the agent
has sufficient staff and:
recommended that the Atlas program (1) review whether the information
assurance and security experience of key IV&V personnel was sufficient
and (2) appoint a full-time COTR before the end of the first quarter of
2007.
The program manager stated that he is currently drafting the response
on behalf of ICE's CIO with the goal of transmitting it to DHS by the
first or second week of February 2007.
Second, as we have previously reported[Footnote 27], the independence
of an IV&V agent is critical to providing management with objective
insight into program processes and associated work products. We also
reported that to be effective, the verification and validation function
is to be performed by an entity that is independent of the processes
and products that are being reviewed. When the agent engaged to provide
the service is a contractor, a practice to help ensure independence is
to include in the agent's contract a provision that prohibits the agent
from soliciting, proposing, or being awarded program work other than
the IV&V services and products.
The contract that Atlas awarded to its IV&V agent (dated Sept. 13,
2006) does not include such a provision. Instead, the contract states
that the contractor will notify the agency of any potential conflicts
of interest in performing work under the contract. The Atlas program
manager told us he did not believe incorporating the contractual
provision called for by the key practice was necessary because the
agent's only line of work is performing IV&V activities, and he agreed
to provide documentation to support this. He also said that in the
event the agent solicited, proposed, or is awarded other work on the
Atlas program, he would replace the agent with one that was
independent. However, the program manager did not provide documentation
to IV&V being the agent's sole line of business. In addition, having to
replace an agent could delay program activities and is not consistent
with the key practice to avoid such problems. Consequently, until the
practice to ensure independence is included in the agent's contract,
there is an increased risk that the agent's independence could become
impaired, resulting in the program being unable to rely on the agent's
work.
Condition 5 satisfied. DHS and OMB satisfied the legislative condition
requiring that the plan be reviewed and approved by the DHS Investment
Review Board, Secretary of Homeland Security, and OMB.
The DHS Deputy Secretary, who chairs the DHS Investment Review Board,
approved the plan on May 11, 2006.
OMB approved the plan on September 14, 2006.
Condition 6 satisfied. GAO satisfied the condition that it review the
plan.
Our review was completed in January 2007.
Objective 2 Results:
Observation 1: Requirements Management:
Observation 1: Atlas did not fully adhere to rigorous requirements
development and management practices:
Federal IT management guidance and relevant best practices[Footnote 28]
recognize the importance of effectively developing and managing system
requirements. According to this criteria, well-defined requirements are
important because they establish agreement among the various
stakeholders on what the system is to do, how well it is to do it, and
how it is to interact with other systems. Effective requirements
development and management involves, among other things:
establishing a requirements management policy;
eliciting desired or required system capabilities from users and
translating them into system requirements and obtaining approval from
all stakeholders before deploying system capabilities;
establishing a system concept of operations, which describes the
business process to be supported and how users will interact with the
proposed system, and defines system requirements;
ensuring that changes to requirements are controlled and approved via a
disciplined change control process as the system is developed and
implemented; and:
maintaining bidirectional traceability, meaning that a given
requirement can be traced backward to systems documentation and forward
to the appropriate contract vehicle and system components that will
satisfy the requirement and the test that will verify that it is
satisfied.
As we reported in July 2006,[Footnote 29] ICE incorporated these
practices in its ICE System Lifecycle Management Handbook,[Footnote 30]
which was issued in February 2005. However, the program did not fully
follow these practices on its three key projects: Common Computing
Environment (CCE), Integration, and ICE Mission Information. For
example, on CCE, while the project established the ICE System Lifecycle
Management Handbook as the policy to guide project efforts in
developing and managing requirements, it did not:
document stakeholder comments as part of the requirements elicitation
process and review and obtain stakeholder approval on requirements
until approximately 1 year after deploying system capabilities;
develop a concept of operations until several years after the project
had been initiated, specifically, the project was initiated in March
2003 and began deploying system capabilities in December 2005, but the
concept of operations was not developed until June 2006;
follow a disciplined change management process for several years
(between 2003 and 2006) after the project was initiated in March 2003
(during this time, requirements were introduced and implemented in an
ad hoc fashion); and:
develop supporting analysis showing traceability between requirements,
system design, and the contract.
Our assessment of the extent to which the program complied with these
practices on the three projects is shown in attachment II.
According to ICE officials, including the program and project managers,
the reason that the Atlas projects did not fully adhere to these
practices is due to the agency's expedited schedule to deploy
modernized IT infrastructure capabilities to users in the wake of the
events of September 11, 2001. Specifically, the officials stated that
in their haste to deliver capabilities, the project manager skipped
certain system management life cycle practices such as these.
The program manager acknowledged that requirements management needs to
be improved and stated that key steps are being taken to ensure the
projects closely follow the practices specified in the ICE System
Lifecycle Management Handbook. He cited recent efforts by the CCE
project manager to follow such practices on work recently initiated to
refresh hardware, such as desktops and laptop computers across the
agency. For example, in January 2007, the project conducted a
workstation requirements analysis survey that was sent out to ICE users
to identify requirements regarding the current workstation being used
and issues that needed to be addressed in order to perform functions
associated with their jobs. In accordance with the handbook, project
staff drafted requirements, which they plan to finalize and approve by
February 2007. While these are steps in the right direction on this
particular project, the program manager was not able to provide us with
a process improvement plan on how the program going forward would
ensure compliance with requirements development and management
practices across all of the projects.
Consequently, until Atlas fully implements effective requirements
development and management practices programwide, it faces the
increased likelihood that its projects will not meet user needs,
operate as intended, or be delivered on time and within budget. In
addition, there is evidence that this has already occurred. For
example, on CCE, the delivery of its initial operating capability (the
date when it was to begin providing common e-mail capabilities to ICE
users) was delayed from September 2005 to December 2005. According to
the project manager, this delay is attributable to, among other things,
the project not following these practices. Specifically, the project
manager told us that the project experienced numerous changes to
requirements when it was transferred from the Department of Justice to
DHS and that these changes were not documented well. As a result, there
was extensive system rework to address missing requirements, which
caused schedule delays.
Objective 2 Results: Observation 2: Contract Management and Oversight:
Observation 2: Key contract management and oversight practices are not
fully implemented:
The Software Engineering Institute's CMMI[Footnote 31] identifies best
practices that are essential to effectively managing and overseeing
contracts that support IT projects. These best practices include:
establishing and maintaining a plan for managing and overseeing
contracts,
assigning responsibility and authority for performing contract
management and oversight,
training staff performing contract management and oversight activities,
documenting each contract, including the statement of work and
acceptance criteria,
verifying and accepting contractor-provided products and services
(i.e., deliverables), and;
conducting reviews with contractors to ensure that cost and schedule
commitments are being met and risks are being managed.
As we reported[footnote 32] in June 2006, the Atlas program has largely
established policies and procedures for these practices. While the
program has implemented two of the practices, it has not fully
implemented the other four. Specifically, our analysis of the program's
performance of these practices, including efforts planned and underway,
are shown in the following table.
Implementation of Contract Management and Oversight Practices for Atlas
Program.
1;
CMMI practice: Establish and maintain a plan for managing and
overseeing contracts;
Implemented: no;
GAO's assessment of compliance: The program office issued a plan in
February 2006 (referred to as an acquisition plan by program
officials), but it does not describe contract management and oversight
processes. Instead, the plan simply states that Atlas is to have a COTR
work with program and contracting officials to administer the
contracts. In November 2006, the DHS Acquisition Policy and Oversight
Office, which is responsible for reviewing and approving the
acquisition plan, stated that the plan lacked details on contract
management and oversight and recommended that Atlas develop, among
other things, a contract management and oversight plan and incorporate
it into the acquisition plan. In addition, we reported in July 2006
that the ICE System Lifecycle Management Handbook, which is used to
help manage the Atlas program, does not address certain key acquisition
management activities, such as contract tracking and
oversight.[Footnote 33] Atlas‘s program manager stated that the
acquisition plan and the handbook are being revised to include contract
management and oversight activities. The acquisition plan is to be
revised by March 2007 and the handbook by June 2007.
2;
CMMI practice: Assign responsibility and authority for performing
contract management and oversight;
Implemented: yes;
GAO's assessment of compliance: The program office hired a COTR in
November 2006, and ICE assigned the individual the responsibility and
authority for Atlas contract management and oversight. The program
office also provided one contractor staff member to assist the COTR.
The contractor‘s responsibilities include helping develop statements of
work, delivery schedules, and inspection and acceptance procedures.
Further, each of the project managers has been tasked by the program
manager to assist the COTR in identifying requirements and providing
assistance in determining whether contractor deliverables meet
requirements.
3;
CMMI practice: Train staff performing contract management and oversight
activities;
Implemented: yes;
GAO's assessment of compliance: The program‘s COTR has extensive
contract oversight training and experience. For example, during 2005
and 2006, the official took 120 hours of training on topics such as
systems acquisition management, COTR responsibilities, the use of
government purchase cards, and contracting officer‘s representative
mentoring. Further, this official‘s prior experience includes 5 years
of managing and overseeing contracts at the Department of Defense, at a
civilian agency, and in private industry. Also, all but one of the
Atlas project managers received contract management and oversight
training in October 2006. The remaining project manager is to receive
the training by June 2007.
4;
CMMI practice: Document each contract, including clearly identifying
the work to be performed and acceptance criteria;
Implemented: no;
GAO's assessment of compliance: On its three key projects”namely CCE,
Integration, and the ICE Mission Information projects”the program did
not satisfy this practice. Specifically, the contracts for these three
projects did not clearly identify the work to be performed and the
criteria for accepting contractor deliverables. Instead, the contracts
identified the work to be performed and the acceptance criteria in
general terms. For example, the contract for deploying CCE (a standard
e-mail system agencywide) states that the contractor is responsible for
providing deployment services, but does not identify that deploying a
standard e-mail system is one of those services. Similarly, regarding
acceptance criteria, the CCE contract states that the contractor will
provide monthly status reports, but does not identify how the program
will determine whether the deliverable is acceptable, such as
describing tests to be performed.
Program officials, including the program manager and the COTR,
acknowledged that the statement of work and the acceptance criteria are
not clearly defined, and that this makes contract administration
difficult. They added that this difficulty extended to the other Atlas
contracts because the program managed these contracts in the same
manner as the other three. To address the problem, the program manager
stated that he has tasked the COTR to work with project managers in
developing clear statements of work and acceptance criteria for new
contracts that are to be issued by late February 2007 to replace
existing Atlas contracts that will begin to expire at that time.
5;
CMMI practice: Verify and accept the deliverables;
Implemented: partially;
GAO's assessment of compliance: The COTR‘s responsibilities include
verifying and accepting contract deliverables. On the contracts of the
three projects reviewed, the COTR, in assessing whether to accept
deliverables, did not compare them with specific criteria in the
contract because none existed (see the previous slide). Instead, the
COTR, in consultation with the staff from the involved projects and ICE
offices of budget and acquisition management, reviewed each
contractor‘s invoices to assess whether it appeared what the contractor
was charging was reasonable for the deliverables provided. If consensus
was reached among these parties, then the COTR would approve the
invoices for payment.
6;
CMMI practice: Conduct reviews with contractors to ensure cost and
schedule commitments are being met and risks are managed;
Implemented: partially;
GAO's assessment of compliance: The program manager stated that he and
the ICE CIO conduct reviews with program contractors on a quarterly
basis to determine whether cost and schedule commitments are being met
and to review project risks. However, he could not provide
documentation that the meetings occurred due to the program not
documenting the meetings. In addition, the contractors provide reports
to program officials on their progress in meeting commitments and to
identify risks related to each project. However, according to the
program manager, assessing meaningful contractor progress from these
reports is difficult without clearly defined work-to-be-performed
statements and acceptance criteria.
Source: GAO analysis of DHS data.
[End of table]
The Atlas program manager stated that the program did not fully adhere
to these practices due to it not having until recently a COTR to manage
and ensure compliance with contract management and oversight
requirements. He also stated that to strengthen the program's
compliance with such practices, he has tasked the COTR with (1)
revising the acquisition plan (by March 2007) to provide detailed
guidance on how Atlas will manage and oversee contracts, (2) revising
the ICE System Lifecycle Management Handbook by June 2007 to
incorporate key contract management activities such as contract
tracking and oversight, and (3) preparing statements of work (beginning
in February 2007) for future contracts to clearly define Atlas work
statements and acceptance criteria. While these are steps in the right
direction, they are works in progress and, as such, have not been
completed.
Until these steps and the other practices are fully implemented and
institutionalized, Atlas faces the risk that it will not deliver
required capabilities and promised benefits on time and within budget.
It also makes the program vulnerable to contract mismanagement.
Objective 2 Results: Observation 3: Risk Management:
Observation 3: Key risk management practices have yet to be
implemented:
Federal guidance and related best practices[Footnote 34] identify key
practices essential for effectively managing risks that may impact an
IT project. They include, among other things:
identifying and prioritizing risks as to their probability of
occurrence and impact, as well as documenting them in an inventory;
developing and implementing the appropriate risk mitigation strategies;
and:
reporting to management on the existence and status of risks and
progress in implementing mitigation strategies.
In July 2006, we reported[Footnote 35] that the program had developed a
risk management plan and process to guide program office staff in
managing risks throughout each project's life cycle. We also reported
that although the program had recently begun to implement the risk
management process, a complete inventory of risks had not been
developed. Since then, the program has taken additional steps to
implement risk management. For example, the program hired a risk
management coordinator to maintain and update the risk management plan,
facilitate risk assessments, track efforts to reduce risks to
acceptable levels, and develop and conduct risk management training. In
addition, the program has conducted periodic risk management meetings
and quarterly program management meetings to review the program's
progress in managing risks.
Further, the program conducted risk management training with project
managers on October 24, 2006. The training included guidance on, among
other things, adhering to key risk management practices such as
preparing risk assessments and developing risk mitigation strategies.
However, the program has yet to implement other key practices. First,
it does not have a transparent, documented, and traceable process for
inventorying and resolving risks. For example, the January 23, 2006,
inventory identified three risks related to data/information, security,
and privacy. These risks were previously reported as having a medium
probability of occurrence and a low impact. However, the January 23,
2007, inventory did not include these risks. Program officials
explained that the discrepancy with the number of risks in the
inventory is that the program periodically closes risks and removes
them from the inventory. However, these officials provided
documentation on only one risk that had been removed from the
inventory, and it did not include a clear rationale for why the risk
had been removed. In addition, the officials were unable to provide
documentation showing that the other two risks had been closed and
removed and the justification for doing so.
Second, while the risk management plan defines a process and associated
practices for developing risk mitigation strategies, the program has
not fully implemented them. Specifically, the plan specifies that
mitigation strategies are to include:
a rationale for mitigation strategy chosen;
an explanation of the impact on program performance;
a proposed schedule showing milestones for initiation, significant risk
mitigation activities, and completion; and:
the official responsible for implementing and tracking mitigation
activities.
Out of the 61 risk mitigation strategies documented in the January 23,
2007, inventory:
all included a rationale for mitigation strategy chosen;
13 did not include an explanation of the impact on program performance;
none included a proposed schedule showing milestones for initiation,
significant risk mitigation activities, and completion; and:
all identified officials responsible for implementing and tracking
mitigation activities.
Third, although the risk management plan calls for developing a process
for elevating risks to management's attention, the program has not
developed and implemented such a process. According to the program
manager, in lieu of a defined escalation process, program officials do
make ICE's deputy CIO (and sometimes the CIO) aware of program risks as
part of quarterly program management meetings, but acknowledged that,
for the most part, such reviews-and thus the escalation of risks-occur
in an ad hoc fashion and are not documented.
The program manager stated that the reason for the current state of the
program's risk management was that the process is new and that it will
take time for the risk program to fully mature. He added that the
program is in the process of addressing these weaknesses by, for
example, revising the risk management plan to include procedures for
(1) reporting to management on the existence and status of risks and
(2) closing risks. More recently, on January 25, 2007, the program
manager provided us with the revised plan; it calls for the risk
management coordinator to, among other things, establish processes for
elevating and reporting to management risks that are likely to occur
and have a high impact. Further, at our February 1, 2007, exit meeting
with program officials, they provided a document, approved on January
25, 2007, defining their processes for elevating risk. While these are
steps in the right direction, the program manager acknowledged that
program staff were just beginning to implement the processes.
Until the Atlas program fully implements and institutionalizes risk
management, there is increased probability that program risks will not
be proactively mitigated and, thus, will become actual program cost,
schedule, and performance shortfalls.
Objective 2 Results: Observation 4: Performance Management:
Observation 4: Atlas program still implementing performance management
practices essential to measuring progress against commitments:
As we have previously reported,[Footnote 36] to enable Atlas program
management to measure progress and make well-informed investment
decisions, it is important for the program to develop and implement
rigorous performance management practices that include, among other
things, properly aligned goals and anticipated achievements that are
defined in measurable terms. In our September 2005 report,[Footnote 37]
we recommended that the Atlas program develop and implement an
effective performance management system that includes such goals and
measures. More recently, we reported[Footnote 38] in July 2006 that
Atlas had begun taking steps to implement performance goals and
measures, but had not yet completed defining and implementing them. For
example, the program had developed goals for four of its seven projects
but had not done so for its three other projects, which were:
Transformation Planning,
Architecture Engineering, and:
Data Center Migration.
We also reported that the program had developed measures to gauge
progress on the goals, but had not yet begun to track their progress.
Since then, the program has taken additional steps to define and
implement performance goals and measures. For example, the program
identified 13 goals for five of its seven projects that it has been
using to measure progress during fiscal year 2006. Twelve of these
goals either were expected to be finished by fiscal year 2006 or were
expected to have some interim measure completed during fiscal year
2006. The expenditure plan identified an additional goal (goal 12) that
did not have an interim measure, but rather was expected to be finished
by December 2006.
In addition, the program now reports it is measuring progress against
these performance goals by, for example, using them to discuss each
project's status and progress at quarterly program management reviews.
The following table provides our analysis of the program's performance
goals against its reported progress (as of Sept. 30, 2006).
Table: Atlas Project Progress in Meeting Fiscal Year 2006 Goals.
Project: Overall Program Management/Transformation Planning;
Goal: Goal 1: 100 percent of Atlas project activities that are managed
with less than a 10 percent variance in cost and schedule per fiscal
year;
Did Atlas meet goal?: no;
If no, what is progress against goal?: 93 percent.
Project: Overall Program Management/Transformation Planning;
Goal: Goal 2: 30 percent reduction in ICE's operations and maintenance
contribution for network services;
Did Atlas meet goal?: yes;
If no, what is progress against goal?: not applicable.
Project: Overall Program Management/Transformation Planning;
Goal: Goal 3: The enterprise project management tool, primavera, was
scheduled for installation, configuration, and deployment in third
quarter of fiscal year 2006;
Did Atlas meet goal?: no;
If no, what is progress against goal?: deployed in 4th quarter of FY
2006.
Project: Overall Program management/Transformation Planning;
Goal: Goal 4: Performance management processes and balanced scorecard
to be developed and implemented in fourth quarter of fiscal year 2006;
Did Atlas meet goal?: no;
If no, what is progress against goal?: *.
Project: Common Computing Environment;
Goal: Goal 5: 100 percent of ICE personnel using the DHS standard
Microsoft Outlook E-mail platform;
Did Atlas meet goal?: no;
If no, what is progress against goal?: 95 percent.
Project: Common Computing Environment;
Goal: Goal 6: 33 percent progress in implementing a refresh of ICE
personnel desktop equipment;
Did Atlas meet goal?: no;
If no, what is progress against goal?: 20 percent.
Project: Common Computing Environment;
Goal: Goal 7: 100 percent of ICE office sites achieving workforce
productivity upgrades through implementation of ICE e-mail standard,
backend server upgrades, and ICENet connections;
Did Atlas meet goal?: no;
If no, what is progress against goal?: 95 percent.
Project: Integration;
Goal: Goal 8: 100 percent of ICE sites using DHS standard high-speed
network circuits;
Did Atlas meet goal?: no;
If no, what is progress against goal?: 73 percent.
Project: Integration;
Goal: Goal 9: 100 percent of ICE office sites achieving workforce
productivity upgrades through implementation of ICE e-mail standard,
backend server upgrades, and ICENet connections;
Did Atlas meet goal?: no;
If no, what is progress against goal?: 73 percent.
Project: ICE Mission Information;
Goal: Goal 10: 14 percent of investigative and enforcement personnel
with access to decision support system data marts;
Did Atlas meet goal?: yes;
If no, what is progress against goal?: not applicable.
Project: ICE Mission Information;
Goal: Goal 11: 8 percent increase in ICE investigative and enforcement
systems incorporated into ICE decision support system consolidated data
marts;
Did Atlas meet goal?: yes;
If no, what is progress against goal?: not applicable.
Project: ICE Mission Information;
Goal: Goal 12: The ICE Mission Information project will implement a
technology model in the first-quarter of FY 2007 for the integration of
systems and information throughout ICE within the framework of the DHS
EA;
Did Atlas meet goal?: Not applicable (target goal set for FY 2007, but
not for FY 2006);
If no, what is progress against goal?: **.
Project: Information Assurance;
Goal: Goal 13: 30 percent of ICE users employing single sign-on
capability for systems access;
Did Atlas meet goal?: no;
If no, what is progress against goal?: 0 percent.
Source: GAO analysis of DHS data.
* While the program did not meet this goal in the fourth quarter of FY
2006 as planned, program officials told us that the program has efforts
underway to reach this goal, but did not provide a date for when it
would be accomplished.
** The program reported that it met this goal in fiscal year 2007, as
planned.
[End of table]
However, while the program has taken additional steps, it has yet to
fully implement and institutionalize its performance goals and
measures. Specifically, the program has not yet developed goals for
these two projects-Architecture Engineering and Data Center Migration.
According to program officials, they have developed draft goals for
these projects and are in the process of having them reviewed by
management, but officials did not provide a date for when the goals are
to be approved and implemented.
In addition, the progress on the performance goals, as reported by the
program, shows mixed results. As shown in the chart, of the 12 goals,
the program has met only 3 of them. Of the remaining 9, Atlas was close
to meeting its fiscal year 2006 targets. For example, on goal 5, the
fiscal year 2006 target was 100 percent of ICE personnel using the DHS
standard Microsoft Outlook e-mail platform, and the program reported 95
percent of users have been migrated. Also, on goal 8, the fiscal year
2006 target was 100 percent of ICE sites using DHS standard high speed
network circuits, and the program reported 73 percent of ICE sites are
using the circuits.
Further, underlying data used by the program to measure performance may
not be reliable in all cases. For example, on goal 1, the program uses
project cost and schedule data from the 2005 Atlas business case in
measuring progress against the goal, including determining whether
variances have occurred. However, these data are out of date (i.e.,
they do not reflect project cost and schedule slippages that have
occurred since 2005) and are currently being revised as part of a
larger effort to update the entire business case to reflect overall
program changes.
In addition, the program relies on its project managers to develop
estimates of where their projects should be (with respect to schedule)
given funds used to date. The program official responsible for
determining progress against goal 1 told us that this is a key weakness
of the performance management process because it allows project
managers to report on their progress without the program having a means
to verify and validate their assessments. The program manager
acknowledged that having and using reliable information to support
measurement against goals is a challenge across all of the program
measures, and the program plans to address the problem by, among other
things, updating the business case and supporting cost and schedule
analyses (which are to be issued in September 2007) and by implementing
and using an enterprise project management tool (planned to start in
October 2007) that calculates progress against funds used for each
project. While these are steps in the right direction, it also means it
will be months before the program has data it can rely on to measure
performance.
According to the program manager, the program's limited progress in
implementing performance management practices is attributable in large
part to (1) the program's evolving of its performance goals as the
program evolves; (2) performance management being new to the program,
and it will take time for the program's processes to fully mature; and
(3) the enterprise project management tool's taking longer to implement
than originally planned. According to the program manager, the program
recently purchased the tool and plans to begin using it for all
projects beginning in October 2007. Nonetheless, until the program
completes implementation and institutionalization of its performance
management capabilities, including well-defined goals and measures, its
managers will not have the necessary information for measuring progress
and making well-informed investment decisions.
Conclusions:
The fiscal year 2006 Atlas expenditure plan, in combination with
related program documentation and program officials' statements,
satisfies or partially satisfies the legislative conditions set forth
by Congress. However, this satisfaction is based on plans and
commitments that provide for meeting these conditions rather than on
completed actions to satisfy the conditions. In addition, while steps
are being initiated that are intended to address significant program
management weaknesses, a number of improvements, including those
recommended in our past reports, have yet to be implemented. Further,
the program has not fully achieved many performance goals that it set
out to accomplish over the past year. These factors continue to put the
program at risk and call for heightened and sustained management
attention to expeditiously address and resolve the issues.
Thus, there is much that still needs to be accomplished to minimize the
risks associated with the program's capacity to deliver promised IT
infrastructure capabilities and benefits on time and within budget.
This includes demonstrating better progress against established
performance goals in the coming year. Given that hundreds of millions
of dollars are to be invested and the program is critical to supporting
the ICE mission, it is essential that DHS follow through on its
commitments to build the capacity to effectively manage the program.
Proceeding without this capacity introduces unnecessary risks to the
program and potentially jeopardizes its viability for future
investment.
Recommendations for Executive Action:
To minimize risks to the Atlas program, we recommend that the Secretary
of Homeland Security direct the Assistant Secretary for Immigration and
Customs Enforcement to ensure that ICE follows through on its
commitments to implement effective management controls and capabilities
by implementing the following five recommendations:
* Employing practices essential to ensuring that the Atlas program's
IV&V agent is and remains independent, including incorporating
requirements in future contracting actions such as the renegotiation or
recompetition of the current independent verification and validation
contract, which will prohibit the agent from soliciting, proposing, or
being awarded program work other than providing independent
verification and validation services and products.
* Fully adhering to requirements development and management practices,
including those specified in ICE's policies and procedures. This should
also include having the Atlas program manager develop a process
improvement plan for all of the projects that is consistent with the
ICE System Lifecycle Management Handbook and provide for making the
program manager and project managers responsible and accountable for
rigorously adhering to the requirements in the handbook.
* Fully implementing key contract management and oversight practices,
including those specified in ICE's policies and procedures. This should
also include ensuring that the Atlas program manager, working with the
program's COTR, follow through on planned efforts to strengthen the
program's compliance with these practices by (1) revising the
acquisition plan by March 2007; (2) revising the ICE System Lifecycle
Management Handbook by June 2007 to incorporate key contract management
activities such as contract tracking and oversight; and (3) preparing
statements of work (beginning in February 2007) for future contracts to
clearly define Atlas work statements and acceptance criteria.
Completing implementation of planned risk management activities. This
should include (1) fully implementing and institutionalizing procedures
for reporting to management on the existence and status of risks and
progress in implementing mitigation strategies and (2) updating the
risk inventory to include risks for all projects and risk areas.
Improving program performance management, including developing
performance goals for projects that do not have goals and reporting on
their progress in the fiscal year 2007 expenditure plan. Further, the
program should assess whether the data being used to measure progress
are reliable, complete, and accurate.
Agency Comments:
In their February 1, 2007, oral comments on a draft of this briefing,
ICE officials, including the Deputy CIO and Atlas program manager,
agreed with our findings, conclusions, and recommendations. These
officials also provided technical comments, which we incorporated in
the briefing as appropriate.
Attachment I: Detailed Scope and Methodology:
To accomplish our objectives, we:
analyzed the fiscal year 2006 Atlas expenditure plan and supporting
documents against legislative conditions and other relevant federal
requirements, guidance, and best practices to determine whether the
conditions were met (in doing so, we considered the conditions met when
the expenditure plan, including supporting program documentation and
program officials' representations, either satisfied or provided for
satisfying the conditions) and:
evaluated supporting documentation and interviewed program and other
involved ICE and DHS officials to determine progress in establishing
capabilities in program management areas, such as acquisition planning,
enterprise architecture, project management, requirements development
and management, contract management and oversight, and risk management
and performance management.
In assessing the management of requirements development and contract
management and oversight, we focused on three projects-Common Computing
Environment, Integration, and ICE Mission Information-that collectively
accounted for approximately 75 percent of the funds provided to the
program and the Atlas program manager identified as being the most
critical to the program's success and ICE's mission. For DHS and ICE
data that we did not substantiate, we made appropriate attribution
indicating the data source. We conducted our work at ICE and DHS
headquarters in Washington, D.C., from November 2006 through January
2007 in accordance with generally accepted government auditing
standards.
Attachment II: GAO Analysis of Requirements Management:
Table: ICE Requirements Management Efforts Compared with Federal IT
Management Guidance and Best Practices.
Establish a requirements management policy.
Project: CCE;
Implemented: yes;
GAO's assessment of compliance: In February 2005, the CCE project
established the ICE System Lifecycle Management Handbook[Footnote 39]
as the policy to guide project efforts in developing and managing
requirements. The handbook requires requirements development and
management activities, such as eliciting desired system capabilities
from users and translating them into requirements and obtaining
approval by all stakeholders.
Project: Integration;
Implemented: yes;
GAO's assessment of compliance: In February 2005, the Integration
project established the ICE System Lifecycle Management Handbook as the
policy to guide project efforts in developing and managing
requirements.
Project: ICE Mission Information;
Implemented: yes;
GAO's assessment of compliance: In February 2005, the ICE Mission
Information project established the ICE System Lifecycle Management
Handbook as the policy to guide project efforts in developing and
managing requirements.
Source: GAO analysis of DHS data.
Legend: Yes = established/implemented; No=not established/implemented;
Partially = some, but not all, actions are implemented or actions are
in progress of being implemented:
[End of table]
Table: ICE Requirements Management Efforts Compared with Federal IT
Management Guidance and Best Practices
Elicit desired or required system capabilities from users and translate
them into system requirements and obtain approval by all stakeholders.
Project: Common Computing Environment;
Implemented: partially;
GAO's assessment of compliance: In 2003, the CCE project elicited
system capabilities from component and departmental users. Although the
capabilities elicited from the users were not documented, project
officials told us they translated the users‘ input into system
requirements, issuing a draft requirements document in June 2004.
Although the draft requirements had not yet been reviewed and approved
by system stakeholders, the project began deploying system capabilities
in December 2005. Project officials told us that they planned to
finalize the requirements document, including obtaining stakeholder
approval, and in January 2007, they provided us with a finalized and
approved version of the requirements document.
Project: Integration;
Implemented: partially;
GAO's assessment of compliance: In 2003, Integration elicited system
capabilities based on a directive from the DHS Deputy Secretary and
departmental users. The capabilities were elicited from users at
requirements workshops and documented. Project officials told us that
they translated the users‘ input into system requirements, issuing a
draft requirements document on June 15, 2005. Although the draft
requirements have not yet been reviewed and approved by system
stakeholders, the project began deploying system capabilities in March
2006. Project officials told us that they have obtained stakeholder
approval on the requirements document; however, they could not provide
documentation showing this had occurred.
Project: ICE Mission Information;
Implemented: partially;
GAO's assessment of compliance: Divided into four phases, the ICE
Mission Information project is currently in the process of eliciting
user system capabilities for each of the respective phases. For the
first phase, the project has elicited system capabilities from its
users, but has not documented them. Project officials told us that they
translated the user capabilities into system requirements, issuing a
draft requirements document dated August 7, 2006. Project officials
stated that they planned to finalize the requirements document,
including obtaining system stakeholder approval, but could not provide
a date by when this was to occur. For the other three phases, the
project has not completed eliciting requirements and documenting
capabilities, but project officials told us they planned to do so in
accordance with the practices. However, they were not able to provide a
plan and associated timetable for when they are to complete the
practices.
Develop a system of concept of operations.
Project: CCE;
Implemented: partially;
GAO's assessment of compliance: CCE did not develop a concept of
operations, which is integral to defining system requirements, until 3
years after the project had been initiated. Specifically, the project
was initiated in March 2003 and began deploying system capabilities in
December 2005, and the concept of operations was developed in June
2006.
Project: Integration;
Implemented: no;
GAO's assessment of compliance: According to the Integration project
manager, a concept of operations was developed. However, Integration
was not able to provide documentation showing it had been developed.
Project: ICE Mission Information;
Implemented: partially;
GAO's assessment of compliance: The project issued a draft concept of
operations on October 12, 2006. The ICE Mission Information project
manager told us that the project is in the process of finalizing the
document but did not provide a date for when the document would be
completed.
Ensure that changes to requirements are controlled and approved via a
disciplined change control process as the system is developed and
implemented.
Project: CCE;
Implemented: partially;
GAO's assessment of compliance: CCE did not follow a disciplined change
management process for 3 years after the project was initiated in March
2003; during this time, requirements were introduced and implemented in
an ad hoc fashion. The program has taken steps to improve change
management efforts by (1) developing an Atlas Configuration Management
Plan on February 15, 2006; (2) developing a CCE change control board
charter on November 1, 2006; and (3) developing a change process that
provides the procedures for reviewing and approving CCE change
requests. The project is currently in the process of implementing these
changes. Project officials told us that the project recently began
conducting CCE change control board meetings in September 2006.
According to the change process, CCE is required to ensure that
technical, cost, and schedule impacts are considered before approval is
granted. Although program documentation showed the technical impact of
the change, the cost and schedule impacts were not documented.
Project: Integration;
Implemented: partially;
GAO's assessment of compliance: The configuration control board and
change control process that Integration follows was developed by DHS‘s
Infrastructure Transformation Program, which is the departmental
organization (commonly referred to as the ’network steward“ by DHS and
ICE officials) responsible for ensuring that all DHS components migrate
from their existing networks to the DHS network. However, Integration
officials could not provide evidence of compliance with the network
steward‘s change control process.
Project: ICE Mission Information;
Implemented: partially;
GAO's assessment of compliance: The project plans to control and
approve changes via a change control process when the requirements are
developed and finalized. The project plans to do so consistent with the
guidance specified in the ICE System Lifecycle Management Handbook. As
previously noted, project officials were not able to identify when the
requirements would be finalized and approved.
Maintain bidirectional traceability, meaning that a given requirement
can be traced backward to systems documentation and forward to the
appropriate contract vehicle and system components.
Project: CCE;
Implemented: partially;
GAO's assessment of compliance: In December 2006, CCE developed a
requirements traceability analysis document”commonly referred to as a
matrix”to show bidirectional traceability between the requirements and
system documentation. While the project identified requirements, it did
not trace requirements to the system documentation, such as the system
design and test plans. Additionally, the project did not trace
requirements to the existing contract task order.
Project: Integration;
Implemented: partially;
GAO's assessment of compliance: In June 2005, Integration developed a
requirements traceability analysis document”commonly referred to as a
matrix”to show bidirectional traceability between the requirements and
system documentation. While the project identified requirements, it did
not trace requirements to the system documentation, such as the system
design and test plans. Additionally, the project did not trace
requirements to the existing contract task order.
Project: ICE Mission Information;
Implemented: partially;
GAO's assessment of compliance: The project plans to ensure
traceability among requirements, system documentation, and the existing
contract task order when the requirements are developed and finalized.
The project intends to do so in accordance with the guidance specified
in the ICE System Lifecycle Management Handbook. As previously noted,
project officials were not able to identify when the requirements would
be finalized and approved.
Source: GAO analysis of DHS data.
[End of table]
[End of section]
Appendix II: Comments from the Department of Homeland Security:
U.S. Department of Homeland Security:
Washington, DC 20528:
April 5, 2007:
Mr. Randolph C. Hite:
Director:
Information Technology Architecture and Systems Issues:
U.S. Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Mr. Hite:
Thank you for the opportunity to review and comment on the Government
Accountability Office's (GAO's) draft report GAO 07-565 entitled
Information Technology: Immigration and Customs Enforcement Needs to
Fully Address Significant Infrastructure Modernization Program
Management Weaknesses.
U.S. Immigration and Customs Enforcement (ICE) concurs with the draft
report's five recommendations. The following describes the actions ICE
has taken or plans to take to implement the recommendations.
Recommendation 1: Employ practices essential to assuring that the Atlas
program's independent verification and validation agent is and remains
independent, including incorporating requirements in future contracting
actions such as the renegotiation or recompetition of the current
independent verification and validation contract, which will prohibit
the agent from soliciting, proposing, or being awarded program work
other than providing independent verification and validation services
and products.
Response: Future Atlas contract actions will prohibit independent
verification and validation (IV&V) agents from soliciting, proposing,
or being awarded non-IV&V Atlas work to prevent a conflict of interest
situation.
Recommendation 2: Fully adhere to requirements development and
management practices, including those specified in ICE's policies and
procedures. This should also include having the Atlas program manager
develop a process improvement plan for all the projects that is
consistent with the ICE System Lifecycle Management Handbook and
provide for making the program manager and project managers responsible
and accountable for rigorously adhering to the requirements in the
handbook.
Response: The Atlas processes are being refined to fully comply with
Systems Lifecycle Management standards. By May 31, 2007, ICE
anticipates the implementation of an improvement plan that will address
requirement development and management processes, training, and the
establishment of a Change Control Board to govern and manage impacts.
Recommendation 3: Fully implement key contract management and oversight
practices, including those specified in ICE's policies and procedures.
This should also include ensuring that the Atlas program manager,
working with the program's contracting officer's technical
representative, follow through on planned efforts to strengthen the
program's compliance with these practices by (1) revising the
acquisition plan by May 2007, (2) revising the ICE System Lifecycle
Management Handbook by June 2007 to incorporate key contract management
activities such as contract tracking and oversight, and (3) preparing
statements of work (beginning in February 2007) for future contracts to
clearly define Atlas work statements and acceptance criteria.
Response: In November 2006, a full-time contracting officer's technical
representative joined the Atlas program to implement industry compliant
contract management and oversight policies. Furthermore, the Atlas
Acquisition Plan is currently being revised and is proceeding ahead of
the May 2007 schedule. The Atlas program will make the newly instituted
contract tracking and management oversight processes available for
incorporation into the next revision of the ICE System Lifecycle
Management Handbook scheduled for release in June 2007. The use of a
standardized statement of work (SOW) template with sections for
detailed requirements and acceptance criteria has been instituted for
the creation of all Atlas SOWs.
Recommendation 4: Complete implementation of planned risk management
activities. This should include (1) updating the risk inventory to
include risks for all projects and risk areas and (2) fully
implementing and institutionalizing procedures for reporting to
management on the existence and status of risks and progress in
implementing mitigation strategies.
Response: The Atlas risk management and escalation processes have been
refined to ensure effective risk management and better communication of
risk, and were deployed in March 2007. Furthermore, the risk inventory
has been updated to re-incorporate all risks previously archived due to
their closed status, to report risk as it relates to impact on cost,
schedule and performance, to detail risk mitigation strategy, and track
status of risks and progress.
Recommendation 5: Improve program performance management, including
developing performance goals for projects that do not have goals and
reporting on their progress in the fiscal year 2007 expenditure plan.
Further, the program should assess that the data being used to measure
progress are reliable, complete, and accurate.
Response: Performance goals have been developed for all Atlas projects.
Standard procedures to include quarterly assessments have been deployed
and are currently employed to ensure measurement accuracy.
Thank you again for the opportunity to comment on this draft report and
we look forward to working with you on future homeland security issues.
Sincerely,
Sincerely,
Steven J. Pecinovsky:
Director:
Departmental GAO/OIG Liaison Office:
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
Randolph C. Hite (202) 512-3439:
Staff Acknowledgments:
In addition to the individual named above, Gary Mountjoy, Assistant
Director; Nancy Glover; James Houtz; Jennifer Mills; Freda Paintsil;
Madhav Panwar; Karl Seifert; and Kelly Shaw made key contributions to
this report.
(310640):
FOOTNOTES
[1] Pub. L. No. 109-90, Oct. 18, 2005.
[2] OMB Circular A-11, part 7, establishes policy for planning,
budgeting, acquiring, and managing federal capital assets.
[3] An enterprise architecture provides a clear and comprehensive
picture of an organization's operations and its supporting systems and
infrastructure. It is an essential tool for effectively engineering
business processes and for implementing and evolving supporting systems
in a manner that maximizes interoperability, minimizes overlap and
duplication, and optimizes performance.
[4] GAO, Homeland Security: Recommendations to Improve Management of
Key Border Security Program Need to Be Implemented, GAO-06-296
(Washington, D.C.: Feb. 14, 2006).
[5] GAO-06-296.
[6] These three key projects are Common Computing Environment,
Integration, and ICE Mission Information.
[7] CMMI is registered in the U.S. Patent and Trademark Office by
Carnegie Mellon University.
[8] GAO, Information Technology: Immigration and Customs Enforcement Is
Beginning to Address Infrastructure Modernization Program Weaknesses
but Key Improvements Still Needed, GAO-06-823 (Washington, D.C.: July
27, 2006).
[9] GAO-06-823.
[10] ICE was formed from the former Immigration and Naturalization
Service, U.S. Customs Service, and other entities. Atlas began in 2002
under the former Immigration and Naturalization Service.
[11] pub. L. No. 109-90, Oct. 18, 2005.
[12] In the act, appropriations for Atlas are under the heading
Automation Modernization for Immigration and Customs Enforcement.
[13] OMB Circular A-11 establishes policy for planning, budgeting,
acquisition, and management of federal capital assets.
[14] The purpose of the Investment Review Board is to integrate capital
planning and investment control, budgeting, and acquisition. It is also
to ensure that spending on investments directly supports and furthers
the mission and that this spending provides optimal benefits and
capabilities to stakeholders and customers.
[15] DHS's fiscal year 2006 appropriations (Pub. L. No. 109-90, Oct.
18, 2005) provided $40.15 million for Atlas. In December 2005, this
amount was reduced to $39.75 million via a 1 percent governmentwide
rescission in Pub. L. No. 109-148, Division B, Dec. 30, 2005.
[16] Partially satisfies=the plan, in combination with supporting
documentation and stated commitments by program officials, either
satisfies or provides for satisfying many, but not all, key aspects of
the condition that we reviewed.
[17] Satisfies=the plan, in combination with supporting documentation
and stated commitments by program officials, either satisfies or
provides for satisfying every aspect of the condition that we reviewed.
[18] GAO, Information Technology. Immigration and Customs Enforcement
Is Beginning to Address Infrastructure Modernization Program Weaknesses
but Key Improvements Still Needed, GAO-06-823 (Washington, D.C.: July
27, 2006).
[19] Areview group made up of subject matter experts that recommends EA
compliance to the DHS Enterprise Architecture Board and ultimately to
the DHS Investment Review Board.
[20] Department of Homeland Security Enterprise Architecture Board: EAB
Governance Process Guide, August 9, 2004, draft version 2.0.
[21] GAO-06-823.
[22] GAO-06-823.
[23] On August 14, 2006, we reported that version 2006 of DHS's EA was
at stage 2 of the Enterprise Architecture Management Maturity
Framework, demonstrating that DHS has established the foundational
commitments and capabilities needed to manage the development of the
architecture. See GAO, Enterprise Architecture: Leadership Remains Key
to Establishing and Leveraging Architectures for Organizational
Transformation, GAO-06-831 (Washington, D.C.: Aug. 14, 2006).
[24] The Tactical Working Group is responsible for developing
strategies to address service oriented architecture design and
operational issues.
[25] Developed by Carnegie Mellon Software Engineering Institute (SEI),
Software Acquisition Capability Maturity Model (SA-CMM' ) version 1.03
(March 2002).
[26] U.S. Immigration and Customs Enforcement: System Lifecycle
Management Handbook, The Systems Development Lifecycle of Immigration
and Customs Enforcement, February 2005, version 1.0.
[27] GAO, Homeland Security: Recommendations to Improve Management of
Key Border Security Program Need to Be Implemented, GAO-06-296
(Washington, D.C.: Feb. 14, 2006).
[28] See, for example, Software Engineering Institute, Capability
Maturity Model Integrated (CMMI), version 1.1 (March 2002).
[29] GAO-06-823.
[30] U. S. Immigration and Customs Enforcement: System Lifecycle
Management, The Systems Development Lifecycle of Immigration and
Customs Enforcement, version 1.0 (Washington, D.C.: February 2005).
[31] CMMI is registered in the U.S. Patent and Trademark Office by
Carnegie Mellon University.
[32] GAO, Homeland Security: Contract Management and Oversight for
Visitor and Immigrant Status Program Need to Be Strengthened, GAO-06-
404, (Washington, D.C.: June 9, 2006).
[33] GAO-06-823.
[34] See, for example, Software Engineering Institute, Capability
Maturity Model Integrated (CMMI), version 1.1 (March 2002).
[35] GAO-06- 823.
[36] GAO-06-823.
[37] GAO, Information Technology. Management Improvements Needed on
Immigration and Customs Enforcement's Infrastructure Modernization
Program, GAO-05-805 (Washington, D.C.: Sept. 7, 2005).
[38] GAO-06-823.
[39] U.S. Immigration and Customs Enforcement. System Lifecycle
Management, The Systems Development Lifecycle of Immigration and
Customs Enforcement, version 1.0. (Washington, D.C.: February 2005).
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site (www.gao.gov). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site.
To have GAO e-mail you a list of newly posted products every afternoon,
go to www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125
Washington, D.C. 20548:
Public Affairs:
Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548:
