Immigration Benefits
Consistent Adherence to DHS's Acquisition Policy Could Help Improve Transformation Program Outcomes Gao ID: GAO-12-66 November 22, 2011Each year, the Department of Homeland Security's (DHS) U.S. Citizenship and Immigration Services (USCIS) processes millions of applications for immigration benefits using a paper-based process. In 2005, USCIS embarked on a major, multiyear program to transform its process to a system that is to incorporate electronic application filing, adjudication, and case management. In 2007, GAO reported that USCIS was in the early stages of the Transformation Program and that USCIS's plans partially or fully met key practices. In 2008, USCIS contracted with a solutions architect to help develop the new system. As requested, GAO evaluated the extent to which USCIS has followed DHS acquisition policy in developing and managing the Transformation Program. GAO reviewed DHS acquisition management policies and guidance; analyzed transformation program planning and implementation documents such as operational requirements; compared schedule and cost information with GAO best practice guidance; and interviewed USCIS officials.
USCIS has not consistently followed the acquisition management approach that DHS outlined in its management directives in developing and managing the Transformation Program. USCIS awarded a solutions architect contract in November 2008, in effect selecting an acquisition approach before completing documents required by DHS management directives. Specifically, DHS's acquisition policy requires that prior to selecting an acquisition approach, programs establish operational requirements, develop a program baseline against which to measure progress, and complete a plan that outlines the program's acquisition strategy. However, USCIS did not complete an Operational Requirements Document until October 2009, which was to inform the Acquisition Program Baseline and the Acquisition Plan. Consequently, USCIS awarded a solutions architect contract to begin capability development activities prior to having a full understanding of the program's operational requirements and the resources needed to execute the program. GAO has previously reported that firm requirements must be established and sufficient resources must be allocated at the beginning of an acquisition program, or the program's execution will be subpar. The lack of defined requirements, acquisition strategy, and associated cost parameters contributed to program deployment delays of over 2 years. In addition, through fiscal year 2011, USCIS estimates it will have spent about $703 million, about $292 million more than the original program baseline estimate. USCIS expects to begin deployment of the first release of the Transformation Program in December 2011. However, USCIS is continuing to manage the program without specific acquisition management controls, such as reliable schedules, which detail work to be performed by both the government and its contractor over the expected life of the program. As a result, USCIS does not have reasonable assurance that it can meet its future milestones. USCIS has established schedules for the first release of the Transformation Program, but GAO's analysis shows that these schedules are not reliable as they do not meet best practices for schedule estimating. For example, program schedules did not identify all activities to be performed by the government and solutions architect. Moreover, as outlined by DHS acquisition management guidance, a life-cycle cost estimate is a required and critical element in the acquisition process. USCIS has developed and updated the $1.7 billion life-cycle cost estimate for the Transformation Program, but USCIS's individual schedules for the Transformation Program did not meet best practices for schedule estimating, raising questions about the credibility of the program's life-cycle cost estimates. Because some program costs such as labor, supervision, and facilities cost more if the program takes longer, reliable schedules can contribute to an understanding of the cost impact if the program does not finish on time. Collectively, and moving forward, not meeting best practices increases the risk of schedule slippages and related cost overruns, making meaningful measurement and oversight of program status and progress, and accountability for results, difficult to achieve. GAO recommends that USCIS ensure its program schedules and life-cycle cost estimates are developed in accordance with best practices guidance. DHS concurred with GAO's recommendations and outlined the actions that USCIS is taking or has taken to address each recommendation.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Richard M. Stana Team: Government Accountability Office: Homeland Security and Justice Phone: (202) 512-8816GAO-12-66, Immigration Benefits: Consistent Adherence to DHS's Acquisition Policy Could Help Improve Transformation Program Outcomes
This is the accessible text file for GAO report number GAO-12-66
entitled 'Immigration Benefits: Consistent Adherence to DHS's
Acquisition Policy Could Help Improve Transformation Program Outcomes'
which was released on November 22, 2011.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
United States Government Accountability Office:
GAO:
Report to Congressional Requesters:
November 2011:
Immigration Benefits:
Consistent Adherence to DHS's Acquisition Policy Could Help Improve
Transformation Program Outcomes:
GAO-12-66:
GAO Highlights:
Highlights of GAO-12-66, a report to congressional requesters.
Why GAO Did This Study:
Each year, the Department of Homeland Security‘s (DHS) U.S.
Citizenship and Immigration Services (USCIS) processes millions of
applications for immigration benefits using a paper-based process. In
2005, USCIS embarked on a major, multiyear program to transform its
process to a system that is to incorporate electronic application
filing, adjudication, and case management. In 2007, GAO reported that
USCIS was in the early stages of the Transformation Program and that
USCIS‘s plans partially or fully met key practices. In 2008, USCIS
contracted with a solutions architect to help develop the new system.
As requested, GAO evaluated the extent to which USCIS has followed DHS
acquisition policy in developing and managing the Transformation
Program. GAO reviewed DHS acquisition management policies and
guidance; analyzed transformation program planning and implementation
documents such as operational requirements; compared schedule and cost
information with GAO best practice guidance; and interviewed USCIS
officials.
What GAO Found:
USCIS has not consistently followed the acquisition management
approach that DHS outlined in its management directives in developing
and managing the Transformation Program. USCIS awarded a solutions
architect contract in November 2008, in effect selecting an
acquisition approach before completing documents required by DHS
management directives. Specifically, DHS‘s acquisition policy requires
that prior to selecting an acquisition approach, programs establish
operational requirements, develop a program baseline against which to
measure progress, and complete a plan that outlines the program‘s
acquisition strategy. However, USCIS did not complete an Operational
Requirements Document until October 2009, which was to inform the
Acquisition Program Baseline and the Acquisition Plan. Consequently,
USCIS awarded a solutions architect contract to begin capability
development activities prior to having a full understanding of the
program‘s operational requirements and the resources needed to execute
the program. GAO has previously reported that firm requirements must
be established and sufficient resources must be allocated at the
beginning of an acquisition program, or the program‘s execution will
be subpar. The lack of defined requirements, acquisition strategy, and
associated cost parameters contributed to program deployment delays of
over 2 years. In addition, through fiscal year 2011, USCIS estimates
it will have spent about $703 million, about $292 million more than
the original program baseline estimate.
USCIS expects to begin deployment of the first release of the
Transformation Program in December 2011. However, USCIS is continuing
to manage the program without specific acquisition management
controls, such as reliable schedules, which detail work to be
performed by both the government and its contractor over the expected
life of the program. As a result, USCIS does not have reasonable
assurance that it can meet its future milestones. USCIS has
established schedules for the first release of the Transformation
Program, but GAO‘s analysis shows that these schedules are not
reliable as they do not meet best practices for schedule estimating.
For example, program schedules did not identify all activities to be
performed by the government and solutions architect. Moreover, as
outlined by DHS acquisition management guidance, a life-cycle cost
estimate is a required and critical element in the acquisition
process. USCIS has developed and updated the $1.7 billion life-cycle
cost estimate for the Transformation Program, but USCIS‘s individual
schedules for the Transformation Program did not meet best practices
for schedule estimating, raising questions about the credibility of
the program‘s life-cycle cost estimates. Because some program costs
such as labor, supervision, and facilities cost more if the program
takes longer, reliable schedules can contribute to an understanding of
the cost impact if the program does not finish on time. Collectively,
and moving forward, not meeting best practices increases the risk of
schedule slippages and related cost overruns, making meaningful
measurement and oversight of program status and progress, and
accountability for results, difficult to achieve.
What GAO Recommends:
GAO recommends that USCIS ensure its program schedules and life-cycle
cost estimates are developed in accordance with best practices
guidance. DHS concurred with GAO‘s recommendations and outlined the
actions that USCIS is taking or has taken to address each
recommendation.
View [hyperlink, http://www.gao.gov/products/GAO-12-66]. For more
information, contact Richard M. Stana at (202) 512-8777 or
stanar@gao.gov.
Contents:
Letter:
Background:
USCIS Did Not Always Follow Acquisition Policy Contributing to Program
Delays, Cost Increases, and Unreliable Schedules:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Detailed Results of GAO Assessment of USCIS's Detailed
Transformation Program Schedules:
Appendix II: Comments from the Department of Homeland Security:
Appendix III: GAO Contact and Staff Acknowledgments:
Related GAO Products:
Tables:
Table 1: USCIS ELIS's Five Core Business Processes:
Table 2: Transformation Program Planned Deployment:
Table 3: Transformation Program Spending, Fiscal Years 2006-2011:
Table 4: Program Milestones and Acquisition Planning Postcontract
Award:
Table 5: Description of GAO Scheduling Best Practices:
Table 6: Transformation Program Schedules' Satisfaction of Schedule
Estimating Best Practices:
Table 7: Detailed Results of Transformation Program's OIT Schedule
Satisfaction of Scheduling Best Practices:
Table 8: Detailed Results of Transformation Program's Contractor
Schedule Satisfaction of Scheduling Best Practices:
Figures:
Figure 1: USCIS ELIS Features from Customer Perspective:
Figure 2: USCIS ELIS Features from Adjudicator Perspective:
Figure 3: Overview of the DHS Acquisition Process:
Abbreviations:
ADE: acquisition decision event:
APB: Acquisition Program Baseline:
ARB: Acquisition Review Board:
DHS: Department of Homeland Security:
ELIS: Electronic Immigration System:
ESAR: Enterprise Segment Activity Roadmap:
EVM: Earned Value Management:
FTE: full-time equivalents:
IMS: Integrated Master Schedule:
LOE: level of effort:
OIT: Office of Information Technology:
OMB: Office of Management and Budget:
ORD: Operational Requirements Document:
PIPT: Program Integrated Product Team:
PMR: Program Management Review:
SDR: System Definition Review:
SNET: Start No Earlier Than:
SRA: Schedule Risk Analysis:
TLT: Transformation Leadership Team:
TPO: Transformation Program Office:
USCIS: United States Citizenship and Immigration Services:
WBS: Work Breakdown Structure:
WIPT: Working Integrated Product Teams:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
November 22, 2011:
Congressional Requesters:
Each year, the U.S. Citizenship and Immigration Services (USCIS),
within the Department of Homeland Security (DHS), processes millions
of applications and petitions for more than 50 types of immigrant and
nonimmigrant-related benefits for persons seeking to study, work,
visit, or live in the United States, and for persons seeking to become
U.S. citizens. Having a system that allows USCIS to accurately grant
immigration and citizenship benefits in a timely manner to eligible
applicants and deny benefits to those who are ineligible, as well as
one that can quickly and accurately identify fraudulent and criminal
activity, is essential for ensuring the integrity of the immigration
process. USCIS has long recognized the need to improve its benefits
application and adjudication processes and underlying technology
infrastructure. Moreover, we have previously reported on benefit
processing inefficiencies that make it difficult to manage and process
immigration benefit applications in a timely manner.[Footnote 1]
Further, the current systems and paper records do not allow USCIS to
easily share information with other government agencies, limiting
their ability to quickly identify criminals and possible terrorists.
We have previously recommended that USCIS improve its quality
assurance program to help ensure that immigration benefits are
provided only to eligible individuals. USCIS generally agreed with
this recommendation and has taken action to address.
To address program inefficiencies, USCIS embarked on a major
initiative in 2005 to transform its current paper-based system into an
electronic account-based system that is to use electronic adjudication
and account-based case management tools, including tools that are to
allow applicants to apply online for benefits. In July 2007, we
reported that USCIS was in the early stages of its Transformation
Program and that its plans partially or fully addressed most key
practices for organizational transformations.[Footnote 2] However, we
identified gaps in USCIS's plans that created risks that could
undermine its success as it began to implement the program, such as
the lack of clear performance measures and targets for the transformed
agency to show progress towards goals. We recommended that USCIS
address gaps in its plans in the areas of performance measurement,
strategic human capital management, communications, and information
technology management practices. USCIS agreed with these
recommendations and has taken actions to address them. In October
2007, USCIS prepared an acquisition plan that described its planned
phased approach for acquiring new electronic tools (i.e.,
capabilities), established key milestones, and included an estimated
program cost. In November 2008, USCIS selected a solutions architect
to help design, build, and implement the Transformation Program.
[Footnote 3]
To help its component agencies manage large-scale acquisitions such as
the Transformation Program, DHS has established an acquisition review
process to provide departmental oversight at key points in an
investment's life-cycle to assess the cost, schedule, and performance
of these acquisitions. At critical steps in the acquisition process,
agency components are required to prepare certain documents and to
obtain approval from DHS's Acquisition Review Board (ARB) before
moving into the next phase of the acquisition process.[Footnote 4] In
2007, DHS established an Acquisition Program Management Division to
support the ARB in managing DHS acquisitions.[Footnote 5]
Given the critical nature of the Transformation Program to USCIS, you
asked that we evaluate USCIS's efforts to implement the program.
Specifically, this report addresses the extent to which USCIS has
followed DHS acquisition policy in developing and managing the
Transformation Program.
To address this objective, we reviewed DHS departmental policies and
guidance, such as DHS acquisition management directives, and relevant
Transformation Program planning documents, including the 2007
Acquisition Plan and Expenditure Plan; 2008, 2010, and 2011
Acquisition Program Baseline; and 2009, 2010, and 2011 Operational
Requirements Document. We also reviewed the program's monthly status
reports dated April 2010, when we began our work, through July 2011;
the solutions architect contract for contractual program milestones;
and program schedules for tracking program milestones and activities.
Further, we reviewed DHS ARB decision memorandums for information on
DHS-level approval of the Transformation Program's key acquisition
events and action items to be addressed by the program. We interviewed
USCIS's Transformation Program officials concerning early program
planning efforts and challenges, schedule estimates and costs, and the
development and planned deployment of the first phase of the program,
among other things. We also interviewed DHS Acquisition Program
Management Division officials about the acquisition process and
decisions related to the Transformation Program. We analyzed two key
Transformation Program schedules as of November 2010, and one updated
schedule as of August 2011, against nine best practices associated
with developing and maintaining reliable schedules in our Cost
Estimating and Assessment Guide.[Footnote 6] We used commercially
available software to determine whether these schedules, among other
best practices, included all major activities and had a logical
sequence of activities and reasonable activity durations.[Footnote 7]
We conducted this performance audit from April 2010 through November
2011, in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on our audit objectives.
We believe that the evidence obtained provides a reasonable basis for
our findings and conclusions based on our audit objectives.
Background:
Overview of Transformation Program:
Once the Transformation Program is completed, USCIS envisions that the
new electronic adjudication capabilities and improved information
technology will improve agency operations and enable greater data
sharing and management of information. USCIS expects the new system,
named the USCIS Electronic Immigration System (ELIS), to have features
that will allow USCIS to meet its transformation goals of enhanced
national security and system integrity, better customer service, and
operational efficiency. For example, once USCIS ELIS is implemented,
USCIS expects that:
* Individuals will be able to establish an account with USCIS and file
applications over the internet, as well as obtain information on the
status of their application.
* USCIS will automatically apply risk-based rules to incoming
applications to identify potentially fraudulent applications and
national security risks.
* Adjudicators will have electronic access to applications, as well as
relevant USCIS policies and procedures and external databases, to aid
in decision making.
* USCIS will have the necessary management information to help it
allocate workload and measure performance.
* USCIS will have electronic linkages with other agencies, such as the
Departments of Justice and State, for data sharing and security
purposes.
Figure 1 depicts the key features of USCIS ELIS, as envisioned, from a
USCIS customer perspective. Figure 2 depicts these key features, as
envisioned, from a USCIS adjudicator perspective.
Figure 1: USCIS ELIS Features from Customer Perspective:
[Refer to PDF for image: illustration]
How customers/third party representatives will file:
1) Create user account:
* Select notification preferences and password.
2) Enter benefit application data:
* Enter data using a wizard that guides the user step=by=step through
the electronic application.
3) Review and submit application to USCIS online and pay online.
4) Receive e-mail notification to access account information to view
recent action on the benefit application, or receive USCIS notices by
mail.
5) Check case status through online account.
6) Manage submitted application:
* Uses account to change customer/representative relationship, change
address, respond to request for evidence and access USCIS notices
online.
Source: USCIS: and Art Explosion (clipart).
[End of figure]
Figure 2: USCIS ELIS Features from Adjudicator Perspective:
[Refer to PDF for image: illustration]
How USCIS Immigration Services Officer (adjudicator) will process
benefits requests:
1) Accesses system online to view a list of all assigned cases.
2) Selects and reviews case. Validates evidence; identifies previous
prerequisite related cases.
3) Reviews risk assessments completed automatically by the system and
reviewed by a background check officer before case is made available
for adjudication.
4) Reviews immigration history and associated policy and resources:
* Accesses related information and resources from existing sources to
support decision making. Documents completion of activities.
5) Assesses eligibility and completeness, based on current law and
policy.
6) Indicates and documents adjudication decision. Generates proof of
benefit and notifies customer.
Source: USCIS; and Art Explosion (clipart).
[End of figure]
Implementation of Transformation Program Business Processes:
The Transformation Program intends to design and develop five core
business processes to form the foundation of USCIS ELIS and process
and manage all applications. Table 1 identifies and briefly describes
the five core business processes.
Table 1: USCIS ELIS's Five Core Business Processes:
Business process: Immigration Account Management;
Description: Enable USCIS customers to establish and maintain their
account information electronically and USCIS employees to access case
information when adjudicating cases.
Business process: Benefits Case Management;
Description: Enable USCIS customers to apply for immigration benefits
online via the internet, and enable USCIS employees to access case
information in one location for more efficient processing and managing
of benefits requests.
Business process: Electronic Content Management;
Description: Enable USCIS employees to access electronic evidence and
information from within USCIS and from agency partners such as the
Departments of State and Labor to support decision making.
Business process: Agency and Knowledge Management;
Description: Enable USCIS management to electronically access and
centrally manage workload, resources, and performance.
Business process: Risk and Fraud Management;
Description: Enable USCIS employees to access actionable risk
information to help quickly identify potential threats, risks, and
fraud. Information to be integrated from across USCIS and from other
agencies.
Source: GAO analysis of USCIS documentation.
[End of table]
USCIS plans to deploy USCIS ELIS in a series of five releases, labeled
A through E.[Footnote 8] Within the first two releases, USCIS ELIS is
to be available to USCIS customers applying for nonimmigrant benefit
types, followed by immigrant benefits, humanitarian benefits, and
citizenship benefits.[Footnote 9] Much of the functionality needed to
operate the five core business processes are to be established during
Releases A and B, with an enhanced level of functionality to be added
during Releases C through E. Table 2 below shows the order in which
ELIS's five releases are to be deployed and available to USCIS
customers.
Table 2: Transformation Program Planned Deployment:
Release: A;
Planned deployment: Five core business processes for individuals
applying for selected nonimmigrant benefits such as application to
change immigration status;
Examples of benefits: Extend/Change nonimmigrant status; Employment
authorization; Temporary protected status.
Release: B;
Planned deployment: Enhanced core business processes' functionality
for USCIS customers applying for nonimmigrant benefit types not
addressed in Release A;
Examples of benefits: Replace permanent resident card; Petition for a
nonimmigrant worker.
Release: C;
Planned deployment: USCIS customers applying for immigrant benefits;
Examples of benefits: Petition for alien relative; Petition for alien
worker; Permanent residency (except for asylees and refugees).
Release: D;
Planned deployment: USCIS customers applying for humanitarian benefits;
Examples of benefits: Permanent residency for asylees and refugees;
Asylum.
Release: E;
Planned deployment: USCIS customers applying for citizenship benefits;
Examples of benefits: Naturalization.
Source: GAO analysis of USCIS data.
[End of table]
Overview of the Transformation Program Management Structure,
Acquisition Approach, and Funding Sources:
In 2006, USCIS drafted a transformation strategic plan to guide its
modernization efforts and established the Transformation Program
Office (TPO) to lead and carry out the effort. By 2007, USCIS
established a governance structure for the overall management,
leadership, decision making, and oversight of the Transformation
Program. The TPO governance structure includes three key groups: (1)
Transformation Leadership Team (TLT) responsible for the overall
program direction and coordination of transformation initiatives
within the agency; (2) Program Integrated Product Teams (PIPT)
responsible for advising on and approving strategy and performance
measures, and overseeing and managing the program, including cost,
schedule, and performance; and, (3) Working Integrated Product Teams
(WIPT), composed of agencywide representatives with expertise to help
define the transformed business processes and its operational aspects.
In addition to these key groups, USCIS also holds Program Management
Review (PMR) meetings to help manage the transformation effort. Each
month, the Transformation Program conducts PMR meetings to assess the
status of the overall program and solutions architect activities in
terms of cost, schedule, performance, and risk. Major program groups
associated with USCIS's transformation efforts, and the solutions
architect, report on the status of activities and deliverables for
which they have responsibility.[Footnote 10] The monthly PMR reports
help provide an up-to-date snapshot of top program risks and concerns
and how they are being mitigated, as well as the overall status of the
program in meeting its milestones, among other information.
In November 2008, USCIS awarded a solutions architect contract for
approximately $500 million to be allocated over a 5-year period, to
design, develop, test, deploy, and sustain the Transformation Program
by November 2013.[Footnote 11] As such, the Transformation Program is
USCIS's largest acquisition and according to USCIS's current Director
"no project is more important to long-term operational improvement and
efficiency than Transformation."[Footnote 12] USCIS has funded the
Transformation Program through both direct legislative appropriations
and revenue from applicants' application fees paid. In fiscal years
2006 and 2007, Congress appropriated a combined total of $71.7 million
to fund Transformation Program efforts. Since fiscal year 2007,
USCIS's premium processing fee revenue has been the primary source of
funding for the Transformation Program.[Footnote 13] In addition, the
program has used funds from its application fee account to pay for the
salaries and benefits of USCIS Transformation Program staff. As shown
in Table 3, USCIS spent about $455 million from fiscal years 2006
through 2010, which includes costs incurred by both the solutions
architect and USCIS. USCIS estimates it will spend approximately $248
million in fiscal year 2011 for an estimated total cost of about $703
million through fiscal year 2011.
Table 3: Transformation Program Spending, Fiscal Years 2006-2011:
Fiscal year: 2006;
Transformation Program spending: $19.3 million.
Fiscal year: 2007;
Transformation Program spending: $26.9 million.
Fiscal year: 2008;
Transformation Program spending: $52.5 million.
Fiscal year: 2009;
Transformation Program spending: $122.6 million.
Fiscal year: 2010;
Transformation Program spending: $233.2 million.
Fiscal year: Total spending 2006-2010;
Transformation Program spending: $454.5 million.
Fiscal year: Estimated spending 2011;
Transformation Program spending: $248.4 million.
Fiscal year: Total estimated spending 2006-2011;
Transformation Program spending: $702.9 million.
Source: GAO analysis of USCIS data.
Note: Transformation Program did not incur solutions architect costs
in fiscal years 2006 through 2007.
[End of table]
Overview of DHS Acquisition Process:
In 2003, DHS established an investment review process to help reduce
risk and increase the chances for successful acquisition outcomes by
providing departmental oversight of major investments throughout their
life-cycles. The process was intended to help ensure that funds
allocated for investments through the budget process were being spent
wisely, efficiently, and effectively. In March 2006, DHS issued
Management Directive No. 1400 that defined and updated DHS's
investment review process. The directive required programs to prepare
certain documents before transitioning to the next acquisition phase
to ensure the program is ready to move to the next phase. To implement
more rigor and discipline in its acquisition processes, DHS created
the Acquisition Program Management Division in 2007 to develop and
maintain acquisition policies, procedures, and guidance as a part of
the system acquisition process.[Footnote 14] In November 2008, DHS
issued an interim acquisition directive and guidebook that superseded
Management Directive No. 1400, which provided programs guidance to use
in preparing key documentation to support component and departmental
decision making.[Footnote 15] In January 2010, DHS finalized the
acquisition directive, which established acquisition life-cycle phases
and senior-level approval of each major acquisition program at key
acquisition decision events during a program's acquisition life-
cycle.[Footnote 16] This directive established the acquisition life-
cycle framework with four phases:
1. identify a capability need (need phase);
2. analyze and select the means to provide that capability (analyze/
select phase);
3. obtain the capability (obtain phase); and:
4. produce, deploy, and support the capability (produce/deploy/support
phase).
Each acquisition phase culminates in a presentation to the DHS ARB,
which is to review each acquisition at least three times at key
acquisition decision events during a program's life-cycle. Figure 3
presents the four DHS acquisition phases, including the documents
presented to ARB and their review as defined in the acquisition
directive.
Figure 3: Overview of the DHS Acquisition Process:
[Refer to PDF for image: illustration]
Need: Define the problem; review mission need statement;
ADE 1;
Analyze/select: Identify the alternatives and resource requirements;
REview planning documents, including operational requirements
document, acquisition program baseline, and acquisition plan;
DHS approves selected acquisition approach;
ADE 2A;
Obtain: DHS approves supporting acquisitions including small-scale
productions;
ADE 2B;
Develop and evaluate capabilities; DHS approves the acquisition to go
into production.
Produce/deploy/support: ADE 3; Produce and maintain those capabilities.
Legend:
ADE: Acquisition Decision Event.
Source: GAO analysis of DHS Acquisition Management Directive 102-01;
and Art Explosion (clipart).
[End of figure]
The Acquisition Decision Authority--the Chief Acquisition Officer or
other designated senior-level official--is to chair ARB and decide
whether the proposed acquisition meets certain requirements necessary
to move onto the next phase and eventually to full production. The
directive outlines the extent and scope of required program, project,
and service management; level of reporting; and the acquisition
decision authority based on whether the acquisition is considered a
major life-cycle cost[Footnote 17] (estimated at or above $300
million) or nonmajor (life-cycle costs estimated to be below $300
million). DHS considers the USCIS Transformation Program a major
acquisition, and as such, the decision authority is the DHS Under
Secretary for Management.[Footnote 18]
Following an ARB meeting, the Acquisition Program Management Division
is to prepare an acquisition decision memorandum as the official
record of the meeting. This memorandum is to be signed by the
acquisition decision authority and must describe the approval or other
decisions made at the ARB and any action items to be satisfied as
conditions of the decision. The ARB reviews provide the department an
opportunity to determine a program's readiness to proceed to the
following life-cycle phase. However, we reported in March 2011 that
the ARB had not reviewed most of DHS's major acquisition programs by
the end of fiscal year 2009, and the programs that were reviewed had
not consistently implemented action items identified in the review by
established deadlines.[Footnote 19] Our prior work has shown that when
these types of reviews are skipped or not fully implemented, programs
move forward with little, if any, early department-level assessment of
the programs' costs and feasibility, which contributes to poor cost,
schedule, and performance outcomes.[Footnote 20] In June 2011, DHS
reported that it was taking action to strengthen its acquisition
management processes by reviewing programs on an ongoing basis rather
than only at key acquisition decision events, and developing decision-
making support tools to aid with oversight. These are positive steps
that if effectively implemented should help strengthen its acquisition
management processes.
USCIS Did Not Always Follow Acquisition Policy Contributing to Program
Delays, Cost Increases, and Unreliable Schedules:
USCIS has not consistently followed the acquisition management
approach that DHS outlined in its management directives in developing
and managing the Transformation Program. Consistent with DHS
acquisition policy, USCIS prepared a Mission Needs Statement to
justify the need and value of the Transformation Program in pursuing
the proposed acquisition. In addition, USCIS identified and analyzed
various alternatives for transforming its business processes. However,
USCIS did not complete several acquisition planning documents required
by DHS policy prior to moving forward with an acquisition approach and
selecting a solutions architect to develop USCIS ELIS's capabilities.
The lack of this program documentation contributed to the
Transformation Program being more than 2 years behind schedule in its
planned initial deployment of USCIS ELIS and increased program costs.
In addition, USCIS has not developed reliable or integrated schedules,
both of which, under DHS acquisition guidance, are required and
essential acquisition management elements. As a result, USCIS cannot
reliably estimate when all releases of the Transformation Program will
be delivered.
USCIS Awarded a Contract to Develop Program Capabilities without Fully
Understanding Requirements and Resources Needed:
USCIS awarded a solutions architect contract to begin capability
development activities prior to having a full understanding of
requirements and resources needed to execute the program. DHS's
acquisition policy requires that programs conduct planning efforts to
establish a program's operational requirements, to develop a program
baseline against which to measure progress, and a plan that outlines
the program's acquisition strategy. These planning efforts are to be
documented in three key documents: the Operational Requirements
Document, the Acquisition Program Baseline, and the Acquisition Plan.
According to DHS policy, these key documents are to be completed
before selecting and moving forward with an acquisition
approach.[Footnote 21] According to agency officials, the goal is to
help ensure that before committing funds to develop a capability, the
program's operational requirements, cost, schedule, and performance
parameters have been fully defined. We have previously reported that
firm requirements must be established and sufficient resources must be
allocated at the beginning of an acquisition program, or the program's
execution will be subpar.[Footnote 22] We have also reported that well-
defined requirements are critical to ensuring communication about what
the government needs from the contractor providing services.[Footnote
23] However, when the solutions architect contract was awarded in
November 2008, one document had not been completed and the other two
did not fully address the program's estimated cost, planned schedule,
or performance parameters. Below is a summary of the three planning
documents that USCIS did not develop according to DHS policy:
Operational Requirements Document (ORD)--According to DHS acquisition
policy, this document is to describe the operational mission,
objectives, capabilities, and operational user key performance
parameters (i.e., the minimum as well as the desired levels of
performance that must be met to provide a useful capability to the
user) and should be completed before an acquisition approach is
selected.[Footnote 24] However, USCIS did not develop the first
version of the ORD until October 2009, almost a year after the award
of the solutions architect contract.[Footnote 25]
Program officials acknowledged that an ORD was not prepared prior to
selecting an acquisition approach but stated that the solutions
architect had sufficient information on the program's operational
requirements to begin work. For example, they stated that the
contractor received USCIS's Enterprise Segment Activity Roadmap
(ESAR), which described various activities related to ELIS's core
business process. However, in a February 2009 memorandum, the USCIS
Chief Information Officer stated that the ESAR did not provide a
realistic capability to guide, constrain, or measure the solutions
architect because the business process mappings were incomplete and
vague, among other reasons.
Acquisition Program Baseline (APB)--This document is to provide cost,
schedule, and performance parameters. DHS policy requires it to be
prepared prior to selecting an acquisition approach. USCIS completed a
draft acquisition program baseline in May 2008 prior to awarding the
solutions architect contract. However, the May 2008 APB did not fully
address cost, schedule, and performance parameters as required by DHS
policy. Regarding cost, the APB estimated the Transformation Program
would cost approximately $410.7 million for fiscal years 2009 through
the second quarter of fiscal year 2013. However, this estimate only
included the estimated contract cost for a solutions architect.
According to program officials, the estimate did not include USCIS
costs for upgrading its information technology infrastructure, such as
upgrading networks and servers or the costs of USCIS Transformation
Program personnel and other support contractors, because these costs
had yet to be defined. Moreover, USCIS had not yet developed a life-
cycle cost estimate, which per DHS acquisition policy, is a source
document used to develop the APB's cost parameters.
Regarding the schedule included in the May 2008 APB, it was a high-
level view of the program's key milestones. The acquisition program
baseline shows that the program's start was expected in fiscal year
2009 and the deployment of all benefit types in USCIS ELIS by fiscal
year 2013. According to DHS acquisition policy, this high-level
schedule is to be based upon a program's integrated master schedule, a
larger and more detailed delineation of program milestones and
associated deliverables. However, USCIS did not complete an integrated
master schedule prior to contract award. In the absence of an
integrated master schedule, program officials were unable to clarify
for us how USCIS determined the program's key milestones, which had
USCIS implementing USCIS ELIS from fiscal years 2009 through 2013.
Lastly, DHS acquisition policy required that performance parameters be
based upon operational requirements. The May 2008 acquisition program
baseline captured performance parameters, but they were not based on
operational requirements since USCIS had not yet developed operational
requirements, as discussed above.
Acquisition Plan--This document is to address, among other things,
technical, business, management, and other significant considerations
affecting the acquisition strategy and contract selection. USCIS
developed an acquisition plan in October 2007; however, this document
did not address all capabilities for sustaining and maintaining the
acquisition, such as certain technical considerations that would
affect the acquisition strategy, as required by DHS acquisition
guidance. For example, USCIS was to upgrade its information technology
infrastructure. However, the October 2007 acquisition plan did not
reflect these technical considerations.
Moreover, cost information in the acquisition plan is not traceable to
other documents, such as a validated life-cycle cost estimate or an
acquisition program baseline, as required by DHS guidance.
Specifically, the October 2007 acquisition plan presented a $3.4
billion estimated cost for the Transformation Program. According to
program officials, the $3.4 billion included information technology
costs and covered the life of the program, which is similar to a life-
cycle cost estimate. However, the $3.4 billion cost had not been
validated as a life-cycle cost estimate by the DHS Cost Analysis
Division. Moreover, the May 2008 acquisition program baseline makes no
reference to the $3.4 billion cost over the life of the program.
However, as required by DHS guidance, the acquisition program baseline
is to reflect all cost parameters.
According to program officials, the solutions architect contract was
performance-based, meaning that USCIS specified the outcomes it was
seeking to achieve and gave the solutions architect responsibility for
identifying and delivering the assets needed to achieve these
outcomes.[Footnote 26] As a result of this approach, many of the
specifics that would affect the program's cost and schedule were to be
determined after the contract was signed. The contract called for the
solutions architect to use the 90-day base period from November 2008
to February 2009 to develop a plan to (1) identify work activities to
be performed; (2) assign resources to these activities; (3) project
the start and completion dates for these activities; (4) provide
deliverables to the TPO; and (5) establish performance measures that
the contractor and USCIS could use to measure progress. For example,
the specific operational requirements and USCIS information technology
upgrades that would be needed would depend upon the solutions
architect plan. However, as early as 2004--4 years prior to the
solutions architect contract--we reported that this type of
acquisition strategy and contracting approach had led to poor
acquisition outcomes at DHS. Specifically, we reported that a
contracting approach that assigned a U.S. Coast Guard contractor
significant responsibilities, such as the identification of work
activities and deliverables had been a primary reason for performance,
cost, and schedule problems, as it had led to incomplete information
about performance and production risks.[Footnote 27] Program officials
emphasized that the work completed during this 90-day base period was
done in conjunction with USCIS, which helped to inform the production
of these deliverables.
Incomplete Planning Contributed to Missed Milestones, Increased Costs,
and Anticipated Benefits That Were Not Achievable:
Incomplete program planning documents at the start of the program
contributed to the delayed deployment of USCIS ELIS, increased costs,
and anticipated benefits not being achieved.
Missed Milestones:
According to the November 2008 solutions architect contract, the
deployment of capabilities was to begin by September 2009 and be
completed by 2013. USCIS did not meet the September 2009 deployment
milestone. In an April 2009 memorandum to USCIS's Acting Deputy
Director and Chief Financial Officer, the program manager stated that
based on the solutions architect's proposal, the program did not have
sufficient staff to provide adequate government oversight of the
solutions architect or funding to support the proposed solution
"rendering the solution unachievable." Consequently, the solutions
architect contract was modified by scaling back the scope to allow the
contractor to focus on work activities necessary to develop the five
core business processes. Accordingly, TPO was not authorized to start
preliminary design work for Release A until December 2009. By December
2009, TPO proposed deployment of this first release by April 2011.
However, as noted in its July 2010 Acquisition Decision Memorandum,
TPO experienced delays while defining Release A requirements. These
delays resulted in a revised deployment milestone to occur between
June and August 2011. Difficulties defining requirements continued and
in November 2010, USCIS revised the deployment milestone to December
2011. By January 2011, the requirements had not yet been completed,
and by April 2011 USCIS reduced the scope of the first release to meet
the newly revised December 2011 deployment time-frame. Operational
requirements were completed in April 2011 and approved by the ARB in
July 2011, nearly 3 years after the solutions contract was awarded.
Table 4 provides information on Transformation Program milestones,
status, and acquisition planning postcontract award.
Table 4: Program Milestones and Acquisition Planning Postcontract
Award:
Milestone date: Sept. 2009;
Milestone: Begin deployment of the first phase of USCIS ELIS to
Citizenship benefit types;
Milestone status: Not met;
Description: USCIS awarded the solutions architect contract in
November 2008, with a contractual milestone to deploy five core
business processes for customers to file, and adjudicators to process,
all benefit applications associated with its Citizenship line of
business;
Acquisition planning postaward of contract: USCIS reviewed staff needs
and costs and scope of upgrading USCIS's technology infrastructure,
among other things, as proposed in the solutions architect's plan--
concluding that USCIS did not have the resources, including funding,
or infrastructure to support the proposed solution.
Milestone date: April 2011;
Milestone: Proposed Release A deployment, to first deliver core
processes to Non-Immigrant benefit types;
Milestone status: Not met;
Description: USCIS requested and was authorized by the ARB in December
2009 to change the order of deployment of its lines of business (start
from Non-Immigrant instead of Citizenship);
Acquisition planning postaward of contract: Costs of digitizing
existing paper files in support of the schedule for the program were
defined, showing that original plans were not achievable within
associated budget and timeframe.
Milestone date: June to Aug. 2011;
Milestone: Revised deployment of Release A, to deliver a release that
was reduced in scope;
Milestone status: Not met;
Description: By July 2010, USCIS had changed the scope of the first
release moving certain capabilities to Release B because of delays in
fully defining Release A requirements;
Acquisition planning postaward of contract: Contract structure was
reviewed, showing there was a cost risk due to lack of alignment
between key events and deliverables for each release.
Milestone date: Dec. 2011;
Milestone: Revised deployment of Release A, to first deliver core
processes to one Non-Immigrant benefit type;
Milestone status: n/a[A];
Description: By February 2011, USCIS proposed reducing the scope of
the first release because the solutions architect indicated that it
was unable to support a timely deployment unless the release was
limited to one benefit type. By April 2011, USCIS reduced the scope of
Release A and expected the additional benefit types associated with
the Non-Immigrant business line to be deployed by October 2012;
Acquisition planning postaward of contract: Requirements were in the
process of being defined, showing that USCIS and the solutions
architect had underestimated the time and effort needed to develop
requirements based on the complexity of the immigration process (i.e.,
workflow process business rules, legacy environment, and subject
matter).
Source: GAO analysis of DHS data.
[A] The December 2011 milestone had not occurred as of the date this
report was issued.
[End of table]
Cost Increases:
Because the acquisition strategy and associated cost parameters were
not fully outlined at the start of the program, costs associated with
the Transformation Program have increased above the original estimate.
The program's May 2008 acquisition program baseline estimated that the
total cost of the program from fiscal years 2009 to 2013 would be
$410.7 million. However, the estimated cost through fiscal year 2011
is about $703 million, about $292 million more than estimated in May
2008. This increase in the cost estimate is due to the fact that
USCIS's original planning efforts did not cover the entire program, as
required by DHS acquisition planning guidance. For example, the
acquisition program baseline did not include USCIS's information
technology enabling costs which, based on data gathered from program
officials, totals approximately $618 million and includes activities
such as upgrading its technology infrastructure. In addition, the
staffing levels have significantly increased from original
projections. At the start of the Transformation Program, USCIS had
allocated funding for 20 full-time equivalent staff assigned to TPO.
As of June 2011, the program had an authorized staffing level of 98.
Other costs not planned for have contributed to the program's overall
cost increases. For example, the cost of an operational testing agent,
who would be responsible for planning, conducting and reporting
independent operational testing and evaluation for Release A, was not
included in the acquisition planning process. USCIS officials from TPO
and the Office of Information Technology (OIT) agreed that an
operational test agent appeared to be a duplicative effort because TPO
had already planned to conduct independent testing.[Footnote 28]
However, DHS denied TPO's request for a waiver of the operational
testing agent. As a result, USCIS contracted with an independent
operational test agent by October 2010, and as of June 2011, TPO has
awarded approximately $1.8 million towards this contract.
Deferred Capabilities and Reduced Scope:
USCIS's Transformation Program planned to deploy USCIS ELIS first to
USCIS customers applying for citizenship benefits. However, once USCIS
defined the costs associated with digitizing (scanning paper documents
into an electronic format) existing records following the June 2009
ARB, USCIS concluded that the original plans were not achievable
within the associated budget. As a result, in December 2009, USCIS
requested--and was authorized by the ARB--to change the order of
deployment and begin with the nonimmigrant instead of citizenship line
of business. Moreover, according to program officials, from June 2010
to March 2011, USCIS worked to fully define the operational
requirements that had not been developed prior to the start of the
solutions architect contract. For example, an operational requirement
of USCIS ELIS is account set up and intake, which is the ability of
USCIS customers to set up accounts and for adjudicators to process
them through one, person-centric account. TPO worked with subject-
matter experts from USCIS's field and headquarters offices who were
most familiar with the adjudication process to map out steps that were
needed to fully define USCIS ELIS's operational requirements. However,
in an ARB meeting held in July 2010, and in a program management
review meeting for January 2011, program officials explained that
defining operational requirements was taking longer than expected due
to the complexity of the rules that needed to be defined in USCIS
ELIS, and the review of and agreement to these rules by all
stakeholders. For example, one requirement--the account set-up and
intake requirement--identified 35 operational functions for USCIS ELIS
to perform this action, including set up account online, schedule an
appointment, and evaluate any identity discrepancy. To enable
completion of the operational requirements needed to move into
subsequent phases of development for Release A, USCIS moved
approximately 10 percent of the capabilities into the second release.
In May 2011, program officials told us they changed the scope of the
first release and that full automation of Release A would not be in
place in December 2011. Further, only one nonimmigrant benefit would
be deployed at that time. Other nonimmigrant benefits were scheduled
to be deployed between January and October 2012.
DHS Increased Its Oversight to Help Ensure Transformation Program's
Compliance with DHS Acquisition Process:
DHS has increased its oversight of the Transformation Program since it
authorized USCIS to award the solutions architect contract in October
2008. In 2008, we reported that DHS's investment review process had
not provided the oversight needed to identify and address cost,
schedule, and performance problems in its major acquisitions,
including ensuring that programs prepared key documents prior to
moving into subsequent phases of program development. At the time, we
made several recommendations aimed at better ensuring DHS fully
implemented and adhered to its acquisition review process, including
tracking major investments.[Footnote 29] DHS generally agreed with our
recommendations and has since taken actions to improve its acquisition
review process, including developing a database to capture and track
key program information, such as cost and schedule performance,
contract awards, and program risks. The database became fully
operational in September 2009. DHS Acquisition Program Management
Division officials acknowledged that there was limited oversight of
the Transformation Program at the time the contract was signed
primarily due to having limited staff to oversee DHS's programs. These
officials further stated that DHS was continuing to develop its
acquisition oversight function and had begun to implement the revised
acquisition management directive that included more detailed guidance
for programs to use when informing component and departmental decision
making.
Since the contract award, the ARB has met six times to review the
Transformation Program's status. At these meetings, the ARB has
directed the TPO to address a number of issues related to cost,
schedule, and performance. For example, in June 2009, the ARB held two
meetings to discuss risks that had been identified during the 90-day
baseline period, such as inadequate staffing levels and delays in
delivering required government-furnished items to the contractor. As a
result of these risks, the ARB authorized USCIS to move forward with
awarding contract options one and two, but restricted the amount that
could be expended. The Transformation Program Office was also required
to return to the ARB for authorization to award any additional
options. According to the ARB Acquisition Decision Memorandum from
December 2009, the program had improved its staffing significantly,
but issues remained, including the need to fully define system
requirements prior to returning to the ARB for authorization to enter
into design, development, and testing phases, as noted in the August
2010 ARB Acquisition Decision Memorandum. As a result of this and
other outstanding action items, the ARB did not grant the program
permission to proceed with development as requested by USCIS at the
July 2010 and November 2010 ARBs. Subsequently, in April 2011, the
program completed development of operational requirements and the
acquisition program baseline. USCIS received departmental approval for
both the requirements and acquisition baseline in July 2011, along
with approval to proceed with development. DHS Acquisition Program
officials stated that USCIS had received approval because they had
fully defined operational requirements for Release A, but that USCIS
was expected to return to the ARB in December 2011 in order to obtain
a decision on whether Release A can be deployed as scheduled at the
end of the year. In addition, DHS officials stated that before the ARB
approves releases beyond Release A, TPO will need to demonstrate that:
* USCIS ELIS's core business processes work in accordance with its
operational requirements, and:
* USCIS can afford to pay for the rest of the program.
In several meetings, the ARB has requested that USCIS refine or
otherwise provide a complete and documented life-cycle cost estimate
for DHS review and validation. USCIS subsequently completed life-cycle
cost estimates in September 2009, November 2010, and an updated
version in March 2011. This most recent version estimated that the
Transformation Program's life-cycle cost would be approximately $1.7
billion from fiscal years 2006 through 2022. However, as referenced in
the life-cycle cost estimate--a planning document--USCIS cannot
estimate several work elements because the program does not have
required information to estimate complete cost, such as requirements
beyond Release A.[Footnote 30] According to the TPO Program Manager,
DHS has reviewed and provided guidance on the development of the life-
cycle cost estimate, but it has not yet validated the life-cycle cost
estimate as being sound and reasonable. Therefore, at this time, the
total expected costs of the program from initiation through completion
remain uncertain. Prior to validation of the life-cycle cost estimate,
program officials stated that TPO and the DHS Cost Analysis Division
are to work closely to ensure the cost estimate is sound and
reasonable. According to best practices in cost estimating, an updated
life-cycle cost estimate is to, among others, show the source of data.
In the case of the Transformation Program, an updated life-cycle cost
estimate should show the source of data underlying the software design
and cost estimating model, and the equations used to estimate the
costs of this large effort. The most recent Acquisition Decision
Memorandum dated July 7, 2011, states that the Transformation Program
Office must work closely with the DHS Cost Analysis Division to
complete a life-cycle cost estimate by September 30, 2011.
Improved Acquisition Planning Could Help USCIS Avoid Further Delays
and Potential Cost Overruns for the Transformation Program:
USCIS Does Not Have Reasonable Assurance That Future Milestones Are
Achievable:
USCIS is continuing to manage the Transformation Program without
specific acquisition management controls such as reliable schedules
and as a result it will be difficult for USCIS to provide reasonable
assurance that it can meet its future milestones. USCIS has
established schedules for the first release of the Transformation
Program, but our analysis shows that these schedules are not reliable
as they do not meet best practices for schedule estimating. For
example, the schedules did not identify all activities to be performed
by the government and solutions architect. Additionally, USCIS has
encountered a number of challenges in implementing the schedules, such
as assumptions that have not been met regarding the time frames that
either the solutions architect or USCIS would complete certain tasks.
For example, according to an April 2010 program management review,
USCIS planned to provide the solutions architect with two technical
environments to conduct production and testing activities by December
2010.[Footnote 31] However, USCIS has since revised the schedule due
to challenges in procuring hardware and software needed before these
two environments were ready for the solutions architect.
Based on the revised schedule, delivery of the technical environments
was delayed to April 2011, according to USCIS, so that OIT could take
actions to address the delay, such as borrowing equipment until a
contract protest was resolved, and providing the solutions architect
with a walk-through of the technical environments to ensure it met
their needs. Program officials stated that factors outside their
control, such as contract protests or review and approval of system
requirements, have contributed to challenges in implementing the
schedules.[Footnote 32] According to program officials, defining and
developing requirements was expected to last about 2 ˝ to 3 months.
However, USCIS completed the requirements in 9 months, which included
review and validation of these requirements by agency leadership.
Program officials stated that detailed reviews and approval by agency
leadership took longer than expected.
Best practices in schedule estimating state that a comprehensive
schedule should include a schedule risk analysis, so that the risk to
the estimate if items are delayed can be modeled and presented to
management including, among others, assumptions on equipment
deliveries or length of internal and external reviews. However,
according to a January 2011 program management review, as changes to
the program were happening rapidly, there was no analysis completed to
determine the impact on the schedule.
In addition to the challenges USCIS has encountered in carrying out
the schedules as originally planned, on the basis of our analysis we
found that the current schedules for the first release of the
Transformation Program are of questionable reliability. Best practices
state that the success of a large-scale system acquisition program,
such as the Transformation Program, depends in part on having reliable
schedules that identify:
* when the program's set of work activities and milestone events will
occur;
* how long they will take; and:
* how they are related to one another.
Among other things, reliable schedules provide a road map for
systematic execution of a program and the means by which to gauge
progress, identify and address potential problems, and promote
accountability. Our research has identified nine best practices
associated with developing and maintaining a reliable
schedule.[Footnote 33] To be considered reliable, a schedule should
meet all nine practices. In a July 2008 memorandum, DHS's Under
Secretary for Management endorsed the use of these scheduling
practices and noted that DHS would be using them. The nine scheduling
best practices are summarized in table 5.
Table 5: Description of GAO Scheduling Best Practices:
Characteristic: Capturing all activities;
Description: The schedule should reflect all activities (steps,
events, outcomes, etc.) as defined in the program's work breakdown
structure, to include activities to be performed by both the
government and its contractors.
Characteristic: Sequencing all activities;
Description: The schedule should be planned so that critical project
dates can be met. To meet this objective, activities need to be
logically sequenced--that is, listed in the order in which they are to
be carried out. In particular, activities that must be completed
before other activities can begin (predecessor activities), as well as
activities that cannot begin until other activities are completed
(successor activities), should be identified. This helps ensure that
interdependencies among activities that collectively lead to the
accomplishment of events or milestones can be established and used as
a basis for guiding work and measuring progress.
Characteristic: Assigning resources to all activities;
Description: The schedule should reflect what resources (e.g., labor,
materials, and overhead) are needed to do the work, whether all
required resources will be available when needed, and whether any
funding or time constraints exist.
Characteristic: Establishing duration of all activities;
Description: The schedule should reflect how long each activity will
take to execute. In determining the duration of each activity, the
same rationale, data, and assumptions used for cost estimating should
be used. Durations should be as short as possible and have specific
start and end dates. Excessively long periods needed to execute an
activity should prompt further decomposition of the activity so that
shorter execution durations will result.
Characteristic: Integrating schedule activities horizontally and
vertically;
Description: The schedule should be horizontally integrated, meaning
that it should link products and outcomes associated with other
sequenced activities. These links are commonly referred to as
"handoffs" and serve to verify that activities are arranged in the
right order to achieve aggregated products or outcomes. The schedule
should also be vertically integrated, meaning that traceability exists
among varying levels of activities and supporting tasks and subtasks.
Such mapping or alignment among levels enables different groups to
work to the same master schedule.
Characteristic: Establishing a critical path;
Description: Scheduling software should be used to identify the
critical path, which represents the longest total duration through the
sequenced list of activities. Establishing a project's critical path
is necessary to examine the effects of any activity slipping along
this path. Potential problems along or near the critical path should
also be identified and reflected in scheduling the duration of high-
risk activities.
Characteristic: Identifying float between activities;
Description: The schedule should identify the float--the amount of
time by which a predecessor activity can slip before the delay affects
successor activities--so that a schedule's flexibility can be
determined. As a general rule, activities along the critical path have
the least float.
Characteristic: Conducting a schedule risk analysis;
Description: A schedule risk analysis should be performed using
statistical techniques to predict the level of confidence in meeting a
project's completion date. This analysis focuses not only on critical
path activities but also on activities near the critical path, since
they can affect the project's status.
Characteristic: Updating the schedule using logic and durations to
determine dates;
Description: The schedule should be continuously updated using logic
and durations to determine realistic start and completion dates for
program activities. The schedule should be analyzed continuously for
variances to determine when forecasted completion dates differ from
planned dates.
Source: GAO.
[End of table]
The Transformation Program has 18 individual schedules. Table 6
summarizes the findings of our assessments of two of these individual
schedules as of November 2010 representing the bulk of the
Transformation Program efforts and those most critical to the
production of USCIS ELIS. Specifically, these schedules track
activities associated with USCIS's OIT and the solutions architect.
TPO is responsible for managing key acquisition functions associated
with the Transformation Program; thus, USCIS is responsible for
tracking and oversight of the OIT and solutions architect's activities
and associated schedules. Appendix I includes a detailed discussion of
our analysis.
Table 6: Transformation Program Schedules' Satisfaction of Schedule
Estimating Best Practices:
Best practice: 1. Capturing all activities;
GAO assessment of USCIS approved OIT schedule: Minimally met;
GAO assessment of USCIS approved solutions architect schedule:
Partially met.
Best practice: 2. Sequencing all activities;
GAO assessment of USCIS approved OIT schedule: Minimally met;
GAO assessment of USCIS approved solutions architect schedule:
Minimally met.
Best practice: 3. Assigning resources to all activities;
GAO assessment of USCIS approved OIT schedule: Minimally met;
GAO assessment of USCIS approved solutions architect schedule:
Minimally met.
Best practice: 4. Establishing the duration of all activities;
GAO assessment of USCIS approved OIT schedule: Partially met;
GAO assessment of USCIS approved solutions architect schedule:
Substantially met.
Best practice: 5. Integrating schedule activities horizontally and
vertically;
GAO assessment of USCIS approved OIT schedule: Partially met;
GAO assessment of USCIS approved solutions architect schedule:
Partially met.
Best practice: 6. Establishing the critical path for all activities;
GAO assessment of USCIS approved OIT schedule: Not met;
GAO assessment of USCIS approved solutions architect schedule: Not met.
Best practice: 7. Identifying float between activities;
GAO assessment of USCIS approved OIT schedule: Minimally met;
GAO assessment of USCIS approved solutions architect schedule:
Minimally met.
Best practice: 8. Conducting a schedule risk analysis;
GAO assessment of USCIS approved OIT schedule: Minimally met;
GAO assessment of USCIS approved solutions architect schedule:
Minimally met.
Best practice: 9. Updating the schedule using logic and durations to
determine dates;
GAO assessment of USCIS approved OIT schedule: Minimally met;
GAO assessment of USCIS approved solutions architect schedule:
Minimally met.
Source: GAO analysis of USCIS data and information.
Note: "Not met" means the program provided no evidence that satisfies
any of the criterion. "Minimally met" means the program provided
evidence that satisfies a small portion of the criterion. "Partially
met" means the program provided evidence that satisfies about half of
the criterion. "Substantially met" means the program provided evidence
that satisfies a large portion of the criterion. "Fully met" means the
program provided evidence that completely satisfies the criterion.
[End of table]
As shown above, the two Transformation Program schedules, for the most
part, did not substantially or fully meet the nine best practices. For
example, neither the OIT nor the USCIS-approved solutions architect
schedule contained detailed information for Release A activities
beyond March 2011. In addition, both schedules were missing a
significant number of logic links between activities which indicate
activities that must finish before others and which activities may not
begin until others have been completed.[Footnote 34] While we cannot
generalize these findings to all 18 schedules, our review raises
questions about the reliability of the program's schedules.
Based on our discussions with the Transformation Program's lead
program scheduler, this condition stems, in part, from the
"aggressiveness of the Transformation Program to implement most of the
capabilities within the first 3 years of the program," and a lack of
program management resources for developing knowledge to create and
maintain schedules. Moreover, and regardless of the aggressiveness of
the solution, our best practices call for schedules to reflect all
activities--government, contractors, and any other necessary external
parties--essential for successful program completion. As such, neither
of the Transformation Program schedules we reviewed substantially met
this practice. Furthermore, as demonstrated in the challenges USCIS
has encountered in carrying out the schedule as originally planned,
not including all work for all deliverables, regardless of whether the
deliverables are the responsibility of the government or contractor,
may result in confusion among team members and lead to management
difficulties because of an incomplete understanding of the plan and of
the progress being made.
Collectively, and moving forward, not meeting the nine key practices
increases the risk of schedule slippages and related cost overruns and
makes meaningful measurement and oversight of program status and
progress, as well as accountability for results, difficult to achieve.
For example, in June 2011, a program management review noted a
schedule risk if the development, testing, and deployment process slip
again. This could result in USCIS being unable to deliver the first
release in December 2011. A schedule risk analysis could be used to
determine the level of uncertainty and to help mitigate this risk.
Similarly, capturing and sequencing all activities, as outlined in
best practices, could help identify the extent to which other
activities linked to this schedule risk are affecting its progress.
Furthermore, without the development of a schedule that meets
scheduling best practices, it will be difficult for USCIS to
effectively monitor and oversee the progress of an estimated $1.7
billion to be invested in the acquisition of USCIS ELIS.
In August 2011, TPO provided us the updated USCIS-approved solutions
architect schedule. Program officials indicated that this updated
schedule addressed some areas in which their previous schedule was
deficient according to our assessment of the nine scheduling best
practices. For example, they said that this schedule included
activities through December 2011 rather than only through March 2011.
In addition, officials indicated that they have confidence in meeting
the December 2011 milestone. Specifically, they said that USCIS and
the solutions architect have tested over 70 percent of the Release A
capabilities that are to be released in December 2011, and
demonstrated these capabilities to the USCIS leadership team in August
2011. On the basis of our analysis of the updated USCIS-approved
solutions architect schedule, we determined that the updated schedule
did not address many of the deficiencies we identified in the earlier
version of the schedule. For example, the USCIS-approved solutions
architect schedule did contain activities through December 2011, but
logical links were missing between activities indicating which
activities must finish before others and which activities may not
begin until others have been completed.[Footnote 35] The schedule's
authorized work has therefore not been established in a way that
describes the sequence of work, which prevents the schedule from
meeting other best practices, such as establishing a critical path or
developing a schedule risk analysis. Thus, the updated USCIS-approved
solutions architect schedule does not fully meet all nine key
practices, making meaningful measure and oversight of program status
and progress difficult to achieve. USCIS did not provide us with an
updated OIT schedule; therefore, we were unable to determine to what
extent many of the deficiencies we identified in the earlier versions
were addressed.
Further, USCIS established the Transformation Program as a long-term
program made up of five releases to procure, test, deploy, and
maintain USCIS ELIS, but USCIS officials confirmed in October 2010
that there was no integrated master schedule for the entire
Transformation Program. In addition, the schedules we received in
August 2011 were also not integrated into a master schedule. According
to best practices, an integrated master schedule is to contain the
detailed tasks necessary to ensure program execution and is a required
document to develop key acquisition planning documents under DHS
acquisition management guidance.[Footnote 36] Among other things, best
practices and related federal guidance call for a program schedule to
be programwide in scope, meaning that it should include the integrated
breakdown of the work to be performed by both the government and its
contractors over the expected life of the program.[Footnote 37]
According to program officials, when the Transformation Program's
planning efforts began, USCIS was unable to develop an integrated
master schedule for the Transformation Program due to the complexity
of integrating the numerous individual schedules and the lack of
skilled staff necessary to develop and manage such an integrated
master schedule. In addition, program officials explained that
scheduling software to develop and maintain individual schedules was
not used by every organization performing transformation work, such as
OIT, even though the program issued guidance in August 2010 to all
organizations on scheduling best practices, including the use of
scheduling software.
As an alternative to an integrated master schedule and for ease of
reporting to the ARB and other senior officials, TPO developed a high-
level tracking tool summarizing dates and activities for the first
release of the program and based on individual schedules such as the
OIT and solutions architect schedule, which are not directly managed
by TPO.[Footnote 38] According to program officials, in a September
2010 briefing to agency leadership, this high-level tracking tool
created capacity for USCIS to analyze the schedule. In this briefing,
program officials stated TPO used the high-level tracking tool to
ensure coordination and alignment of activities by collaborating with
staff responsible for the management of individual schedules. However,
this tracking tool is not an integrated master schedule as it does not
integrate all activities necessary to meet the milestones for Release
A; rather, it is a selection of key activities drawn from the
individual schedules maintained by USCIS components and the solutions
architect. Moreover, the Transformation Program Manager expressed
concern in a May 2011 program management review that the information
reported in the high-level tracking tool was not being reported in the
individual schedules. In addition, this tracking tool is not an
integrated master schedule because it does not show activities over
the life of the program. That is, there are no dates or activities for
when the other four releases' set of work activities will occur, how
long they will take, and how they are related to one another. As a
result, it will be difficult for program officials to predict, with
any degree of confidence, how long it will take to complete all five
releases of the Transformation Program. It will also be difficult for
program officials to manage and measure progress in executing the work
needed to deliver the program, thus increasing the risk of cost,
schedule, and performance shortfalls. Lastly, USCIS's ability to
accurately communicate the status of Transformation Program efforts to
key stakeholders such as its employees, Congress and the public will
be hindered.
Unreliable Schedules Affect USCIS's Ability to Develop Reliable Life-
Cycle Cost Estimates:
Because USCIS lacks reliable schedules, its ability to develop
reliable life-cycle cost estimates is hampered. As outlined by DHS
acquisition management guidance, a life-cycle cost estimate is a
required and critical element in the acquisition process. USCIS has
developed and updated the life-cycle cost estimate for the
Transformation Program, but USCIS's individual schedules for the
Transformation Program do not meet best practices for schedule
estimating, thus raising questions about the credibility of the
program's life-cycle cost estimates. For example, neither the OIT nor
the solutions architect schedule fully captured all activities to be
performed by the government and contractor. Therefore, when USCIS is
developing the life-cycle cost estimate there is an incomplete
understanding of the work necessary to accomplish the five releases of
the Transformation Program. Further, in the case of both individual
schedules, the absence of a schedule risk analysis makes it difficult
for officials to account for the cost effects of schedule slippage
when developing the life-cycle cost estimate. Further, a reliable life-
cycle cost estimate is essential for helping the program determine how
much funding is needed and whether it will be available to achieve the
Transformation Program's goals.
Best practices that we have previously identified for cost estimation
state that because some program costs such as labor, supervision,
rented equipment, and facilities cost more if the program takes
longer, a reliable schedule can contribute to an understanding of the
cost impact if the program does not finish on time.[Footnote 39]
Meeting planned milestones and controlling costs are both dependent on
the quality of a program's schedule. An integrated schedule is key to
managing program performance and is necessary for determining what
work remains and the expected cost to complete the work.
Conclusions:
USCIS's effort to develop a modern, automated system for processing
benefit applications and addressing the many current program
inefficiencies has been in progress for nearly 6 years. The program is
now more than 2 years behind its planned deployment schedule for
implementing the agencywide transformed business process, and given
the enormity, significance, and complexity of this transformation, it
is essential that USCIS ensures it takes the proper steps for
implementation. Although only one benefit type is expected to be
available for online account management and adjudication in December
2011, the decision to channel resources and efforts to focus on
ensuring the core businesses are ready for a December 2011 launch
prior to making other application types available for online-
processing appears to be prudent.
Moving forward, it is essential that USCIS consistently follows DHS
acquisition management guidance to best position the department to
develop and share information within the department and with Congress
and the public that can be relied upon for purposes of informed
decision making. Moreover, ensuring that the program's schedules are
consistent with schedule estimating best practices and integrated
through an integrated master schedule would better position USCIS to
reliably estimate the amount of time and effort needed to complete the
program. Reliable schedules could also assist USCIS in developing and
maintaining a complete and reliable life-cycle cost estimate for the
program which is essential for helping the program determine how much
funding is needed and whether it will be available to achieve the
Transformation Program's goals.
Recommendations for Executive Action:
To help ensure that USCIS takes a comprehensive and cost-effective
approach to the development and deployment of transformation efforts
to meet the agency's goals of improved adjudications and customer
services processes, we recommend that the Director of USCIS take the
following three actions:
1. Ensure program schedules are consistent with the nine estimating
best practices.
2. Develop and maintain an Integrated Master Schedule consistent with
these same best practices for the Transformation Program.
3. Ensure that the life-cycle cost estimate is informed by milestones
and associated tasks from reliable schedules that are developed in
accordance with the nine best practices we identified.
Agency Comments and Our Evaluation:
We provided a draft of this report to DHS for comment. DHS provided
written comments, which are reprinted in Appendix II. In commenting on
this report, DHS, including USCIS, concurred with the recommendations.
DHS's letter outlined the actions that USCIS is taking action or has
taken to address each recommendation. Regarding the first
recommendation to ensure program schedules are consistent with best
practices, DHS stated that USCIS is incorporating the nine schedule
estimating best practices we identified into Transformation Program
management reviews, as well as the Acquisition Review Board review.
Regarding the second recommendation to develop and maintain an
Integrated Master Schedule consistent with these same best practices
for the Transformation Program, DHS stated that USCIS will develop an
Integrated Master Schedule to depict the multiple tasks,
implementation activities, and interrelationships needed to
successfully develop and deploy the Transformation Program. Regarding
the third recommendation to ensure that life-cycle cost estimates are
developed in accordance with the nine best practices, DHS stated that
they will refine the Transformation Program life-cycle cost estimate
in accordance with GAO's 12-Step Process for Cost Estimation. In
addition, DHS noted that their revised master schedule will clearly
identify work elements to ensure a reasonable and cost-effective
timeframe for accomplishing the five releases associated with the
program. If fully implemented, we believe that the actions that DHS
identified will address our recommendations.
DHS also provided technical comments, which we have incorporated, as
appropriate.
We are sending copies of this report to the Secretary of Homeland
Security and other interested parties. The report also will be
available at no charge on the GAO Web site at [hyperlink,
http://www.gao.gov].
If you or your staff members have any questions about this report,
please contact me at (202) 512-8777 or stanar@gao.gov. Contact points
for our Offices of Congressional Relations and Public Affairs may be
found on the last page of this report. GAO staff who made major
contributions to this report are listed in appendix III.
Sincerely,
Richard M. Stana:
Director, Homeland Security and Justice Issues:
List of Requesters:
The Honorable Joseph I. Lieberman:
Chairman:
Committee on Homeland Security and Governmental Affairs:
United States Senate:
The Honorable Charles Grassley:
Ranking Member:
Committee on the Judiciary:
United States Senate:
The Honorable Charles E. Schumer:
Chairman:
Subcommittee on Immigration, Refugees and Border Security:
Committee on the Judiciary:
United States Senate:
The Honorable John Conyers, Jr.
Ranking Member:
Committee on the Judiciary:
House of Representatives:
The Honorable Zoe Lofgren:
Ranking Member:
Subcommittee on Immigration Policy and Enforcement:
Committee on the Judiciary:
House of Representatives:
[End of section]
Appendix I: Detailed Results of GAO Assessment of USCIS's Detailed
Transformation Program Schedules:
In prior work, we have identified nine best practices associated with
effective schedule estimating.[Footnote 40] These are (1) capturing
all activities; (2) sequencing all activities; (3) assigning resources
to all activities; (4) establishing the duration of all activities;
(5) integrating activities horizontally and vertically; (6)
establishing the critical path for all activities; (7) identifying
float time between activities; (8) conducting a schedule risk
analysis; and (9) updating the schedule using logic and durations. We
assessed the extent to which two detailed schedules, Office of
Information Technology (OIT) and the solutions architect, dated
November 2010, met each of the nine practices. We characterized
whether the schedules met each of the nine best practices as follows:
* Not met--the program provided no evidence that satisfies any portion
of the criterion.
* Minimally met--the program provided evidence that satisfies less
than one-half of the criterion.
* Partially met--the program provided evidence that satisfies about
one-half of the criterion.
* Substantially met--the program provided evidence that satisfies more
than one-half of the criterion.
* Met--the program provided evidence that satisfies the entire
criterion.
Tables 7 and 8 provide the detailed results of our analysis of these
schedules.
Table 7: Detailed Results of Transformation Program's OIT Schedule
Satisfaction of Scheduling Best Practices:
Best practice: 1. Capturing all activities;
Explanation: The schedule should reflect all activities as defined in
the project's work breakdown structure, which defines in detail the
work necessary to accomplish a project's objectives, including
activities to be performed by the government;
Criterion met (November 2010): Minimally met;
GAO analysis: The schedule only reflects government effort for the
completion of Release A. Because the OIT program schedule does not
account for all planned government work for the five phases of the
program to be fully deployed by 2014, program officials are not able
to reliably estimate planned finish dates beyond the schedule's
current end date of April 20, 2012. Moreover, the schedule does not
include a standardized Work Breakdown Structure (WBS). The WBS should
be the basis of all project schedules. Aligning the schedule to the
program WBS will ensure that the total scope of work is accounted for
within the schedule. These shortcomings in this best practice will
have cascading effects on the remaining best practices. Unless all
activities are accounted for, it is uncertain whether all activities
are properly sequenced, resources are properly assigned, the critical
path is valid, or a Schedule Risk Analysis (SRA) accounts for all
risk. If the schedule does not fully and accurately reflect the
project, it will not serve as an appropriate basis for analysis and
may result in unreliable completion dates, time extension requests,
and delays.
Best practice: 2. Sequencing all activities;
Explanation: The schedule should be planned so that critical project
dates can be met. To meet this objective, activities need to be
logically sequenced--that is, listed in the order in which they are to
be carried out. In particular, activities that must be completed
before other activities can begin (predecessor activities), as well as
activities that cannot begin until other activities are completed
(successor activities), should be identified. This helps ensure that
interdependencies among activities that collectively lead to the
accomplishment of events or milestones can be established and used as
a basis for guiding work and measuring progress;
Criterion met (November 2010): Minimally met;
GAO analysis: Our analysis of the OIT program schedule shows that 377
of the remaining 618 activities (or 61 percent) have missing
predecessor or successor logic. The detailed planning period leading
up to System Definition Review (SDR) shows 233 of the remaining 317
activities (74 percent) have missing predecessor or successor logic.
Missing predecessors or successors reduce the credibility of the
calculated dates. If an activity that has no logical successor slips,
the schedule will not reflect the effect on the critical path, float,
or scheduled start dates of downstream activities. We also found 293
activities with Start No Earlier Than (SNET) constraints. These are
considered "soft" date constraints in that they allow the activity to
slip into the future based on what happens to their predecessor
activities. However, while activities may be soft constrained, for
example, to represent receipt of delivery of equipment, in general
constraining an activity's start date prevents managers from
accomplishing work as soon as possible and consumes flexibility in the
project.
Best practice: 3. Assigning resources to all activities;
Explanation: The schedule should reflect what resources (e.g., labor,
materials, and overhead) are needed to do the work, whether all
required resources will be available when needed, and whether any
funding or time constraints exist;
Criterion met (November 2010): Minimally met;
GAO analysis: According to program officials, resources are not
assigned to activities in the schedule. In lieu of putting resources
in the schedule, program officials said they perform a full-time
equivalents (FTE) projection where they work with each division and
ask them to estimate the number of FTEs. However, assigning resources
to activities in the schedule ensures that resources are used to
determine activity durations because resource requirements may
directly relate to the duration of an activity. Furthermore, if the
current schedule does not allow for insight into current or projected
over-allocation of resources, then the risk of the program slipping is
significantly increased.
Best practice: 4. Establishing the duration of all activities;
Explanation: The schedule should realistically reflect how long each
activity will take to execute. In determining the duration of each
activity, the same rationale, data, and assumptions used for cost
estimating should be used. Durations should be as short as possible
and have specific start and end dates. Excessively long periods needed
to execute an activity should prompt further decomposition of the
activity so that shorter execution durations will result;
Criterion met (November 2010): Partially met;
GAO analysis: The majority of remaining activities in the schedule
meet best practices for durations; however, leading up to SDR, we
found that majority of the remaining activities did not meet best
practices for durations 44 days or less. This does not support program
officials' statement that activities leading up to SDR are detail
planned and with durations of 44 days or less.
Best practice: 5 Integrating schedule activities horizontally and
vertically;
Explanation: The schedule should be horizontally integrated, meaning
that it should link products and outcomes associated with other
sequenced activities. These links are commonly referred to as
"handoffs" and serve to verify that activities are arranged in the
right order to achieve aggregated products or outcomes. The schedule
should also be vertically integrated, meaning that the dates for
starting and completing activities in the integrated master schedule
should be aligned with the dates for supporting tasks and subtasks.
Such mapping or alignment among levels enables different groups to
work to the same master schedule;
Criterion met (November 2010): Partially met;
GAO analysis: Vertical integration is partially demonstrated in the
schedule. While we were able to trace some activities between the OIT
and USCIS high-level schedules, the program's WBS numbers did not
match. In addition, the name and WBS numbers for the SDR activity,
which is a critical milestone necessary for detail planning to
continue, is also not consistent across the OIT, contractor and USCIS
high-level master schedules. Without a standardized WBS, identifying
activities across different schedules is hampered, if not impossible.
Issues with missing dependencies, activities with dangling logic, and
overuse of date constraints prevent the schedule from fully complying
with the requirement of horizontal integration--that is, the overall
ability of the schedule to depict relationships between different
program elements and product handoffs.
Best practice: 6. Establishing the critical path for all activities;
Explanation: Scheduling software should be used to identify the
critical path, which represents the chain of dependent activities with
the longest total duration. Establishing a project's critical path is
necessary to examine the effects of any activity slipping along this
path. Potential problems along or near the critical path should also
be identified and reflected in scheduling the duration of high-risk
activities;
Criterion met (November 2010): Not met;
GAO analysis: Our analysis could not determine a valid critical path
within the schedule, particularly because over 61 percent of remaining
activities have missing or incomplete logic. Unless all activities are
included and properly linked, it is not possible to generate a true
critical path. Program Management Office officials acknowledged that a
critical path cannot be calculated within the schedule.
Best practice: 7. Identifying reasonable float between activities;
Explanation: The schedule should identify the float--the amount of
time by which a predecessor activity can slip before the delay affects
successor activities--so that a schedule's flexibility can be
determined. As a general rule, activities along the critical path have
the least float;
Criterion met (November 2010): Minimally met;
GAO analysis: Our analysis found that float calculations within the
OIT schedule are not reliable because of missing logic links and the
high number of date constraints. In addition, because the schedule is
missing dependencies, float estimates will be miscalculated since
float is directly related to the logical sequencing of events. Because
the critical path is directly related to the amount of float in the
schedule, excessive float will cause an invalid critical path.
Best practice: 8. Conducting a schedule risk analysis;
Explanation: A schedule risk analysis should be performed using
statistical techniques to predict the level of confidence in meeting a
project's completion date. This analysis focuses not only on critical
path activities but also on activities near the critical path, since
they can affect the project's status;
Criterion met (November 2010): Minimally met;
GAO analysis: The agency has not performed a schedule risk analysis on
the schedule. However, program officials provided risk management
documentation that identified the list of top risks, their impacts,
probability of occurrence, mitigation strategy and actions, and risk
status. It appears that USCIS is managing risk, but these program
management reviews do not provide management with the necessary
program information to determine if the program will meet its planned
completion date. Moreover, before a schedule risk analysis can be
performed and produce realistic results, the agency must fix the
missing dependencies, remove the date constraints, break down the long
durations and examine and address unrealistic float values.
Best practice: 9. Updating the schedule using logic and durations to
determine dates;
Explanation: The schedule should be continuously updated using logic
and durations to determine realistic start and completion dates for
program activities. The schedule should be analyzed continuously for
variances to determine when forecasted completion dates differ from
planned dates;
Criterion met (November 2010): Minimally met;
GAO analysis: Our analysis shows that the schedule does not have a
valid status date. Both the initial and updated schedules provided by
the program office showed a status date of July 5, 2011, which was
more than 7 months in the future relative to our November 2010
analysis. A status date denotes the date of the latest update to the
schedule and thus defines the point in time at which completed work
and remaining work are calculated. As a result of this incorrect
status date, we found several activities in the schedule that should
have started in the past with no actual start dates and several
activities that should have finished in the past with no actual finish
dates. As a best practice, maintaining the integrity of the schedule
logic is not only necessary to reflect true status, but is also
required before conducting a schedule risk analysis.
Source: GAO analysis of USCIS data.
[End of table]
Table 8: Detailed Results of Transformation Program's Contractor
Schedule Satisfaction of Scheduling Best Practices:
Best practice: 1. Capturing all activities;
Explanation: The schedule should reflect all activities as defined in
the project's work breakdown structure, which defines in detail the
work necessary to accomplish a project's objectives, including
activities to be performed by both the owner and contractors;
Criterion met (November 2010): Partially met;
GAO analysis: The contractor's schedule accounts for key activities to
be performed by the government and contractor across all five phases,
but officials stated that work beyond SDR is not sufficiently planned
because they were not authorized to do so by the government. Without
an IMS that accounts for all planned effort, management is not able to
reliably estimate planned dates beyond the current schedule's end date
of August 8, 2014. In addition, activities in the schedule are not
aligned with government master schedules via a work breakdown
structure (WBS). Because the contractor activities are not aligned to
a program WBS we cannot be certain that the schedule captures the work
necessary to accomplish the program's objectives.
Best practice: 2. Sequencing all activities;
Explanation: The schedule should be planned so that critical project
dates can be met. To meet this objective, activities need to be
logically sequenced--that is, listed in the order in which they are to
be carried out. In particular, activities that must be completed
before other activities can begin (predecessor activities), as well as
activities that cannot begin until other activities are completed
(successor activities), should be identified. This helps ensure that
interdependencies among activities that collectively lead to the
accomplishment of events or milestones can be established and used as
a basis for guiding work and measuring progress;
Criterion met (November 2010): Minimally met;
GAO analysis: Specifically, 2,812 of the 7,053 activities (40 percent)
have missing logic. The detailed planning period leading up to SDR
shows 587 (38 percent) of the 1,561 remaining activities have missing
logic. The high number of activities with missing logic is cause for
concern because missing predecessors or successors links reduce the
credibility of the calculated dates. We also found 935 activities (13
percent) with Start No Earlier Than (SNET) constraints, including 251
activities (16 percent) with SNET constraints within the detail
planning period. SNET constraints are considered "soft" date
constraints in that they allow the activity to slip into the future
based on what happens to their predecessor activities. However, while
activities may be soft constrained, for example, to represent receipt
of delivery of equipment, in general constraining an activity's start
date prevents managers from accomplishing work as soon as possible and
consumes flexibility in the project. The schedule includes 1,284
activities (18 percent) that are linked to their successor activities
with lags, including 498 negative lags. The detail planning period
leading up to SDR shows 190 activities (12 percent), including 51
negative lags. Lags are often used to put activities on a specific
date or to insert a buffer for risk; negative lags predict when the
successor activity will start prior to the finish of its predecessor;
however, these lags persist even when predecessor activities are
delayed (that is, when the buffer should be consumed). Lags should be
justified because they cannot have risk or uncertainty. There are also
221 activities (3 percent) in the schedule that have dangling logic.
Of these, 82 activities are missing logic that would determine their
start dates and 139 are missing a successor from their finish dates.
The detail planning period leading up to SDR shows 51 activities with
dangling logic, of which 22 are missing logic that would determine
their start and 29 are missing a successor from their finish date.
Regarding activities with dangling logic, activities missing
predecessors to their start date would have to start earlier in order
to finish on time if they ran longer than their planned durations;
and activities missing successors from their finish date could
continue indefinitely and not affect the start or finish dates of
future activities.
Best practice: 3. Assigning resources to all activities;
Explanation: The schedule should reflect what resources (e.g., labor,
materials, and overhead) are needed to do the work, whether all
required resources will be available when needed, and whether any
funding or time constraints exist;
Criterion met (November 2010): Minimally met;
GAO analysis: The schedule does not directly assign resources needed
to complete the captured activities. Instead, the contractor's
resource data are maintained separately as part of its Earned Value
Management (EVM) system. According to program and contract
representatives, the contractor manages resources through detailed EVM
reporting on a monthly basis. Program officials agreed to provide
documentation showing that the contractor has associated control
accounts with schedule activity codes. However, the documentation we
received did not show the mapping between control accounts and
schedule activities. Without this information, we were unable to
reconcile the resource information in the EVM system with activities
planned in the schedule.
Best practice: 4. Establishing the duration of all activities;
Explanation: The schedule should realistically reflect how long each
activity will take to execute. In determining the duration of each
activity, the same rationale, historical data, and assumptions used
for cost estimating should be used. Durations should be as short as
possible and have specific start and end dates. Excessively long
periods needed to execute an activity should prompt further
decomposition of the activity so that shorter execution durations will
result;
Criterion met (November 2010): Substantially met;
GAO analysis: The durations of the majority of remaining activities
(82 percent) met best practices for durations, being less than 44 days
(or 2 working months). In addition, the majority of the long-duration
activities (462 activities or 7 percent), that is, activities longer
than 100 days, occur after SDR in the nondetail planning period.
Program officials stated that all activities prior to SDR are detail
planned and that long duration activities are level of effort (LOE)
activities. However, many of these long duration activities are
included in schedules to represent effort that has no measurable
output and cannot be associated with any one single product. We also
found 150 remaining activities that are scheduled to start on a
Saturday or Sunday. Officials stated these activities are related to
the start and finish dates of specific contract periods;
however, we found that only 14 of the 150 activities were associated
with a 7-day workweek calendar.
Best practice: 5. Integrating schedule activities horizontally and
vertically;
Explanation: The schedule should be horizontally integrated, meaning
that it should link products and outcomes associated with other
sequenced activities. These links are commonly referred to as
"handoffs" and serve to verify that activities are arranged in the
right order to achieve aggregated products or outcomes. The schedule
should also be vertically integrated, meaning that the dates for
starting and completing activities in the integrated master schedule
should be aligned with the dates for supporting tasks and subtasks.
Such mapping or alignment among levels enables different groups to
work to the same master schedule;
Criterion met (November 2010): Partially met;
GAO analysis: Our analysis determined that the schedule is not fully
vertically integrated. While the schedule is vertically integrated
within itself because low-level tasks and milestones are traceable to
higher-level summary tasks, it does not roll up into an overall
government integrated master schedule. In addition, and similar to the
OIT schedule, the name and WBS numbers for the SDR activity, which is
a critical milestone necessary for detail planning to continue, is
also not consistent. Without a standardized WBS, identifying
activities across different schedules is hampered, if not impossible.
The schedule is also not fully horizontally integrated. The horizontal
traceability is hampered due to the issues noted in Best Practice 2
with incomplete logic and reliance on date constraints. Unless the
schedule is fully horizontally integrated, the effects of slipped
tasks on downstream work cannot be determined. Further, when schedules
are not horizontally integrated, relationships between different
program teams cannot be seen and product handoffs cannot be identified.
Best practice: 6. Establishing the critical path for all activities;
Explanation: Scheduling software should be used to identify the
critical path, which represents the chain of dependent activities with
the longest total duration. Establishing a project's critical path is
necessary to examine the effects of any activity slipping along this
path. Potential problems along or near the critical path should also
be identified and reflected in scheduling the duration of high-risk
activities;
Criterion met (November 2010): Not met;
GAO analysis: The schedule does not reflect a valid critical path for
several reasons. First, the schedule does not include all logic links
between activities. Second, there are excessive constraints, lags and
open-ends in the schedule. Unless all activities are included and
properly linked, it is not possible to generate a true critical path.
Without clear insight into a critical path at the project level,
management will not be able to monitor critical or near-critical
detail activities that may have a detrimental impact on downstream
activities if delayed.
Best practice: 7. Identifying reasonable float between activities;
Explanation: The schedule should identify the float--the amount of
time by which a predecessor activity can slip before the delay affects
successor activities--so that a schedule's flexibility can be
determined. As a general rule, activities along the critical path have
the least float;
Criterion met (November 2010): Minimally met;
GAO analysis: We found a relatively high number of remaining
activities 1,282 (18 percent) with negative float ranging from -308
days to -2 days. Negative float means that an activity must be
completed ahead of schedule in order for the overall program to be on
time. However, the float calculations within the contractor schedule
are not reliable because of missing logic links and a high number of
date constraints. In addition, because the critical path is directly
related to the amount of float in the schedule, excessive float will
cause an invalid critical path.
Best practice: 8. Conducting a schedule risk analysis;
Explanation: A schedule risk analysis should be performed using
statistical techniques to predict the level of confidence in meeting a
project's completion date. This analysis focuses not only on critical
path activities but also on activities near the critical path, since
they can affect the project's status;
Criterion met (November 2010): Minimally met;
GAO analysis: Based on documentation provided by program officials and
contract representatives, we found that the contractor has not
performed a schedule risk analysis. Program officials provided
cumulative schedule variance data from their monthly EVM reports and a
Risk Report, which identifies risk. However, these risks are not tied
to activities in the schedule and therefore have no direct impact on
the schedule's forecasted completion date. Moreover, before a schedule
risk analysis can be credible, the program must have a quality
schedule that reflects reliable logic and clearly identifies the
critical path--conditions that the solutions architect schedule does
not meet.
Best practice: 9. Updating the schedule using logic and durations to
determine dates;
Explanation: The schedule should be continuously updated using logic
and durations to determine realistic start and completion dates for
program activities. The schedule should be analyzed continuously for
variances to determine when forecasted completion dates differ from
planned dates;
Criterion met (November 2010): Minimally met;
GAO analysis: Contractor and USCIS officials review the schedule
during the weekly and monthly Program Management Reviews to discuss
updates, identify critical work and ensure schedule coordination. In
addition, the program schedule team, including contractor officials,
provides weekly status updates to management. However, despite these
status updates, data anomalies exist. For example, 107 tasks that
should have started in the past have no actual start dates and 116
tasks that should have finished in the past have no actual finish
dates. As a best practice, the schedule should be continually
monitored to determine when forecasted completion dates differ from
the planned dates, which can be used to determine whether schedule
variances will affect downstream work. Maintaining the integrity of
the schedule logic is not only necessary to reflect true status, but
is also required before conducting a schedule risk analysis.
Source: GAO analysis of USCIS data.
[End of table]
Appendix II: Comments from the Department of Homeland Security:
U.S. Department of Homeland Security:
Washington, DC 20528:
November 16, 2011:
Richard Stana:
Director, Homeland Security and Justice:
U.S. Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Re: Draft Report GAO-12-66, "Immigration Benefits: Consistent
Adherence to DHS's Acquisition Policy Could Help Improve
Transformation Program Outcomes"
Dear Mr. Stana:
Thank you for the opportunity to review and comment on this draft
report. The U.S. Department of Homeland Security (DHS) appreciates the
U.S. Government Accountability Office's (GAO's) work in planning and
conducting its review and issuing this report.
The Department is pleased to note GAO's positive acknowledgment of the
actions DHS has taken in establishing key institutional acquisition
and Information Technology (IT) investment oversight controls and
implementing them in major programs. The Department and The U.S.
Citizenship and Immigration Services (USCIS) are committed to
continuing efforts to instill more discipline and rigor in acquisition
and IT processes.
The draft report contained three recommendations directed at DHS with
which the Department concurs. Specifically, GAO recommended that the
Director of USCIS:
Recommendation 1: Ensure program schedules are consistent with the
nine estimating best practices.
Response: Concur. USCIS is incorporating the nine schedule estimating
best practices into Transformation Program management reviews, as well
as the Acquisition Review Board process administered under Acquisition
Management Directive 102-1, and will work with the Acquisition Program
Management Division to ensure Directive 102-1 reflects focus in its
Directive 102-1 Instructions and Guideline documents. A more detailed
management response describing specific implementation actions USCIS
will take related to each of the nine best practices can be found in
the attachment to this memorandum.
The USCIS Transformation Program will also be transitioning the second
release from a traditional waterfall software methodology (a classical
linear and sequential approach of software design and systems
development) to an agile development methodology (a parallel approach
that emphasizes working in weekly or monthly cycles to quickly
demonstrate working software). An agile process will allow USCIS to
focus on delivering the highest business value in the shortest amount
of time. The features and functions of the program will be identified
and prioritized by the business, and the highest priorities developed
first, enabling the program to produce a consumable solution on a
regular basis and providing greater value to stakeholders. Beginning
in Fiscal Year (FY) 2012, the program will incorporate the agile
project management concepts using cross-functional teams to design,
test, and implement the features and functions. A product-oriented
work breakdown structure and schedule will be developed to (1) capture
the necessary activities for each build, (2) allocate the necessary
resources for each product, (3) establish a duration for each build or
iteration, and (4) identify integration points throughout the process.
By implementing the nine best practices and the agile project
management concepts, the program will ensure consistent schedules.
Recommendation 2: Develop and maintain an Integrated Master Schedule
consistent with these same best practices for the Transformation
program.
Response: Concur. We agree that an Integrated Master Schedule (IMS)
must be developed to depict the multiple tasks, implementation
activities, and interrelationships needed to successfully develop and
deploy the Transformation Program. USCIS will first validate the Work
Breakdown Structure (WBS) as a basis for developing a schedule for
reliable estimates of all program activities. To help ensure no
extended activities exist that may cause slippage in the schedule,
USCIS will analyze and evaluate activities within the duration of the
schedule, further defining more granular activities into a manageable
timeframe, as necessary.
Improvement in the WBS will correct IMS related deficiencies the GAO
identified in the areas of: resource allocation, materials cost,
overhead expenses, interdependencies, missing dependencies, and
"handoffs." The result will be the Transformation Program's improved
ability to meet planned milestones and control program costs.
Successful completion of the program IMS will enable USCIS to adopt an
industry standard tracking tool for purposes of integrating the
numerous individual program schedules into a single IMS. This calendar-
based IMS will be used in program management reviews, Acquisition
Review Boards, and ad hoc meetings and will be capable of identifying
required tasks to support program execution, support risk management
activities, and identify the program's critical path.
Modifying the IMS and using an industry-proven tracking tool will
demonstrate relationships within activities creating dependencies and
further identify and define required pre-requisite activities. IMS
Report Generation will include the "forecasted" versus "actual"
percentage complete in tracking the progress of each activity and
associated performance measures.Periodic analysis of the program's
progress will be performed using the IMS Report, alleviating potential
risks while providing USCIS the ability to immediately mitigate risks
observed "cradle-to-grave" across the total schedule and program.
Recommendation 3: Ensure that the life-cycle cost estimate is informed
by milestones and associated tasks from reliable schedules that are
developed in accordance with the nine best practices identified.
Response: Concur. A refined life-cycle cost estimate (LCCE) will
evolve in accordance with GAO's 12-Step Process for Cost Estimation.
Moreover, in relation to the LCCE, the improved IMS will clearly
identify work elements contributing to the depiction of a reasonable
and cost-effective timeframe for accomplishing the five releases
associated with the program, through 2014 and beyond. Specifically:
* The level of detail within the estimate will be consistent with the
level of detail available for the five releases, and this relationship
will be clear and included in the life cycle costs.
* USCIS will utilize additional cost analysis resources offered at the
Department level to assist with the estimation planning, which will
include the IMS.
* The LCCE will identify requirements, purpose; technical
characteristics (including predecessor or similar systems), physical
characteristics and parameters, relationships to other systems,
development plan and schedule, acquisition strategy, operational plan,
and risk (including technology implications). Moreover, USCIS will
identify any technical, programmatic, or schedule changes that have
taken place to date.
* A clearly defined WBS will be developed to the appropriate level of
detail, at minimum level 3.[Footnote 1] The WBS will be developed in a
standard format for future use and easily aligned to cost analysis
data. All changes to the WBS will be documented in real time, and a
WBS dictionary will be developed.
* All ground rules and assumptions will be documented and contain risk
analysis, identification of possible budget constraints, program
delays, program dependencies, and technology maturity. Moreover,
approval authority will be identified in relation to each action and
this will be vetted within the program as well as with the cost
analyst at the Department level.
* Historical actual cost, schedule, program, technical sources (i.e.,
past invoices related to the program as the basis of cost), and
government personnel costs will additionally be used, and the data
collection method and source will be identified.
* Cost data will be segregated into nonrecurring and recurring costs,
where applicable.
* The cost estimating methodology will be documented to include the
reports to be used on a recurring basis for analysis and statistical
purposes.
* If resources are available, a sensitivity analysis will be conducted
and documented and all outcomes will be evaluated for the parameters
most sensitive to change.
* Risk analysis will be conducted, to include cost drivers (e.g.,
requirements, cost estimation errors, and uncertainty in business,
technology, schedule, program, and software).
* All information, sources, and approvals will be documented,
evaluated for risk, and made available for review and reference, as
appropriate, once finalized.
USCIS is also committed to vetting its cost estimate documentation
with the DHS Cost Analysis Division to demonstrate accuracy of source
data and validity of the LCCE.
Again, thank you for the opportunity to review and comment on this
draft report. General, technical, and sensitivity comments were
previously provided under separate cover. We look forward to working
with you on future Homeland Security issues.
Sincerely,
Signed by:
Jim H. Crumpacker:
Director:
Departmental GAO-OIG Liaison Office:
[End of letter]
DHS Detailed Management Response to GAO's Nine Schedule Estimating
Best Practices:
Best Practice: 1. Capturing all activities;
Description: The schedule should reflect all activities (steps,
events, outcomes, etc.) as defined in the program's work breakdown
structure to include activities to be performed by both the
government and its contractors;
Response: Extend the IMS to include the five phases of the program to
full deployment by 2014. Develop and sequence program activities
within the IMS via use of a Work Breakdown Structure; align the IMS to
ensure total scope of program work is accounted for, resources are
properly assigned, critical path is validated, and a Schedule Risk
Analysis is documented.
Best Practice: 2. Sequencing all activities;
Description: The schedule should be planned so that critical project
dates can be met. To meet this objective, activities need to be
logically sequenced”-that is, listed in the order in which they are to
be carried out. In particular, activities that must be completed
before other activities can begin (predecessor activities), as well as
activities that cannot begin until other activities are completed
(successor activities), should be identified. This helps ensure that
interdependencies among activities that collectively lead to the
accomplishment of events or milestones can be established and used as
a basis for guiding work and measuring progress;
Response: Identify and reconcile interdependencies to avoid program
slippage and improve credibility of schedule.
Best Practice: 3. Assigning resources to all activities;
Description: The schedule should reflect what resources (e.g., labor,
materials, and overhead) are needed to do the work, whether all
required resources will be available when needed, and whether any
funding or time constraints exist;
Response: Utilize project management tools to effectively plan and
allocate resources calculating labor hours, cost of material, and
other project related expenses.
Best Practice: 4. Establishing the duration;
Description: The schedule should reflect of all activities how long
each activity will take to execute. In determining the duration of
each activity, the same rationale, data, and assumptions used for cost
estimating should be used. Durations should be as short as possible
and have specific start and end dates. Excessively long periods needed
to execute an activity should prompt further decomposition of the
activity so that shorter execution durations will result;
Response: Further the use of the WBS to analyze any potential
prolonging in remaining activities by decomposing to make efficient
execution durations.
Best Practice: 5. Integrating schedule activities horizontally and
vertically;
Description: The schedule should be horizontally integrated, meaning
that it should link products and outcomes associated with other
sequenced activities. These links are commonly referred to as
"handoffs" and serve to verify that activities are arranged in the
right order to achieve aggregated products or outcomes. The schedule
should also be vertically integrated, meaning that traceability exists
among varying levels of activities and supporting tasks and sub-tasks.
Such mapping or alignment among levels enables different groups to
work to the same master schedule;
Response: Identify logical and vertical dependencies of tasks and
subtasks, and integrate all other programs schedules for seamless
transitioning.
Best Practice: 6. Establishing the critical path for all activities;
Description: Scheduling software should be used to identify the
critical path, which represents the longest total duration through the
sequenced list of activities. Establishing a project's critical path
is necessary to examine the effects of any activity slipping along
this path. Potential problems along or near the critical path should
also be identified and reflected in scheduling the duration of high-
risk activities;
Response: Adopt the use of a project scheduling tool to upload
schedule dates for the purpose of identifying and resolving critical
path issues.
Best Practice: 7. Identifying reasonable float between activities;
Description: The schedule should identify the float-”the amount of
time by which a predecessor activity can slip before the delay affects
successor activities”-so that a schedule's flexibility can be
determined. As a general rule, activities along the critical path have
the least float;
Response: Adopt the use of a project scheduling tool to upload
schedule dates for the purpose of identifying and adjusting float
estimates.
Best Practice: 8. Conducting a schedule risk analysis;
Description: A schedule risk analysis should be performed using
statistical techniques to predict the level of confidence in meeting a
project's completion date. This analysis focuses not only on critical
path activities but also on activities near the critical path, since
they can affect the project's status;
Response: Correct missing dependencies, remove date constraints,
decompose extended activity durations, analyze, and evaluate float
values as inclusive of developing Schedule Risk Analysis to be
included in the Risk Management documentation.
Best Practice: 9. Updating the schedule using logic and durations to
determine dates;
Description: The schedule should be continuously updated using logic
and durations to determine realistic start and completion dates for
program activities. The schedule should be analyzed continuously for
variance to determine when forecasted completion dates differ from
planned dates;
Response: Review program schedule activities in weekly, monthly, and
ad hoc meetings, ensuring current depiction of "start," "end," "actual
start," "actual end," and variances of planned versus forecasted
dates; ensure accuracy in Status Date.
[End of table]
Appendix II Footnote:
[1] GAO, Cost Estimating and Assessment Guide: Best Practices for
Developing and Managing Capital Program Costs, GAO-09-
35P (Washington, D.C.: March 2009) Figure 10, page 66; "level 3 would
be the lowest level of the breakdown; for others, still lower levels
would be required" page 67.
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
Richard M. Stana at (202) 512-8777 or stanar@gao.gov:
Staff Acknowledgments:
In addition to the contact named above, Mike Dino, Assistant Director;
Kathryn Bernet, Assistant Director; and Carla Brown, Analyst-in-
Charge; managed this assignment. Sylvia Bascopé, Jim Russell, and
Ulyana Panchishin made significant contributions to the work. Nate
Tranquilly and Bill Russell provided expertise on acquisition issues.
Tisha Derricotte and Jason Lee provided expertise on scheduling best
practices. Frances Cook provided legal support. Linda Miller and
Labony Chakraborty provided assistance in report preparation, and
Robert Robinson developed the report's graphics.
[End of section]
Related GAO Products:
Acquisition Planning: Opportunities to Build Strong Foundations for
Better Services Contracts. [hyperlink,
http://www.gao.gov/products/GAO-11-672]. Washington, D.C.: August 9,
2011.
Opportunities to Reduce Potential Duplication in Government Programs,
Save Tax Dollars, and Enhance Revenue. [hyperlink,
http://www.gao.gov/products/GAO-11-318SP]. Washington, D.C.: March 1,
2011.
Secure Boarder Initiative: DHS Needs to Strengthen Management and
Oversight of its Prime Contractor. [hyperlink,
http://www.gao.gov/products/GAO-11-6]. Washington, D.C.: October 18,
2010.
Department of Homeland Security: Assessments of Selected Complex
Acquisitions. [hyperlink, http://www.gao.gov/products/GAO-10-588SP].
Washington, D.C.: June 30, 2010.
GAO Cost Estimating and Assessment Guide: Best Practices for
Developing and Managing Capital Program Costs. [hyperlink,
http://www.gao.gov/products/GAO-09-3SP]. Washington, D.C.: March 2009.
Department of Homeland Security: Billions Invested in Major Programs
Lack Appropriate Oversight. [hyperlink,
http://www.gao.gov/products/GAO-09-29]. Washington, D.C.: November 18,
2008.
Defense Acquisitions: Realistic Business Cases Needed to Execute Navy
Shipbuilding Programs. [hyperlink,
http://www.gao.gov/products/GAO-07-943T]. Washington, D.C.: July 24,
2007.
USCIS Transformation: Improvements to Performance, Human Capital, and
Information Technology Management Needed as Modernization Proceeds.
[hyperlink, http://www.gao.gov/products/GAO-07-1013R]. Washington,
D.C.: July 17, 2007.
Immigration Benefits: Additional Efforts Needed to Help Ensure Alien
Files Are Located when Needed. [hyperlink,
http://www.gao.gov/products/GAO-07-85]. Washington, D.C.: October 27,
2006.
Information Technology: Near-Term Effort to Automate Paper-Based
Immigration Files Needs Planning Improvements. [hyperlink,
http://www.gao.gov/products/GAO-06-375]. Washington, D.C.: March 31,
2006.
Immigration Benefits: Improvements Needed to Address Backlogs and
Ensure Quality of Adjudications. [hyperlink,
http://www.gao.gov/products/GAO-06-20]. Washington, D.C.: November 21,
2005.
Contract Management: Coast Guard's Deepwater Program Needs Increased
Attention to Management and Contractor Oversight. [hyperlink,
http://www.gao.gov/products/GAO-04-380]. Washington, D.C.: March 9,
2004.
[End of section]
Footnotes:
[1] GAO, Immigration Benefits: Additional Efforts Needed to Help
Ensure Alien Files Are Located when Needed, [hyperlink,
http://www.gao.gov/products/GAO-07-85] (Washington, D.C.: Oct. 27,
2006); Information Technology: Near-Term Effort to Automate Paper-
Based Immigration Files Needs Planning Improvements, [hyperlink,
http://www.gao.gov/products/GAO-06-375] (Washington, D.C.: Mar. 31,
2006); and Immigration Benefits: Improvements Needed to Address
Backlogs and Ensure Quality of Adjudications, [hyperlink,
http://www.gao.gov/products/GAO-06-20] (Washington, D.C.: Nov. 21,
2005).
[2] GAO, USCIS Transformation: Improvements to Performance, Human
Capital, and Information Technology Management Needed as Modernization
Proceeds, [hyperlink, http://www.gao.gov/products/GAO-07-1013R]
(Washington, D.C.: July 17, 2007).
[3] A solutions architect is a person or company contracted by an
agency to help with the overall execution and organization of a large-
scale technology development effort.
[4] DHS Acquisition Directive 102-01 established the ARB as a cross-
component within the department that determines whether a proposed
acquisition has met the requirements of key phases in the acquisition
life-cycle framework and is able to proceed to the next phase and
eventual full production and deployment. The board is comprised of the
Acquisition Decision Authority (chair of the ARB); the Under Secretary
for Management; the Under Secretary for Science and Technology; the
Assistant Secretary for Policy; the General Counsel; the Chief
Financial Officer; the Chief Procurement Officer; the Chief
Information Officer; the Chief Human Capital Officer; the Chief
Administrative Officer; the Chief Security Officer; user
representatives from components sponsoring the capability; and other
officials within the department determined to be appropriate to the
subject matter by the Acquisition Decision Authority.
[5] The Acquisition Program Management Division has recently been
replaced by the Program Accountability and Risk Management office.
[6] In March 2009, we published our Cost Estimating and Assessment
Guide that identifies best practices for developing and managing
capital program costs. Agencies can follow the 12-step process which
addresses best practices in cost estimating, including defining the
program's purpose; developing the estimating plan; defining the
program's characteristics; determining the estimating approach;
identifying ground rules and assumptions; obtaining data, developing
the point estimate; conducting sensitivity analysis (examine the
effects of changing assumptions and ground rules); performing a risk
or uncertainty analysis; documenting the estimate; presenting it to
management for approval; and updating it to reflect actual costs and
changes. In addition, within this guide, we have developed nine best
practices for schedule estimating, which, when followed, should result
in reliable and valid schedules that management can use to make
informed decisions. GAO, Cost Estimating and Assessment Guide: Best
Practices for Developing and Managing Capital Program Costs,
[hyperlink, http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.:
Mar. 2009).
[7] The USCIS Program schedule consists of 18 individual program
schedules and one high-level tracking tool that summarizes activities
from these 18 individual schedules, called a "Critical Task Schedule."
To independently determine which schedules to review, we requested all
program schedules from the Transformation Program Office. We reviewed
the schedules and related documentation, and interviewed program
officials on the content of the schedules. We selected two individual
program schedules for assessment: the USCIS Office of Information
Technology schedule and the solutions architect's schedule. We
selected these two schedules for assessment because the work that the
solutions architect and the Office of Information Technology are doing
make up the bulk of the work and they are the most critical drivers
affecting the overall Transformation Program time frames.
Specifically, the Office of Information Technology schedule is a
summary of 10 of the 18 individual schedules and the solutions
architect schedule represents their effort. We did not review the high-
level tracking tool as it did not integrate all activities necessary
to meet the milestones for Release A; rather, the tracking tool was a
selection of key activities drawn from the individual schedules
maintained by USCIS components and the solutions architect.
[8] Within Release A there are to be several phases of deployment. The
first phase of Release A will be deployed to customers applying for an
extension of stay or change of status, which are nonimmigrant
benefits. Subsequent phases of Release A and Release B will
incorporate additional nonimmigrant benefit types.
[9] Nonimmigrant benefits are for individuals seeking to enter the
United States temporarily for a specific purpose, such as tourism or
temporary employment; Immigrant benefits are for foreign nationals
(citizens of another country) seeking to live or work in the United
States permanently; Humanitarian benefits are for persons who are
brought to the United States or are currently in the United States,
who are fleeing persecution, require temporary protection from
removal, or need an extended stay due to emergent circumstances (e.g.,
those placed in Temporary Protected Status, seeking asylum or entering
as refugees, granted significant public benefit parole), as a form of
humanitarian aid, such as those in need of shelter or aid from
disaster, oppression, or other specific circumstances; and,
Citizenship benefits can be granted to a noncitizen who meets certain
eligibility requirements and seeks to become a United States citizen.
[10] There are six program groups associated with the Transformation
Program: Increment Release Management; Business Integration;
Organizational Change Management; Program Management and Integration;
Regulatory, Privacy, and Policy; and Office of Information Technology.
[11] The solutions architect contract was awarded with a 90-day
initial base period and five option periods for a total of 5 years.
[12] Testimony of USCIS Director before the House Subcommittee on
Homeland Security on the U.S. Citizenship and Immigration Services
Fiscal Year 2011 Budget Request; March 16, 2010.
[13] The premium processing fees are part of USCIS's Immigration
Examination Fee Account, and are a fee that certain USCIS customers
pay in addition to the base filing fee. Premium processing guarantees
that USCIS will process an application within 15 days.
[14] System acquisition process means the sequence of acquisition
activities starting from the agency's reconciliation of its mission
need with its capabilities, priorities and resources, and extending
through the introduction of a system into operational use or the
otherwise successful achievement of program objectives. OMB Circular A-
109, Major System Acquisitions.
[15] The department operated under the March 2006 Management Directive
No. 1400, Investment Review Process, until November 2008 when DHS
issued Acquisition Management Directive 102-01, Interim Version, which
superseded Management Directive No. 1400. DHS Acquisition Instruction/
Guidebook 102-01, Interim Version 1.9 (Nov. 7, 2008).
[16] DHS Management Directive No. 102-01, January 20, 2010.
Acquisition Decision Events occur when the Acquisition Review Board
meets to determine whether a program has all of the necessary
acquisition documents and other DHS requirements to move to the next
phase in the acquisition process, such as when a program wants to move
from analyzing and selecting a desired capability (phase 2) to
actually acquiring that capability (phase 3). Key acquisition decision
events are labeled 1, 2A, 2B, and 3. The 2A and 2B acquisition events
may be combined into one acquisition decision event. See figure 2 for
additional details.
[17] Life-cycle costs represent all resources and associated cost
elements required to develop, produce, deploy, and sustain a
particular program from initial concept through operations, support,
and disposal.
[18] According to agency officials, until late August 2010, the DHS
Deputy Secretary was the decision authority and chaired the ARB.
[19] GAO, Opportunities to Reduce Potential Duplication in Government
Programs, Save Tax Dollars, and Enhance Revenue, [hyperlink,
http://www.gao.gov/products/GAO-11-318SP] (Washington, D.C.: Mar. 1,
2011).
[20] GAO, Department of Homeland Security: Assessments of Selected
Complex Acquisitions, [hyperlink,
http://www.gao.gov/products/GAO-10-588SP] (Washington, D.C.: June 30,
2010).
[21] These planning documents were required by the March 2006
Management Directive No. 1400 Investment Review Process and continued
to be required in November 2008 when DHS issued Acquisition Management
Directive 102-01, Interim Version which superseded Management
Directive No. 1400.
[22] GAO, Defense Acquisitions: Realistic Business Cases Needed to
Execute Navy Shipbuilding Programs, [hyperlink,
http://www.gao.gov/products/GAO-07-943T] (Washington, D.C.: July 24,
2007).
[23] GAO, Acquisition Planning: Opportunities to Build Strong
Foundations for Better Services Contracts, [hyperlink,
http://www.gao.gov/products/GAO-11-672] (Washington, D.C.: Aug. 9,
2011).
[24] For example, one key performance parameter of the Transformation
Program is that USCIS ELIS will be able to establish only one account
per identical set of key personal data.
[25] USCIS's Component Acquisition Executive signed the first version
of the USCIS Transformation Program's Operational Requirements
Document on October 21, 2009.
[26] According to program officials, all vendors received a Statement
of Objectives, which enabled vendors to propose widely varying
solutions that could meet the stated performance objectives.
[27] GAO, Contract Management: Coast Guard's Deepwater Program Needs
Increased Attention to Management and Contractor Oversight,
[hyperlink, http://www.gao.gov/products/GAO-04-380] (Washington, D.C.:
Mar. 9, 2004), and [hyperlink, http://www.gao.gov/products/GAO-11-6].
[28] OIT officials explained that TPO had planned for 5 weeks of
Independent Verification and Validation as well as 6 weeks of
Operational Testing and Evaluation. Both of these activities are an
examination of the system and are to be performed by an independent
organization. Since these efforts were to overlap, OIT officials
agreed with TPO officials that including an operational test agent in
the Operational Test and Evaluation process was a duplicative effort
because Independent Verification and Validation was already going to
take place. However, according to DHS acquisition policy, an
operational test agent is a required element in the deployment of a
system.
[29] GAO, Department of Homeland Security: Billions Invested in Major
Programs Lack Appropriate Oversight, [hyperlink,
http://www.gao.gov/products/GAO-09-29] (Washington, D.C.: Nov. 18,
2008).
[30] Per the USCIS planning document, Transformation Program Life
Cycle Cost Estimate, March 2011, version 1.5, the present estimate's
margin of error remained in the low to moderate range despite certain
costs not being included.
[31] The technical environments are physical locations equipped with
hardware and software, and which USCIS is making available for the
solutions architect to conduct production and testing activities prior
to full deployment.
[32] A contract award protest was filed with GAO by a vendor that
competed for the solutions architect contract. This protest was filed
in November 2008, and required that the winning vendor, IBM, stop work
on the contract until the protest was resolved. The vendor withdrew
its protest in December 2008, and the contract moved forward by
January 2009.
[33] [hyperlink, http://www.gao.gov/products/GAO-09-3SP], 218-224.
[34] Our analysis of the OIT program schedule showed 74 percent of
activities were missing predecessor or successor logic. In addition,
the USCIS-approved solutions architect schedule contained 38 percent
of activities with missing logic.
[35] The updated USCIS-approved solutions architect schedule contained
over 40 percent of activities with missing predecessor or successor
logic. Therefore, in both the original and updated USCIS-approved
solutions architect schedules, more than one-third of the remaining
activities are missing logic links.
[36] For example, an acquisition program baseline requires that an
updated integrated master schedule be used to support the schedule
parameters, and a life-cycle cost estimate is to be developed
following GAO's Cost Assessment Guide: Best Practices for Estimating
and Managing Program Costs, which includes the development of an
integrated master schedule.
[37] See, for example, [hyperlink,
http://www.gao.gov/products/GAO-09-3SP] and OMB, Capital Programming
Guide V 2.0, Supplement to the Office of Management and Budget
Circular A-11, Part 7: Planning, Budgeting, and Acquisition of Capital
Assets (Washington, D.C.: June 2006).
[38] USCIS refers to this tracking tool as a Critical Task Schedule.
The Critical Task Schedule consists of the dates for major milestones
and the activities for the Transformation Program, but program
officials stated that the Critical Task Schedule is not an integrated
master schedule.
[39] See [hyperlink, http://www.gao.gov/products/GAO-09-3SP].
[40] GAO Cost Estimating and Assessment Guide, [hyperlink,
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009).
[End of section]
GAO‘s Mission:
The Government Accountability Office, the audit, evaluation, and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the
performance and accountability of the federal government for the
American people. GAO examines the use of public funds; evaluates
federal programs and policies; and provides analyses, recommendations,
and other assistance to help Congress make informed oversight, policy,
and funding decisions. GAO‘s commitment to good government is
reflected in its core values of accountability, integrity, and
reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO‘s website [hyperlink, http://www.gao.gov]. Each
weekday afternoon, GAO posts on its website newly released reports,
testimony, and correspondence. To have GAO e mail you a list of newly
posted products, go to [hyperlink, http://www.gao.gov] and select ’E-
mail Updates.“
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black
and white. Pricing and ordering information is posted on GAO‘s
website, [hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
Connect with GAO:
Connect with GAO on facebook, flickr, twitter, and YouTube.
Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts.
Visit GAO on the web at www.gao.gov.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm];
E-mail: fraudnet@gao.gov;
Automated answering system: (800) 424-5454 or (202) 512-7470.
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov, (202) 512-4400
U.S. Government Accountability Office, 441 G Street NW, Room 7125
Washington, DC 20548.
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, DC 20548.
