2010 Census
Census Bureau Has Made Progress on Schedule and Operational Control Tools, but Needs to Prioritize Remaining System Requirements
Gao ID: GAO-10-59 November 13, 2009
To carry out the decennial census, the U.S. Census Bureau (Bureau) conducts a sequence of thousands of activities and numerous operations. As requested, The Government Accountability Office (GAO) examined (1) the Bureau's use of scheduling tools to maintain and monitor progress and (2) the status of two systems key to field data collection: the control system the Bureau will use to manage the work flow for paper-based operations, including nonresponse follow-up, and the system used to manage quality control of two major field operations. GAO applied schedule analysis tools; reviewed Bureau evaluations, planning documents, and other documents on work flow management; and interviewed Bureau officials.
The Bureau's master schedule provides a useful tool to gauge progress, identify and address potential problems, and promote accountability as the Bureau carries out the census. GAO found that the Bureau's use of its master schedule generally follows leading scheduling practices that enable such high-level oversight. However, errors GAO found in the Bureau's schedule hinder the Bureau's ability to identify the effects of activity delays and to plan for the unexpected. The Bureau has recently begun taking systematic steps to identify and correct remaining errors. However, within its schedule, the Bureau does not identify the resources needed to complete activities, making it difficult for the Bureau to evaluate the costs of schedule changes or the resource constraints that may occur at peak levels of activity. Leveraging the 2010 scheduling experience and including resource needs in the 2020 schedule should facilitate planning for the 2020 Census, already underway. The automated control system that the Bureau plans to use to help manage major field data collection operations has significant development and testing milestones remaining, with some scheduled to finish shortly before the system needs to be deployed. This aggressive schedule leaves little time for resolving problems that may arise, and without prioritized and final software specifications and reliable progress measures, the Bureau may not get what it needs from the system to conduct the operations. Additionally, development of quality control software for two major field operations faces delays, although detailed specifications and test plans are final.
Recommendations
Our recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director:
Team:
Phone:
GAO-10-59, 2010 Census: Census Bureau Has Made Progress on Schedule and Operational Control Tools, but Needs to Prioritize Remaining System Requirements
This is the accessible text file for GAO report number GAO-10-59
entitled '2010 Census: Census Bureau Has Made Progress on Schedule and
Operational Control Tools, but Needs to Prioritize Remaining System
Requirements' which was released on November 13, 2009.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
United States Government Accountability Office:
GAO:
November 2009:
2010 Census:
Census Bureau Has Made Progress on Schedule and Operational Control
Tools, but Needs to Prioritize Remaining System Requirements:
GAO-10-59:
GAO Highlights:
Highlights of GAO-10-59, a report to congressional requesters.
Why GAO Did This Study:
To carry out the decennial census, the U.S. Census Bureau (Bureau)
conducts a sequence of thousands of activities and numerous operations.
As requested, GAO examined (1) the Bureau‘s use of scheduling tools to
maintain and monitor progress and (2) the status of two systems key to
field data collection: the control system the Bureau will use to manage
the work flow for paper-based operations, including nonresponse follow-
up, and the system used to manage quality control of two major field
operations.
GAO applied schedule analysis tools; reviewed Bureau evaluations,
planning documents, and other documents on work flow management; and
interviewed Bureau officials.
What GAO Found:
The Bureau‘s master schedule provides a useful tool to gauge progress,
identify and address potential problems, and promote accountability as
the Bureau carries out the census. GAO found that the Bureau‘s use of
its master schedule generally follows leading scheduling practices that
enable such high-level oversight. However, errors GAO found in the
Bureau‘s schedule hinder the Bureau‘s ability to identify the effects
of activity delays and to plan for the unexpected. The Bureau has
recently begun taking systematic steps to identify and correct
remaining errors. However, within its schedule, the Bureau does not
identify the resources needed to complete activities, making it
difficult for the Bureau to evaluate the costs of schedule changes or
the resource constraints that may occur at peak levels of activity.
Leveraging the 2010 scheduling experience and including resource needs
in the 2020 schedule should facilitate planning for the 2020 Census,
already underway.
The automated control system that the Bureau plans to use to help
manage major field data collection operations has significant
development and testing milestones remaining, with some scheduled to
finish shortly before the system needs to be deployed. This aggressive
schedule leaves little time for resolving problems that may arise, and
without prioritized and final software specifications and reliable
progress measures, the Bureau may not get what it needs from the system
to conduct the operations. Additionally, development of quality control
software for two major field operations faces delays, although detailed
specifications and test plans are final.
Figure: Development and Testing Operations Control System Proceeding on
a Tight Schedule:
[Refer to PDF for image: timeline]
April 1, 2010: Census Day;
December 31, 2010: Delivery of apportionment counts to the President.
Release Number: Release 1;
Operation name and description:
Remote Alaska Enumeration: Field staff visit limited-access households
in Alaska‘s isolated areas before the spring thaw;
Development dates: April-September, 2009;
Testing dates: May-December, 2009;
Operation dates: February-May, 2010.
Release Number: Release 1;
Operation name and description:
Group Quarters Advance Visit: Field staff visit probable group quarters
to confirm location and establish a contact person;
Development dates: April-September, 2009;
Testing dates: May-December, 2009;
Operation dates: February-March, 2010.
Release Number: Release 1;
Operation name and description:
Update/Leave: Field staff update addresses and leave census forms at
housing units in areas that primarily do not have house numbers and/or
street names;
Development dates: April-September, 2009;
Testing dates: May-December, 2009;
Operation dates: March, 2010.
Release Number: Release 1;
Operation name and description:
Enumeration of Transitory Locations: Field staff conduct personal
interviews with individuals who do not have a usual home elsewhere;
Development dates: April-September, 2009;
Testing dates: May-December, 2009;
Operation dates: March-April, 2009.
Release Number: Release 2;
Operation name and description:
Remote Update/Enumerate: Field staff update addresses and complete
census forms in sparsely inhabited areas of Maine and Alaska;
Development dates: September, 2009-January, 2010;
Testing dates: October, 2009-January 2010;
Operation dates: April-June, 2010.
Release Number: Release 2;
Operation name and description:
Update/Enumerate: Field staff update addresses and complete census
forms in areas with special enumeration needs, such as American Indian
reservations and seasonal resort areas;
Development dates: September, 2009-January, 2010;
Testing dates: October, 2009-January 2010;
Operation dates: April-June, 2010.
Release Number: Release 2;
Operation name and description:
Group Quarters Enumeration: Field staff visit group housing such as
prisons and nursing facilities;
Development dates: September, 2009-January, 2010;
Testing dates: October, 2009-January 2010;
Operation dates: April-June, 2010.
Release Number: Release 2;
Operation name and description:
Nonresponse Follow-up: Field staff follow up in person with
nonresponding households;
Development dates: September, 2009-January, 2010;
Testing dates: October, 2009-March 2010;
Operation dates: May-July, 2010.
Release Number: Release 3;
Operation name and description:
Vacant Delete Check: Field staff confirm earlier assessments that a
housing unit is vacant or that it is no longer a housing unit;
Development dates: January-April, 2010;
Testing dates: February-June, 2010;
Operation dates: August-September, 2010.
Release Number: Release 3;
Operation name and description:
Field Verification: Field staff verify the existence of housing units
whose addresses do not match an address in the Bureau‘s master address
file;
Development dates: January-April, 2010;
Testing dates: February-June, 2010;
Operation dates: September-October, 2010.
Source: GAO summary of 2010 U.S. Census Master Activity Schedule
(9/25/2009).
[End of figure]
What GAO Recommends:
GAO recommends, among other things, that the Bureau include resource
needs in its schedule for 2020 and implement reliable progress measures
of the development of a key system.
In commenting on a draft of this report, the Department of Commerce
provided additional details on steps the Bureau is already taking and
plans to take to address the intent of GAO‘s recommendations. GAO
agrees that the monitoring efforts the department describes can help
assess progress on key system development. GAO maintains that without
prioritized requirements and reliable progress measures, the Bureau is
unable to fully gauge its progress on system development.
View [hyperlink, http://www.gao.gov/products/GAO-10-59] or key
components. For more information, contact Robert Goldenkoff at (202)
512-2757 or goldenkoffr@gao.gov.
[End of section]
Contents:
Letter:
Background:
The Bureau Is Successfully Using Schedule Tools to Track the
Implementation of the Census, but Opportunities Exist for Improvement
in 2010 and Beyond:
Bureau Plans to Deliver Two Key Systems on a Tight Schedule:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology for Reviewing the Bureau's Schedule:
Appendix II: Comments from the Department of Commerce:
Appendix III: GAO Contact and Staff Acknowledgments:
Related GAO Products:
Figure:
Figure 1: Development and Testing Schedule for Paper-Based Operations
Control System Is Proceeding on a Tight Schedule:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
November 13, 2009:
The Honorable Thomas R. Carper:
Chairman:
The Honorable John McCain:
Ranking Member:
Subcommittee on Federal Financial Management, Government Information,
Federal Services, and International Security:
Committee on Homeland Security and Governmental Affairs:
United States Senate:
The Honorable William Lacy Clay:
Chairman:
The Honorable Patrick T. McHenry:
Ranking Member:
Subcommittee on Information Policy, Census, and National Archives:
Committee on Oversight and Government Reform:
House of Representatives:
The Honorable Tom Coburn:
United States Senate:
The Constitution mandates that the federal government count the U.S.
population every 10 years--a large and complex undertaking known as the
decennial census. To carry out the census, the U.S. Census Bureau
(Bureau) conducts numerous operations and coordinates a sequence of
thousands of interdependent activities. Given mandated deadlines that
the Bureau faces for delivering census tabulations, the Bureau uses a
master activity schedule to help managers stay on track and alert them
to potential delays. With less than 5 months until Census Day, April 1,
2010, there is little time remaining for the Bureau to deal with any
unexpected problems that may arise, so early recognition of potential
delays is essential.
The largest census field operation is Nonresponse Follow-up (NRFU),
where field staff visit households that have not mailed back their
census forms. The Bureau will rely on a Paper-Based Operations Control
System (PBOCS) to manage this follow-up operation and other activities
scheduled to be completed in the coming months. Field office managers
will use PBOCS to assign work to roughly 1.2 million temporary
employees in about 500 offices nationwide. Among other activities,
field office managers will also track response data collected by field
staff from an estimated 47 million census forms from households that
did not return their forms by mail.
Another key part of managing upcoming census field operations is
ensuring the quality of the data collected. The quality control
operation helps the Bureau determine whether the first interviews held
to collect census data were done correctly, look for workers who may
need additional training, and identify workers who are falsifying
census data. For some operations, the quality control workload will be
determined by matching and coding software that will identify cases
that need follow-up and provide that case selection information to
PBOCS. While the Bureau developed a similar operations control system
for the 2000 Census and has used the core functionality of the matching
software in census-like tests before, neither PBOCS nor the matching
software was used in a "dress rehearsal" for the decennial held in
2008. PBOCS and its integration with the matching software has not been
fully tested in a census-like environment.
Because of the importance of staying on schedule to meet mandated
deadlines, effectively managing field operations, and ensuring the
quality of the data collected, as you requested, we examined (1) the
Bureau's use of scheduling tools to maintain and monitor progress and
(2) the status of two systems critical to conducting field data
collection: the control system the Bureau will use to manage the
workflow for paper-based operations including NRFU, and the system used
to manage quality control of two major field operations.
To meet these objectives, we reviewed Bureau planning documents,
evaluations, schedules, scheduling best practices, and operational and
software specifications. We also interviewed Bureau and contractor
staff regarding the planning, scheduling, and status of selected areas
related to field office workflow management. To address the first
objective, we examined the schedule, comparing its estimates to
relevant best practices and testing its data for reliability using
methods described in detail in appendix I. To address the second
objective, we compared the schedule of Bureau operations to the testing
schedule of systems needed to conduct those operations, examined
previous evaluations of those systems, and evaluated the status of
operational and software specifications. We also reviewed past GAO
reports as appropriate to meet our objectives.
We conducted this performance audit from October 2008 to November 2009
in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.
Background:
The Secretary of Commerce is legally required to (1) conduct the census
on April 1 of the decennial year, (2) report the state population
counts to the President for purposes of congressional apportionment by
December 31 of the decennial year, and (3) send population tabulations
to the states for purposes of redistricting no later than April 1 of
the year following Census Day. The Bureau has defined over 40 different
operations in its high-level requirements document, describing all of
the planned operations and systems needed to meet these mandates.
[Footnote 1]
For the 2010 Census, the Bureau is using a comprehensive master
schedule to integrate the work to be carried out in the dozens of
operations. The schedule provides a high-level roadmap for Bureau
executives and is used to alert executives to activities that are
behind schedule or experiencing issues, allowing problems to be
addressed so the census can continue to proceed on track. Staying on
schedule is crucial to accomplishing all of the tasks involved in
conducting the census. In fact, scheduling and planning are so
important that the Bureau has already established a high-level schedule
for planning the 2020 Census.
While the schedule can be used to manage census operations at a high
level and dictates major time allocations and deadlines, local census
offices across the nation require more detailed plans to conduct
enumeration that exceed the detail included in the master schedule. A
successful census depends, in large part, on the field work carried out
in these local census offices where employees on the ground in local
communities build a list of where to count people and count people who
do not return their census forms. As we have previously reported
[Footnote 2], the Bureau had initially planned to carry out major field
data collection activities using hand held computing devices.
Development and performance problems with the hand held device led the
Secretary of Commerce in April 2008 to abandon using the device for
most of its intended operations and resulted in the Bureau removing the
NRFU operation from the 2008 Dress Rehearsal. As a result, the Bureau
was not able to use the dress rehearsal as a comprehensive end-to-end
test of the interoperability of all of its planned systems, and the
Bureau has had to develop plans to support and conduct the affected
operations on paper as it did for the 2000 Census.
For the 2010 Census, the Bureau will manage remaining fieldwork
activities with PBOCS. This system is intended to provide managers with
essential real-time information, such as worker productivity and
completion rates for field operations. It also allows managers in the
field to assign or reassign cases among workers. If the system does not
work as intended, it could hinder or delay field operations and
introduce errors into files containing collected data.
Another responsibility of field offices is implementing the quality
control process established by the Bureau to ensure that correct
information is collected by field staff and data are not falsified. To
ensure data quality and consistency of quality control procedures, the
Bureau will manage, track, match, and review answers provided during re-
interview operations using its Census Matching Review and Coding System
(Census MaRCS). This system is also to designate quality control
assignments where selected households will be re-interviewed in order
to determine that the original enumerator correctly conducted the
interview. Census MaRCS is also to assist in identifying interviews
where the data from the re-interview do not match the data from the
original interview, indicating that a mistake has been made.
Both PBOCS and Census MaRCS are key systems that have not been fully
tested. Since 2005, we have reported concerns with the Bureau's
management and testing of key information technology systems.[Footnote
3] In March 2009, we reviewed the status of and plans for the testing
of key 2010 Census systems.[Footnote 4] We reported that while the
Bureau has made progress in conducting systems, integration, and end-to-
end testing, critical testing against baseline requirements still
remained to be performed before systems would be ready to support the
2010 Census and the planning for the testing needed much improvement.
While the Bureau has made noteworthy progress in gearing up for the
enumeration, with less than a year remaining until Census Day,
uncertainties surround the Bureau's overall readiness for 2010.
[Footnote 5]
The Bureau Is Successfully Using Schedule Tools to Track the
Implementation of the Census, but Opportunities Exist for Improvement
in 2010 and Beyond:
The Bureau has implemented processes around its master schedule that
comply with a number of scheduling process criteria that are important
to maintaining a schedule that is a useful management tool. Such a
schedule can provide a road map for systematic execution of a program
and the means by which to gauge progress, identify and address
potential problems, and promote accountability. We have documented the
importance of adhering to these criteria and of implementing associated
best practices in our GAO Cost Estimating and Assessment Guide.
[Footnote 6] According to these criteria, a schedule should be:
* comprehensive, with logically sequenced activities spanning the scope
of work to be performed so that the full picture is available to
managers;
* current, with the progress on ongoing activities updated regularly so
that managers can readily know the status of the project; and:
* controlled, with a documented process for changes to the schedule so
that the integrity of the schedule is assured.
The Bureau's master schedule represents all 44 of the operations
described by the broad requirements for the census in its 2010 Census
Operational Plan.[Footnote 7] While the Bureau continues to add
activities to its central schedule, by including at least all the
activities described in these broad requirements, the Bureau is
ensuring that it has a comprehensive schedule that will be less likely
to miss critical interactions between operations. The Bureau ensured a
complete scope of the schedule with input from stakeholders throughout
the agency, with reviews of previous schedules, and building on a
number of census tests during the decade. As a result, the schedule is
the primary source for senior managers, on a weekly basis, to determine
what census activity is ahead of or behind schedule and provides a
resource for determining any impact to the overall project of delays in
major activities.
The Bureau has documented and implemented a formal process for keeping
the data in the schedule current. Staff within each Bureau division are
responsible for ensuring that schedule activities within their division
have their status updated on a weekly basis. Staff update the actual
start and finish dates, the percentage of an activity completed so far,
and estimates of the time remaining to complete each activity in
progress. The Bureau is recording status information on an average of
more than 1,300 activities in the schedule ongoing during any given
week, generating historical data that could provide valuable input to
future schedule estimates.
Finally, the Bureau has implemented a formal change control process
that preserves a baseline of the schedule so that progress can be
meaningfully measured. The Bureau's criteria for justifying changes are
clearly documented and require approval by a team of senior managers
and acknowledgment of the impact by each affected team within the
Bureau. Since the master schedule was baselined in May 2008, about 300
changes have been approved. Even corrections to the schedule for known
errors, such as incorrect links between activities, must be approved
through the change control process, helping to ensure the integrity of
the schedule.
In addition to these practices, the Bureau has positioned itself to
monitor the schedule regularly to help ensure that the census is
progressing and that work is being completed as planned. A central team
of staff working with the schedule implements a process that begins
with the weekly updates of the schedule status and involves subject
matter experts from multiple divisions, and monitors and resolves
schedule-related issues, resulting in a weekly briefing to the Deputy
Director and the Director of the Census, which includes documented
explanations for critical activities scheduled to start late. For
example, the central team began reporting from the master schedule in
March 2009 that the printing of questionnaires for a field operation to
validate locations of group quarters[Footnote 8] in September and
October 2009 might be running late. The schedule showed that late
printing of the questionnaires would trigger their late delivery and
the late assembly of job assistance kits needed to support the
operation, and thus put the timeliness of the operation in danger.
According to a Bureau official, the Bureau then addressed the issue by
deciding to unlink the kit assembly from the questionnaire printing,
allowing kits to begin assembly on time and having questionnaires
delivered directly to field offices when they were ready, letting the
operation begin on time.
Bureau Is Removing Logic Errors from Its Master Schedule to Improve Its
Reliability:
When we began analyzing the Bureau's master schedule, we discovered a
significant number of activities in the schedule that had either
missing or inaccurate information describing their relationships with
other activities in the schedule. We brought these to the Bureau's
attention, and the Bureau has begun systematically identifying such
activities and correcting their information in the schedule.
In accordance with scheduling best practices, activities in the
schedule should be linked logically with relationships to other
activities that precede or follow them, and they should be linked in
the correct order. Since reports that the Bureau uses to manage the
census depend on the schedule having been built properly, inconsistent
adherence to these scheduling practices has occasionally created false
alarms about the schedule and created unnecessary work for those who
have had to resolve them.
In our analysis of the Bureau's schedule, we found that nearly all
relationships between activities are generally in place in the schedule
and some activities in the schedule do not need relationships. However,
many activities appeared in the schedule missing one of their logical
relationships. From January 2009 through August, an average of more
than 1,200 of the more than 11,000 activities in the entire schedule
were missing relationships to other activities from either their start
or end dates. Each month, on average, over 1,100 of the over 6,100 as
yet not completed activities were missing relationships. For example,
within the Bureau's master schedule, an activity listed for receiving
finished materials from the NRFU re-interview operation appeared in the
schedule with no relationship to subsequent activities, making it
appear that any delays in its completion would have no impact on
subsequent census activities. While the absence of such a relationship
in the schedule does not imply that the Bureau would miss the potential
impact of any delays in the completion of this activity, the incidence
of a large number of such missing relationships can confound attempts
to trace the chain of impacts that any delays may have throughout the
schedule.
Similarly, we found a small number of activities in the schedule that
had been linked together in the wrong order, so that one activity might
appear to finish before a necessary prior activity had been completed.
Such an incorrect relationship can unnecessarily complicate the use of
the schedule to guide work or measure progress. The number of such
apparent out-of-sequence activities in the entire schedule has
decreased from on average more than 100 each month in January through
March to 60 in August.
Since June 2009, Bureau staff have been running structured queries on
the data supporting the master schedule to identify activities with
missing or incorrect data; researching each activity to determine what,
if any, corrections to the data are needed; forwarding proposed changes
to affected activities, operation by operation, to program officials
for review; and submitting changes to the Bureau's formal change
control process. The Bureau reports that since this concerted effort
began to correct such errors that affect activities in 37 different
census operations, the Bureau had completed research for activities in
15 of the operations and approved changes in 12 of them by early
October 2009. The Bureau also informed us that this review process
involving the program officials responsible for the logic errors has
provided an educational opportunity for the officials to see how their
programs can directly affect others, and as a result has heightened
awareness about the importance of getting schedule information keyed in
correctly.
Improvements to the Schedule Could Help the Bureau Manage the 2020
Census:
A schedule provides an estimate of how long a given work plan will take
to complete. Since the duration of the work described by the activities
listed in a schedule is generally uncertain, a schedule can be analyzed
for the amount of risk that its underlying work plan is exposed to.
Schedule risk analysis--the systematic analysis of the impact of a
variety of "what if" scenarios--is an established best practice to help
identify areas of a schedule that need additional management attention.
Conducting a schedule risk analysis helps establish the level of
confidence in meeting scheduled completion dates and the amount of
contingency time needed for given levels of confidence, and helps
identify high-priority risks to a schedule. The Bureau is tracking
risks to the census and managing those risks on a regular basis, as
documented in the 2010 Census Risk Management Plan,[Footnote 9] but
these data are not being mapped into the schedule at a level that can
be used for a systematic schedule risk analysis.
A well-defined schedule should help identify the amount of human
capital and financial resources that are needed to execute the programs
within the scope of the schedule, providing a real-time link between
time and cost and helping to reduce uncertainty in cost estimates and
the risk of cost overruns. However, the Bureau does not link within its
schedule estimates of resource requirements--such as labor hours and
materials--to respective activities. Having this information linked in
a schedule enhances an organization's capability to monitor, manage,
and understand resource productivity; plan for the availability of
required resources; and understand and report cost and staffing
requirements. For example, if the Bureau were to find itself behind
schedule with major operations to be completed, and resource
requirements were linked in the schedule, the Bureau could then better
assess the trade-offs between either adding more resources or reducing
the scope of the operations. In addition, when resources are linked to
activities in the schedule, scheduling tools can identify periods of
their peak usage and assist managers with reordering activities to
level out demands on potentially scarce or costly resources. When we
met with Bureau officials and discussed this, they pointed out that
incorporating this schedule best practice would be difficult to do late
in the preparations for 2010, but they expressed interest in
incorporating this schedule best practice as a step forward in the
Bureau's use of the schedule to manage decennial censuses.
Finally, the Bureau's use of a master schedule in 2010 that is,
according to the Bureau, more highly integrated into the management of
the decennial census provides an opportunity to draw many potential
lessons for 2020. The Bureau learned lessons from its use of the 2000
master schedule as documented in a 2003 Bureau management evaluation,
[Footnote 10] in particular with its adoption of the formal change
control process implemented for 2010. Yet as noted in the evaluation,
there were questions about the quality of the data maintained in the
schedule. Without a reliable change control process, the schedule did
not provide a reliable baseline, making evaluation of schedule and
activity duration estimates difficult, if not impossible. The Bureau is
generating a large amount of data--and experience--with its efforts in
developing, maintaining, and using the 2010 master schedule. Unless the
Bureau prioritizes the need for documenting lessons learned from the
current experience--as it did for the 2000 Census--and formally puts in
place an effort to capture and analyze schedule data, changes to
baselines, and variances between estimated and actual durations, it
runs the risk of missing out on another opportunity for using
additional lessons some of its staff may already be learning.
Bureau Plans to Deliver Two Key Systems on a Tight Schedule:
The automated control system that the Bureau plans to use to help
manage the data collection operations of the decennial census still
faces significant development and testing milestones, some of which are
scheduled to be completed just before the system needs to be deployed
before respective field operations begin. As a result, should the
Bureau encounter any significant problems during final testing, there
will be little time to make changes before systems are needed to
support field operations. PBOCS will help manage both paper, and
people, and it needs to exchange data successfully with several other
Bureau systems, such as one used for processing payroll. The Bureau
plans to complete development and testing of PBOCS in three major
releases, grouping the releases of parts of PBOCS together loosely by
the timing of the field operations those parts are needed to support.
The Bureau already completed a preliminary release of PBOCS with
limited functionality in June 2009 to support some initial testing.
Figure 1 shows the development, testing, and operation periods for the
three remaining releases and the operations that PBOCS supports.
According to the baseline of the Bureau's master schedule, PBOCS should
be deployed and operational anywhere from 1 to 6 weeks before each
operation begins for operations leading up to and including NRFU.
According to the Bureau, the system should ideally be ready for use
during training periods so that managers can familiarize themselves
with the system they will have to use and can begin using the system to
assign work to new staff. This also requires that PBOCS be ready in
time so that production data can be loaded into the system, as well as
information about the employees to whom work will be assigned. For
example, PBOCS for NRFU is to finish its final testing in March 2010,
about 9 weeks before NRFU is scheduled to begin on May 1, 2010, and
deployment is scheduled to take place about 6 weeks before NRFU starts,
leaving 3 weeks of contingency time in the event that unexpected
problems arise during PBOCS development. This means that if any
significant problems are identified during the testing phases of PBOCS,
there is generally little time to resolve the problems before the
system needs to be deployed. In addition, it will be more difficult for
the Bureau to integrate into PBOCS training for users on any late
changes in the PBOCS software. While the Bureau relies on last-minute
additions to training and procedures documents to communicate late
changes to workers, Bureau officials agreed that it can be difficult to
incorporate such last-minute additions into training sessions and for
users to learn them, and doing so should be avoided if possible.
Figure 1: Development and Testing Schedule for Paper-Based Operations
Control System Is Proceeding on a Tight Schedule:
[Refer to PDF for image: timeline]
April 1, 2010: Census Day;
December 31, 2010: Delivery of apportionment counts to the President.
Release Number: Release 1;
Operation name and description:
Remote Alaska Enumeration: Field staff visit limited-access households
in Alaska‘s isolated areas before the spring thaw;
Development dates: April-September, 2009;
Testing dates: May-December, 2009;
Operation dates: February-May, 2010.
Release Number: Release 1;
Operation name and description:
Group Quarters Advance Visit: Field staff visit probable group quarters
to confirm location and establish a contact person;
Development dates: April-September, 2009;
Testing dates: May-December, 2009;
Operation dates: February-March, 2010.
Release Number: Release 1;
Operation name and description:
Update/Leave: Field staff update addresses and leave census forms at
housing units in areas that primarily do not have house numbers and/or
street names;
Development dates: April-September, 2009;
Testing dates: May-December, 2009;
Operation dates: March, 2010.
Release Number: Release 1;
Operation name and description:
Enumeration of Transitory Locations: Field staff conduct personal
interviews with individuals who do not have a usual home elsewhere;
Development dates: April-September, 2009;
Testing dates: May-December, 2009;
Operation dates: March-April, 2009.
Release Number: Release 2;
Operation name and description:
Remote Update/Enumerate: Field staff update addresses and complete
census forms in sparsely inhabited areas of Maine and Alaska;
Development dates: September, 2009-January, 2010;
Testing dates: October, 2009-January 2010;
Operation dates: April-June, 2010.
Release Number: Release 2;
Operation name and description:
Update/Enumerate: Field staff update addresses and complete census
forms in areas with special enumeration needs, such as American Indian
reservations and seasonal resort areas;
Development dates: September, 2009-January, 2010;
Testing dates: October, 2009-January 2010;
Operation dates: April-June, 2010.
Release Number: Release 2;
Operation name and description:
Group Quarters Enumeration: Field staff visit group housing such as
prisons and nursing facilities;
Development dates: September, 2009-January, 2010;
Testing dates: October, 2009-January 2010;
Operation dates: April-June, 2010.
Release Number: Release 2;
Operation name and description:
Nonresponse Follow-up: Field staff follow up in person with
nonresponding households;
Development dates: September, 2009-January, 2010;
Testing dates: October, 2009-March 2010;
Operation dates: May-July, 2010.
Release Number: Release 3;
Operation name and description:
Vacant Delete Check: Field staff confirm earlier assessments that a
housing unit is vacant or that it is no longer a housing unit;
Development dates: January-April, 2010;
Testing dates: February-June, 2010;
Operation dates: August-September, 2010.
Release Number: Release 3;
Operation name and description:
Field Verification: Field staff verify the existence of housing units
whose addresses do not match an address in the Bureau‘s master address
file;
Development dates: January-April, 2010;
Testing dates: February-June, 2010;
Operation dates: September-October, 2010.
Source: GAO summary of 2010 U.S. Census Master Activity Schedule
(9/25/2009).
[End of figure]
The Bureau also faces the significant challenge of developing the
detailed specifications for the software to be developed. As of early-
September 2009, the Bureau had established high-level requirements for
PBOCS and has reported completing development of release 1 of PBOCS.
The Bureau reports that as of late-October its requirements
development, system development, and system testing for phase 1 is
largely completed. However, the Bureau has not yet finalized the
detailed requirements for this release or for later releases. High-
level requirements describe in general terms what functions the system
will accomplish, such as producing specific management reports on the
progress of specific paper-based operations. Detailed requirements
describe more specifically what needs to be done in order to accomplish
such functions. For example, a detailed requirement would specify the
specific data that should be pulled from the specific data set to
produce specific columns of a specific report. While high-level
requirements provide software programmers with general guidelines on,
for example, what types of reports should be produced, without a clear
understanding of the detailed requirements, the programmers cannot be
sure that they are identifying the correct source of information for
producing such reports, and reports can thus be inaccurate. According
to Bureau officials, previous contract programmers with little
decennial census experience and no involvement with current development
efforts made erroneous assumptions about which data to use when
preparing some quality control reports that became problematic in the
dress rehearsal.
Without detailed requirements, the Bureau also cannot be sure how
frequently such reports should be updated or which staff should have
access to which reports. Further, software developers may not have the
required information to meet the Bureau's needs. Also, as we have
previously reported, detailed operational requirements determine system
development, and without well-defined requirements, systems are at risk
of cost increases, schedule delays, or performance shortfalls. As we
have reported and testified numerous times, the Bureau experienced this
with an earlier contract to automate the support of its field data
collection activity, which included the failed handheld computing
device.[Footnote 11]
The Bureau's PBOCS development managers have told us that they are
working closely with stakeholders in an iterative process of short
development cycles to help mitigate PBOCS development risks caused by
not having detailed requirements written in advance. Embedding subject
matter experts within the software development process can help
mitigate risk inherent in the short time frame the Bureau has remaining
to develop and test PBOCS. Yet the absence of well-documented and
prioritized detailed requirements for PBOCS, which still need to be
developed and tested, remains among the most significant risks to
getting PBOCS ready on time. Furthermore, the Bureau lacks reliable
development progress measures that permit estimating which requirements
may not get addressed and that are important to ensuring the visibility
of the development program to Bureau leadership. Aggressive monitoring
of system development and testing progress and of the effort remaining
will help ensure that program officials who will rely on these systems
can anticipate what risks they face and what mitigation activities they
may need for shortfalls in the final systems.
In recognition of the serious implications that a failed PBOCS would
have for the conduct of the 2010 Census, and to see whether there were
additional steps that could be taken to mitigate the outstanding risks
to successful PBOCS development and testing, in June 2009 the Bureau
chartered an assessment of PBOCS. The assessment team, chaired by the
Bureau's Chief Information Officer (CIO), reported initially in late
July 2009 and provided an update report in late August 2009. According
to the August update and our discussion of it with the CIO, the team
increased its risk rating in two areas of PBOCS that it is monitoring
in part because of the absence of fully documented requirements,
testing plans, progress measures, and deployment plans.
In its comments on the draft of this report, the Department of Commerce
provided information describing several steps the Bureau was taking to
monitor the progress of PBOCS development. According to Commerce, the
Bureau already does the following:
* Daily Project Management Standup meetings, which cover action item
management, calendar review, activity sequencing, and any threats or
action-blocking issues.
* Daily Architecture Review Board and team leads meetings.
*Weekly Program Management Review Board meetings, and thrice-weekly
Product Architecture Review Board meetings.
* At least weekly review of progress by the PBOCS Internal Assessment
Team--chaired by the CIO. The team briefs the Bureau Director at least
monthly.
* Monthly Quality Assurance Board meetings and twice-monthly Risk
Review Board meetings.
At the end of our review, the PBOCS development team demonstrated two
software tools it said it was using to help manage its iterative
process of short development cycles. We did not fully assess their use
of the tools. The progress measures they demonstrated with one of the
tools predicted that the completion dates would be missed. It also
showed that the development team was underestimating the development
effort required to achieve its iterative development goals. When we
noted this, the presenters told us that the information in their
management system was not current. Until the Bureau completes the
detailed requirements for PBOCS and prioritizes them and its PBOCS
development monitoring is relying on current and reliable progress
measures, such as those the development team attempted to demonstrate
to us for estimating the effort needed to complete remaining
development, it will not be able to fully gauge the PBOCS development
progress and have reasonable assurance that PBOCS will meet the
program's needs. The Bureau is continuing to examine how improvements
will be made.
Quality Control Matching and Coding Software for Nonreponse Follow-up
and Update/Enumerate Faces Testing Delays and Revisions to
Requirements:
The Bureau has experienced delays in the development and testing of
software that will play a key role along with PBOCS in controlling and
managing field data collection activity for the quality assurance
programs of NRFU and Update/Enumerate. Census MaRCS will help manage
the process of identifying systematic or regular violations in the door-
to-door data collection procedures. In particular, Census MaRCS will be
a tool to help target additional households needing reinterview as part
of the quality assurance program for these two major census data
collection operations in the field. Therefore, fully developing and
testing Census MaRCS will be important to the successful conduct of the
census.
Like other systems at the Bureau, Census MaRCS had to undergo design
changes when the Bureau made the April 2008 decision to switch to paper-
based operations. Detailed performance requirements--such as the
information to be included on reports and its sources, including
performance metrics, such as the number of users the system is designed
to handle--are documented and were baselined in May 2009. However,
specifications have been added or clarified as software development has
progressed and improvements have been suggested.
Software development has at times been slower than expected, leading to
delays in some testing. Test plans for Census MaRCS software and
interfaces are in place, having been documented in May 2009. According
to those plans, the Bureau is in the second of three stages of testing
Census MaRCS and is scheduled to complete its final stage in December
2009, almost 2 months before its first deployment for the
Update/Enumerate operation. Slower-than-expected development has
delayed some parts of the second phase of testing, which will thus
finish late according to the Bureau, but Bureau officials have
indicated that they believe that delay can be absorbed into the
schedule, and that the system will be delivered as scheduled in
February 2010.
The compressed testing schedule leaves little time for additional
delays in writing software or conducting tests. The Bureau is working
on additional plans to test the interfaces between systems like PBOCS
and Census MaRCS to ensure that they work together, but those test
plans have not yet been finalized. Since Census MaRCS was not used in
the dress rehearsal, and a full end-to-end system test--that is, a test
of whether all interrelated systems collectively work together as
intended in an operational environment--is not planned for in the time
remaining before the system is required to be deployed, successful
testing of the interfaces with other systems is critical to the
system's readiness. If development or testing delays persist, it will
be more important than ever that system requirements be prioritized so
that effort is spent on the "must haves" necessary for system
operation.
Conclusions:
Our review of the Bureau's master schedule for conducting the 2010
Census and the processes the Bureau uses to manage it suggests that it
is doing a commendable job conducting such a large and complex
undertaking consistent with leading scheduling practices. Furthermore,
the Bureau's systematic effort to correct errors that we have
identified in the schedule will further improve the ability of the
master schedule to support senior management oversight and decision
making as 2010 approaches. Other improvements, such as embedding
estimates of resource needs into the schedule, may take more time to
implement. Yet, the Bureau's generally well-defined and integrated
schedule provides an essential road map for the systematic execution of
the census and the means by which to gauge progress, identify and
address potential problems, and promote accountability. Leveraging the
Bureau's experience with scheduling for 2010 by documenting it should
provide lessons learned for similar efforts in 2020 as well.
Moreover, since we testified on the status of the 2010 Census in March
2009,[Footnote 12] the Bureau has made progress on a number of key
elements needed to manage the work flow in field operations. In
particular, the Bureau has made progress in developing and testing
systems to support paper-based operations in the wake of the Secretary
of Commerce's April 2008 decision to switch to paper-based operations
for most field data collection activity. That said, some delays are
also occurring, and since so much still remains to be done in the
months leading up to Census Day, the Bureau has limited time to fix any
potential problems that arise in systems that are not thoroughly
tested.
While the Bureau has made significant progress in developing test
schedules for key systems, careful monitoring of the progress in
addressing, and setting priorities among, the remaining detailed
requirements for the control system supporting paper-based operations
is critical for the Bureau to anticipate what risks it faces and
mitigations it may need for shortfalls in the final system. Based on
the challenges faced in the earlier program implementing handheld
computing devices, the Bureau has already experienced the ill effect of
having to change its plans when a system does not fully meet planned
program needs. With limited time before implementation, it is uncertain
whether the Bureau may be able to complete development and fully test
all key aspects of its systems, like PBOCS and Census MaRCS, which are
still under development. Continued aggressive monitoring by the Bureau
and improvements in the progress measures on system development and
testing, of effort remaining, and of the risks relating to these
efforts is needed. Such effort will help ensure that Bureau leadership,
as well as Bureau program officials who will rely on these systems,
have early warning on what, if any, desired system features will be
unavailable in the final systems, maximizing time available to
implement mitigation strategies as needed.
Recommendations for Executive Action:
We recommend that the Secretary of Commerce require the Director of the
U.S. Census Bureau to take the following three actions:
To improve the Bureau's use of its master schedule to manage the 2020
decennial census:
* Include estimates of the resources, such as labor, materials, and
overhead costs, in the 2020 integrated schedule for each activity as
the schedule is built, and prepare to carry out other steps as
necessary to conduct systematic schedule risk analyses on the 2020
schedule.
* Take steps necessary to evaluate the accuracy of the Bureau's
baselined schedule and determine what improvements to the Bureau's
schedule development and management processes can be made for 2020.
To improve the Bureau's ability to manage paper-based field operations
in the 2010 Decennial Census,
* finalize and prioritize detailed requirements and implement reliable
progress reporting on the development of the paper-based operations
control system, including estimates of effort needed to complete
remaining development.
Agency Comments and Our Evaluation:
The Secretary of Commerce provided written comments on a draft of this
report on November 3, 2009. The comments are reprinted in appendix II.
Commerce did not comment on the first two recommendations, but provided
additional information on steps it had already been taking to monitor
progress of PBOCS development related to the third recommendation.
Commerce commented on how we characterized the status of system
development and testing, and provided additional statements about the
status. Commerce also made some suggestions where additional context or
clarification was needed, and where appropriate we made those changes.
With respect to our third recommendation to finalize and prioritize
detailed requirements and implement reliable progress reporting on
PBOCS, including estimates of effort needed to complete remaining
development, Commerce described numerous regular meetings that the
development team holds, as well as efforts by others in the Bureau to
monitor and report on PBOCS development progress. We added references
to these monitoring efforts within the report. We agree that the
monitoring efforts the department describes can help assess the
progress being made; however, the monitoring efforts can only assess
the progress being reported to them. First, a complete set of
requirements is needed to understand the work that has been
accomplished and the work remaining. As we have noted, the Bureau has
not yet developed a full set of requirements. Second, the management
tool's measurement of the development activity successfully addressing
the requirements is directly related to the effectiveness of test cases
developed for those requirements. However, as we have previously noted,
if a requirement has not been adequately defined, it is unlikely that a
test will discover a defect. Accordingly, until the Bureau completes
the detailed requirements for PBOCS and prioritizes them it is unable
to use these tools to fully gauge its progress toward meeting the
overall project's goals and objectives of system development. We
clarified our discussion of this in the report and reworded the
recommendation to better focus on the need for reliable information.
Commerce maintained that our draft report implied that no PBOCS
development had begun and that no testing would be completed until
March 2010. The draft report described development and testing dates
that clearly illustrate that both development and testing have been
occurring over several months leading up to their conclusion. We have
included additional language in the text to clarify that development
and testing take place over a period of time. We have also included a
statement from Commerce that much of this activity has largely been
completed for the first of three phases.
Commerce commented on the accuracy of the dates used in the figures in
the draft report. We verified that the dates we used were the correct
dates that the Bureau had provided to us earlier. We made minor
adjustments to some of the gridlines in the graphic for presentation
purposes.
The Bureau also provided additional information on the prior testing of
matching software, the context for NRFU being dropped from the dress
rehearsal, how PBOCS errors could potentially introduce errors into
census files, and contract programmers we reported being involved in
dress rehearsal PBOCS not being the same ones helping the Bureau with
PBOCS development now. We revised the report as appropriate in
response.
Commerce commented on our discussion of the unreliability of
information that the Bureau PBOCS development team provided us during a
demonstration of two tools that it uses to help manage its iterative
process of short development cycles. Commerce described the use of one
of the two tools but not the one whose progress tracking measures were
demonstrated to us and for which the team told us that data were not
current. We have revised the text to more clearly state that more than
one tool was used to demonstrate how development and testing was being
managed by the PBOCS development team, and we have added additional
language to describe the progress information that should have been
current but that was not.
Finally, as we had noted in the draft report, the Bureau has already
begun taking action to address errors we had identified in its master
schedule. Since we sent a draft of this report to Commerce, the Bureau
provided additional information on the status of that effort, and we
have updated this report accordingly.
We are sending copies of this report to the Secretary of Commerce, the
Director of the U.S. Census Bureau, and interested congressional
committees. The report also is available at no charge on GAO's Web site
at [hyperlink, http://www.gao.gov].
If you have any questions about this report please contact me at (202)
512-2757 or goldenkoffr@gao.gov. Contact points for our Offices of
Congressional Relations and Public Affairs may be found on the last
page of this report. GAO staff who made major contributions to this
report are listed in appendix III.
Signed by:
Robert Goldenkoff:
Director Strategic Issues:
[End of section]
Appendix I: Scope and Methodology for Reviewing the Bureau's Schedule:
We reviewed the U.S. Census Bureau (Bureau) program's schedule
estimates and compared them with relevant best practices in GAO Cost
Estimating and Assessment Guide: Best Practices for Developing and
Managing Capital Program Cost[Footnote 13] to determine the extent to
which they reflect key practices that are fundamental to having a
reliable schedule. These practices address whether the schedule is:
* comprehensive, with logically sequenced activities spanning the scope
of work to be performed so that the full picture is available to
managers;
* current, with progress on ongoing activities updated regularly so
that managers can readily know the status of the project; and:
* controlled, with a documented process for changes to the schedule so
that the integrity of the schedule is ensured.
In doing so, we independently assessed a copy of the program's
integrated master schedule and its underlying schedules against our
best practices. We also interviewed knowledgeable program officials to
discuss their use of best practices in creating the program's current
schedule and we attended a schedule walk-through to better understand
how the schedule was constructed and maintained.
We tested the Bureau's schedule data for reliability by:
* running a schedule check report in Pertmaster which is a scheduling
analysis software tool that identifies missing logic, constraints, and
so forth;
* using the schedule information from Pertmaster, copying the schedule
data into Excel, and checking for specific problems that could hinder
the schedule's ability to dynamically respond to changes;
* examining whether there were any open-ended activities (i.e.,
activities with no predecessors, successors, or both);
* searching for activities with poor logic;
* identifying whether there were any lags or leads that should only be
used to show how two tasks interact and not to represent work;
* determining if activities were resource loaded, which helps to cost
out the schedule and examine whether resources are overstretched or not
available when needed;
* examining whether the schedule was baselined, when it had its status
updated, and what deviations there were from the plan; and:
* examining if there were any actual start or finish dates recorded in
the future and whether there was any broken logic.
[End of section]
Appendix II: Comments from the Department of Commerce:
United States Department Of Commerce:
The Secretary of Commerce:
Washington, D.C. 20230:
November 2, 2009:
Mr. Robert Goldenkoff:
Director, Strategic Issues:
Government Accountability Office:
Washington, DC 20548:
Dear Mr. Goldenkoff:
I appreciate the opportunity to comment on the U.S. Government
Accountability Office's draft report, "2010 Census: Census Bureau Has
Made Progress on Schedule and Operational Control Tools, but Needs to
Prioritize Remaining System Requirements" (GAO-10-59).
The Department of Commerce's comments on this report are enclosed.
Sincerely,
Signed by:
Gary Locke:
Enclosure:
[End of letter]
U.S. Department of Commerce:
Comments on the United States Government Accountability Office Draft
Report Entitled "2010 Census – Census Bureau Has Made Progress on
Schedule and Operation Control Tools, but Needs to Prioritize Remaining
System Requirements" (GAO-10-59):
October 23, 2009:
The U.S. Census Bureau appreciates the Government Accountability
Office's (GAO) efforts in examining our 2010 Census schedule and
operational control system, and for the opportunity to comment on the
observations and recommendations in this report.
We have several comments about statements and conclusions in this
report, and specifically about the recommendation to finalize and
prioritize requirements”and to monitor progress”for development of the
Paper-Based Operational Control System (PBOCS) for the 2010 Census.
In the fall of 2008, the Census Bureau made a decision to de-scope
PBOCS development from the Harris contract, and to instead have
experienced Census Bureau staff take over the development and testing
of this system. We recognized at that time, and have since been fully
candid about, the challenging nature of such a takeover.
Thus, regarding the discussion beginning at the top of page 14 and the
recommendation on page 17 concerning the Census Bureau's need to
monitor the PBOCS development process, we believe that we are doing so,
as follows:
* On a daily basis, we conduct Project Management Standup Meetings,
which cover action item management, calendar review, activity
sequencing, and any threats or blocking issues. We also conduct daily
Architecture Review Board and Team Leads meetings.
* On a weekly basis, we conduct Program Management Review Board
Meetings, which cover Program Quad Analysis, Schedule Status, Exception
and Issue Management, and Opportunities/Threats Management. Three times
each week, we conduct Product Architecture Review Board Meetings. We
also conduct weekly Schedule Reporting and Schedule Exception
Management meetings.
* At least weekly, the PBOCS Internal Assessment Team”chaired by the
Census Bureau's Associate Director for Information Technology (and
Chief Information Officer)”reviews progress on these matters. At least
monthly, this team briefs the Census Bureau's Director on progress and
issues.
* Risk Review Board meetings are conducted twice a month, and Quality
Assurance Board meetings are held each month.
* On a quarterly basis, PBOCS progress and issues are reviewed by the
Census Integration Group Management Review.
Also, regarding the discussion on pages 11, 13, and 14 concerning the
completion of testing, we note that we will be finishing beta testing
(the final stage of testing) in March 2010, but we will have already
finished development, alpha testing, and user testing. As written, the
report implies that no testing will be completed until March.
Similarly, at the bottom of page 12, the report states that we face a
challenge developing detailed specifications for PBOCS. It is important
to note that much of this has been done. As written, the sentence
implies that we have not yet begun the development process.
Finally, we believe that the characterization of our requirement
process on pages 12-13 is an understatement of our progress and current
status. The requirements development is largely completed, as is much
of the system development and testing.
We also have the following comments on some specific statements in the
report:
* The chart on the Highlights page, repeated on page 12, is incorrect.
We provided a revised version of this information to the GAO on
September 29, 2009, and we request that the chart and related
discussions be updated before this report is issued.
* The statement that the MaRCS has not been fully tested in a census-
like environment is inaccurate. In fact, the MaRCS application was used
in both the 2004 and 2006 Census tests. In those tests, the NRFU
operation was automated using a handheld computer, but the core
functionality of the MaRCS software remains the same for automated and
paper operations. In 2006, we also used the MaRCS for the
Update/Enumerate operation, which was a paper operation.
The biggest challenge is the interface of MaRCS with the DRIS and PBOCS
systems that were introduced as a result of the switch to paper NRFU.
We have similar comments about the last paragraph on page 15: the
second sentence should be revised to say, "Test plans for Census MaRCS
software and interfaces are in place, having been documented in May
2009." The third sentence should be changed to, "According to those
plans, the Bureau is in the second of three stages of testing Census
MaRCS and is scheduled to complete its final stage, including testing
of interfaces, in December 2009 (two months prior to the first
deployment of Census MaRCS for the Update/Enumerate operation)."
* The discussion at the bottom of page 3 is incorrect as to why we
dropped many Dress Rehearsal operations. Most of these operations were
dropped when we did not receive funding under the Continuing Resolution
at the beginning of FY 2008 (October 2007). By April 2008, when the
Secretary of Commerce made the decision to drop the use of handheld
computers (HHCs) for the 2010 Census NRFU operation, we had completed
the Dress Rehearsal Address Canvassing operation using HHCs. The NRFU
operation was the only Dress Rehearsal operation still to be done using
HHCs, and was thus the only operation dropped once we stopped
development of the HHC approach.
* Please clarify the statement in the first paragraph on page 4 that
says that if PBOCS does not work, this can introduce errors in data
collection. We agree that the operation could be delayed if the system
is not working, but we do not agree that this could introduce errors in
the data that we collect during the operations.
* On page 13, the statement attributed to Bureau officials about
contract programmers was, a reference to Harris staff, not to the
Census Bureau staff now leading the development teams or to the
contract programmers hired to assist them and working under Census
Bureau supervision. We de-scoped the PBOCS from Harris's contract and
returned development of that system to experienced Census Bureau
developers and managers. As written, this sentence incorrectly implies
that these errors were made by contractors hired to assist the Census
Bureau-led PBOCS development.
* Regarding the discussion on page 14 of incomplete information used
for the mid-September demo for GAO of our tool (Mercury Quality Center)
used to manage and prioritize requirements, the data in our Mercury
Quality Center is populated and updated on an ongoing basis as we move
through the development life cycle. The data were current as of the
demonstration; however, because we are still developing the PBOCS, the
data would necessarily be incomplete.
In conclusion, we want to acknowledge the GAO's extensive work in
reviewing these activities and appreciate their ongoing efforts to help
us implement a successful 2010 Census.
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
Robert Goldenkoff, (202) 512-2757 or goldenkoffr@gao.gov:
Acknowledgments:
In addition to the contact named above, Ty Mitchell, Assistant
Director; Virginia Chanley; Vijay D'Souza; Jason Lee; Andrea Levine;
Donna Miller; Crystal Robinson; Jessica Thomsen; Jonathon Ticehurst;
and Katherine Wulff made key contributions to this report.
Related GAO Products:
2010 Census: Census Bureau Continues to Make Progress in Mitigating
Risks to a Successful Enumeration, but Still Faces Various Challenges.
[hyperlink, http://www.gao.gov/products/GAO-10-132T]. Washington, D.C.:
October 7, 2009.
2010 Census: Fundamental Building Blocks of a Successful Enumeration
Face Challenges. [hyperlink, http://www.gao.gov/products/GAO-09-430T].
Washington, D.C.: March 5, 2009.
Information Technology: Census Bureau Testing of 2010 Decennial Systems
Can Be Strengthened. [hyperlink,
http://www.gao.gov/products/GAO-09-262]. Washington, D.C.: March 5,
2009.
Census 2010: Census Bureau's Decision to Continue with Handheld
Computers for Address Canvassing Makes Planning and Testing Critical.
[hyperlink, http://www.gao.gov/products/GAO-08-936]. Washington, D.C.:
July 31, 2008.
Census 2010: Census at Critical Juncture for Implementing Risk
Reduction Strategies. [hyperlink,
http://www.gao.gov/products/GAO-08-659T]. Washington, D.C.: April 9,
2008.
Information Technology: Census Bureau Needs to Improve Its Risk
Management of Decennial Systems. [hyperlink,
http://www.gao.gov/products/GAO-08-79]. Washington, D.C.: October 5,
2007.
2010 Census: Basic Design Has Potential, but Remaining Challenges Need
Prompt Resolution. [hyperlink, http://www.gao.gov/products/GAO-05-9].
Washington, D.C.: January 12, 2005.
[End of section]
Footnotes:
[1] U.S. Census Bureau, 2010 Census Operational Plan, 2010 Census
Information Memoranda Series No. 2 (July 7, 2009).
[2] GAO, 2010 Census: Fundamental Building Blocks of a Successful
Enumeration Face Challenges, [hyperlink,
http://www.gao.gov/products/GAO-09-430T] (Washington, D.C.: Mar. 5,
2009).
[3] GAO, 2010 Census: Basic Design Has Potential, but Remaining
Challenges Need Prompt Resolution, [hyperlink,
http://www.gao.gov/products/GAO-05-9] (Washington, D.C.: Jan. 12,
2005).
[4] GAO, Information Technology: Census Bureau Testing of 2010
Decennial Systems Can Be Strengthened, [hyperlink,
http://www.gao.gov/products/GAO-09-262] (Washington, D.C.: Mar. 5,
2009).
[5] [hyperlink, http://www.gao.gov/products/GAO-09-430T].
[6] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for
Developing and Managing Capital Program Costs, [hyperlink,
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: Mar. 2009).
[7] U.S. Census Bureau, 2010 Census Operational Plan.
[8] The Bureau conducts an operation where field staff visit potential
group quarters--locations of group housing, such as prisons and nursing
facilities--to confirm each location and establish a contact person for
later enumeration.
[9] U.S. Census Bureau, 2010 Census Risk Management Plan, 2010 Census
Information Memoranda Series No. 5 (June 6, 2008).
[10] U.S. Census Bureau, Management Evaluation of Census 2000, Census
2000 Evaluation Q.1 (Oct. 8, 2003).
[11] For example, see GAO, Information Technology: Census Bureau Needs
to Improve Its Risk Management of Decennial Systems, [hyperlink,
http://www.gao.gov/products/GAO-08-79] (Washington, D.C.: Oct. 5,
2007); Census 2010: Census at Critical Juncture for Implementing Risk
Reduction Strategies, [hyperlink,
http://www.gao.gov/products/GAO-08-659T] (Washington, D.C.: Apr. 9,
2008); and GAO-09-262.
[12] [hyperlink, http://www.gao.gov/products/GAO-09-430T].
[13] [hyperlink, http://www.gao.gov/products/GAO-09-3SP].
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548: