2000 Census
Coverage Measurement Programs' Results, Costs, and Lessons Learned Gao ID: GAO-03-287 January 29, 2003To help measure the quality of the 2000 Census and to possibly adjust for any errors, the U.S. Census Bureau (Bureau) conducted the Accuracy and Coverage Evaluation (A.C.E.) program. However, after obligating around $207 million for A.C.E. and its predecessor program, Integrated Coverage Measurement (I.C.M.), from fiscal years 1996 through 2001, the Bureau did not use either program to adjust the census numbers. Concerned about the amount of money the Bureau spent on I.C.M. and A.C.E. programs and what was produced in return, the subcommittee asked us to review the objectives and results of the programs, the costs of consultants, and how best to track future coverage measurement activities.
The two programs the Bureau employed to measure the quality of the 2000 Census population data did not meet their objectives. The A.C.E. program achieved results other than those laid out in the Bureau's formal objectives that highlight important lessons learned. They include (1) developing a coverage measurement methodology that is both operationally and technically feasible, (2) determining the level of geography at which coverage measurement is intended, (3) keeping stakeholders, particularly Congress, informed of the Bureau's plans, and (4) adequately testing coverage measurement methodologies. It will be important for the Bureau to consider these as its current plans for the 2010 Census include coverage evaluation to measure the accuracy of the census but not necessarily to adjust the results. Of the roughly $207 million the Bureau obligated for I.C.M./A.C.E. programs from fiscal years 1996 through 2001, we identified about $22.3 million that was obligated for contracts involving over 170 vendors. We could not identify any obligations prior to 1996 in part because the Bureau included them with its general research and development efforts and did not assign the I.C.M./A.C.E. operations unique project codes in its financial management system. To track these costs in the future, it will be important for the Bureau to (1) have a financial management system that has specific project codes to capture coverage measurement costs, (2) establish the project codes as early in the planning process as possible, and (3) monitor the usage of the codes to ensure that they are properly charged.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-03-287, 2000 Census: Coverage Measurement Programs' Results, Costs, and Lessons Learned
This is the accessible text file for GAO report number GAO-03-287
entitled '2000 Census: Coverage Measurement Programs‘ Results, Costs,
and Lessons Learned' which was released on January 29, 2003.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products‘ accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
Report to the Chairman, Subcommittee on Civil Service, Census, and
Agency Organization, Committee on Government Reform, House of
Representatives:
January 2003:
2000 Census:
Coverage Measurement Programs‘ Results, Costs, and Lessons Learned:
GAO-03-287:
GAO Highlights:
Highlights of GAO-03-287, a report to
Chairman, Subcommittee on Civil Service,
Census, and Agency Organization,
Committee on Government Reform,
House of Representatives
2000 CENSUS
Coverage Measurement Programs‘
Results, Costs, and Lessons Learned
Why GAO Did This Study:
To help measure the quality of the 2000 Census and to possibly adjust
for any errors, the U.S. Census Bureau (Bureau) conducted the
Accuracy and Coverage Evaluation (A.C.E.) program. However, after
obligating around $207 million for A.C.E. and its predecessor
program, Integrated Coverage Measurement (I.C.M.), from fiscal
years 1996 through 2001, the Bureau did not use either program
to adjust the census numbers. Concerned about the amount of
money the Bureau spent on I.C.M. and A.C.E. programs and what was
produced in return, the subcommittee asked us to review
the objectives and results of the programs, the costs of consultants,
and how best to track future
coverage measurement activities.
What GAO found:
As shown below, the two programs the Bureau employed to measure the
quality of the 2000 Census population data did not meet their
objectives.
Coverage Measurement Programs Did Not Achieve Objectives
Program/objectives.
[See PDF for Image]
[End of Figure]
The A.C.E. program achieved results other than those laid out in the
Bureau‘s formal objectives that highlight important lessons learned.
They
include (1) developing a coverage measurement methodology that is both
operationally and technically feasible, (2) determining the level of
geography at which coverage measurement is intended, (3) keeping
stakeholders, particularly Congress, informed of the Bureau‘s plans,
and (4) adequately testing coverage measurement methodologies. It will
be important for the Bureau to consider these as its current plans for
the 2010 Census include coverage evaluation to measure the accuracy of
the census but not necessarily to adjust the results.
Of the roughly $207 million the Bureau obligated for I.C.M./A.C.E.
programs
from fiscal years 1996 through 2001, we identified about $22.3 million
that
was obligated for contracts involving over 170 vendors. We could not
identify any obligations prior to 1996 in part because the Bureau
included
them with its general research and development efforts and did not
assign
the I.C.M./A.C.E. operations unique project codes in its financial
management system. To track these costs in the future, it will be
important
for the Bureau to (1) have a financial management system that has
specific
project codes to capture coverage measurement costs, (2) establish
the
project codes as early in the planning process as possible, and (3)
monitor
the usage of the codes to ensure that they are properly charged.
What GAO Recommends:
The Secretary of Commerce should direct the Bureau to (1) work with
Congress and other stakeholders and soon decide on whether and
how coverage measurement will be used in 2010, (2) adopt lessons
learned from its 2000 Census experience, and (3) ensure that its
financial management systems can capture and report program
activities early and that projects‘ costs are monitored. The Bureau
agreed with our recommendations but noted that for the 2000 Census,
it followed the steps we identified as lessons learned. It also took
exception to how we presented our conclusions concerning its ability
to properly classify certain costs.
To view the full report, including the scope
and methodology, click on the link above.
For more information, contact Patricia A.
Dalton at (202) 512-6806 or
daltonp@gao.gov
Contents:
Letter:
Results in Brief:
Background:
Scope and Methodology:
Coverage Measurement
Programs for the 2000 Census
Did Not Meet Bureau Objectives:
I.C.M./A.C.E. Contractor Costs Are Not Complete:
Tracking Future Coverage Measurement Costs:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix:
Appendix I: Comments from the Department of Commerce:
Related GAO Products:
Tables :
Table 1: Coverage Measurement Programs Did Not Achieve Objectives:
Table 2: Census 2000 I.C.M./A.C.E. Contractor Costs for Fiscal Years
1996 through 2001 (Dollars in thousands):
Letter January 29, 2003:
The Honorable Dave Weldon, M.D.
Chairman
Subcommittee on Civil Service,
Census, and Agency Organization
Committee on Government Reform
House of Representatives:
Dear Mr. Chairman:
To assess the quality of population data for the 2000 Census and to
possibly adjust for any errors, the U.S. Census Bureau (Bureau)
conducted the Accuracy and Coverage Evaluation (A.C.E.) program. The
A.C.E. program was first included in Bureau program documents in
November 1998 and funded for fiscal years 2000 through 2002 with
proposed funding through December 31, 2002. Its predecessor, Integrated
Coverage Measurement (I.C.M.), began in May 1995 and was funded by the
Bureau for fiscal years 1996 through 1999. The Bureau obligated about
$207 million to both programs from fiscal years 1996 through
2001,[Footnote 1] which was about 3 percent of the $6.5 billion total
estimated cost of the 2000 Census. However, neither program was used to
adjust the 2000 Census population count.
Concerned about the amount of money the Bureau spent on both programs
as well as what was received in return for its investment, you and
former Vice Chairman Dan Miller asked us to examine (1) the Bureau‘s
objectives for the I.C.M./A.C.E. programs and the extent to which those
objectives were met, (2) the cost of consultants and technical studies,
and (3) ways to track the costs of coverage measurement activities in
future censuses. This report responds to that request.
In October 2002, we issued a report that provides additional
information on the cost of the I.C.M./A.C.E. programs.[Footnote 2] Both
reports are part of our ongoing series on the results of the 2000
Census and the lessons learned for planning a more cost-effective
census in 2010. (See the Related GAO Products section at the end of
this report for the assessments issued to date.):
Results in Brief:
The coverage measurement programs the Census Bureau planned for the
2000 Census--I.C.M. and its successor, A.C.E.--did not meet their
intended objectives. Although the Bureau designed I.C.M. to measure
census coverage; generate data for apportionment, redistricting, and
federal programs using statistical sampling and estimation; and produce
a ’one-number“ census based on statistical sampling and estimation, the
Bureau abandoned the program following a Supreme Court ruling that the
Census Act prohibited the use of statistical sampling to generate
population data for reapportioning the House of
Representatives.[Footnote 3] Because its replacement--A.C.E.--did not
provide a reliable measure of census accuracy in time to meet legally
mandated deadlines for releasing redistricting data, the Bureau decided
against using it to adjust the census data for nonapportionment
purposes.
The difficulties the Bureau encountered in trying to implement I.C.M.
and A.C.E. underscore the importance of (1) developing a coverage
measurement methodology that is both technically and operationally
feasible, (2) determining the level of geography at which coverage
measurement is intended, (3) keeping stakeholders, particularly
Congress, informed of the Bureau‘s plans, and (4) adequate testing. It
will be important for the Bureau to address these lessons learned as
its planning efforts for the 2010 Census continue. Those plans
currently call for a coverage measurement program to evaluate the
accuracy of the census, whereas the issue of whether coverage
measurement will be used to adjust the numbers has not yet been
resolved.
Concerning the cost of contracts, including the consultants and
technical studies related to the I.C.M./A.C.E. programs, we identified
about
$22.3 million of obligated amounts for over 170 vendors for fiscal year
1996 through fiscal year 2001 from unaudited Bureau financial
management reports. This amount does not represent the complete
contract costs for the I.C.M./A.C.E. programs for three reasons. First,
the Bureau considered costs from earlier years to be part of its
general research and development efforts and did not assign unique
project codes to identify I.C.M./A.C.E. programs and related costs in
its financial management system. Second, although $182,000 of fiscal
year 1996 obligated costs were identified in the Bureau‘s financial
management systems as contract costs for an I.C.M. special test, the
Bureau did not consider these costs as part of the I.C.M./A.C.E.
programs and classified these costs as general research. We disagreed
with the Bureau on this point and have included this amount in our
report as part of the I.C.M./A.C.E. contract costs that we could
identify from Bureau records. Finally, certain costs, such as program
evaluations, were Bureau-wide in nature, and the portion attributable
to I.C.M./A.C.E could not be separated out.
The Bureau‘s ability to track future costs of coverage measurement
activities is primarily dependent on its ability to (1) ensure that its
financial management system accurately and completely captures the
accounting and reporting of project codes, (2) design its project codes
to capture coverage measurement activities as early in the planning
process as possible, even though the activities‘ names may change as
the programs evolve, and (3) correctly charge the project codes
established.
Although the Bureau has never used the results of its coverage
measurement programs to adjust census numbers, we believe that an
evaluation of the accuracy of the census is essential given the
importance of the data, the need to know the nature of any errors, and
the cost of the census overall. Whether the results of the evaluation
should be used to adjust the census is still an open question, the
answer to which should involve discussions between the Bureau,
Congress, and other stakeholders, and be based on detailed data and a
convincing demonstration of the feasibility of the Bureau‘s proposed
approach. Regardless of the outcome of the decision, it is critical
that it be made soon so that the Bureau can proceed with its planning.
The longer the Bureau goes without a firm decision on the role of
coverage measurement, the greater the risk of wasted resources and
disappointing results. In light of the challenges facing the Bureau as
it prepares for the next decennial census in 2010, we recommend that
the Secretary of Commerce direct the U.S. Census Bureau to:
* in conjunction with Congress and other stakeholders, decide soon on
whether and how coverage measurement will be used in the 2010 Census;
* consider incorporating lessons learned from its coverage measurement
experience during the 2000 Census, such as (1) demonstrating both the
operational and technical feasibility of its coverage measurement
methods, (2) determining the level of geography at which coverage can
be reliably measured, (3) keeping Congress and other stakeholders
informed of its plans, and (4) adequately testing coverage measurement
prior to full implementation; and:
* ensure that the Bureau‘s financial management systems can capture and
report program activities early in the decennial process and that
project costs are monitored for accuracy and completeness.
The Secretary of Commerce forwarded written comments from the Bureau of
the Census on a draft of this report (see app. I). The Bureau agreed
with our recommendations noting that they were important steps that
should be followed in the development of a coverage measurement
methodology for the 2010 Census. However, the Bureau maintained that
most of these steps such as keeping Congress and other stakeholders
informed of its plans were followed for the 2000 Census. We disagree
because, as noted in the report, the lack of information contributed to
stakeholders‘ skepticism surrounding the Bureau‘s plans. The Bureau
also took exception to the way we presented our conclusions concerning
its ability to properly classify certain costs associated with the
development of the Bureau‘s coverage measurement programs. Our
perspective on the Bureau‘s position is detailed in the ’Agency
Comments and Our Evaluation“ section at the end of this report.
Background:
The Bureau puts forth tremendous effort to conduct a complete and
accurate count of the nation‘s population. However, some degree of
error in the form of persons missed or counted more than once is
inevitable because of limitations in census-taking methods. Because
census results are used, among other purposes, to apportion Congress,
redraw congressional districts, and allocate federal aid to state and
local governments, the size and demographic composition of these
coverage errors have become increasingly sensitive since the Bureau was
first able to generate detailed data on them during the 1980 Census.
However, the Bureau has never used the results of its coverage
measurements to correct estimated coverage errors.
The Bureau first attempted to measure the accuracy of the census in the
1940s when it compared the census numbers to birth and death
certificates and other administrative data using a procedure called
demographic analysis. Modern coverage measurement began with the 1980
Census when the Bureau compared census figures to the results of an
independent sample survey of the population. Using statistical methods,
the Bureau generated detailed measures of the differences among
undercounts of particular ethnic, racial, and other groups. In the
months that followed, many lawsuits were filed, most contending that
the results of the 1980 coverage measurement should have been used to
adjust the census. However, the Bureau designed the evaluation to
measure errors, not to correct the census results, and the Director of
the Census Bureau decided against adopting the adjusted numbers, as
they were deemed flawed due to missing and inaccurate data.
The quality of the coverage measurement data improved for the 1990
Census, and the Bureau recommended statistically adjusting the results.
However, the Secretary of Commerce determined that the evidence to
support an adjustment was inconclusive and decided not to adjust the
1990 Census. The adjustment decision was complicated by the fact that
the 1990 Census figures had already been released when the coverage
measurement results became available in the spring of 1991. The
Secretary of Commerce was concerned that two sets of numbers--the
actual census results and the adjusted figures--could create confusion
and might allow political considerations to play a part in choosing
between sets of numbers when the outcome of the choices, such as
congressional apportionment, could be known in advance of a decision.
Scope and Methodology:
To determine the objectives of 2000 Census I.C.M./A.C.E. programs and
their results, we reviewed Bureau and other documents that included
Federal Register notices; Census Operational Plans; reports to
Congress; internal memorandums; research and feasibility studies; and
reports of the Executive Steering Committee for Accuracy and Coverage
Policy (ESCAP) I and II, which assessed the results of the A.C.E.
program and recommended how they should be used.
To determine costs for consultants and technical studies for 2000
Census I.C.M./A.C.E. programs, we focused on object class code 25 from
the financial management reports to obtain contract data. With Bureau
assistance, we identified I.C.M./A.C.E. project accounts and analyzed
amounts by fiscal year using the financial management reports generated
by the Department of Commerce‘s Administrative Management System
(CAMS). We reviewed and analyzed obligated and expended data for all
coverage measurement programs that existed during the 2000 Census for
fiscal years 1991 to 2003. We did not audit financial data provided by
the Bureau.
To determine ways to track future costs, we reviewed current Bureau
financial management reports and considered established standards of
accounting, auditing, and internal controls.
In addition, we met with key Bureau officials to discuss the results of
our analysis and obtain their observations and perspectives. The
limitations we encountered in the scope of our work on this assignment
are as follows.
* We were unable to determine the complete contractual and technical
studies costs of the I.C.M./A.C.E. programs because the Bureau
considered any I.C.M./A.C.E.-related costs from fiscal years 1991
through 1995 as part of its general research and development programs
and thus did not separately track these costs. Although some costs were
tracked in fiscal year 1996, the Bureau still considered these costs as
research and development and did not include these costs as I.C.M./
A.C.E. program costs.
* We were unable to identify I.C.M./A.C.E. portions of costs from
projects that covered the entire census, such as the 2000 Census
Evaluation program.
* We did not evaluate the propriety of contracts for I.C.M./A.C.E.
programs.
Our work was performed in Washington, D.C., and at U.S. Census Bureau
headquarters in Suitland, Maryland, from June 2002 through October 2002
in accordance with generally accepted government auditing standards. On
January 7, 2003, the Secretary of Commerce provided written comments on
a draft of this report. We address these comments in the ’Agency
Comments and Our Evaluation“ section, and have reprinted them in
appendix I.
Coverage Measurement
Programs for the 2000 Census
Did Not Meet Bureau Objectives:
In planning the 2000 Census, the Bureau developed a new coverage
measurement program, I.C.M., that was designed to address the major
shortcomings of the 1990 coverage measurement program. However, as
shown in table 1, much like similar programs in earlier censuses, the
Bureau did not use I.C.M. and its successor program, A.C.E., to adjust
the census because of legal challenges, technical obstacles, and the
inability to resolve uncertainties in the data in time to meet the
deadlines for releasing the data.
Table 1: Coverage Measurement Programs Did Not Achieve Objectives:
[See PDF]
[End of Figure]
Source: GAO.
Note: This table reflects GAO‘s analysis of U.S. Census Bureau data.
[End of table]
The Bureau Canceled
I.C.M. in Response to a
Supreme Court Ruling:
In designing I.C.M., the Bureau‘s goal was to produce a single,
consolidated count or ’one-number“ census and thus avoid the
controversy of having two sets of census results as occurred during the
1990 Census. Thus, as shown in table 1, the objectives of I.C.M. were
to (1) measure census coverage, (2) generate, using statistical
sampling and estimation methods, the detailed data required for
apportionment, congressional redistricting, and federal program
purposes, and (3) produce a one-number census.
The Bureau‘s plans for I.C.M. emerged in response to the unsatisfactory
results of the 1990 Census. Although the 1990 headcount was, at that
time, the most costly in U.S. history, it produced data that were less
accurate than those from the 1980 Census. The disappointing outcome was
due in large part to the Bureau‘s efforts to count housing units that
did not mail back their census questionnaires. The operation, known as
nonresponse follow-up, where enumerators visited and collected
information from each nonresponding housing unit, proved to be costly
and error-prone when a higher-than-expected workload and a shortage of
enumerators caused the operation to fall behind schedule. The final
stages of nonresponse follow-up were particularly problematic. Indeed,
while enumerators finished 90 percent of the follow-up workload within
8 weeks (2 weeks behind schedule), it took another 6 weeks to resolve
the remaining 10 percent. Moreover, in trying to complete the last
portion of nonresponse follow-up cases, the Bureau accepted less
complete responses and information from nonhousehold members such as
neighbors, which may have reduced the quality of the data.
In the years following the 1990 Census, Congress, the Bureau, several
organizations, and GAO, concluded that fundamental design changes were
needed to reduce census costs and improve the quality of the data. In
response, the Bureau reengineered a number of operations for the 2000
Census.
For example, to save time and reduce its nonresponse follow-up
workload, the Bureau planned to enumerate a sample of the last
remaining portion of nonresponse follow-up cases instead of visiting
every nonresponding household as it had done in previous censuses. To
adjust for enumeration errors, the Bureau developed I.C.M., which was
intended to reconcile the original census figures with data obtained
from a separate, independent count of a sample of 750,000 housing units
using a statistical process called Dual System Estimation. The Bureau
believed that this approach offered the best combination of reduced
costs, improved accuracy expected at various geographic levels, and
operational feasibility.
However, concerned about the legality of the Bureau‘s planned use of
sampling and estimation, members of Congress challenged the Bureau‘s
use of I.C.M. in court. In January 1999, the Supreme Court ruled that
the Census Act[Footnote 4] prohibited the use of statistical sampling
to generate population data for reapportioning the House of
Representatives.[Footnote 5]
A.C.E. Did Not Meet Bureau Objectives:
Following the Supreme Court ruling, the Bureau planned to produce
apportionment numbers using traditional census-taking methods, and
provide statistically adjusted numbers for nonapportionment uses of the
data such as congressional redistricting and allocating federal funds.
The Bureau initiated the A.C.E. program, which was designed to take a
national sample of approximately 300,000 housing units to evaluate
coverage errors among different population groups and statistically
correct for them. Thus, as shown in table 1, the Bureau‘s objectives
for A.C.E. were to (1) measure how many people were missed in the
census and how many were erroneously included and (2) produce the
detailed data required in time for redistricting and federal program
purposes.
However, while the Bureau generally conducted A.C.E. in accordance with
its plans,[Footnote 6] the Bureau later determined that the A.C.E.
results did not provide a reliable measure of census accuracy and could
not be used to adjust the nonapportionment census data.
The first decision against A.C.E. occurred in March 2001, when the
Acting Director of the Census Bureau recommended to the Secretary of
Commerce that the unadjusted census data be used for redistricting
purposes. He cited as a primary reason an apparent inconsistency
between the population growth over the prior decade, as implied by
A.C.E. results, and demographic analysis, which estimated the
population using birth, death, and other administrative records. The
inconsistency raised the possibility of an unidentified error in either
the A.C.E. or census numbers. He reported that the inconsistency could
not be resolved prior to April 1, 2001, the legally mandated deadline
for releasing redistricting data.[Footnote 7]
The second decision against A.C.E. came in October 2001 when, based on
a large body of additional research, ESCAP decided against adjusting
census data for allocating federal aid and other purposes, because
A.C.E. failed to identify a significant number of people erroneously
included in the census, and other remaining uncertainties. According to
Bureau officials, it might be possible to use adjusted data to produce
intercensal population estimates for federal programs that require this
information; however, the Bureau would need to revise the A.C.E.
results before any use of the data could be considered.
The Bureau‘s Experience in Implementing
Coverage Measurement Programs
Highlights Important Lessons Learned:
Although I.C.M. and A.C.E. did not meet their formal objectives, they
did produce a body of important lessons learned. As the Bureau‘s
current approach for the 2010 Census includes coverage measurement to
assess the accuracy of the census (but not necessarily to adjust the
numbers themselves), it will be important for the Bureau to consider
these lessons as its planning efforts continue. The lessons include (1)
developing a coverage measurement methodology that is both technically
and operationally feasible, (2) determining the level of geography at
which coverage measurement is intended, (3) keeping stakeholders,
particularly Congress, informed of the Bureau‘s plans, and (4)
adequately testing the eventual coverage measurement program.
1. A.C.E. demonstrated operational, but not technical feasibility.
According to Bureau officials, an important result of the A.C.E.
program was that it demonstrated, from an operational perspective only,
the feasibility of conducting a large independent field check on the
quality of the census. The Bureau canvassed the entire A.C.E. sample
area to develop an address list, collected census response data for
persons living in the sample areas on census day, and conducted an
operation to try and match A.C.E. respondents to census respondents,
all independent of the regular census operations and within required
time frames.
Our separate reviews of two of these operations--interviewing
respondents and matching A.C.E. and census data--while raising
questions about the impact on final A.C.E. results due to apparently
small operational deviations, also concluded that the Bureau
implemented those two operations largely as planned.[Footnote 8]
Nevertheless, while the Bureau demonstrated that it could execute
A.C.E. field operations using available resources within required time
frames, as the Bureau has noted, feasibility also consists of a
technical component--that is, whether the A.C.E. methodology would
improve the accuracy of the census. Although the Bureau clearly stated
in its justification for A.C.E. that the effort would make the census
more accurate, as noted earlier, because of unresolved data
discrepancies, its experience in 2000 proved otherwise. Moreover,
according to the Bureau, because the A.C.E. was designed to correct a
census with a net coverage error similar to that observed in previous
censuses, the Bureau commented that applying the methodology to the
historically low levels of net error observed in the 2000 Census
represented a unique and unanticipated challenge for A.C.E. Thus, it
will be important for the Bureau to refine its coverage measurement
methodology to ensure that it is technically feasible.
2. The level of geography at which the Bureau can successfully measure
coverage is unclear. Since the October 2001 decision to not rely on
adjusted census data for nonapportionment and nonredistricting
purposes, Bureau officials have told us that they now doubt whether
census data can reliably be improved down to the level of geography for
which A.C.E. was intended to improve the accuracy--the census tract
level (neighborhoods that typically contain around 1,700 housing units
and 4,000 people). The Bureau‘s current position differs from that
taken in 2000, when it reported to Congress that it expected accuracy
at the tract level to be improved, on average, by A.C.E. statistically
adjusting numbers at an even lower level of geography--the census block
level.[Footnote 9] Uncertainty in the level of geography at which
accuracy is to be measured or improved can affect the overall design of
coverage measurement, as well as its technical feasibility. Therefore,
it will be important for the Bureau to determine the level of geography
at which it intends to measure accuracy as it decides the role and
design of future coverage measurement programs.
3. Keeping stakeholders informed is essential. Throughout the 1990s,
Congress and other stakeholders, including GAO, expressed concerns
about the Bureau‘s planned use of sampling and statistical estimation
procedures to adjust the census. A key cause of this skepticism was the
Bureau‘s failure to provide sufficiently detailed data on the effects
that I.C.M. would have at different levels of geographic detail.
Information was also lacking on the various design alternatives being
considered, their likely implications, and the basis for certain
decisions. As a result, it was difficult for Congress and other
stakeholders to support the Bureau‘s coverage measurement initiatives.
For example, on September 24, 1996, the House Committee on Government
Reform and Oversight issued a report that criticized the Bureau‘s
initiatives for sampling and statistical estimation. Among other
things, the Committee found that the Bureau had not clarified issues of
accuracy, particularly for small geographic areas, raised by the
sampling initiative. Congress‘s perspective on the process was later
reflected in its enactment of legislation in 1997 that included
provisions requiring the Department of Commerce to provide Congress
with comprehensive information on its planned use of statistical
estimation within 30 days.[Footnote 10]
4. Adequate testing of coverage measurement methodologies is critical.
Although the Bureau conducted a dress rehearsal for the census in three
locations across the country that was intended to demonstrate the
overall design of the census, the 1998 operation did not reveal the
problems that the Bureau encountered in dealing with the discrepancies
between the 2000 A.C.E. results and its benchmarks. According to Bureau
officials, this was partly because the sites were not representative of
the nation at large. Additionally, as a result of a compromise between
Congress and the administration to simultaneously prepare for a
nonsampling census, the I.C.M. was tested at only two of the three
dress rehearsal sites--an urban area and an Indian reservation--but was
not tested in a rural location as was originally planned. An earlier
test in 1995 was also not comprehensive in that it did not test a
sampling operation designed to help determine whether nonresponse
follow-up of the magnitude projected by the Bureau‘s current plan could
be completed in time for the I.C.M to be done on schedule.
I.C.M./A.C.E. Contractor Costs Are Not Complete:
From fiscal year 1996 through fiscal year 2001, the Bureau obligated
about $207 million for I.C.M./A.C.E. activities.[Footnote 11] As shown
in table 2, of that
$207 million, we identified about $22.3 million (11 percent) in
obligated amounts for contracts involving more than 170 vendors. These
contracts were primarily for technical advisory and assistant services,
computer systems support, and training.
Table 2: Census 2000 I.C.M./A.C.E. Contractor Costs for Fiscal Years
1996 through 2001 (Dollars in thousands):
Project description: A.C.E. coverage management; FY96: 0; FY97: 0 ;
FY98: 0; FY99: 0; FY00: $1,458; FY01: 0; Total: $1,458.
Project description: A.C.E. operations; FY96: 0; FY97: 0; FY98: 0;
FY99: 0; FY00: 249; FY01: ($2); Total: 247.
Project description: I.C.M. collection; FY96: 0; FY97: 0; FY98: $201;
FY99: $41; FY00: 6,272; FY01: 467; Total: 6,981.
Project description: Framework 3 total; FY96: 0 ; FY97: 0; FY98: 201;
FY99: 41; FY00: 7,979; FY01: 465; Total: 8,686.
Project description: I.C.M. procedures and training; FY96: 0; FY97:
$249; FY98: 594; FY99: 1,065; FY00: 1,766; FY01: 814; Total: 4,488.
Project description: I.C.M. processing; FY96: 0; FY97: 0; FY98: 0;
FY99: 2,412; FY00: 287; FY01: 574; Total: 3,273.
Project description: Framework 5 total; FY96: 0; FY97: 249; FY98: 594;
FY99: 3,477; FY00: 2,053; FY01: 1,388; Total: 7,761.
Project description: I.C.M. dress rehearsal; FY96: 0; FY97: 0; FY98:
502; FY99: 20; FY00: 0; FY01: (10); Total: 512.
Project description: I.C.M. special test; FY96: $182; FY97: 628; FY98:
0; FY99: 0; FY00: 0; FY01: 0; Total: 810.
Project description: I.C.M. coverage measurement; FY96: 0; FY97: 409;
FY98: 1,257; FY99: 1,991; FY00: (130); FY01: 999; Total: 4,526.
Project description: Framework 6 total; FY96: 182; FY97: 1,037; FY98:
1,759; FY99: 2,011; FY00: (130); FY01: 989; Total: 5,848.
Project description: Total; FY96: $182; FY97: $1,286; FY98: $2,554;
FY99: $5,529; FY00: $9,902; FY01: $2,842; Total: $22,295.
Source: GAO.
Note: This table reflects GAO‘s analysis of U.S. Census Bureau
financial management reports.
[End of table]
Although the Bureau tracked some costs of contracts for the I.C.M./
A.C.E. programs, we found that the $22.3 million did not represent the
complete contractor costs of the programs because of the following
three factors.
* First, the Bureau only tracked the contractor costs associated with
conducting the I.C.M./A.C.E. programs, which covers the period from
fiscal year 1997 through 2003. Although life cycle costs for the 2000
Census cover a 13-year period from fiscal years 1991 through 2003,
senior Bureau officials said that the I.C.M./A.C.E. program was not
viable for implementation until fiscal year 1997. Therefore, the Bureau
considered contractor costs from earlier years as part of its general
research and development programs, and the Bureau did not assign unique
project codes to identify I.C.M./A.C.E. programs and related costs in
its financial management system.
* Second, although $182,000 of fiscal year 1996 obligated contractor
costs were identifiable in the Bureau‘s financial management system as
an I.C.M. special test, the Bureau did not consider these costs as part
of the I.C.M./A.C.E. programs. Instead, these costs were considered
general research and development. However, because the Bureau
separately identified these costs as I.C.M. program contractor costs,
we have included the $182,000 as part of the I.C.M./A.C.E. program
contractor costs in this report.
* Finally, we were unable to identify the I.C.M./A.C.E. portions of
costs that were part of other programs. For example, in late fiscal
year 2000 and after, the Bureau did not separate A.C.E. evaluations
from its other 2000 Census evaluations in its financial management
systems. Bureau officials stated that the contracts for evaluations
included overall 2000 Census and A.C.E. evaluations, and did not have a
separate code identifying A.C.E. costs.
Tracking Future Coverage Measurement Costs:
During the 2000 Census, the Bureau, its auditors, and GAO, found
extensive weaknesses in the Bureau‘s financial management system, the
components of which include hardware, software, and associated
personnel. The weaknesses included difficulties in providing reliable
and timely financial information to manage current government
operations and preparing financial statements and other reports.
Together, they affected the completeness, accuracy, and timeliness of
data needed for informed management decisions and effective oversight.
In light of these weaknesses, the Bureau‘s ability to track future
costs of coverage measurement activities will largely depend on three
factors.
* First, a sound financial management system is critical. As discussed
in our December 2001 report, the Bureau‘s core financial management
system, CAMS, had persistent internal control weaknesses in fiscal year
2000.[Footnote 12] In its latest financial report, the Bureau indicated
that these weaknesses have continued through fiscal year 2001.[Footnote
13] The Bureau expects to issue its fiscal year 2002 financial report
shortly.
* Second, it would be important to set up project codes to capture
coverage measurement activities as early in the planning process as
possible. The Bureau did not set up a specific project code to identify
I.C.M. program costs until 1996 because, according to Bureau officials,
the I.C.M. program was not viable until 1997 and all costs up to this
point were considered general research.
* Finally, it would be important for Bureau personnel to correctly
charge the project codes established for the coverage measurement
program activities. During the 2000 Census, for example, while the
Bureau established a project code and a budget for the remote Alaska
enumeration, the project costs were erroneously charged to and
commingled with a project code for enumerating special populations. As
a result, the actual costs for remote Alaska enumeration were reported
by the Bureau‘s financial management system as zero and are unknown,
while enumerating special population costs are overstated.
Conclusions:
The Bureau‘s 2000 Census coverage measurement programs did not achieve
their primary objectives of measuring the accuracy of the census and
adjusting the results because of legal challenges, technical hurdles,
and questionable data. However, beyond these formal objectives, there
emerged several important lessons learned that Bureau managers should
consider because current plans for the 2010 Census include coverage
measurement. At the same time, it will also be important for the Bureau
to be capable of fully tracking the money it spends on coverage
measurement and other census activities so that Congress and other
stakeholders can hold the Bureau accountable for achieving intended
results.
Although the Bureau has never used the results of its coverage
measurement programs to adjust census numbers, we believe that an
evaluation of the accuracy and completeness of the census is critical
given the many uses of census data, the importance of identifying the
magnitude and characteristics of any under-and overcounts, and the cost
of the census overall. Less clear is whether the results of the
coverage measurement should be used to adjust the census. Any Bureau
decisions on this matter should involve close consultation with
Congress and other stakeholders, and be based on detailed data and a
convincing demonstration of the feasibility of the Bureau‘s proposed
approach. Whatever the decision, it is imperative that it be made soon
so that the Bureau can design appropriate procedures and concentrate on
the business of counting the nation‘s population. The longer the 2010
planning process proceeds without a firm decision on the role of
coverage measurement, the greater the risk of wasted resources and
disappointing results.
Recommendations for Executive Action:
To help ensure that any future coverage measurement efforts achieve
their intended objectives and costs can be properly tracked, we
recommend that the Secretary of Commerce direct the Bureau to:
* in conjunction with Congress and other stakeholders, come to a
decision soon on whether and how coverage measurement will be used in
the 2010 Census;
* consider incorporating lessons learned from its coverage measurement
experience during the 2000 Census, such as (1) demonstrating both the
operational and technical feasibility of its coverage measurement
methods, (2) determining the level of geography at which coverage can
be reliably measured, (3) keeping Congress and other stakeholders
informed of its plans, and (4) adequately testing coverage measurement
prior to full implementation; and:
* ensure that the Bureau‘s financial management systems can capture and
report program activities early in the decennial process and ensure
that project costs are monitored for accuracy and completeness.
Agency Comments and Our Evaluation:
The Secretary of Commerce forwarded written comments from the Census
Bureau on a draft of this report, which are reprinted in appendix I.
The Bureau agreed with our recommendations highlighting the steps that
should be followed in the development of a coverage measurement
methodology for the 2010 Census and acknowledged their importance.
However, the Bureau maintained that it followed most of these steps for
the 2000 Census including (1) keeping stakeholders, particularly
Congress, informed of the Bureau‘s plans, (2) determining the level of
geography at which coverage measurement is intended, and (3) adequately
testing coverage measurement methodologies. The Bureau also maintained
that throughout the 1990s, it had an open and transparent process for
implementing the coverage measurement program, including the levels of
geography to which its results would be applied.
We disagree. As we stated in our report, the Bureau‘s failure to
provide important information was a key cause of congressional
skepticism over the Bureau‘s coverage measurement plans. In fact,
Congress was so concerned about the lack of comprehensive information
on the Bureau‘s proposed approach that in July 1997, it passed a law
that included provisions requiring the Department of Commerce to
provide detailed data on the Bureau‘s planned use of statistical
estimation within 30 days.[Footnote 14] We revised the report to
include this, and provide other examples to further support our
position that the Bureau‘s I.C.M. and A.C.E. planning and development
processes were less than fully open and transparent.
The Bureau also commented that each major component of the I.C.M./
A.C.E. program underwent ’rigorous“ testing in the middle of the decade
as well as during the dress rehearsal for the 2000 Census held in 1998.
We believe this overstates what actually occurred. As we noted in the
report, the dress rehearsal failed to detect the problems that A.C.E.
encountered during the 2000 Census because the sites were not
representative of the nation. Additionally, because of an agreement
between Congress and the administration to simultaneously prepare for a
census that did not include sampling, the I.C.M. was only tested at two
of the three dress rehearsal sites--an urban area and an Indian
reservation--but was not tested in a rural location as was originally
planned. We made this and other revisions to strengthen our point.
Because the A.C.E. was designed to correct a census with a net coverage
error similar to that observed in previous censuses, the Bureau
commented that applying the methodology to the historically low levels
of net error observed in the 2000 Census represented a unique and
unexpected challenge for A.C.E. We revised the report to reflect this
additional context.
The Bureau took exception to the way we presented our conclusions
concerning its ability to properly classify certain costs associated
with the development of the Bureau‘s coverage measurement programs. The
Bureau noted that it decided not to separately track coverage
measurement development costs in 1994, because there was no internal or
external request for a separate cost accounting of the program.
Our report does not make interpretive conclusions or qualitative
judgments about which coverage measurement program costs the Bureau
decided to track. Instead, the report (1) points out that we could not
identify all of the contractor costs associated with the I.C.M./A.C.E.
programs because of the three factors described in the report, and (2)
underscores the importance of a sound financial management system for
tracking, planning, and development costs for the 2010 Census.
We are sending copies of this report to other interested congressional
committees, the Secretary of Commerce, and the Director of the U.S.
Census Bureau. Copies will be made available to others upon request.
This report will also be available at no charge on GAO‘s home page at
http://www.gao.gov.
Please contact Patricia A. Dalton on (202) 512-6806 or by E-mail at
daltonp@gao.gov if you have any questions. Other key contributors to
this report were Robert Goldenkoff, Roger Stoltz, Carolyn Samuels,
Cindy Brown-Barnes, Ty Mitchell, and Linda Brigham.
Sincerely yours,
Patricia A. Dalton
Director, Strategic Issues:
Signed by Patricia A. Dalton:
McCoy Williams
Director
Financial Management and Assurance:
Signed by McCoy Williams:
[End of section]
Appendixes:
Appendix I: Comments from the Department of Commerce:
THE SECRETARY OF COMMERCE Washington D.C. 20230:
JAN 7 2003:
Ms. Patricia A Dalton Director:
Strategic Issues:
United States General Accounting Office Washington, DC 20548:
Dear Ms. Dalton:
The Department of Commerce appreciates the opportunity to comment on
the General Accounting Office (GAO) draft document entitled 2000
Census: Coverage Measurement Programs‘ Results, Costs and Lessons
Learned (GAO-03-287)_ The Department‘s comments on this report are
enclosed.
Sincerely,
Donald L. Evans:
Signed by Donald L. Evans:
Enclosure:
Comments from the U.S. Department of Commerce U.S. Census Bureau:
U.S. General Accounting Office draft report entitled 2000 Census:
Coverage Measurement Programs‘ Results, Costs, and Lessons Learned:
Comments on the Teat of the Report:
In its analysis of the coverage measurement program implemented by the
U.S. Census Bureau during Census 2000, the General Accounting Office
{GAO) reaches conclusions that undcrscore the importance of finalizing
the census design early in the decade, continuing to refine coverage
measurement methodology based on lessons learned from Census 2000, and
thoroughly testing coverage measurement operations prior to
implementing them in 2010. We take exception with the way in which
these conclusions are presented Moreover, the GAO asserts that the
Census Bureau did not maintain a financial accounting system that could
properly classify certain costs attributable to the development of the
integrated Coverage Measurement (ICM)/Accuracy and Coverage Evaluation
(A.C.E.) programs. We fundamentally disagree with this assertion, as we
stressed in our response to the GAO‘s earlier discussion of this issue
(see GAO-03-41).
The GAO states that the Census Bureau did not adequately account for
the costs of the ICM/A.C.E. program. This conclusion was correct only
at the beginning of the Census 2000 cycle. At the beginning of each
census, we identify the specific activities for which we think costs
should be separately tracked. The decision not to separately track
coverage measurement development costs was made in 1994, when there was
no internal or extetnal request for a separate cost accounting of this
program. The GAO‘s suggestion after Census 2000 that we should have
separately tracked coverage measurement activities early in the census
cycle is far different from the implication that we had no system in
place for tracking those costs. The Census Bureau plans to track
coverage measurement expenditures earlier in the 2010 census cycle and,
in fact, will be happy to begin tracking costs in whatever additional
categories the GAO identifies.
(2) The GAO also identifies a series of ’lessons learned“ from Census
2000 that include the following:
’keeping stakeholders, particularly Congress, informed of the Bureau‘s
plans“ ’determining the level of geography at which coverage
measurement is intended“ ’adequately testing coverage measurement
methodologies‘*:
It is worth underscoring that each of these steps was taken during
Census 2000. Throughout the census, and in the years leading up to it,
the Census Bureau was unwavering in its maintenance of an open and
transparent process for implementing the coverage measurement program,
including the levels of geography to which its remits:
See the one-page overview ar the beginning of the report and the
’Results in Brief“ section of the text.
would be applied, in the face of unprecedented scrutiny from census
stakeholders. Moreover, each major component of the ICM/A.C.E. program
underwent rigorous testing in the middle of the decade and in the dress
rehearsal conducted in 1998.
Finally, while shortcomings in the data from the A.C.E. prevented their
use for improving the counts from Census 2000, as originally intended,
the A.C.E. itself was designed to correct a census with a net coverage
error based on the level of error observed in previous decennial
censuses. Applying the dual system estimation methodology central to
the A.C.E. to improve upon a level of net coverage error at the
historically low levels observed in Census 2000 represented a unique
and unexpected challenge for this program. This new circumstance will
be taken into consideration as coverage measurement methodologies are
developed for the 2010 census.
In short, the GAO has identified important steps that should be
followed in the development of a coverage measurement methodology for
the 2010 census. It is important to stress, however, that most of these
steps were followed in Census 2000, as well.
[End of Section]
Related GAO Products:
2000 Census: Complete Costs of Coverage Evaluation Programs Are Not
Available. GAO-03-41. Washington, D.C.: October 31, 2002.
2000 Census: Lessons Learned for Planning a More Cost-Effective 2010
Census. GAO-03-40. Washington, D.C.: October 31, 2002.
2000 Census: Refinements to Full Count Review Program Could Improve
Future Data Quality. GAO-02-562. Washington, D.C.: July 3, 2002.
2000 Census: Coverage Evaluation Matching Implemented as Planned, but
Census Bureau Should Evaluate Lessons Learned. GAO-02-297. Washington,
D.C.: March 14, 2002.
2000 Census: Best Practices and Lessons Learned for More Cost-Effective
Nonresponse Follow-up. GAO-02-196. Washington, D.C.: February 11, 2002.
2000 Census: Coverage Evaluation Interviewing Overcame Challenges, but
Further Research Needed. GAO-02-26. Washington, D.C.:
December 31, 2001.
2000 Census: Analysis of Fiscal Year 2000 Budget and Internal Control
Weaknesses at the U.S. Census Bureau. GAO-02-30. Washington, D.C.:
December 28, 2001.
2000 Census: Significant Increase in Cost Per Housing Unit Compared to
1990 Census. GAO-02-31. Washington, D.C.: December 11, 2001.
2000 Census: Better Productivity Data Needed for Future Planning and
Budgeting. GAO-02-4. Washington, D.C.: October 4, 2001.
2000 Census: Review of Partnership Program Highlights Best Practices
for Future Operations. GAO-01-579. Washington, D.C.: August 20, 2001.
Decennial Censuses: Historical Data on Enumerator Productivity Are
Limited. GAO-01-208R. Washington, D.C.: January 5, 2001.
2000 Census: Information on Short-and Long-Form Response Rates. GAO/
GGD-00-127R. Washington, D.C.: June 7, 2000.
FOOTNOTES
[1] At the time of our report, obligated costs after fiscal year 2001
were not final.
[2] U.S. General Accounting Office, 2000 Census: Complete Costs of
Coverage Evaluation Programs Are Not Available, GAO-03-41 (Washington,
D.C.: Oct. 31, 2002).
[3] Department of Commerce v. United States House of Representatives,
525 U.S. 316 (1999).
[4] 13 U.S.C. 195.
[5] Department of Commerce v. United States House of Representatives
525, U.S. 316 (1999).
[6] See, for example, U.S. General Accounting Office, 2000 Census:
Coverage Evaluation Interviewing Overcame Challenges, but Further
Research Needed, GAO-02-26 (Washington, D.C.: Dec. 31, 2001), and 2000
Census: Coverage Evaluation Matching Implemented as Planned, but Census
Bureau Should Evaluate Lessons Learned, GAO-02-297 (Washington, D.C.:
Mar. 14, 2002).
[7] March 1, 2001, Memorandum to Secretary Donald Evans from Acting
Director William Barron, Jr. (ESCAP I decision memo.)
[8] See GAO-02-26 and GAO-02-297.
[9] U.S. Census Bureau, ’Report to Congress--The Plan for Census 2000,“
44-46 and Accuracy and Coverage Evaluation--Statement on the
Feasibility of Using Statistical Methods to Improve the Accuracy of the
Census 2000 (June 2000), 19, fn. 19.
[10] Pub. L. 105-18, Title VIII, June 12, 1997.
[11] At the time of our report, obligated costs after fiscal year 2001
were not final.
[12] U.S. General Accounting Office, 2000 Census: Analysis of Fiscal
Year 2000 Budget and Internal Control Weaknesses at the U.S. Census
Bureau, GAO-02-30 (Washington, D.C.: Dec. 28, 2001).
[13] U.S. Census Bureau, 2001 Financial Report (Suitland, Md.: May
2002).
[14] Pub. L. 105-18, Title VIII, June 12, 1997.
GAO‘s Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO‘s commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO‘s Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as ’Today‘s Reports,“ on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select ’Subscribe to daily E-mail alert for newly
released products“ under the GAO Reports heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office
441 G Street NW,
Room LM Washington,
D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.
General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.
20548:
