2010 Census
Plans for Census Coverage Measurement Are on Track, but Additional Steps Will Improve Its Usefulness Gao ID: GAO-10-324 April 23, 2010Assessing the accuracy of the census is essential given that census data are used to apportion seats in Congress, to redraw congressional districts, and for many other public and private purposes. The U.S. Census Bureau's (Bureau) Census Coverage Measurement program (CCM) is to assess the accuracy of the 2010 Census and improve the design of operations for the 2020 Census. In April 2008, GAO recommended that the Bureau identify how it would relate CCM results--where the 2010 Census was accurate and inaccurate--to census operations to improve future censuses. Knowing where the 2010 Census was inaccurate can help inform research to improve the 2020 Census. GAO was asked to examine (1) the status of CCM planning and (2) the effects of design decisions since GAO issued its April 2008 report. GAO reviewed Bureau documents related to CCM design and National Academy of Sciences reports, and interviewed responsible Bureau officials.
Since GAO's April 2008 report, the Bureau has finalized plans for 2010 CCM goals, the timing of operations, and the types of results to be produced. Planning continues in other areas, such as developing estimation methods, evaluating the CCM program, and implementing its Master Trace Project, which would enable the Bureau to link its datasets and systems to support a broad range of research. The deadlines for some of these plans have not yet passed, but the Bureau already has default plans in place in case further changes do not occur. In mid-December, the Director decided to make some additional changes to the CCM program to improve the quality of CCM results. GAO found that additional actions on Bureau decisions may make CCM more useful in informing Bureau decisions on future census and coverage measurement efforts: (1) The Bureau's 2020 planning efforts are described in a series of decision memoranda issued in the summer of 2009. However, the Bureau has not yet taken steps to integrate CCM results with early 2020 planning to prepare for a census test in 2014. By describing, for example, what the Bureau might learn from CCM or how the results might feed into 2020 Census planning, the Bureau will better ensure that there are no gaps or overlaps in the use of CCM for early 2020 planning. (2) In September 2009, the Bureau began its Master Trace Project, which is intended to ensure that its datasets and systems can be used together to support detailed research into the causes of census coverage problems and facilitate research on the possible interactions of future operations. At the time of this review, the Bureau had not yet completed an inventory of the census databases that might be of potential interest for future research, identified which archived versions might be most useful, or mapped out how they might be archived and linked. Doing this quickly will be important as the census is already underway and it will be difficult to make changes to database structures or archival and data storage plans if the Bureau's assessments determine that changes are necessary. (3) The Bureau reviewed its previous decision to start CCM's Person Interviewing operation later than it did in 2000, and decided in June 2009 not to change it. However, the Bureau does not have a plan to assess the trade-offs in error between earlier and later start dates. Additional research on the trade-offs of different start dates could help the Bureau more fully understand the implications of CCM timing decisions on the resulting estimates of coverage error and better determine the optimal timing of Person Interviewing in future censuses.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Robert N. Goldenkoff Team: Government Accountability Office: Strategic Issues Phone: (202) 512-2757GAO-10-324, 2010 Census: Plans for Census Coverage Measurement Are on Track, but Additional Steps Will Improve Its Usefulness
This is the accessible text file for GAO report number GAO-10-324
entitled '2010 Census: Plans for Census Coverage Measurement Are on
Track, but Additional Steps Will Improve Its Usefulness' which was
released on April 23, 2010.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
United States Government Accountability Office:
GAO:
April 2010:
2010 Census:
Plans for Census Coverage Measurement Are on Track, but Additional
Steps Will Improve Its Usefulness:
GAO-10-324:
GAO Highlights:
Highlights of GAO-10-324, a report to congressional requesters.
Why GAO Did This Study:
Assessing the accuracy of the census is essential given that census
data are used to apportion seats in Congress, to redraw congressional
districts, and for many other public and private purposes. The U.S.
Census Bureau's (Bureau) Census Coverage Measurement program (CCM) is
to assess the accuracy of the 2010 Census and improve the design of
operations for the 2020 Census. In April 2008, GAO recommended that
the Bureau identify how it would relate CCM results”where the 2010
Census was accurate and inaccurate”to census operations to improve
future censuses. Knowing where the 2010 Census was inaccurate can help
inform research to improve the 2020 Census.
GAO was asked to examine (1) the status of CCM planning and (2) the
effects of design decisions since GAO issued its April 2008 report.
GAO reviewed Bureau documents related to CCM design and National
Academy of Sciences reports, and interviewed responsible Bureau
officials.
What GAO Found:
Since GAO‘s April 2008 report, the Bureau has finalized plans for 2010
CCM goals, the timing of operations, and the types of results to be
produced. Planning continues in other areas, such as developing
estimation methods, evaluating the CCM program, and implementing its
Master Trace Project, which would enable the Bureau to link its
datasets and systems to support a broad range of research. The
deadlines for some of these plans have not yet passed, but the Bureau
already has default plans in place in case further changes do not
occur. In mid-December, the Director decided to make some additional
changes to the CCM program to improve the quality of CCM results.
GAO found that additional actions on Bureau decisions may make CCM
more useful in informing Bureau decisions on future census and
coverage measurement efforts:
* The Bureau‘s 2020 planning efforts are described in a series of
decision memoranda issued in the summer of 2009. However, the Bureau
has not yet taken steps to integrate CCM results with early 2020
planning to prepare for a census test in 2014. By describing, for
example, what the Bureau might learn from CCM or how the results might
feed into 2020 Census planning, the Bureau will better ensure that
there are no gaps or overlaps in the use of CCM for early 2020
planning.
* In September 2009, the Bureau began its Master Trace Project, which
is intended to ensure that its datasets and systems can be used
together to support detailed research into the causes of census
coverage problems and facilitate research on the possible interactions
of future operations. At the time of this review, the Bureau had not
yet completed an inventory of the census databases that might be of
potential interest for future research, identified which archived
versions might be most useful, or mapped out how they might be
archived and linked. Doing this quickly will be important as the
census is already underway and it will be difficult to make changes to
database structures or archival and data storage plans if the Bureau‘s
assessments determine that changes are necessary.
* The Bureau reviewed its previous decision to start CCM‘s Person
Interviewing operation later than it did in 2000, and decided in June
2009 not to change it. However, the Bureau does not have a plan to
assess the trade-offs in error between earlier and later start dates.
Additional research on the trade-offs of different start dates could
help the Bureau more fully understand the implications of CCM timing
decisions on the resulting estimates of coverage error and better
determine the optimal timing of Person Interviewing in future censuses.
What GAO Recommends:
GAO is making recommendations to improve the Bureau's use of CCM in
planning for future operations, with which the Department of Commerce
generally agreed. Commerce stated that it is taking action to ensure
data preservation. Further, Commerce stated that although it considers
a 2000 contamination study comprehensive, a new recall bias study is
planned for 2010.
View [hyperlink, http://www.gao.gov/products/GAO-10-324] or key
components. For more information, contact Robert Goldenkoff at (202)
512-2757 or goldenkoffr@gao.gov.
[End of section]
Contents:
Letter:
Background:
The Bureau Has Finalized Decisions in Some Key Areas Since Our 2008
Report:
The Bureau Needs to Take Additional Actions to Improve the Usefulness
of CCM:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Comments from the Department of Commerce:
Table:
Table 1: Status of the Bureau's Decisions in Key Areas of the Census
Coverage Measurement Program:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
April 23, 2010:
The Honorable Darryl E. Issa:
Ranking Member:
Committee on Oversight and Government Reform:
House of Representatives:
The Honorable Patrick T. McHenry:
Ranking Member:
Subcommittee on Information Policy, Census, and National Archives:
Committee on Oversight and Government Reform:
House of Representatives:
Data from the decennial census--a constitutionally mandated count of
the national population--are used to apportion seats in Congress,
redraw congressional districts, allocate billions of dollars in
federal assistance to state and local governments each year, and
inform the planning and investment decisions of numerous public and
private sector entities. The census aims to locate and count people--
only once--in the right place, and collect complete and correct
information about them. Because census data are central to so many
critical functions, it is essential to assess census accuracy and
improve the process when needed.
In April 2008, we reported that the U.S. Census Bureau (Bureau) needed
to finalize plans for its Census Coverage Measurement (CCM) program--
the effort intended to assess the accuracy of the 2010 Census and
improve the design of operations for the 2020 Census--and made related
recommendations to the Bureau's parent agency, the Department of
Commerce.[Footnote 1] In particular, we noted that the Bureau should
produce plans that include (1) a description of when it will provide
CCM results, (2) how it plans to report its CCM results in relation to
census operations, and (3) key decision points and plans for
evaluating aspects of the CCM. Commerce has taken steps to implement
actions for each recommendation. Since we issued our report, the
National Academy of Sciences panel on Correlation Bias and Coverage
Measurement in the 2010 Decennial Census released its final report on
the Bureau's plans for CCM,[Footnote 2] recommending numerous steps to
enhance the value of the program.
You asked us to examine the current plan for the CCM program and to
provide (1) an update on the status of the planning since our April
2008 report, and (2) the potential effects of major CCM decisions on
the quality and usefulness of CCM data. To meet these objectives, we
reviewed Bureau documents related to CCM design and National Academy
of Sciences (NAS) reports, and we interviewed Bureau officials
responsible for CCM. We focused primarily on the Bureau's decisions in
the following areas: 2010 CCM goals, timing of operations and
reporting results, estimation methods, results to be produced, program
evaluation, and implementing its Master Trace Project. Specifically,
to update the status of CCM, we reviewed scheduling documents and
decision memorandums. To identify the potential effects we reviewed
decision memorandums and related justifications, prior Bureau and NAS
research and our reports related to CCM and evaluation. We conducted
our work from June 2009 to February 2010 in accordance with generally
accepted government auditing standards. Those standards require that
we plan and perform the audit to obtain sufficient, appropriate
evidence to provide a reasonable basis for our findings and
conclusions based on our audit objectives. We believe that the
evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.
Background:
The Bureau puts forth tremendous effort to conduct a complete and
accurate count of the nation's population; nonetheless, some degree of
coverage error is inevitable because of the inherent complexity of
counting the nation's large and diverse population and limitations in
census-taking methods. These census coverage errors can take a variety
of forms, including a person missed (an undercount), a person counted
more than once (an overcount), or a person who should not have been
counted, such as a child born after Census Day (another type of
overcount). To further understand and to inform users about the
quality of the census, the Bureau has been evaluating coverage
measurement for more than 50 years. While initial evaluations relied
solely on demographic analysis--population estimates based on birth
and death rates as well as immigration estimates--modern coverage
measurement began with the 1980 Census when the Bureau began also
comparing census counts to survey results from an independent coverage
measurement sample of the population. Using statistical methods, the
Bureau generated detailed measures of the differences among
undercounts of particular ethnic, racial, and other groups, which have
been referred to as "differential undercounts." These measures were
also generated for the 1990 and 2000 censuses.
Although the Bureau considered doing so in earlier decades, it has
never used its estimates of coverage error to adjust census data. In
1980, the Director of the Census Bureau decided that potential
adjustments would be flawed due to missing and inaccurate data. In
1990, the Bureau recommended statistically adjusting census data;
however, the Secretary of Commerce determined that the evidence to
support an adjustment was inconclusive and decided not to adjust. For
the 2000 Census, a 1999 Supreme Court ruling held that the Census Act
prohibited the use of statistical sampling to generate population data
for apportioning the House of Representatives.[Footnote 3] The Bureau
had planned to produce apportionment numbers using traditional census-
taking methods, and provide statistically adjusted numbers for non-
apportionment uses of the data such as congressional redistricting and
allocating federal funds. The Bureau later determined that its
statistical estimates did not provide a reliable measure of census
accuracy and could not be used to adjust the non-apportionment census
data.
The Bureau is not planning to use CCM to adjust the 2010 Census.
Instead, CCM will be used to evaluate coverage error to improve the
2020 and future censuses, and will focus on estimating various
components of census coverage in addition to net coverage errors--the
net effect on coverage after undercounts and overcounts are
considered. These components of coverage include correct enumerations,
erroneous enumerations (people or housing units that were counted but
should not have been), and omissions (people or housing units that
were not counted but should have been). The Bureau also plans to
include imputations (counts of people and their characteristics that
are provided for nonresponding households, usually based on responses
from others under similar circumstances, such as from surrounding
households).
Statistical measurements of census coverage are obtained by comparing
and matching the housing units and people counted by the independent
coverage measurement sample to those counted by the census in and
around the sample areas. The Bureau has developed separate address
lists--one for the entire nation of over 134 million housing units
that it will use to conduct the census and one for coverage
measurement sample areas--and will collect each set of data through
independent operations. For the 2010 Census, census operations began
collecting population data from households in January 2010 and will
continue through the end of July, while CCM operations will collect
data by visiting each of the housing units in the coverage measurement
sample during an operation called Person Interviewing from August
through October.
The statistical methodology the Bureau uses to estimate net coverage
errors relies on an assumption that the chance that a person is
counted by the census is not affected by whether he or she is counted
in the independent coverage measurement sample, or vice versa. Because
violating this "independence" assumption can bias coverage estimates,
the Bureau takes special measures to maintain CCM's separation from
the census, such as developing a separate address list for the
coverage measurement sample discussed above.
The Bureau Has Finalized Decisions in Some Key Areas Since Our 2008
Report:
Since our April 2008 report, the Bureau has finalized its plans in key
areas of the CCM program including CCM's goals, the timing of
operations, and the timing and types of results to be produced.
Planning continues in other areas, such as developing estimation
methods, evaluating the CCM program, and implementing its Master Trace
Project. Continued progress and adherence to schedule will be
important to ensure that the Bureau carries out CCM in order to meet
its goal of improving the 2020 Census.
For example, in our 2008 report, we recommended that the Bureau
provide decision points and plans for evaluating CCM. In September
2009, the Bureau finalized its list of 22 planned evaluations for the
2010 Census, which included five that address specific methodological
or procedural topics within the CCM program. However, all study plans
are not due to be completed until April 2010. In addition, while the
deadlines for finalizing CCM estimation methods have not yet passed,
the Bureau has many of its default plans already in place. Default
plans allow the Bureau to move forward on schedule even if new plans
have not been developed. Table 1 shows the status of the Bureau's
plans for the design of CCM in each of these areas.
Table 1: Status of the Bureau's Decisions in Key Areas of the Census
Coverage Measurement Program:
CCM plan area: Goals;
Decision status: Finalized;
Decision date: Sept. 2005;
Comment: The goals have not substantially changed and are subject to
refinement as research on how to meet them progresses.
CCM plan area: Timing: Of Person Interviewing;
Decision status: Finalized;
Decision date: June 2009;
Comment: In 2009, the Bureau reconsidered and stayed with its earlier
decision on the timing of this operation.
CCM plan area: Timing: Of releasing results;
Decision status: Finalized;
Decision date: June 2009;
Comment: The Bureau decided to move reporting forward by about 3
months from initial baseline schedule.
CCM plan area: Estimation Methods;
Decision status: In progress;
Decision date: est. April 2010;
Comment: Default plans are largely in place, but the Bureau is
researching additional technical improvements.
CCM plan area: Results to Produce: By level of geography;
Decision status: In progress;
Decision date: Spring 2010;
Comment: The Bureau will publicize the levels of estimates planned for
below the state level.
CCM plan area: Results to Produce: By demographic groups;
Decision status: Finalized;
Decision date: June 2009;
Comment: The Bureau plans public dissemination of the planned
reporting groups in early 2010.
CCM plan area: Evaluations: Topics;
Decision status: Finalized;
Decision date: Sept. 2009;
Comment: Additional studies are expected outside the formal evaluation
program. The Bureau also plans a series of technical memorandums
documenting CCM process and results, similar to its approach for the
2000 Census.
CCM plan area: Evaluations: Study plans;
Decision status: In progress;
Decision date: est. April 2010;
Comment: Subject area experts will complete study plans on a rolling
basis until the deadline.
CCM plan area: Master Trace Project;
Decision status: In progress;
Decision date: est. Sept. 2010;
Comment: The Director of the Census Bureau recently initiated the
Master Trace Project.
Source: GAO analysis of Bureau documentation and schedule.
[End of table]
Recent Changes to CCM Data Collection Plan Could Improve Quality of
CCM Data:
In September 2009, shortly after taking office, the Director of the
Census Bureau asked the staff responsible for CCM to review its CCM
design and propose specific changes that would reduce the introduction
of nonsampling error--such as human errors made when recording data
during interviews--into CCM and its resulting estimates. The staff
proposed numerous changes intended to reduce error in collected data.
They also proposed an additional research study. The Director approved
all of these proposals in mid-December 2009. Key changes included:
* increasing the reinterview rates for CCM field work to improve
quality assurance;
* increasing training time for short-term workers hired to conduct
door-to-door visits during the Person Interviewing operation to
improve interview techniques for local or other special situations due
to current economic conditions (such as people who became homeless or
have had to move frequently during the housing crisis);
* increasing supervisor-to-employee field staffing ratios to improve
quality and monitoring of field work at each level; and:
* adding a telephone-based study to collect information about how well
respondents recall information about their residence and possible
movement since Census Day.
In addition, the decision authorized a nearly 45 percent reduction in
the CCM sample size that the Bureau believes would generate the cost
savings to pay for the other changes. Our understanding of the issues
suggests that these are reasonable efforts to improve survey quality.
The Bureau's reduction in sample size will reduce precision of the
estimates, yet the proposed changes should reduce nonsampling errors
and thus provide users with more reliable estimates. For example, the
Bureau expects short-term CCM workers to make fewer mistakes in
identifying temporary or unconventional housing units when they have
received additional training specific to their local circumstances,
such as in areas with large numbers of seasonal or displaced workers.
The Bureau Needs to Take Additional Actions to Improve the Usefulness
of CCM:
The Bureau's actions to finalize some areas of CCM program planning
are important steps in the right direction. Still, in some cases, it
will be important for the Bureau to take additional actions to help
ensure the results of CCM are as useful as they could be to inform
Bureau decisions on improving future censuses and coverage measurement
efforts.
For example, the Bureau could better document how CCM results will be
used as part of the planning process for the 2020 Census. Indeed, the
Bureau has already begun laying the foundation for its 2020 planning
efforts. These early planning efforts are described in a series of
decision memorandums issued in the summer of 2009, and include
milestones leading up to a census test in April 2014, descriptions of
planning phases, and a list of the various organizational components
that conduct the census. Although these planning documents explicitly
state the importance of relying on the 2010 Census Evaluation and
Testing program--an ongoing assessment effort separate from CCM that,
like CCM, is designed to improve future operations--the Bureau has not
yet taken similar steps to integrate the CCM program with 2020
planning. In addition, the Bureau does not have specific plans in its
CCM program goals to do anything beyond producing CCM results.
Bureau officials have maintained that until it produces CCM results,
it is difficult to determine how to use CCM data to improve the design
of future decennials. While we agree with the Bureau that the results
will determine the specifics of any potential design improvements, it
is not premature to consider how the earliest results from CCM--
scheduled for early 2012--could help inform early planning and
decisions. Importantly, by creating a "roadmap" that describes, for
example, what the Bureau might learn from CCM or how the results might
feed into early 2020 Census planning, the Bureau will better ensure
that there are no gaps or overlaps in the use of CCM in early 2020
planning.
The Bureau's Master Trace Project is another area where additional
efforts are needed to ensure useful CCM results. The Bureau initiated
the Master Trace Project in September 2009, to facilitate the use of
census and CCM data for future research. Currently, Bureau data are
collected and archived in different types of datasets and systems. The
Master Trace Project is intended to ensure that these datasets and
systems can be used together, or linked, to support detailed research
into the causes of census coverage problems and facilitate research on
the possible interactions of future operations. For example, a
researcher might want to see if there is a relationship between the
Bureau's employment practices and the magnitude of an undercount in a
particular area. In so doing, the researcher may want to compare
census payroll, overtime, and other human capital data to the data
from that region collected and processed by census and CCM. Such
datasets would not ordinarily be linked during the census.
The Bureau has not yet taken the steps needed to ensure that such
research across different data systems would be possible. The Bureau
held a meeting in December 2009 with staff responsible for many major
decennial systems and obtained agreement about the importance of data
retention for this project; however, the Bureau has not yet resolved
how it would make the project happen. In particular, the Bureau has
not yet completed an inventory of the census databases that might be
of potential interest for future research, identified which archived
versions might be most useful, or mapped out how they might be
archived and linked. Until this is done, it is unclear that Bureau or
other researchers will have access to census operational data that
they need to fully analyze the census coverage errors that CCM may
uncover. Moving forward, it will be important for the Bureau to
perform the initial assessment of its data systems, identify gaps in
data collection, and identify any other related steps to ensure that
key data can be linked. Doing this quickly will also be important as
Census 2010 is underway and it could become increasingly difficult to
make changes to database structures or archival and data storage plans
if the Bureau's assessments determine that changes are necessary.
A third area where the Bureau needs to do additional work is in
assessing how the timing of CCM data collection might adversely affect
CCM findings. When planning CCM, the Bureau faced the challenge of
determining the optimal time to launch the CCM data collection
operation, known as Person Interviewing (PI). If the Bureau starts PI
too early, it increases the chance that it overlaps with census data
collection, possibly compromising the independence of the two
different operations and introducing a "contamination bias" error into
CCM data. If the Bureau starts PI too late, it increases the chance
that respondents will not accurately remember household information
from Census Day, April 1, introducing error (known as "recall bias")
in the CCM count. Both types of errors--contamination bias and recall
bias--could affect the Bureau's conclusions about the accuracy of the
census. An understanding of the trade-offs between these two types of
biases would be important in future decisions regarding the optimal
timing of PI.
In early 2009, based on concerns by the National Academy of Sciences
(NAS) and other stakeholders about the relative lateness in the start
date of PI and its possible impact on the quality of CCM findings, the
Bureau considered whether to start PI 6 weeks earlier than planned. In
June 2009, the Bureau decided to keep the originally scheduled start
on August 14, 2010. Bureau memorandums and officials justified the
decision largely because of concern that it was too late in the
planning process to make a change in the complex CCM schedule. The
memorandums cited gaps in knowledge about the impact of timing on
recall bias, presented research with differing conclusions about the
extent of contamination in prior census tests, and justified the
recommendation to not change the start date by the operational
challenges faced to make the change.
Bureau officials have also explained that the goal of using coverage
measurement in 2000 to possibly adjust the census-created time
pressures in 2000 that forced an early PI, and because such time
pressures do not exist for PI in 2010, it is scheduled to begin more
than 4 months after Census Day.
By comparison, during the 2000 Census, the Bureau launched PI in April
2000 and had completed about 99 percent of its data collection by the
end of the first week of August 2000, a week earlier than the
scheduled 2010 PI start date. An extensive 2000 Census evaluation
found no evidence of contamination bias caused by the earlier start of
PI in 2000. Related Bureau research since then has also found no
significant evidence of contamination bias during census tests,
although one test found that census results could be affected. Yet
Bureau officials remained concerned about the possibility, since the
CCM questions are similar to follow-up questions used in one of the
2010 census follow-up operations. Furthermore, parts of this census
operation are new in 2010, and end later than similar operations did
in 2000.
Moving forward, additional research on the trade-offs between recall
bias and contamination errors could help the Bureau more fully
understand the implications of choosing various start times for PI on
the resulting estimates of coverage error and better determine the
optimal timing of PI in future censuses. Currently, the Bureau has a
telephone-based study planned in order to measure recall errors, which
could provide additional information about when recall errors are more
likely to occur. However, this study is limited to certain types of
recall error, and the Bureau does not have an evaluation planned to
measure possible contamination between the new, much later, parts of
census follow-up and CCM data collection or to assess the trade-offs
between the biases from starting earlier compared to starting later.
Such additional study after the 2010 Census could provide the Bureau
better information about the trade-offs in data quality from potential
contamination and recall biases and provide a better basis for
determining the optimal scheduling of coverage measurement operations.
Conclusions:
Assessing the accuracy of the census is an essential step in improving
current and future censuses. The Bureau has made progress on designing
and planning for its CCM program and continues activity to complete
the plan. Additional actions in three CCM planning areas may further
improve CCM or its usefulness to the 2020 Census.
Specifically, the Bureau has stated the importance of using 2010
evaluation data such as CCM's for 2020 Census design, but has not yet
taken steps to link CCM data to an improved 2020 design. If the Bureau
is to best achieve its goal of using CCM to help improve the 2020
Census, it will need to integrate planning for any follow-up work on
CCM results or data with the other early planning already underway for
Census 2020.
Second, the Bureau has many different processes that come together in
the conduct of a decennial census, and archived data on those
processes could provide useful information to researchers trying to
figure out what worked well and what did not. The Master Trace Project
can help researchers link CCM results and data to potential design
changes for Census 2020. Determining which data need to be linked or
archived to enable future linkage within the project can help prevent
gaps in 2010 data that might hinder the project's ability to help
identify improvements for the 2020 Census.
Third, the timing of CCM's primary data collection operation--Person
Interviewing--involves trade-offs between reducing contamination bias
and reducing recall error that the Bureau did not have conclusive
information on. Since 2010 Person Interviewing is starting 1 week
after a similar operation ended in 2000, the chance of introducing
recall bias errors into CCM data is higher in 2010 than it was in
2000. Although the Bureau has a study planned to measure some recall
errors, there is no study planned to measure contamination between the
new parts of census follow-up--which use questions similar to those
asked by CCM and finish much later than follow-up did in 2000--and CCM
or to assess the trade-offs between the two types of biases in timing
decisions. Targeted research after the 2010 Census on the relationship
between the timing of data collection and the trade-offs between these
types of errors before the 2020 Census and its coverage measurement
efforts could help the Bureau better determine the optimal timing of
future data collection operations.
Recommendations for Executive Action:
We recommend that the Secretary of Commerce require the Director of
the U.S. Census Bureau to take the following three actions to improve
the usefulness of CCM for 2020:
To help the Bureau achieve its goal of using CCM to improve the 2020
Census, better document links between the 2010 CCM program and 2020
Census planning, integrating the goal of using the CCM program to
improve Census 2020, such as with CCM results and data, into those
broader plans for 2020.
To ensure that Bureau datasets from the 2010 Census can be used with
other Bureau datasets to support research that could improve the
census and CCM, complete the Master Trace Project's assessment of how
key census and CCM data systems are, or can be, linked to each other;
identify any potential data gaps; and identify other related steps for
future action.
To help the Bureau better determine the optimal timing of future
coverage measurement data collection, fully assess the trade-offs
between starting the data collection earlier, with the possibility of
introducing contamination errors, and starting later, with the
possibility of introducing recall errors.
Agency Comments and Our Evaluation:
The Secretary of Commerce provided written comments on a draft of this
report on April 5, 2010. The comments are reprinted in appendix I.
Commerce generally agreed with the overall findings and
recommendations and appreciated our efforts in helping the Census
Bureau develop a successful evaluation plan for the 2020 Census.
Commerce also provided additional information and comments on certain
statements and conclusions in the report.
With respect to our second recommendation to complete the Master Trace
Project's assessment of linking key census and CCM data systems, to
identify any potential data gaps, and to identify other related steps
for future action, Commerce maintained that it would be taking action
to preserve adequate documentation and maximize the amount of data
retained from each major decennial system. We commend the Bureau for
committing to these steps and encourage its follow-through on them and
its identification of remaining data gaps and additional steps needed.
With respect to our third recommendation to fully assess the trade-
offs between two types of error related to starting CCM data
collection either earlier or later relative to Census Day, Commerce
responded that (1) it is too late to create a new study for 2010
Census; (2) it considers a Bureau contamination study from 2000 to be
definitive; and (3) it has recently developed a study on recall bias
to try to measure some of the effects of scheduling CCM data
collection at various periods of time following the census
enumeration. We agree that it is too late to attempt any additional
unplanned data collection during the 2010 Census, and we revised our
discussion to clarify our intent that the recommended research be
conducted after the 2010 Census. We also recognize the thoroughness of
the 2000 contamination study the Bureau cites, commend the Bureau on
undertaking additional study of recall bias, and look forward to
reviewing its study plans when they are available. However, we
recommended research comparing trade-offs between the two types of
errors at a variety of start dates for CCM data collection--something
the 2000 study did not discuss and something it is unclear that a
study of only recall bias will achieve. Furthermore as we discussed in
our draft report, the Bureau expressed concerns over possible
contamination between CCM and new parts of census follow-up in 2010--
parts that were introduced after the 2000 study and that were not
included in the scope of the 2000 study. We clarified our discussion
of this in the report to better focus on the need for research that
relates the trade-offs between the two types of error at different
timing of data collection.
Commerce provided additional information that in response to advice
from various advisory panels and after additional research, it would
soon make public its proposed geographic levels for CCM estimates. We
reflected this decision in table 1 of our report.
Finally, Commerce provided additional information about its plans to
produce highly technical documentation of the results of CCM
estimation including modeling, missing data, and errors in the
estimates in a series of memorandums as it did for Census 2000. We
reflected this decision in table 1 of this report.
We are sending copies of this report to the Secretary of Commerce, the
Director of the U.S. Census Bureau, and interested congressional
committees. The report also is available at no charge on GAO's Web
site at [hyperlink, http://www.gao.gov].
If you have any questions about this report please contact me at (202)
512-2757 or goldenkoffr@gao.gov. Contact points for our Offices of
Congressional Relations and Public Affairs may be found on the last
page of this report. Key contributors to this report were Jeff Dawson,
Dewi Djunaidy, Ron Fecso (Chief Statistician), Andrea Levine, Ty
Mitchell, Melanie Papasian, and Tamara F. Stenzel.
Signed by:
Robert Goldenkoff:
Director:
Strategic Issues:
[End of section]
Appendix I: Comments from the Department of Commerce:
United States Department Of Commerce:
The Secretary of Commerce:
Washington, D.C. 20230:
April 5, 2010:
Mr. Robert Goldenkoff:
Director:
Strategic Issues:
U.S. Government Accountability Office:
Washington, DC 20548:
Dear Mr. Goldenkoff:
The Department of Commerce (Department) appreciates the opportunity to
comment on the U.S. Government Accountability Office's draft report
entitled "2010 Census: Plans for Census Coverage Measurement Are on
Track, but Additional Steps Will Improve Its Usefulness" (GA040-324).
The Department's comments on this report are enclosed.
Sincerely,
Signed by:
Gary Locke:
Enclosure:
[End of letter]
U.S. Department of Commerce Comments on the United States Government
Accountability Office Draft Report Entitled "2010 Census: Plans for
Census Coverage Measurement Are on Track, but Additional Steps Will
Improve Its Usefulness," GAO 10-324.
March 2010:
The U.S. Census Bureau would like to thank the Government
Accountability Office (GAO) for its efforts in examining the 2010
Census Coverage Measurement (CCM) program plans for assessing the
accuracy of the 2010 Census, and for the opportunity to comment on the
observations and recommendations in this report.
We have no fundamental disagreements with the overall findings or the
recommendations regarding improvements in our evaluation processes for
the 2020 Census cycle. We do, however, offer the following comments on
certain statements and conclusions in this report.
* Page 6, second bullet: "The Bureau will not decide the level of
geography for which to release estimates before knowing the quality of
the results."
Census Bureau response: In response to advice from various advisory
panels and after additional research in the spring of 2010, we will
make public the proposed geographic levels of the estimates below the
state level (already planned).
* Page 6, third bullet: "Additional studies are expected outside the
formal evaluation program."
Census Bureau response: We have recently developed a study on recall
bias to try to measure some of the effects of scheduling Person
Interviewing at various periods of time following the census
enumeration. This study is not yet part of the formal evaluation
program. In addition, to document the results from CCM estimation”
including modeling, missing data, and errors in the estimates”we will
produce a series of memorandums. The results will not come from
experiments or evaluations but will be based on actual data and
estimates from the CCM. This highly technical documentation is similar
to our approach for Census 2000.
* Page 9: "The Bureau's Master Trace Project is another area where
additional efforts are needed to ensure useful CCM results."
Census Bureau response: The Census Bureau is currently taking actions
in accordance with GAO's recommendation on retaining and linking 2010
Census data (including CCM data) for purposes of analysis and 2020
census planning. In addition to preserving adequate
documentation, the Census Bureau's immediate focus is to maximize the
amount of data retained from each major decennial system. As the
Census Bureau can strategically leverage the time of key staff members
during the implementation of the 2010 Census, the polishing of data
documentation and the development of guidelines on how to link data
from different decennial systems will take final form. As part of this
process, we plan to save all data collection and processing files from
the CCM.
* Page 11: "Yet Bureau officials remained concerned about the
possibility [of contamination], since the CCM questions are similar to
follow-up questions used in one of the 2010 follow-up operations."
Census Bureau response: Parts of this census operation are new in 2010
and end later than similar operations did in 2000. Their timing also
pushed back the start of Person Interviewing.
* Page 13: "To help the Bureau ...fully assess the tradeoffs between
starting the data collection earlier ...and starting later...."
Census Bureau response: We ran a careful, and what we consider
definitive, contamination study in 2000. We cannot create a new study
for the 2010 Census now, as its too late,. We would have had to create
a separate Person interviewing panel to undergo an early interview and
measure its contamination. For reasons of cost and response burden, we
do not want to repeat studies unless we expect to learn more from
them. Furthermore, we would not have had an instrument ready to
conduct a contamination test.
In conclusion, we acknowledge the GAO's extensive work in reviewing
these activities, and we appreciate its ongoing efforts to help us
develop a successful evaluation plan for the
2020 Census.
[End of section]
Footnotes:
[1] [hyperlink, http://www.gao.gov/products/GAO-08-414], 2010 Census:
Bureau Needs to Specify How It Will Assess Coverage Follow-up
Techniques and When It Will Produce Coverage Measurement Results
(April 25, 2008).
[2] Robert M. Bell and Michael L. Cohen, eds., Coverage Measurement in
the 2010 Census (Washington, D.C.: The National Academies Press, 2009).
[3] Dep't of Commerce v. U.S. House of Representatives, 525 U.S. 316
(1999) (citing 13 U.S.C. §195.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
