2000 Census
Refinements to Full Count Review Program Could Improve Future Data Quality Gao ID: GAO-02-562 July 3, 2002To ensure the completeness and accuracy of the 2000 census data, Bureau of the Census analysts were to identify, investigate, and document suspected data discrepancies or issues to clear census data files and products for subsequent processing or public release. They were to determine whether and how to correct the data by weighing quality improvements against time and budget constraints. Because the bureau lacked sufficient staff to conduct a full count review on its own, it contracted out some of the work to members of the Federal-State Cooperative Program for Population Estimates (FSCPE). FSCPE documented 1,402 data issues, 29 percent of the 4,809 issues identified by both FSCPE and bureau analysts during the full count review. Of the 4,809 issues, 1,599 dealt with "group quarters," where counts for prisons, nursing homes, dormitories, and other group living facilities differed from what analysts expected. Of the 1,599 group quarters issues, FSCPE identified 567. Discrepancies relating to housing unit counts, population data, and demographic characteristics accounted for 1,150 issues, 375 of which were identified by FSCPE. Overall, of the 4,809 issues identified during review, 4,267 were not subjected to further investigation by the bureau because of insufficient documentation. Because the bureau's preliminary plans for the 2010 Census include a Full Count Review program, several areas warrant improvement. Foremost among these is the need for the bureau to investigate and resolve a larger number of issues before releasing the public law data.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-02-562, 2000 Census: Refinements to Full Count Review Program Could Improve Future Data Quality
This is the accessible text file for GAO report number GAO-02-562
entitled '2000 Census: Refinements to Full Count Review Program Could
Improve Future Data Quality' which was released on July 3, 2002.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
United States General Accounting Office:
GAO:
Report to Congressional Addressees:
July 2002:
2000 Census:
Refinements to Full Count Review Program Could Improve Future Data
Quality:
Contents:
Letter:
Results in Brief:
Scope and Methodology:
Background:
The Numbers and Kinds of Issues Identified During Full Count Review:
The Bureau Resolved Few Issues Prior to Releasing Public Law Data:
Lessons Learned That Can Inform Future Data Clearance Reviews:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Disposition of Data Issues Identified During Full Count
Review by State:
Appendix II: Comments From the Department of Commerce:
Appendix III: GAO Products on the Results of the 2000 Census and
Lessons Learned for a More Cost-Effective Census in 2010:
Tables:
Table 1: Types of Issues Flagged During Full Count Review:
Table 2: Disposition of Data Issues Identified During Full Count
Review:
[End of section]
United States General Accounting Office:
Washington, DC 20548:
July 3, 2002:
The Honorable Henry A. Waxman:
Ranking Minority Member:
Committee on Government Reform:
House of Representatives:
The Honorable William Lacy Clay, Jr.
The Honorable Carolyn B. Maloney:
House of Representatives:
Demographic Full Count Review was one of a number of quality assurance
programs the Bureau of the Census developed for the 2000 decennial
headcount to help ensure the completeness and accuracy of census data.
Analysts were to identify, investigate, and document suspected data
discrepancies or ’issues“ in order to clear census data files and
products for subsequent processing or public release. Bureau reviewers
were to determine whether and how to correct the data by weighing
quality improvements against time and budget constraints.
According to bureau officials, because the bureau lacked sufficient
staff to conduct Full Count Review on its own, it contracted out some
of the analysts‘ work to members of the Federal-State Cooperative
Program for Population Estimates (FSCPE), an organization composed of
state demographers that, since its inception in 1973, has worked with
the bureau to ensure accurate state and local population estimates. The
bureau believed that FSCPE members‘ knowledge of the demographic
characteristics of their respective states could help the bureau review
data files and products, including politically sensitive apportionment
and redistricting data files known as public law data. [Footnote 1]
This letter responds to your request to review the FSCPE members‘
participation in the Full Count Review program. As agreed with your
offices, we provide information on (1) the number and kinds of data
issues identified by FSCPE members and bureau analysts, (2) how the
bureau used the information developed during Full Count Review, and (3)
lessons learned from the conduct of Full Count Review that can help the
bureau plan for the 2010 Census. This report is the latest in a series
of reviews that examine the results of key census-taking operations and
highlight opportunities for reform (see app. III for a list of products
issued to date).
Results in Brief:
FSCPE members documented 1,402 data issues, which was about 29 percent
of the 4,809 issues identified by both FSCPE and bureau analysts during
Full Count Review. Of the 4,809 issues, 1,599 (33 percent) dealt with
’group quarters,“ where the location or population counts for prisons,
nursing homes, dormitories, and other group living facilities differed
from what analysts expected. Of the 1,599 group quarters issues, FSCPE
members identified 567 (35 percent). Discrepancies relating to housing
unit counts, population data, and demographic characteristics accounted
for 1,150 issues (24 percent), 375 of which (33 percent) were
identified by FSCPE members. The bureau was unable to classify 2,060
issues (43 percent) because of insufficient documentation.
According to bureau officials, 5 of the 4,809 issues identified during
Full Count Review were corrected prior to the December 31, 2000,
release of apportionment data, and the April 1, 2001, release of
redistricting data. All five were group quarters issues where the
bureau had the correct population count for each facility but placed
them in the wrong locations. The ’misplaced“ group quarters included a
military base, a federal medical center, and multiple facilities at two
prisons and a college. The bureau was able to correct these issues
because FSCPE members identified them early in the Full Count Review
process and thoroughly documented them so that they did not require
additional research or field verification. Because the bureau lacked
the time and field staff from its regional offices to do any further
investigative work, the 4,804 remaining issues went unresolved prior to
the release of the redistricting data. As a result, uncertainties
surround the accuracy of the census data for the affected localities.
Overall, of the 4,809 issues identified during Full Count Review, 4,267
(89 percent) were not subjected to further investigation by the bureau
in large part because of insufficient documentation. The bureau plans
to review 537 issues as part of a subsequent effort called the Count
Question Resolution (CQR) program, which the bureau developed to
respond to challenges to the census data brought by state, local, or
tribal governments (see app. I for the disposition of Full Count Review
data issues by state).
The bureau‘s preliminary plans for the 2010 Census include a Full Count
Review program. Our examination identified several areas where
improvements are warranted. Foremost among these is that it will be
important for the bureau to investigate and resolve a larger number of
issues prior to the release of the public law data. We found three
factors that limited the bureau‘s ability to do so for the 2000 Census.
First, bureau officials said that correcting individual issues was
beyond the scope of the Full Count Review program. They noted that the
program was developed in February 1999, just 14 months prior to Census
Day 2000 and, as a result, the bureau was unable to test the program,
or integrate it with other census operations and units that could have
investigated the issues and made corrections.
Second, the bureau‘s requirements for documenting data issues were not
clearly defined. As a result, there was considerable variation in the
quality of the documentation analysts used to support issues, and in a
number of cases, the bureau had difficulty determining the precise
nature of an issue, or if in fact an issue even existed.
A third, and related item that limited the bureau‘s ability to resolve
a larger number of issues was the fact that the bureau had no mechanism
for managing its workload. Unlike the CQR program where the bureau
requires specific documentation before committing resources to
investigate local issues, the Full Count Review program had no filter
for screening submissions. Had the bureau first categorized issues
based on the quality and precision of the documentation provided, the
bureau could have prioritized its investigative workload and perhaps
reconciled a larger number of data issues.
Another area where there is room for improvement concerns the
consistency and clarity of the Full Count Review program‘s objectives.
For example, training materials noted that one purpose of the Full Count
Review program was to document issues and ’fix what can be fixed.“
However, this conflicts with statements from bureau managers that
correcting issues was outside the scope of the Full Count Review
program. The different messages may have raised FSCPE members‘
expectations that the bureau was going to correct a larger number of
data issues than it actually did.
That the apportionment and redistricting data were released with around
4,800 unresolved data issues of unknown validity, magnitude, and impact
is cause for concern. The bureau did not fully exploit the Full Count
Review program‘s potential, and missed an opportunity to verify and
possibly improve the quality of the public law data. To help develop the
Full Count Review program into a more effective tool for enhancing data
quality in the future, as well as to make better use of FSCPE members‘
input, we recommend that the Secretary of Commerce explore ways of
reconciling a larger number of data issues prior to the release of
public law data. Specific steps include:
(1) planning the Full Count Review program early in the census cycle and
testing procedures under conditions as close to the actual census as
possible;
(2) integrating the Full Count Review program with other census
organizational units and operations to ensure the bureau has sufficient
time and field support to investigate issues;
(3) developing clear guidelines on the minimum documentation needed for
the bureau to investigate individual data issues;
(4) categorizing issues on the basis of the quality and precision of the
documentation, and investigating first those issues that are best
documented and thus more easily resolved, and;
(5) exploring the feasibility of using staff from the bureau‘s regional
offices to help investigate data issues in the field prior to the
release of public law data.
The Secretary of Commerce should also ensure that the bureau clarifies
and consistently communicates to FSCPE members the objectives of the
Full Count Review program and how the bureau plans to use the
information derived from it.
The Secretary of Commerce forwarded written comments from the Bureau
of the Census on a draft of this report (see app. II). The bureau
concurred with all of our recommendations and had no comments on them.
The bureau also provided minor technical corrections that we
incorporated in our report as appropriate.
Scope and Methodology:
To obtain information on the number and kinds of issues identified by
the FSCPE and bureau analysts and to determine how the bureau used the
information developed during the Full Count Review program, we analyzed
the work papers submitted by FSCPE members and other participants in
the Full Count Review program. We also analyzed data from the bureau‘s
Count Review Information System, a database that the bureau used to
track issues flagged during the review process. We did not
independently verify the information it contained.
To identify lessons learned for future improvements, we examined bureau
training manuals, statements of work, process models, and other
documents that described the objectives, processes, and decision-making
criteria. We also reviewed the results of a survey the bureau conducted
of FSCPE members that asked them to rate their experience with Full
Count Review processes and tools, bureau staff, and the overall
effectiveness of the Full Count Review program. In addition, we
interviewed managers in the bureau‘s Population Division and other
officials responsible for implementing the Full Count Review program,
as well as three FSCPE members.
We performed our audit in Washington, D.C., and the bureau‘s
headquarters in Suitland, Maryland, between May 2001 and April 2002. Our
work was done in accordance with generally accepted government
auditing standards.
On April 26, 2002, we requested comments on a draft of this report from
the Secretary of Commerce. The Secretary forwarded the bureau‘s written
comments on June 11, 2002 (see app. II). We address them in the ’Agency
Comments and Evaluation“ section of this report.
Background:
Accurate census results are critical because the data are used to
reapportion seats in the House of Representatives and for congressional
redistricting. Moreover, census data remain an important element in
allocating federal aid to state and local governments. With billions of
dollars at stake, the data are scrutinized intensely for accuracy.
To help ensure the accuracy of census data, the bureau conducted a
number of quality assurance programs throughout the course of the
census. One such program was the Full Count Review program, which was
designed to rapidly examine, rectify if possible, and clear census data
files and products for subsequent processing or public release. The
bureau expected data analysts to identify data discrepancies,
anomalies, and other data ’issues“ by checking the data for its overall
reasonableness, as well as for its consistency with historical and
demographic data, and other census data products. The Full Count Review
program ran from June 2000 through March 2001.
According to bureau officials, because the bureau could not complete the
Full Count Review workload without a costly staff increase, some of the
analysts‘ work was contracted to members of the FSCPE, an organization
composed of state demographers that works with the bureau to ensure
accurate state and local population estimates.
The bureau contracted with 53 FSCPE members who reviewed data for 39
states and Puerto Rico. Bureau employees reviewed data for the 11
remaining states and the District of Columbia without FSCPE
representation in Full Count Review. Bureau and FSCPE analysts were to
ensure that (1) group quarters were correctly placed or ’geocoded“ on
census maps, and that their population counts and demographic
characteristics appeared reasonable and (2) population counts of other
areas were in line with population estimates. They were to describe each
issue flagged and provide supporting documentation derived from bureau
resources and/or resources of the respective state government.
Additionally, bureau officials stated that staff from the regional
offices reviewed demographic data from the 50 states, Puerto Rico, and
the District of Columbia. They focused on identifying inconsistent
demographic characteristics and did not necessarily concentrate on any
one particular state or locality. The bureau reimbursed state
governments for wages and expenses FSCPE members incurred.
A separate set of employees from the bureau‘s Population Division
assessed issues identified by Full Count Review analysts based on (1)
the adequacy of the documentation supporting each issue, and (2)
whether or not they believed the issue to be resolvable through follow-
up research by the bureau. Those issues deemed to have adequate
documentation were classified as a ’group quarters,“ ’housing unit,“ or
’household“ or ’other“ issue. Bureau officials told us that the
remaining issues could not be categorized because the nature of the
issue could not be determined from the documentation.
The Numbers and Kinds of Issues Identified During Full Count Review:
Bureau data show that after reviewing census data for 39 states and
Puerto Rico, FSCPE members identified a total of 1,402 issues, or about
29 percent of the 4,809 issues collectively flagged during Full Count
Review (see table 1). Since the bureau has yet to resolve most of these
issues, it is not known whether they are necessarily errors.
Table 1: Types of Issues Flagged During Full Count Review:
Type of issue: Group quarters;
Issues identified by FSCPE members: 567;
Issues identified by bureau personnel: 1,032;
Total: 1,599.
Type of issue: Housing unit;
Issues identified by FSCPE members: 203;
Issues identified by bureau personnel: 276;
Total: 479.
Type of issue: Household;
Issues identified by FSCPE members: 134;
Issues identified by bureau personnel: 154;
Total: 288.
Type of issue: Other;
Issues identified by FSCPE members: 38;
Issues identified by bureau personnel: 345;
Total: 383.
Type of issue: No type assigned;
Issues identified by FSCPE members: 460;
Issues identified by bureau personnel: 1,600;
Total: 2,060.
Type of issue: Total;
Issues identified by FSCPE members: 1,402;
Issues identified by bureau personnel: 3,407;
Total: 4,809.
Note: 201 issues were placed in more than one category.
Source: GAO analysis of U.S. Census Bureau data.
[End of table]
Table 1 also shows that group quarters issues were those most frequently
identified by the bureau, accounting for 1,599 of the 4,809 issues
identified (33 percent). Group quarters issues relate to suspected
discrepancies in the population counts and locations of prisons,
dormitories, nursing homes, and similar group living arrangements.
Analysts also identified 479 housing unit issues (10 percent of the
total), and 288 household issues (6 percent of the total). With housing
unit issues, the count of occupied housing units differed from what
analysts expected while household issues had population data for
occupied residences that differed from what analysts expected. There
were also 383 issues (8 percent) that the bureau classified as ’other“.
They contained questions concerning the demographic characteristics of
the data such as age, race, and gender. The bureau was unable to
classify 2,060 issues (43 percent). Bureau officials told us that in
these cases, analysts did not provide sufficient documentation for the
bureau to determine the nature of the issue.
According to bureau officials, bureau analysts identified a larger
number of issues than FSCPE members”and a far larger number of issues
for which the bureau could not assign a type”because bureau analysts
used an automated process that compared data from the 2000 Census to
independent benchmarks such as the 1990 Census, and flagged any
anomalies. This process alerted bureau officials that there were data
discrepancies, but did not indicate their nature. By comparison, FSCPE
members compared census data to administrative records and other data,
and were better able to document specific issues.
Examples of the three issue categories and how they were found include:
Group quarters issues: Analysts noticed that the group quarters
population count in a particular census tract of a large midwestern
city appeared to be too high, while a neighboring tract had a
correspondingly low group quarters population count. By comparing state
administrative records to information obtained from bureau resources,
analysts determined that bureau data had placed college dormitories in
the wrong tract.
Housing unit issues: An urban area had a large amount of redevelopment
since the 1990 census. As part of this, several condominiums and
apartment complexes were built which substantially increased the number
of housing units in a particular census tract. However, when the analyst
compared population data from the 1990 Census and 2000 Census, the
2000 Census did not appear to reflect this increase, and it was
flagged.
Household issues: Data from the 2000 Census appeared to accurately
reflect the large amount of new house construction that had taken place
within a specific census tract. However, because the population count
differed from that indicated by other data sources, the analyst flagged
it as an issue to avoid undercounting the population.
The Bureau Resolved Few Issues Prior to Releasing Public Law Data:
Bureau officials told us that they used the Full Count Review program to
identify systemic errors such as those that could be produced by
software problems. None were found. The officials noted that the bureau
generally did not use the Full Count Review program to resolve
individual issues. According to bureau officials, the bureau corrected
data for 5 of the 4,809 issues prior to the December 31, 2000, release
of reapportionment data and the April 1, 2001, release of redistricting
data.
According to bureau officials, FSCPE members identified the five issues,
all of which involved group quarters that were placed in the wrong
locations, but the population counts were correct. They included (1) a
military base in Nevada, (2) 10 facilities at a college in Wisconsin,
(3) 9 facilities at a prison in New York City, (4) 14 facilities at a
Washington prison, and (5) a federal medical center in Massachusetts.
Bureau officials said that the bureau was able to correct these issues
for two reasons. First, FSCPE analysts found them early in the Full
Count Review program, while the bureau was processing a key geographic
data file and was thus able to incorporate the corrections before the
data were finalized. Second, the FSCPE analysts had thoroughly
documented the issues and recommended how the bureau should correct the
errors. The five errors did not require additional research or field
verification.
Bureau officials told us that they lacked the time to research the
remaining issues, as well as field staff to inspect purported
discrepancies prior to the release of the public law data. As a result,
the bureau missed an important opportunity to verify and possibly
improve the quality of the data, and instead the apportionment and
redistricting data were released with more than 4,800 unresolved
issues. Until these issues are resolved, uncertainties will surround
the accuracy of the census data for the affected localities.
Some of the issues might be resolved under the CQR program, which the
bureau designed to respond to challenges to housing unit and group
quarters population counts received from state, local, or tribal
governments. [Footnote 2] However, as shown in table 2, of the 4,804
issues remaining after Full Count Review, 1,994 (42 percent) were
referred to CQR, and of these, 537 (11 percent) were accepted for
further investigation. The remaining 1,457 issues referred to CQR did
not meet the bureau‘s documentation requirements and consequently, the
bureau took no further action on them (see app. 1 for the disposition
of Full Count Review data issues by state).
Table 2: Disposition of Data Issues Identified During Full Count
Review:
Action: Identified during Full Count Review;
Number of issues: 4,809.
Action: Corrected during Full Count Review;
Number of issues: 5.
Action: Identified during Full Count Review but not referred to CQR;
Number of issues: 2,810.
Action: Referred to CQR;
Number of issues: 1,994.
Action: Accepted by CQR;
Number of issues: 537.
Action: Rejected by CQR;
Number of issues: 1,457.
Source: GAO analysis of U.S. Census Bureau data.
[End of table]
Lessons Learned That Can Inform Future Data Clearance Reviews:
The overall results of the Full Count Review program and FSCPE members‘
participation appear to be mixed. On the one hand, the bureau reported
that the Full Count Review program was successful in that it met a
number of performance goals. For example, the bureau reported that the
Full Count Review program was comprehensive in its review of geography
and content, and was completed in time to release the public law data on
schedule.
Moreover, between January and February 2001, the bureau surveyed the
40 entities that participated in Full Count Review and the results
suggest that most FSCPE members were satisfied with their Full Count
Review experience. For example, respondents indicated that they were
generally satisfied with such aspects of the program as its processes
and technical tools, bureau staff, and the overall effectiveness of the
review in terms of positioning states to use and understand census
data. In addition, bureau officials believe the Full Count Review
program benefited from FSCPE members‘ local demographic knowledge.
Nevertheless, our review of the Full Count Review program highlighted
several areas where there is room for future improvement. It will be
important for the bureau to address these shortcomings as its
preliminary plans call for a similar operation as part of the 2010
Census. According to bureau officials, the bureau plans to include a
Full Count Review program in census tests it expects to conduct later
in the decade.
Foremost among the areas in need of improvement is resolving, to the
extent practical, a larger number of data issues prior to the release of
apportionment data by December 31 of the census year, and redistricting
data by April 1 of the following year. We found three factors that
limited the bureau‘s ability to do so.
First, according to bureau officials, resolving individual issues was
outside the scope of the Full Count Review program. They explained that
the program was poorly integrated with other census operations and
units that could have investigated the issues and corrected the data if
warranted. This was because the Full Count Review program, with FSCPE
participation, was not conceived until February 1999, which was
extremely late in the census cycle, coming just 14 months before Census
Day, April 1, 2000.
The timing of the decision stemmed from the Supreme Court‘s January
1999 ruling [Footnote 3] that prohibited the bureau from using
statistical sampling for purposes of congressional apportionment (the
bureau originally planned a ’one-number“ census that would have
integrated the results of a sample survey with the traditional census
to provide one adjusted set of census numbers). Faced with the larger
workload of reviewing two sets of data”adjusted and unadjusted”the
bureau decided to enlist the help of FSCPE members in order to meet the
deadlines for releasing the public law data. Additionally, the bureau‘s
decision came after the 1998 dress rehearsal for the 2000 Census, which
meant that the bureau had no opportunity to test the Full Count Review
program in an operational environment.
Bureau officials explained that if more time or staff were available in
the future, it would be possible to correct a larger number of
individual issues prior to the release of the public law data. They
noted that field staff would be needed to help verify issues, and the
effort would require close coordination with several bureau units.
A second factor that affected the bureau‘s ability to correct a larger
number of issues was that the bureau‘s requirements for documenting data
issues were not clearly defined. For example, the training materials we
examined did not provide any specific guidance on the type of evidence
analysts needed to support data issues. Instead, the training materials
told analysts to supply as much supporting information as necessary.
This could help explain the variation that we observed in the quality
of the documentation analysts provided. Indeed, while some analysts
provided only minimal data, others supported issues with state and local
administrative records, historical data, photographs, and maps. In some
cases, the bureau had difficulty determining the precise nature of an
issue or if in fact an issue even existed.
In contrast, the CQR program provides comprehensive guidelines on the
documentation required for making submissions. The guidance available
on the bureau‘s CQR web site notes that before the bureau will
investigate concerns raised by government and tribal officials, such
officials must first supply specific information. The guidance then
details the information needed to support boundary corrections,
geocoding and coverage corrections, and group quarters population
corrections.
A third, and related factor that affected the bureau‘s ability to
resolve a larger number of issues stemmed from the fact that the bureau
had no mechanism for managing the Full Count Review workload. Unlike the
CQR program, where the bureau required local governments to provide
specific documentation before it would commit resources to investigate
local data issues, the Full Count Review program had no filter for
screening submissions based on the quality of the documentation. Better
guidance on documenting issues for the Full Count Review program could
make the bureau‘s follow-up investigations more efficient.
Another area where there is room for improvement concerns the
consistency and clarity in which the bureau communicated the objectives
of the Full Count Review program and how the bureau planned to use
analysts‘ input. For example, materials used to train FSCPE members
noted that one purpose of Full Count Review was to document issues and
’fix what can be fixed.“ However, this appears to be inconsistent with
statements made by bureau officials, who noted that resolving individual
issues was beyond the scope of the Full Count Review program. Moreover,
according to one bureau official, it was not clear internally what was
meant by ’fix what can be fixed.“
None of the bureau‘s documentation or training manuals that we reviewed
explicitly stated that the bureau would only check for systemic errors.
Because of the inconsistent message on the purpose of the Full Count
Review program, the bureau may have set up the expectation that a larger
number of issues would be resolved during Full Count Review. For
example, one FSCPE member told us that he expected FSCPE members would
identify any geographic discrepancies that contrasted with preliminary
census data, and the bureau would investigate and make the necessary
changes. He noted that both he and his staff were very ’dismayed“ to
find out that certain discrepancies involving group quarters were not
resolved prior to the release of the public law data. Another FSCPE
member told us that participants were strongly motivated by the
expectation that everything would be done to correct the census data.
Conclusions:
The Full Count Review program was one of a series of quality assurance
efforts the bureau implemented throughout the census that helped ensure
the bureau released accurate data. Moreover, FSCPE members‘
participation, and specifically their expertise and knowledge of local
geography, demographics, and housing arrangements, had the potential to
identify data issues that the bureau might have otherwise missed.
However, the fact that the apportionment and redistricting data were
released with around 4,800 unresolved data issues of unknown validity,
magnitude, and impact, is cause for concern, and indicates that the
bureau missed an opportunity to verify and possibly improve the quality
of the public law data. Given the importance of accurate census data
and the resources that bureau staff and FSCPE members invest in the
Full Count Review program, it will be important for the bureau to
explore how to make better use of the program for correcting potential
errors in census data in the future.
It will also be important for the bureau to clarify the purpose of the
Full Count Review program and convey that purpose clearly and
consistently to FSCPE members. Doing so could help ensure that the
bureau meets FSCPE members‘ expectations.
Recommendations for Executive Action:
To help ensure the accuracy and completeness of census data and take
full advantage of the Full Count Review program and FSCPE members‘
participation, we recommend that the Secretary of Commerce direct the
bureau to develop ways to resolve a larger number of data issues prior
to the release of the public law data. Specifically, consideration
should be given to:
(1) planning the Full Count Review program early in the census cycle and
testing procedures under conditions as close to the actual census as
possible;
(2) integrating the Full Count Review program with other census
organizational units and operations to ensure the bureau has sufficient
time and field support to investigate issues;
(3) developing clear guidelines on the minimum documentation needed for
the bureau to investigate individual data issues;
(4) categorizing issues on the basis of the quality and precision of the
documentation, and investigating first those issues that are best
documented and thus more easily resolved, and;
(5) exploring the feasibility of using staff from the bureau‘s regional
offices to help investigate data issues in the field prior to the
release of public law data.
Moreover, to ensure no expectation gaps develop between the bureau and
FSCPE members, the Secretary of Commerce should also ensure that the
bureau clarifies and consistently communicates to participants the
objectives of the Full Count Review program and how the bureau plans to
use the information derived from it.
Agency Comments and Our Evaluation:
The Secretary of Commerce forwarded written comments from the Bureau
of the Census on a draft of this report (see app. II). The bureau
concurred with all of our recommendations and had no comments on them.
The bureau also provided minor technical corrections that we
incorporated in our report as appropriate.
As agreed with your office, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days
from its issue date. At that time, we will send copies to other
interested congressional committees, the Secretary of Commerce, and the
Director of the Bureau of the Census. Copies will be made available to
others upon request. In addition, the report will be available at no
charge on the GAO Web site at [hyperlink, http://www.gao.gov]. Corinna
Wengryn, Ty Mitchell, and Robert Goldenkoff made major contributions to
this report. If you have any questions concerning this report, please
contact me on (202) 512-6806.
Signed by:
Patricia A. Dalton:
Director:
Strategic Issues:
[End of section]
Appendix I: Disposition of Data Issues Identified During Full Count
Review by State:
State: Alabama;
Issues identified by FSCPE members: 18;
Issues identified by bureau personnel: 54;
Total accepted for research by Count Question Resolution program: 3.
State: Alaska;
Issues identified by FSCPE members: 21;
Issues identified by bureau personnel: 77;
Total accepted for research by Count Question Resolution program: 4.
State: Arizona;
Issues identified by FSCPE members: 17;
Issues identified by bureau personnel: 57;
Total accepted for research by Count Question Resolution program: 5.
State: Arkansas;
Issues identified by FSCPE members: 65;
Issues identified by bureau personnel: 72;
Total accepted for research by Count Question Resolution program: 16.
State: California;
Issues identified by FSCPE members: 20;
Issues identified by bureau personnel: 71;
Total accepted for research by Count Question Resolution program: 12.
State: Colorado;
Issues identified by FSCPE members: 36;
Issues identified by bureau personnel: 29;
Total accepted for research by Count Question Resolution program: 5.
State: Connecticut;
Issues identified by FSCPE members: 0;
Issues identified by bureau personnel: 70;
Total accepted for research by Count Question Resolution program: 16.
State: Delaware;
Issues identified by FSCPE members: 0;
Issues identified by bureau personnel: 67;
Total accepted for research by Count Question Resolution program: 0.
State: District of Columbia;
Issues identified by FSCPE members: 18;
Issues identified by bureau personnel: 61;
Total accepted for research by Count Question Resolution program: 1.
State: Florida;
Issues identified by FSCPE members: 39;
Issues identified by bureau personnel: 61;
Total accepted for research by Count Question Resolution program: 14.
State: Georgia;
Issues identified by FSCPE members: 23;
Issues identified by bureau personnel: 42;
Total accepted for research by Count Question Resolution program: 6.
State: Hawaii;
Issues identified by FSCPE members: 0;
Issues identified by bureau personnel: 127;
Total accepted for research by Count Question Resolution program: 5.
State: Idaho;
Issues identified by FSCPE members: 0;
Issues identified by bureau personnel: 62;
Total accepted for research by Count Question Resolution program: 2.
State: Illinois;
Issues identified by FSCPE members: 112;
Issues identified by bureau personnel: 52;
Total accepted for research by Count Question Resolution program: 15.
State: Indiana;
Issues identified by FSCPE members: 21;
Issues identified by bureau personnel: 55;
Total accepted for research by Count Question Resolution program: 9.
State: Iowa;
Issues identified by FSCPE members: 71;
Issues identified by bureau personnel: 38;
Total accepted for research by Count Question Resolution program: 21.
State: Kansas;
Issues identified by FSCPE members: 19;
Issues identified by bureau personnel: 46;
Total accepted for research by Count Question Resolution program: 6.
State: Kentucky;
Issues identified by FSCPE members: 16;
Issues identified by bureau personnel: 60;
Total accepted for research by Count Question Resolution program: 11.
State: Louisiana;
Issues identified by FSCPE members: 25;
Issues identified by bureau personnel: 66;
Total accepted for research by Count Question Resolution program: 3.
State: Maine;
Issues identified by FSCPE members: 0;
Issues identified by bureau personnel: 67;
Total accepted for research by Count Question Resolution program: 1.
State: Maryland;
Issues identified by FSCPE members: 8;
Issues identified by bureau personnel: 39;
Total accepted for research by Count Question Resolution program: 0.
State: Massachusetts;
Issues identified by FSCPE members: 29;
Issues identified by bureau personnel: 38;
Total accepted for research by Count Question Resolution program: 14.
State: Michigan;
Issues identified by FSCPE members: 23;
Issues identified by bureau personnel: 51;
Total accepted for research by Count Question Resolution program: 32.
State: Minnesota;
Issues identified by FSCPE members: 69;
Issues identified by bureau personnel: 44;
Total accepted for research by Count Question Resolution program: 30.
State: Mississippi;
Issues identified by FSCPE members: 0;
Issues identified by bureau personnel: 123;
Total accepted for research by Count Question Resolution program: 1.
State: Missouri;
Issues identified by FSCPE members: 21;
Issues identified by bureau personnel: 37;
Total accepted for research by Count Question Resolution program: 7.
State: Montana;
Issues identified by FSCPE members: 27;
Issues identified by bureau personnel: 69;
Total accepted for research by Count Question Resolution program: 12.
State: Nebraska;
Issues identified by FSCPE members: 0;
Issues identified by bureau personnel: 93;
Total accepted for research by Count Question Resolution program: 20.
State: Nevada;
Issues identified by FSCPE members: 29;
Issues identified by bureau personnel: 40;
Total accepted for research by Count Question Resolution program: 5.
State: New Hampshire;
Issues identified by FSCPE members: 0;
Issues identified by bureau personnel: 39;
Total accepted for research by Count Question Resolution program: 0.
State: New Jersey;
Issues identified by FSCPE members: 58;
Issues identified by bureau personnel: 52;
Total accepted for research by Count Question Resolution program: 13.
State: New York;
Issues identified by FSCPE members: 33;
Issues identified by bureau personnel: 100;
Total accepted for research by Count Question Resolution program: 33.
State: North Carolina;
Issues identified by FSCPE members: 0;
Issues identified by bureau personnel: 149;
Total accepted for research by Count Question Resolution program: 31.
State: North Dakota;
Issues identified by FSCPE members: 19;
Issues identified by bureau personnel: 47;
Total accepted for research by Count Question Resolution program: 11.
State: Ohio;
Issues identified by FSCPE members: 0;
Issues identified by bureau personnel: 174;
Total accepted for research by Count Question Resolution program: 25.
State: Oklahoma;
Issues identified by FSCPE members: 14;
Issues identified by bureau personnel: 41;
Total accepted for research by Count Question Resolution program: 13.
State: Oregon;
Issues identified by FSCPE members: 7;
Issues identified by bureau personnel: 57;
Total accepted for research by Count Question Resolution program: 3.
State: Pennsylvania;
Issues identified by FSCPE members: 51;
Issues identified by bureau personnel: 56;
Total accepted for research by Count Question Resolution program: 12.
State: Puerto Rico;
Issues identified by FSCPE members: 20;
Issues identified by bureau personnel: 57;
Total accepted for research by Count Question Resolution program: 18.
State: Rhode Island;
Issues identified by FSCPE members: 25;
Issues identified by bureau personnel: 26;
Total accepted for research by Count Question Resolution program: 7.
State: South Carolina;
Issues identified by FSCPE members: 43;
Issues identified by bureau personnel: 51;
Total accepted for research by Count Question Resolution program: 11.
State: South Dakota;
Issues identified by FSCPE members: 33;
Issues identified by bureau personnel: 50;
Total accepted for research by Count Question Resolution program: 11.
State: Tennessee;
Issues identified by FSCPE members: 31;
Issues identified by bureau personnel: 55;
Total accepted for research by Count Question Resolution program: 7.
State: Texas;
Issues identified by FSCPE members: 86;
Issues identified by bureau personnel: 93;
Total accepted for research by Count Question Resolution program: 1.
State: Utah;
Issues identified by FSCPE members: 19;
Issues identified by bureau personnel: 59;
Total accepted for research by Count Question Resolution program: 8.
State: Vermont;
Issues identified by FSCPE members: 112;
Issues identified by bureau personnel: 35;
Total accepted for research by Count Question Resolution program: 12.
State: Virginia;
Issues identified by FSCPE members: 0;
Issues identified by bureau personnel: 135;
Total accepted for research by Count Question Resolution program: 8.
State: Washington;
Issues identified by FSCPE members: 59;
Issues identified by bureau personnel: 52;
Total accepted for research by Count Question Resolution program: 8.
State: West Virginia;
Issues identified by FSCPE members: 7;
Issues identified by bureau personnel: 59;
Total accepted for research by Count Question Resolution program: 4.
State: Wisconsin;
Issues identified by FSCPE members: 0;
Issues identified by bureau personnel: 124;
Total accepted for research by Count Question Resolution program: 11.
State: Wyoming;
Issues identified by FSCPE members: 26;
Issues identified by bureau personnel: 39;
Total accepted for research by Count Question Resolution program: 10.
State: Total;
Issues identified by FSCPE members: 1,402;
Issues identified by bureau personnel: 3,407;
Total accepted for research by Count Question Resolution program: 537.
Note: The table includes 201 issues that were placed in more than one
issue category (the bureau had five categories of issues depending on
the nature of the discrepancy).
Source: GAO analysis of U.S. Census Bureau data.
[End of table]
[End of section]
Appendix II: Comments From the Department of Commerce:
The Secretary Of Commerce:
Washington, D.C. 20230:
June 11, 2002:
Ms. Patricia A. Dalton:
Director, Strategic Issues:
U.S. General Accounting Office:
Washington, DC 20548:
Dear Ms. Dalton:
The Department of Commerce appreciates the opportunity to comment on
the General Accounting Office draft report entitled 2000 Census:
Refinements to Full Count Review Program Could Improve Future Data
Quality. The Department's comments on this report are enclosed.
Warm regards,
Signed by:
Donald L. Evans:
Enclosure:
Comments from the U.S. Department of Commerce:
U.S. Census Bureau:
U.S. General Accounting Office draft report entitled 2000 Census:
Refinements to Full Count Review Program Could Improve Future Data
Quality:
Comments on the Text of the Report:
1. Section: Page 5, last paragraph, continued on page 6 ” "The Full
Count Review program ran from June through August, 2000."
Comment: The period June through August 2000 was the training phase of
the Full Count Review program. The program ran from June 2000 through
March 2001.
2. Section: Page 6, second full paragraph, first sentence ” "The bureau
contracted with 53 SCPE members who reviewed data for 38 states, the
District of Columbia, and Puerto Rico."
Comment: The 53 FSCPE members represented 39 states and Puerto Rico.
The District of Columbia did not participate in the Full Count Review
program.
3. Section: Page 6, second full paragraph, second sentence ” "Bureau
employees reviewed data for the 12 remaining states without FSCPE
representation in Full Count Review."
Comment: The sentence should read "... remaining 11 states ...."
4. Section: Page 7, first paragraph -- "Bureau data show that after
reviewing census data for 38 states, the District of Columbia, and
Puerto Rico, FSCPE members identified ...."
Comment: See comment in response to Item #2 above.
5. Section: Page 7, Table 1.
Comment: In the GAO analysis in Table 1, 201 of the issues were placed
in more than one category. For this reason, the Census Bureau cannot
independently verify the accuracy of these data.
6. Section: Page 9, last paragraph, last sentence ” "In all, of the
4,809 issued [sic] identified during Full Count Review, 4,267 (89
percent), were not subjected to any further review by the bureau ...."
Comment: The 1,457 issues that were referred to the Count Question
Resolution (CQR) program, but rejected, were "subjected to further
review," in the context of the CQR program criteria. That they failed
to meet the criteria for processing under the CQR, and no further
action was taken with respect to these issues, does not mean that no
additional review was undertaken after they were identified during the
Full Count Review program.
7. Section : Page 10, third paragraph, last sentence – "According to
bureau officials, if funding permits, the bureau will include a Full
Count Review program as part of a 2004 test of census-taking
activities."
Comment: the Census Bureau is still developing its plan for the 2004
Census Test and has not determined at this time whether it will include
a Full Count Review program. However, we certainly plan to include such
a program in census tests later in the decade.
Responses to GAO Recommendations:
Census Bureau Response: The Census Bureau concurs with the
recommendations and has no specific comments on them at this time.
[End of enclosure]
[End of section]
Appendix III: GAO Products on the Results of the 2000 Census and
Lessons Learned for a More Cost-Effective Census in 2010:
U.S. General Accounting Office. 2000 Census: Coverage Evaluation
Matching Implemented As Planned, but Census Bureau Should Evaluate
Lessons Learned. GAO-02-297. Washington, D.C.: March 14, 2002.
U.S. General Accounting Office. 2000 Census: Best Practices and Lessons
Learned for a More Cost-Effective Nonresponse Follow-Up. GAO-02-196.
Washington, D.C.: February 11, 2002.
U.S. General Accounting Office. 2000 Census: Coverage Evaluation
Interviewing Overcame Challenges, but Further Research Needed. GAO-
02-26. Washington, D.C.: December 31, 2001.
U.S. General Accounting Office. 2000 Census: Analysis of Fiscal Year
2000 Budget and Internal Control Weaknesses at the U.S. Census Bureau.
GAO-02-30. Washington, D.C.: December 28, 2001.
U.S. General Accounting Office. 2000 Census: Significant Increase in
Cost Per Housing Unit Compared to 1990 Census. GAO-02-31. Washington,
D.C.: December 11, 2001.
U.S. General Accounting Office. 2000 Census: Better Productivity Data
Needed for Future Planning and Budgeting. GAO-02-4. Washington, D.C.:
October 4, 2001.
U.S. General Accounting Office. 2000 Census: Review of Partnership
Program Highlights Best Practices for Future Operations. GAO-01-579.
Washington, D.C.: August 20, 2001.
U.S. General Accounting Office. Decennial Censuses: Historical Data on
Enumerator Productivity Are Limited. GAO-01-208R. Washington, D.C.:
January 5, 2001.
U.S. General Accounting Office. 2000 Census: Information on Short- and
Long-Form Response Rates. GAO/GGD-00-127R. Washington, D.C.: June 7,
2000.
[End of section]
Footnotes:
[1] The Census Act (13 USC sec. 141(a,b)) requires the Secretary of
Commerce to deliver state population counts to the President within 9
months after the census date (for the 2000 Census, this meant no later
than December 31, 2000). State population counts are used to
reapportion seats in the U.S. House of Representatives. The Census Act
also requires the Secretary of Commerce to send census population
tabulations to the states no later than 1 year after the April 1
decennial census date (13 USC sec. 141 (c)). These numbers are used
for redistricting.
[2] CQR began in late June 2001, and is scheduled to end in September
2003.
[3] Department of Commerce v. United States House of Representatives,
525 U.S. 316 (1999).
[End of section]
GAO‘s Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO‘s commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO‘s Web site [hyperlink,
http://www.gao.gov] contains abstracts and fulltext files of current
reports and testimony and an expanding archive of older products. The
Web site features a search engine to help you locate documents using
key words and phrases. You can print these documents in their entirety,
including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as ’Today‘s Reports,“ on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
[hyperlink, http://www.gao.gov] and select ’Subscribe to daily E-mail
alert for newly released products“ under the GAO Reports heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov:
(202) 512-4800:
U.S. General Accounting Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
