Bureau of Justice Statistics
Quality Guidelines Generally Followed for Police-Public Contact Surveys, but Opportunities Exist to Help Assure Agency Independence Gao ID: GAO-07-340 March 30, 2007The Bureau of Justice Statistics (BJS), a statistical agency of the Department of Justice's Office of Justice Programs, produces a recurring national Police-Public Contact Survey documenting contacts between the police and the public, including instances involving the use or threat of force by police. BJS issues public reports and sometimes press releases from survey results. For reports and a press release issued from the 1999 and 2002 surveys (the most recent available), GAO reviewed (1) the extent to which BJS followed quality guidelines to ensure the accuracy and integrity of its survey-related products, and (2) factors that affected whether and how BJS followed available guidelines. GAO reviewed applicable federal data quality guidelines, policy and procedure documents, and interviewed current and former officials familiar with BJS.
BJS followed nearly all quality guidelines for its 1999 and 2002 Police-Public Contact Surveys. Specifically, for the four public reports issued from these surveys, BJS fully followed all data quality guidelines available for reviewing statistical information, obtaining the approval of key decision makers, and publicly disseminating information. These guidelines were issued by the National Research Council, Department of Justice, Justice's Office of Justice Programs, and BJS itself. GAO believes that because BJS followed these guidelines, proper steps were taken to help ensure the accuracy and integrity of the reports. BJS followed 7 of the 10 quality guidelines available for the one press release issued from its 1999 survey, but was not in a position to fully follow 3 other guidelines for reasons discussed below. Two key factors affected whether and how BJS followed quality guidelines. The first concerned different interpretations about certain guideline applicability. BJS considered its survey-related reports--but not its press releases--to be statistical products covered by the National Research Council's guidelines. BJS cited a lack of specificity in these guidelines, which did not specifically state that they were applicable to statistical agency press releases, as a basis for concluding that the survey press releases need not conform to guidelines for statistical products. We believe BJS's position was not unreasonable, and did not find fault with the agency. However, we determined nonetheless that the single press release issued from the 1999 survey was a statistical product, and therefore believe the council's guidelines appropriately applied. Second, certain noncareer appointees outside BJS may, in accordance with Justice Department policy, make decisions about the review, approval, and dissemination of press releases, and BJS press releases are jointly issued with the Justice Department, with input from its Office of Justice Programs. Both conditions can potentially affect BJS's independence. Owing to these conditions, BJS was not, in our view, in a position to meet 3 council quality guidelines related to statistical agency independence, including that it be able to issue statistical products without prior clearance, and control the scope and content of its products. Justice affirmed several of GAO's findings but disagreed with certain GAO conclusions about the applicability of guidelines to a press release. Justice's detailed comments and GAO's response are contained in the report.
GAO-07-340, Bureau of Justice Statistics: Quality Guidelines Generally Followed for Police-Public Contact Surveys, but Opportunities Exist to Help Assure Agency Independence
This is the accessible text file for GAO report number GAO-07-340
entitled 'Bureau of Justice Statistics: Quality Guidelines Generally
Followed for Police-Public Contact Surveys, but Opportunities Exist to
Help Assure Agency Independence' which was released on April 30, 2007.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
United States Government Accountability Office:
GAO:
March 2007:
Bureau Of Justice Statistics:
Quality Guidelines Generally Followed for Police-Public Contact
Surveys, but Opportunities Exist to Help Assure Agency Independence:
GAO-07-340:
GAO Highlights:
Highlights of GAO-07-340, a report to congressional requesters
Why GAO Did This Study:
The Bureau of Justice Statistics (BJS), a statistical agency of the
Department of Justice‘s Office of Justice Programs, produces a
recurring national Police-Public Contact Survey documenting contacts
between the police and the public, including instances involving the
use or threat of force by police. BJS issues public reports and
sometimes press releases from survey results.
For reports and a press release issued from the 1999 and 2002 surveys
(the most recent available), GAO reviewed (1) the extent to which BJS
followed quality guidelines to ensure the accuracy and integrity of its
survey-related products, and (2) factors that affected whether and how
BJS followed available guidelines.
GAO reviewed applicable federal data quality guidelines, policy and
procedure documents, and interviewed current and former officials
familiar with BJS.
What GAO Found:
BJS followed nearly all quality guidelines for its 1999 and 2002 Police-
Public Contact Surveys. Specifically, for the four public reports
issued from these surveys, BJS fully followed all data quality
guidelines available for reviewing statistical information, obtaining
the approval of key decision makers, and publicly disseminating
information. These guidelines were issued by the National Research
Council, Department of Justice, Justice‘s Office of Justice Programs,
and BJS itself. GAO believes that because BJS followed these
guidelines, proper steps were taken to help ensure the accuracy and
integrity of the reports. BJS followed 7 of the 10 quality guidelines
available for the one press release issued from its 1999 survey, but
was not in a position to fully follow 3 other guidelines for reasons
discussed below.
Two key factors affected whether and how BJS followed quality
guidelines. The first concerned different interpretations about certain
guideline applicability. BJS considered its survey-related reports”but
not its press releases”to be statistical products covered by the
National Research Council‘s guidelines. BJS cited a lack of specificity
in these guidelines, which did not specifically state that they were
applicable to statistical agency press releases, as a basis for
concluding that the survey press releases need not conform to
guidelines for statistical products. We believe BJS‘s position was not
unreasonable, and did not find fault with the agency. However, we
determined nonetheless that the single press release issued from the
1999 survey was a statistical product, and therefore believe the
council‘s guidelines appropriately applied. Second, certain noncareer
appointees outside BJS may, in accordance with Justice Department
policy, make decisions about the review, approval, and dissemination of
press releases, and BJS press releases are jointly issued with the
Justice Department, with input from its Office of Justice Programs.
Both conditions can potentially affect BJS‘s independence. Owing to
these conditions, BJS was not, in our view, in a position to meet 3
council quality guidelines related to statistical agency independence,
including that it be able to issue statistical products without prior
clearance, and control the scope and content of its products.
Justice affirmed several of GAO‘s findings but disagreed with certain
GAO conclusions about the applicability of guidelines to a press
release. Justice‘s detailed comments and GAO‘s response are contained
in the report.
Table: Guidelines BJS Followed for Products Issued from 1999 through
2002 Police-Public Contact Surveys:
Reports: Number of reports issued: 4;
Reports: Number of guidelines available: Up to 23[A];
Reports: Number of guidelines fully followed: All;
Press releases: Number of press releases issued: 1[B];
Press releases: Number of guidelines available: 10;
Press releases: Number of guidelines fully followed: 7.
Source: GAO analysis of agency data.
[A] Total guidelines were 24 across the different points in time; 23
was maximum at any given time.
[B] From 1999 survey information.
[End of table]
What GAO Recommends:
In a May 2006 report, GAO recommended that the Office of Management and
Budget (OMB) issue a directive to further safeguard the integrity of
federal statistical data and improve guideline clarity. Such a
directive could help address conflicting interpretations about the
applicability of quality guidelines. Since OMB is still working on this
directive, GAO makes no new recommendations in this report.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-340].
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Brian Lepore at (202) 512-
4523 or leporeb@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
BJS Followed Nearly All Available Quality Guidelines to Help Ensure
Accuracy and Integrity of Products Issued from Police-Public Contact
Surveys:
Differing Views on Applicability of Guidelines Raised Questions about
Clarity, while Roles of Certain Noncareer Appointees during Product
Issuance Process Affected BJS's Independence:
Agency Comments and Our Evaluation:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Applicable Quality Guidelines for Statistical Products
Issued by the Bureau of Justice Statistics and Other Agencies:
Appendix III: Press Release Issued in 2001 Based on BJS's 1999 Police-
Public Contact Survey:
Appendix IV: BJS Processes for the Review, Approval, and Dissemination
of Police-Public Contact Survey Products:
Appendix V: Changes in the Scope and Methodology of the Police-Public
Contact Surveys:
Appendix VI: Comments from the Department of Justice:
Appendix VII: GAO Contact and Acknowledgments:
Tables:
Table 1: Public Products Issued Based on 1999 and 2002 Police-Public
Contact Surveys:
Table 2: Guidelines That BJS Fully Followed for the Reports Issued from
Its 1999 and 2002 Police-Public Contact Surveys:
Table 3: Noncareer Appointees' Roles in the Review, Approval, and
Dissemination of BJS Press Releases:
Table 4: Extent to Which BJS Followed Available Quality Guidelines for
1999 and 2002 Police-Public Survey Products:
Table 5: Police-Public Contact Survey (PPCS) Methodologies for the Four
Surveys: 1996, 1999, 2002, and 2005:
Figures:
Figure 1: Office of Justice Programs Organizational Structure:
Figure 2: Review, Approval, and Dissemination Process for BJS Survey
Reports:
Figure 3: Review, Approval, and Dissemination Process for BJS Survey
Press Releases:
Abbreviations:
BJS: Bureau of Justice Statistics:
CATI: computer-assisted telephone interview:
DOJ: Department of Justice:
FBI: Federal Bureau of Investigation:
NRC: National Research Council:
OCOM: Office of Communications:
OJP: Office of Justice Programs:
OMB: Office of Management and Budget:
PAPI: paper-and-pencil in-person face-to-face interview:
United States Government Accountability Office:
Washington, DC 20548:
March 30, 2007:
Congressional Requesters:
The Bureau of Justice Statistics (BJS), a statistical agency within the
Department of Justice's Office of Justice Programs, collects, analyzes,
publishes, and disseminates information on crime, criminal offenders,
victims of crime, and the operation of justice systems at all levels of
government. A critical aspect of the agency's mission is to produce
impartial, timely, and accurate statistics. An ongoing BJS data
collection activity is the Police-Public Contact Survey--one of many
statistical surveys the agency administers. This recurring, nationally
representative survey of the public is designed to document contacts
between the police and the public and the outcomes of those contacts,
including instances involving the use of force or the threat of force
by the police. BJS issues reports to the public based on the results of
this survey, and these may be accompanied by a press release. To date,
BJS has conducted the survey four times--a pilot survey was conducted
in 1996, and more extensive surveys were done in 1999, 2002, and
2005.[Footnote 1]
The Police-Public Contact Survey involves conducting interviews in
person and by telephone with tens of thousands of individuals to obtain
information about their encounters with police officers. The most
frequently reported reason for such contacts has involved traffic
stops. In this context, interviewers have gathered self-reported data
from motorists on such factors as the reason for the traffic stop and
what occurred after the traffic stop--for example, whether the
individual was ticketed, handcuffed, or searched, and whether the
police used or threatened to use force.
In the spring of 2005, a disagreement arose between the then-director
of BJS and an official outside of BJS but within the Department of
Justice about the statistical content of a draft press release
concerning the results of the 2002 Police-Public Contact Survey (which
at the time was the most recent survey to have been completed), and no
press release was ultimately issued from this survey.
As a result of this incident, some members of Congress, the research
community, and the media raised questions about the methods and
procedures BJS follows in reviewing and approving reports and other
products based on the agency's statistical information, and the extent
of BJS's independence as a statistical agency.
In light of the questions raised about BJS's ability to meet its self-
stated mission as a statistical agency committed to maintaining public
trust and confidence in its statistics, we reviewed what processes are
in place at BJS to help the agency ensure the integrity and
independence of its statistical studies. This report specifically
addresses the following two questions for the 1999 and 2002 Police-
Public Contact Surveys, the two surveys for which products had been
issued as of February 2007: (1) To what extent did BJS follow
guidelines to help ensure the accuracy and integrity of the review,
approval, and dissemination of reports and press releases based on its
surveys? (2) What key factors affected whether and how BJS followed
available guidelines?
To assess the extent to which BJS followed guidelines to help ensure
the accuracy and integrity of its product issuance processes for the
1999 and 2002 Police-Public Contact Surveys, we reviewed quality
guidelines pertaining to the review, approval, and dissemination of
written products issued by the National Research Council (a private,
nonprofit institution that is a component of the congressionally
chartered National Academies), the Department of Justice, the Office of
Justice Programs, and BJS itself. The guidelines that we reviewed
covered the period between February 2001, when the first product based
on the 1999 survey was issued, and June 2006, when the most recent
product based on the 2002 survey was issued. We initially identified a
total of 63 guidelines from the four organizations. Because the
organizations had a number of similar guidelines, we eliminated the
overlap by reducing the list to 24 nonduplicative guidelines. To
determine the extent to which BJS followed each guideline, we developed
a data collection instrument for recording information on whether BJS
fully, partially, or did not at all follow the guideline. For BJS's
1999 and 2002 Police-Public Contact Survey reports and the one press
release that was issued, we asked BJS to assess the extent to which it
followed each of 15 guidelines, and provide us with documentary
evidence supporting its assessment. We completed the instrument for the
remaining 9 guidelines for which we already had sufficient information
from BJS and Office of Justice Programs documents and interviews, and
asked BJS to confirm or not confirm our assessments. Two GAO analysts
reviewed BJS's responses and all available supporting documentary and
testimonial evidence, and made a final determination on the extent to
which we believed BJS followed each guideline. We provided our findings
to BJS for review and comment. We also reviewed guidelines issued by
the Office of Management and Budget (OMB), but did not specifically
assess BJS's practice with respect to following OMB's guidelines,
because OMB issued governmentwide policy and procedural guidance to
federal agencies that called for agencies to develop their own agency-
specific guidelines. To determine key factors that affected whether and
how BJS followed applicable guidelines, we reviewed BJS procedural
documents and interviewed both current and former officials involved in
the review, approval, and dissemination of BJS products, including past
and present BJS directors and assistant attorneys general within the
Office of Justice Programs. In addition, we reviewed pertinent
statutory provisions relating to the roles and responsibilities of
officials with respect to BJS. It was beyond the scope of this review
to address any personnel issues that may have arisen in connection with
the disagreement over the content of the draft press release. Appendix
I provides additional details on our scope and methodology. We
conducted our work between April 2006 and February 2007 in accordance
with generally accepted government auditing standards.
In addition, in response to your interest regarding changes that have
occurred in the scope and methodology of the Police-Public Contact
Surveys between 1996 and 2005, and the reasons for any changes,
appendix V provides detailed information on this issue.
Results in Brief:
For the four issued reports and the one issued press release based on
the 1999 and 2002 Police-Public Contact Surveys, BJS followed nearly
all quality guidelines related to product issuance developed by various
organizations. These reports and press release were issued between 2001
and 2006, and are the most recent products available from the surveys.
Specifically, for the four reports issued from these surveys, we found
that BJS fully followed all product issuance guidelines of the National
Research Council, the Department of Justice, the Office of Justice
Programs, and BJS itself, pertaining to how agencies should review
statistical information, obtain the approval of key decision makers,
and publicly disseminate the information. Because BJS followed these
guidelines, we believe it took proper steps to help ensure the accuracy
and integrity of the processes associated with issuing public reports
based on Police-Public Contact Surveys. BJS concurred with our
analysis. With respect to the single press release issued based on the
Police-Public Contact Survey, we found that BJS was in a position to
fully follow 7 of 10 quality guidelines available for the one press
release issued from the 1999 survey.[Footnote 2] BJS did not, in our
view, fully follow 3 other guidelines pertaining to agency independence
for reasons discussed below.
Two key factors had an impact onaffected whether and how BJS followed
quality guidelines during the review, approval, and dissemination
processes followed for of the press releases that could beproducts
issued from the 1999 and 2002 Police-Public Contact Surveys. First, BJS
believed certain quality guidelines, including those issued by the
National Research Council, applied to its survey-related reports, but
not to press releases. This was did not apply to survey-related press
releases because it BJS considered its reports to be statistical
products covered by the guidelines, and did not consider thempress
releases not to be statistical products, and therefore not covered by
the quality guidelines. BJS cited a lack of specificity in the National
Research Council's guidelines available at the time as a basis for this
conclusion. We believe, however, that while we believe that BJS's
conclusion was not unreasonable, our analysis of the press release
issued led us to conclude--based on its content as opposed to its label
as a press release, and regardless of the procedures used to develop
and issue it--that it was primarily a statistical product. there was
nonetheless sufficient evidence to indicate that the press release
issued from the 1999 survey (a release was not issued from the 2002
survey) was a statistical product and that in our view, it was
therefore appropriate to apply the council's guidelines to it. We
acknowledge that, given the lack of specificity that existed at the
time in the that the National Research Council's councils printed
guidelines did not specifically state that they covered press releases
based on the statistical reports of statistical agencies, , BJS could
reach a different conclusion, and therefore we do not fault BJS for
concluding that the 2001 press release based on the 1999 Police-Public
Contact Survey was exempt from the guidelines. Indeed, we noted in our
May 2006 report on data quality that 2 of 14 statistical agencies we
surveyed stated that there was ambiguity as to whether a statistical
press release was a statistical product. BJS was among the 14
statistical agencies surveyed, but it was not 1of the 2 agencies
reporting ambiguity in whether a statistical press release was a
statistical product. The second factor that has had an impact on
affected whether and how BJS followed guidelines in its product
issuance process concerned is the role that certain noncareer
appointees outside BJS can play in the product issuance process., in
part, as a matter of Justice Department policy--which in turn affects
BJSs position as an independent statistical agency. In particular, the
Assistant Attorney General in the Office of Justice Programs has the
ability to certain noncareer appointees outside of BJS, including the
Assistant Attorney General in the Office of Justice Programs, have the
ability under Department of Justice policy to become involved in the
review, approval, and dissemination of press releases; and press
releases are issued jointly by BJS, the Department of Justice, and the
Office of Justice Programs. Owing to these circumstances, BJS was not
in a position to follow the National Research Council guidelines
recommending that statistical agencies issue statistical products
without prior clearance; and control the scope, content, and timing of
their products--issues relating to BJS's independence as a statistical
agency.
To address conflicting interpretations of certain quality guidelines,
we recommended in a May 2006 report on the quality of federal data that
OMB issue a directive that, among other things, would provide clear
definitions about what guidelines cover.[Footnote 3] As of January
2007, according to an OMB analyst assigned to the effort, OMB was still
working on this directive. Because our May 2006 recommendation to OMB
has not yet been implemented, we are not making any new recommendations
in this report. We continue to believe, however, that implementing our
recommendation could help safeguard the integrity of federal
statistical data. We also believe it would reduce the likelihood that
the type of disagreement discussed in this report would recur, and help
assure the independence of BJS as a statistical agency.
We provided a draft of this report to the Department of Justice and
received written comments from the Justice Department's Office of
Justice Programs' Assistant Attorney General. The Assistant Attorney
General affirmed several of our findings and agreed that a need exists
for clear definitions about what constitutes a statistical product and
that the federal statistical community would benefit from clarity in
this area. The agency disagreed, however, with our characterization of
the 2001 Police-Public Contact Survey press release as a statistical
product, and therefore did not believe that the National Research
Council's or BJS's own quality guidelines were applicable to it. We
maintain that we made a sound decision in applying these guidelines to
the Police-Public Contact Survey press release because we felt that the
content of the press release, which was almost entirely statistical in
nature, was a more important determinant of whether or not it was a
statistical product than the label attached to it. The Assistant
Attorney General's comments appear in appendix VI and our detailed
response to these comments is contained in the report.
Background:
Overview of BJS Mission and Organization:
BJS was established by the Justice Systems Improvement Act of
1979.[Footnote 4] In 1995, OMB identified BJS as one of 10 principal
statistical agencies within the federal government. As defined by
OMB,[Footnote 5] the statistical activities of statistical agencies
include the planning of statistical surveys and studies; and the
collection, processing, or tabulation of statistical data for
publication, dissemination, research, analysis, or program management
and evaluation. BJS publishes annual data on criminal victimization,
populations under correctional supervision, and federal criminal
offenders and case processing. It provides periodic data series on the
administration of law enforcement agencies and correctional facilities,
prosecutorial practices and policies, state court case processing,
felony convictions, the characteristics of correctional populations,
criminal justice expenditure and employment, civil case processing in
state courts, and special studies on other criminal justice topics.
BJS is organizationally located within the Department of Justice's
Office of Justice Programs (see fig. 1). The highest-ranking executives
of BJS (BJS Director) and the department's Office of Justice Programs
(Assistant Attorney General) are both noncareer officials appointed by
the President and confirmed by the Senate. Within BJS, only the
Director is a noncareer appointee.
Figure 1: Office of Justice Programs Organizational Structure:
[See PDF for image]
Source: BJS.
[End of figure]
The BJS Police-Public Contact Survey:
BJS initiated the Police-Public Contact Survey pursuant to a mandate in
the Violent Crime Control and Law Enforcement Act of 1994,[Footnote 6]
which required the Attorney General to collect information on the use
of excessive force by law enforcement officers. The data were to be
used only for research or statistical purposes and were not to contain
any information that could reveal the identity of the victim or any law
enforcement officer. BJS fielded its first pilot survey in 1996 with
the goal of better understanding the types and frequency of contacts
between the police and the public, and the conditions under which force
may be threatened or used. The pilot survey consisted of 6,421
respondents. The three subsequent surveys (in 1999, 2002, and 2005)
consisted of 80,543, 76,910, and 63,943 respondents, respectively.
Multiple reports and press releases may be issued in connection with
any of the surveys. The years in which reports and a single press
release associated with the 1999 and 2002 surveys were issued are shown
in table 1.
Table 1: Public Products Issued Based on 1999 and 2002 Public-Police
Contact Surveys:
Date of survey: 1999;
Years reports issued: 2001, 2002[A,B];
Year press release issued: 2001[E].
Date of survey: 2002;
Years reports issued: 2005, 2006[C,D];
Year press release issued: None issued.
Sources: GAO, BJS.
[A] Contacts between Police and the Public, Findings from the 1999
National Survey, February 2001.
[B] Characteristics of Drivers Stopped by Police, 1999, March 2002.
[C] Contacts between Police and the Public, Findings from the 2002
National Survey, April 2005.
[D] Characteristics of Drivers Stopped by Police, 2002, June 2006.
[E] Force or Threatened Force Used in Less than 1 Percent of All Police-
Public Interactions, March 2001.
[End of table]
Quality Guidelines for Statistical Products:
Over the last several years, various types of guidance have been
developed to help federal agencies such as BJS ensure the integrity of
statistical information. In 1992, in response to requests from Congress
and others as to what constitutes an effective statistical agency, the
National Research Council began issuing best-practice
guidelines.[Footnote 7] According to the Committee on National
Statistics,[Footnote 8] which authored the guidelines, the guidelines
have been widely cited and used by Congress and federal agencies, and
have shaped legislation and executive actions to establish and evaluate
statistical agencies. These recommended guidelines, which BJS and other
statistical agencies may choose to voluntarily follow, cover the
review, approval, and dissemination processes of products issued by
federal statistical agencies. In its guideline document, Principles and
Practices for Federal Statistical Agencies, the National Research
Council indicated, among other things, that statistical agencies should
provide high-quality data, take a strong position of independence, be
perceived to be free of political interference and policy advocacy, and
strive for wide dissemination of their results.[Footnote 9] In
particular, according to the National Research Council, the quality
guidelines are to cover the:
* review process and include verification of sources and results,
disclosure of limitations, and accuracy of results;
* approval process and include who has authority over the content and
timing of the release of a product, and separation of policy from
statistical information; and:
* dissemination process and include the usability of information and
its accessibility to a wide range of people.
In February 2002 and September 2006, pursuant to the Information
Quality Act of 2001, OMB issued policy and procedural guidance to
federal agencies, including statistical agencies such as BJS, directing
them to develop their own quality guidelines to help maximize the
quality, objectivity, utility, and integrity of the information they
disseminate.[Footnote 10] OMB stated that it was essential that federal
statistics be collected, processed, and published in a manner that
guarantees and inspires confidence in their reliability.[Footnote 11]
Specifically, OMB directed federal agencies to "adopt a basic standard
of quality — as a performance goal," and "take appropriate steps to
incorporate information quality criteria into agency dissemination
practices."
In response to OMB's February 2002 guidance, the Department of Justice,
Office of Justice Programs, and BJS issued their own guidelines later
that year. BJS issued a second edition of its guidelines in 2005. In
formulating its guidelines, BJS stated that it sought to provide the
public with additional information regarding its methods for ensuring
the quality, utility, objectivity, and integrity of the statistics it
publicly disseminates. As a component of the Department of Justice, BJS
is governed by its own data quality guidelines, as well as the
information quality guidelines promulgated by the Office of Justice
Programs, Department of Justice, and Office of Management and Budget.
The Department of Justice's Information Quality Guidelines are intended
to (1) provide the department's components with a foundation for
developing their own, more detailed procedures, (2) provide guidance to
component staff, and (3) inform the public of the agency's policies and
procedures.
The Office of Justice Programs' information quality guidelines require
its components--including BJS--to (1) assess the usefulness of the
information to be disseminated to the public by continuously monitoring
information needs, developing new information sources, or revising
existing methods, models, and information products where appropriate;
(2) ensure disseminated information is accurate, clear, complete,
reproducible, and presented in an unbiased manner by using reliable
data sources, sound analytical techniques, and documenting methods and
data sources; and (3) protect information from unauthorized access,
corruption, or revision. As in the case of the Department of Justice's
guidelines, the Office of Justice Programs provides guidance to its
components in developing their own, more specific quality guidelines.
BJS Followed Nearly All Available Quality Guidelines to Help Ensure
Accuracy and Integrity of Products Issued from Police-Public Contact
Surveys:
BJS Fully Followed All Applicable Quality Guidelines for the Four
Reports Issued from the 1999 and 2002 Police-Public Contact Surveys:
For all four reports issued from the two Police-Public Contact Surveys,
we found that BJS fully followed all of the review, approval, and
dissemination guidelines available at the time of issuance. We
considered a guideline to have been fully followed if our independent
analysts determined that all aspects of the guideline were followed.
(Our methodology for how we determined the extent to which BJS followed
the guidelines is explained in app. I.) The extent to which BJS
followed applicable, available guidelines when it issued its Police-
Public Contact Survey reports is shown in table 2.
Table 2: Guidelines That BJS Fully Followed for the Reports Issued from
Its 1999 and 2002 Police-Public Contact Surveys:
Year survey conducted: 1999;
Year report issued: 2001;
Number of applicable guidelines available when report was issued: 10;
Number of applicable guidelines that BJS fully followed: 10.
Year survey conducted: 1999;
Year report issued: 2002;
Number of applicable guidelines available when report was issued: 12;
Number of applicable guidelines that BJS fully followed: 12.
Year survey conducted: 2002[A];
Year report issued: 2005;
Number of applicable guidelines available when report was issued: 23;
Number of applicable guidelines that BJS fully followed: 23.
Year survey conducted: 2002[A];
Year report issued: 2006;
Number of applicable guidelines available when report was issued: 23;
Number of applicable guidelines that BJS fully followed: 23.
Source: GAO analysis:
[A] The number of guidelines varied across the time periods covered.
Across the time periods covered, there were a total of 24 guidelines
applicable to the review, approval, and dissemination of Police-Public
Contact Surveys products. In general, the number of guidelines
increased between February 2001, when the first survey report was
issued, and June 2006, when the latest survey report was issued.
However, one National Research Council guideline was dropped when the
council issued the second edition of its guidelines. Had this guideline
not been dropped, 13 guidelines would have been applicable to BJS's
2002 survey report, instead of 12, and 24 guidelines would have been
applicable to the 2005 and 2006 survey reports, instead of 23. Table 4
in appendix II reflects which guidelines were applicable to each of the
survey products.
[End of table]
For the first report issued from the 1999 survey, we found that BJS
voluntarily followed the National Research Council's 10 applicable
existing guidelines; for the second report, we found that BJS
voluntarily followed those 10, as well as 2 additional guidelines
issued since the first report, for a total of 12. For each of the two
reports based on the 2002 survey, we found that BJS followed all 23
available data quality guidelines that had by then been issued by the
National Research Council, the Department of Justice, the Office of
Justice Programs, and BJS itself.[Footnote 12]
The data quality guidelines that BJS followed describe how agencies
should review statistical information, obtain the approval of key
decision makers, and publicly disseminate the information. While not
all of the guideline-issuing organizations addressed the review,
approval, and dissemination process, in total across the four
organizations--the National Research Council, Department of Justice,
Office of Justice Programs, and Bureau of Justice Statistics--all three
areas were addressed. Some examples of the guidelines that BJS fully
followed in its report issuance process are listed below. (For a
complete list of all available data quality guidelines, see appendix
II.)
* Components of the Department of Justice and Office of Justice
Programs will review all information dissemination products for their
quality (including objectivity, utility, and integrity) before they are
disseminated.
* All BJS reports and other statistical products must be subject to an
objective and appropriate verification process conducted by qualified
BJS staff other than the author of the report.
* The statistical agency has recognition by policy officials outside
the statistical agency of its authority to release statistical
information without prior clearance.
* The statistical agency has authority for professional decisions over
the scope, content, and frequency of data compiled, analyzed, or
published.
* The objectivity of BJS statistics must be vigilantly protected at all
times by BJS staff.
On the basis of our analysis, BJS successfully followed all applicable
quality guidelines for these survey-based statistical reports, which
both BJS and we consider to be statistical products covered by the
guidelines. Thus, we believe the agency took proper steps to help
ensure the accuracy and integrity of the review, approval, and
dissemination processes associated with issuing public reports based on
the two surveys we reviewed. BJS concurred with our analysis. All of
the reports were posted to the BJS Web site, where the information is
to be accessible to the general public.[Footnote 13]
BJS Fully Followed 7 of 10 Applicable Data Quality Guidelines for the
Single Press Release Issued from Its Police-Public Contact Surveys:
For the single press release that was issued--that is, the 2001 press
release based on BJS's 1999 Police-Public Contact Survey--we determined
that BJS fully followed 7 of the 10 applicable National Research
Council guidelines available at the time.[Footnote 14] The 7 federal
data quality guidelines that BJS fully followed are listed below.
(1) A statistical agency should develop an understanding of the
validity and accuracy of its data and convey the resulting measures of
quality to users in ways that are comprehensible to nonexperts.
(2) A statistical agency should use modern statistical theory and sound
statistical practice in all technical work.
(3) A statistical agency has maintenance of a clear distinction between
statistical information and policy interpretations of such information
by the President, the secretary of the department, or others in the
executive branch.
(4) A statistical agency should follow good practice, in reports and
other data releases, in documenting concepts, definitions, data
collection methodologies, and measures of uncertainty, and in
discussing possible sources of error.
(5) Effective dissemination programs include policies for the
preservation of data that guide what data to retain and how they are to
be archived for future secondary analysis.
(6) An agency should have an established publications policy that
describes, for a data collection program, the types of reports and
other data released to be made available, the audiences served, and the
frequency of release.
(7) Dissemination of data and information (basic series, analytic
reports, press releases, public use tapes) should be timely and public.
Avenues of dissemination should be chosen to reach as broad a public as
reasonably possible.
There were 3 other applicable quality guidelines that we determined BJS
was not in a position to follow in connection with this issued press
release, and a press release based on the 2002 Police-Public Contact
Survey findings was not issued. It is important to note that, for
reasons discussed later in this report, BJS officials did not believe
these guidelines were applicable to its press releases in the first
place.
Differing Views on Applicability of Guidelines Raised Questions about
Clarity, Differing Views on Guideline Applicability Raised Questions
About Their Clarity, while the Roles of Certain Noncareer Appointees
during During the Product Issuance Process Affected BJS's Independence:
Two key factors affected whether and how BJS followed quality
guidelines during the review, approval, and dissemination of products
issued from the 1999 and 2002 Police-Public Contact Surveys. Two key
factors had an impact on whether and how BJS voluntarily followed
quality guidelines during the review, approval, and dissemination
processes followed for the press releases that could be issued from the
1999 and 2002 Police-Public Contact Surveys. First, while BJS believed,
as noted earlier, that its survey reports were statistical products
covered by the quality guidelines, it did not believe that BJS believed
certain quality guidelines did not apply to the survey-related press
releases was a statistical product because it did not consider them to
be statistical products covered by the quality guidelines, and . BJS
cited a lack of specificity in the National Research Council's
guidelines available at the time as a basis for this conclusion. We
believe, however, that while BJS's interpretation of the guidelines was
not unreasonable, there was nonetheless sufficient evidence to indicate
for a different interpretation; namely, that this press release was a
statistical product, that the available guidelines did apply, and that
BJS was not in a position to meet three3 of 10 guidelines, in
particular, for the single press release issued from the 1999 survey,
owing to a second factor. This second factor was the role that certain
noncareer appointees outside BJS have the ability to play, pursuant to
Department of Justice policy, in the product issuance process. In
certain instances, the roles of these non-BJS officials meant that BJS
was not in a position to fully follow all guidelines related to agency
independence, and this holds the potential for future actual or
perceived political interference in BJS's product issuance process for
statistical products.
BJS Interpreted Guidelines as Not Applying to Press Releases, Raising
Questions about the Applicability and Clarity of Certain Guidelines:
In both written documentation and oral comments, BJS officials stated
that they believed they were in full conformance with the National
Research Council's guidelines and disagreed with our determination that
the agency was not in a position to follow 3 of 10 guidelines for the
2001 Police-Public Contact Survey press release that was issued from
the 1999 survey. The guidelines that we determined BJS was not in the
position to fully follow all pertain to the agency's independence and,
in particular, to its control over the issuance of press releases.
These guidelines were:
(1) The statistical agency has recognition by policy officials of its
authority to release statistical information without prior clearance.
(2) The statistical agency has authority for professional decisions
over the scope, content, and frequency of data compiled, analyzed, or
published.
(3) The release of information should not be subject to actual or
perceived political interference. In particular, the timing of the
public release of data should be the responsibility of the statistical
agency.
BJS officials asserted that, based on their interpretation of the
National Research Council's guidelines, BJS press releases did not
qualify as statistical products and, therefore, press releases did not
fall within the purview of the council's guidelines. They also asserted
that neither BJS's own quality guidelines, nor those issued by the
Department of Justice and the Office of Justice Programs, apply to BJS
press releases. Both BJS and Office of Justice Programs officials
stated that the applicability of the council's guidelines to BJS press
releases was, at a minimum, open to question because the council did
not state that press releases are data disseminations. In other words,
according to BJS and the Office of Justice Programs, press releases are
not publications of data, but rather they are simply announcements that
a data publication is forthcoming. In its communications with us, BJS
stated that many of the guidelines do not apply to press releases but
apply only to statistical products.
Based on its content rather than its label as a press release, and
notwithstanding that the policies and procedures for developing and
issuing products labeled by the Office of Justice Programs as press
releases differed from policies and procedures for products it labeled
as statistical products, we believe there is sufficient evidence for us
to conclude that the press release issued that BJS jointly issued from
the 1999 Police-Public Contact Survey qualified as a statistical
product to which the National Research Council's quality guidelines
appropriately apply. Our analysis of this press release indicated that
it was a data-based statistical product, more than simply an
announcement that a data publication was forthcoming. In its entirety,
the press release consisted of 20 sentences and one table describing
the survey's statistical findings; 3 sentences on the survey's
methodology; and 5 sentences on who prepared the report and how to
obtain copies. We found that this press release was a compilation of
statistical data that contained no interpretations, conclusions, or
policy statements.[Footnote 15] (See Appendix III for a reproduction of
the press release.) In accordance with the council's guidelines, the
release maintained "a clear distinction between statistical information
and policy interpretations of such information."
To understand whether the National Research Council was purposeful in
not stating that its guidelines were applicable to statistical agency
press releases, we contacted the council to seek clarification.
Officials from the council's Committee on National Statistics, which
authored the data quality guidelines, stated that although the
Principles and Practices document did not specifically state that the
guidelines covered the content, scope, and timing of press releases
issued by statistical agencies, it was not the committee's intent to
exclude press releases from the guidelines. They stated that, in their
view, press releases issued by BJS are statistical products to which it
is appropriate to apply the guidelines.
BJS and we agree that the National Research Council's guidelines apply,
in general, to statistical products. In asserting that the press
release that BJS jointly issued with the Department of Justice and
Office of Justice Programs was not a statistical product, BJS correctly
noted that the National Research Council did not explicitly state that
the guidelines covered press releases. However, given the strong
statistical content of the Police-Public Contact Survey press release,
we did not believe that such an explication was necessary. Nonetheless,
we acknowledge that it is not unreasonable for BJS to could reach a
different conclusion given the lack of specificity that existed in the
council's printed guidelines. Because BJS's own data quality
guidelines, issued in 2002, state that they "govern all justice
statistics that BJS produces and disseminates for the general public,
including all statistics that are featured in BJS publications, on the
website, and in BJS press releases," we considered the BJS guidelines
to be applicable to press releases, as well. BJS, however, did not hold
this view.
It is important to note that we are not finding fault with BJS for the
conclusions it drew with respect to the applicability of the quality
guidelines to its press release issuance process because the National
Research Council's guidelines were not explicit on this matter. Indeed,
we noted in our May 2006 report on data quality that 2 of 14
statistical agencies we surveyed stated that there was ambiguity as to
whether a statistical press release was a statistical product, and if
so, whether statistical agencies could issue them without first getting
releases cleared at the departmental level. BJS was among the 14
statistical agencies surveyed, but it was not one of the two agencies
reporting ambiguity in whether a statistical press release was a
statistical product. Overall, we believe that BJS made a good faith
effort to follow the guidelines it deemed to be applicable to the
Police-Public Contact Survey products.
Deciding which guidelines a statistical agency like BJS should follow
is further complicated by the fact that BJS's parent organizations--the
Department of Justice and Office of Justice Programs--have explicitly
stated that their own guidelines do not apply to press releases.
However, these organizations' guidelines are intended to be broadly
applicable to both statistical and nonstatistical agencies. For
example, the Department of Justice comprises 38 separate component
organizations that produce a variety of types of information, both
statistical and nonstatistical in nature. The Office of Justice
Programs is composed of 6 bureaus and program offices, and these, too,
produce both statistical and nonstatistical information. Because we
believe that press releases issued by the department and the Office of
Justice Programs may in some, but not all, instances be statistical
products, we do not hold the view that statistical guidelines should be
universally applicable to all press releases issued by the Department
of Justice and Office of Justice Programs. However, because different
interpretations can arise, we believe that clarification regarding
which guidelines should be applied under which circumstances--and,
specifically, to press releases--would be helpful to statistical
agencies that are in situations similar to BJS's.
To address potential discrepancies such as these, in a May 2006 report
on the quality of federal data,[Footnote 16] we recommended that to
help improve governmentwide data dissemination practices that would
further safeguard the integrity of federal statistical data, OMB should
consider how best to address the gaps we identified between agencies'
data dissemination practices and the National Research Council's
guidelines. We noted in that report that OMB, in concert with federal
statistical agencies, was developing a governmentwide directive on the
release and dissemination of statistical products that, according to
OMB officials, parallels the council's and other generally accepted
dissemination practices. We pointed out that it will be important for
OMB's directive to consider, for example, whether the directive should
cover principal statistical agencies only, the statistical functions of
all agencies, or only statistical products. OMB officials indicated
that the guidance is intended to help ensure that statistical products
are policy-neutral, timely, and accurate. We recommended that, among
other things, OMB include in this directive (1) clear definitions of
what is and is not covered, (2) the extent to which agencies should
document their data dissemination guidance and how often the guidance
should be reviewed, (3) the amount of flexibility agencies have in
implementing OMB's guidance, and (4) procedures for monitoring
agencies' adherence to its directive. To the extent that statistical
agencies appropriately follow these practices, the directive could
promote more consistent adherence to practices that facilitate broader
dissemination of statistical data and enhance its credibility. Although
OMB did not provide comments on the recommendations in our 2006 report,
an OMB official told us that as of January 2007, OMB was still working
on this directive.[Footnote 17] We believe it remains important for OMB
to complete its directive on the release and dissemination of
statistical products in order to help safeguard the integrity of
federal statistical data, reduce the likelihood that the type of
disagreement discussed in this report would recur, and help assure both
the actual and perceived independence of BJS.
The Roles Played by Certain Outside Noncareer Appointees in the Press
Release Process Affected BJS Carrying Out Its Role as an Independent
Statistical Agency:
The second key In addition to the role that lack of specificity played
in BJSs actions with respect to following certain guidelines for its
issued press release, we believe there is another Another factor that
contributed to whether that affected whether and how had an impact on
affected BJS followed guidelines was in a position to follow these
guidelines in carrying out its role as an independent statistical
agency concerned the involvement of noncareer appointees outside of BJS
in the press release issuance process, and had implications for BJS's
independence as a statistical agency. of noncareer appointees outside
of BJS . Specifically, we determined that BJS was not in a position to
fully follow the 3 National Research Council guidelines listed above in
the previous section for the 2001 press release based on the 1999
survey (the only applicable, available data quality guidelines in place
in 2001) because certain noncareer appointees outside of BJS and within
the Department of Justice, are vested--pursuant to the Department of
Justice's and Office of Justice Programs' policies defining the roles
and responsibilities of their noncareer appointees--with the ability to
participate in the review, approval, and dissemination of press
releases.[Footnote 18] In certain cases, the roles and responsibilities
of these noncareer appointees precluded BJS from being in the position
to fully follow certain guidelines. Specifically, the Assistant
Attorney General within the Department of Justice's Office of Justice
Programs has general statutory responsibilities with respect to
coordinating the activities of that office and its various components,
such as BJS.[Footnote 19] These statutory provisions do not
specifically address the Office of Justice Programs's role with respect
to the review, approval, and dissemination of press releases. However,
under departmental policy, noncareer appointees within the Department
of Justice and outside of BJS have the ability to participate in the
press release issuance process.
Table 3 shows the type of involvement that the Assistant Attorney
General in the Office of Justice Programs and other noncareer
appointees generally have had in the press release review, approval,
and dissemination process. Appendix IV describes in more detail the
responsibilities of these various officials associated with review,
approval, and dissemination procedures for both BJS reports and press
releases.
Table 3: Noncareer Appointees' Roles in the Review, Approval, and
Dissemination of BJS Press Releases:
Area of involvement: Review;
Entity involved in press release process: Bureau of Justice Statistics:
Director: X;
Entity involved in press release process: Office of Justice Programs:
Chief of Staff: X;
Entity involved in press release process: Office of Justice Programs:
Deputy Assistant Attorney General: X;
Entity involved in press release process: Office of Justice Programs:
Assistant Attorney General: X;
Entity involved in press release process: Department of Justice: Office
of Public Affairs Director: X.
Area of involvement: Approval;
Entity involved in press release process: Bureau of Justice Statistics:
Director: X;
Entity involved in press release process: Office of Justice Programs:
Chief of Staff: X;
Entity involved in press release process: Office of Justice Programs:
Deputy Assistant Attorney General: X;
Entity involved in press release process: Office of Justice Programs:
Assistant Attorney General: X;
Entity involved in press release process: Department of Justice: Office
of Public Affairs Director: X.
Area of involvement: Dissemination;
Entity involved in press release process: Bureau of Justice Statistics:
Director: X[A];
Entity involved in press release process: Office of Justice Programs:
Chief of Staff: [Empty];
Entity involved in press release process: Office of Justice Programs:
Deputy Assistant Attorney General: [Empty];
Entity involved in press release process: Office of Justice Programs:
Assistant Attorney General: [Empty];
Entity involved in press release process: Department of Justice: Office
of Public Affairs Director: X[B].
Source: BJS and Office of Justice Programs.
[A] To the BJS web site, after approval by the BJS Director.
[B] To Congress, the media, and executive department press offices,
after verification by the BJS Director.
[End of table]
With respect to the first of the three guidelines, which calls for a
statistical agency to have authority to release information without
prior clearance, it is our view that BJS was not in a position to
follow this independence-related guideline at all because it did not
have the ability to do so. This is because press releases are subject
to review and approval by not only the BJS Director, but also by other
Department of Justice noncareer appointees.[Footnote 20] Outside of
BJS, the noncareer appointees participating in the clearance process
are located in the Department of Justice's Office of Justice Programs
(these include the Office's Chief of Staff, Deputy Assistant Attorney
General, and Assistant Attorney General) and Office of Public Affairs.
The current Assistant Attorney General and two former Assistant
Attorneys General in the Office of Justice Programs told us that there
is no written, formal policy or guidance that bounds their input and
decision-making roles and responsibilities with respect to BJS press
releases. BJS and OJP officials indicated that the Office of Justice
Programs' Assistant Attorney General has ultimate responsibility for
the review and approval of BJS press releases. Press releases are
issued jointly on letterhead listing BJS and the Department of Justice.
The current BJS Director confirmed that publication and dissemination
functions for press releases are considered to be within the Assistant
Attorney General's oversight authority. Because the National Research
Council stated that an aspect of independence includes "recognition by
policy officials outside the statistical agency of its authority to
release statistical information without prior clearance," we concluded
that BJS was not in the position to follow this guideline because, as
we have stated, we believe the Police-Public Contact Survey press
release was a statistical product that BJS could not issue
independently.
In practice, the ways in which Assistant Attorneys General of the
Office of Justice Programs have exercised their authority have varied.
For instance, one former Office of Justice Programs' Assistant Attorney
General told us that she placed "self-imposed" limits on her decisions
to modify the content of a BJS press release based on her awareness of
congressional support for, and her own belief in, the independence of
statistical agencies. The current Office of Justice Programs' Assistant
Attorney General told us that she reviews only press releases that
contain quotes from the Attorney General. She said that since she
assumed her position in 2005, there have been no BJS press releases
that have quoted the Attorney General, and she has relied on her Deputy
Assistant Attorney General, the BJS Director, and others to ensure the
accuracy and clarity of press releases. Nevertheless, the BJS Director
must obtain the approval of the Office of Justice Programs' Assistant
Attorney General and other Justice noncareer appointees to issue a
press release.
With respect to the second guideline, pertaining to the agency's
decisions over the scope, content, and frequency of data compiled,
analyzed, or published, we found that BJS was not in a position to
fully follow this independence-related guideline. Specifically, we
found that BJS could exercise professional decisions about the
frequency of data analyzed and published (within available budgets),
but did not always have complete control over the scope and content of
survey press releases to be issued. As noted above, this was due to the
fact that press releases are joint products of BJS, the Office of
Justice Programs, and the Department of Justice, and noncareer
appointees outside of BJS can become involved in the press release
process. BJS's situation with respect to this second guideline came to
the fore during the drafting of a press release in 2005 based on the
2002 Police-Public Contact Survey. The press release that BJS sought to
publish would have included the following statistical findings from the
accompanying Police-Public Contact Survey report: (1) there was no
statistically significant difference between the rates that white and
minority drivers reported being stopped by police, and (2) once
stopped, a larger percentage of black and Hispanic minority drivers
reported police using or threatening to use force against them than did
whites. The then-BJS Director and the then-Acting Assistant Attorney
General had a difference of opinion regarding the presentation of the
second statistical finding, which was included in the Police-Public
Contact Survey report. Despite reported efforts on the part of both
parties to negotiate alternative language with respect to the content
of the press release, they could not resolve their differences and the
BJS Director decided that a press release would not be issued. The
current BJS Director told us that it is "inconceivable" that the
Assistant Attorney General would issue a press release without the BJS
Director's prior approval.
According to current BJS officials (both career and noncareer) and the
Office of Justice Programs' Office of Communications staff, during the
period 1996-2006, this was the only instance in which a BJS press
release was prepared but not issued because the Office of Justice
Programs and BJS could not agree on the contents. In all other
instances during this period, according to these officials, when the
parties disagreed on the content of a press release, they were able to
resolve their differences.
With respect to the third guideline, pertaining to actual or perceived
political interference and the timing of a release, we similarly
believe BJS was in not in a position to fully follow this independence-
related guideline for the 2001 press release, which, as discussed
earlier, we believe to be a statistical product. Although we found no
evidence of political interference with the timing of the 2001 survey
press release issued from the 1999 survey, we found that BJS does not
have complete control over the timing of press releases, as recommended
by the National Research Council. Since both noncareer appointees and
career officials in the Office of Justice Programs and the Department
of Justice have a role in reviewing and approving BJS press releases,
they can affect the date that a press release is issued. According to
BJS, career and noncareer appointees outside of BJS can delay the
issuance of a press release for reasons having nothing to do with
political interference, such as a determination that the press release
is not sufficiently newsworthy at the time that it was designated to be
issued.
On balance, we believe that the noncareer appointees who played
decision-making roles in the Police-Public Contact Survey press release
process that we reviewed acted within the scope of the roles and
responsibilities accorded them under Department of Justice policies,
and that BJS made a reasonable effort to adhere to all applicable data
quality guidelines. The fact that certain noncareer officials have the
ability to make decisions that affect BJS's ability to fully meet
federal data quality guidelines suggests, however, that the potential
exists for BJS's review, approval, and dissemination process for
statistical products to be subject to political interference. Thus,
certain actions by noncareer appointees--though made on the basis of
professional judgment--could put them at odds with the very guidelines
designed to ensure the statistical independence and integrity of
agencies such as BJS.
Agency Comments and Our Evaluation:
We provided a draft of this report to the Department of Justice for
review and comment. On March 13, we received written comments on the
draft report from the Office of Justice Programs' Assistant Attorney
General, and the comments are reproduced in full in appendix VI.
In her letter, the Assistant Attorney General affirmed several of our
findings and agreed that a need exists for clear definitions about what
quality guidelines cover. She noted that competing interpretations
exist about what constitutes a statistical product and that the federal
statistical community would benefit from clarity in this area.
However, the Assistant Attorney General disagreed with our
characterization of the 2001 Police-Public Contact Survey press release
as a statistical product and, therefore, with our conclusion that the
National Research Council's quality guidelines applied to this press
release. The Assistant Attorney General stated that "a press release —
is a public relations announcement issued to encourage media coverage.
The mere presence of statistics in a press release does not transform a
press release into a statistical product." We do not believe and have
not stated that the mere presence of statistics in a press release in
and of itself transforms it into a statistical product any more than we
believe or have stated that labeling a document lacking in statistics
but called a statistical product necessarily transforms it into one.
The Assistant Attorney General also stated that we "mischaracterized"
BJS's data quality guidelines as applying to press releases because the
guidelines apply only to the statistics contained in BJS press
releases, and because BJS conforms with OMB, the Department of Justice,
and the Office of Justice Programs in considering press releases to be
outside the scope of the guidelines. For the following reasons, we
maintain that we made a sound decision in applying BJS's guidelines to
the Police-Public Contact Survey press release: (1) BJS's guidelines
state that they "govern all justice statistics that BJS produces and
disseminates for the general public, including all statistics that are
featured in BJS publications, on the website, and in BJS press
releases;" and (2) the Police-Public Contact Survey press release was
made up almost entirely of survey statistics, indicating to us that it
was a statistical product. In determining that the Police-Public
Contact Survey was a statistical product, we felt that the content of
the press release was a more important determinant than the label
attached to it, or the fact that the processes and staff involved in
developing the press release were different from those in BJS reports.
The Assistant Attorney General also noted that the National Research
Council's written guidelines did not explicitly cover press releases.
Because we agree, we contacted the National Research Council and
consulted with officials of the Council's Committee on National
Statistics (the authoring committee of the Principles and Practices).
The officials concurred with our view that BJS press releases referring
to statistical products (as opposed to press releases about the
announcement of a new agency head, for example) are statistical
products to which it is appropriate to apply the guidelines. Although
the Principles and Practices document did not specifically state that
the guidelines covered the content, scope, and timing of press releases
issued by statistical agencies, according to these officials it was not
the Committee's intent to exclude such press releases from the
guidelines.
The Assistant Attorney General also felt that the draft report
overstated the potential threats to BJS's independence because we used
the term "statistical products" to refer to press releases. She was
concerned with our observation that the potential exists for BJS's
review, approval, and dissemination process for statistical products to
be subject to political interference since noncareer officials can
affect BJS's ability to meet federal data quality guidelines. We stand
by this conclusion. Department of Justice policy permits noncareer
appointees within the Department but outside of BJS to participate in
the press release process. At the same time however, we are unaware of
anything that prevents future modifications to that policy to similarly
allow noncareer appointees to participate in BJS's report issuance.
Thus, we believe that we have correctly assessed the risk of potential
or actual threats to BJS's independence.
Finally, the Assistant Attorney General stated that even if the
council's written guidelines explicitly applied to press releases, the
BJS director would not adhere to them and no current law can make him
do so. We recognize that they are voluntary and not legally required
and never have said otherwise.
As agreed with your offices, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days
from its issue date. At that time, we will send copies of the report to
the Attorney General, the Director of the Office of Management and
Budget, and other interested parties. In addition, the report will be
available at no charge on GAO's home page at http://www.gao.gov.
Please contact Brian Lepore at (202) 512-4523 or leporeb@gao.gov if you
or your staff have any questions concerning this report. Contact points
for our Offices of Congressional Relations and Public Affairs may be
found on the last page of this report. Key contributors to this report
are listed in appendix VII.
Signed by:
Brian Lepore, Acting Director:
Homeland Security and Justice Issues:
List of Congressional Addressees:
The Honorable John Conyers:
Chairman:
Committee on the Judiciary:
House of Representatives:
The Honorable William Delahunt:
The Honorable Sheila Jackson Lee:
The Honorable Zoe Lofgren:
The Honorable Martin Meehan:
The Honorable Jerrold Nadler:
The Honorable Linda Sanchez:
The Honorable Debbie Wasserman Schultz:
The Honorable Robert C. "Bobby" Scott:
The Honorable Mel Watt:
The Honorable Robert Wexler:
House of Representatives:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
This report addresses the following two objectives for the 1999 and
2002 Police-Public Contact Surveys, the two surveys for which products
had been issued as of February 2007: (1) To what extent did the Bureau
of Justice Statistics (BJS) follow available guidelines to help ensure
the accuracy and integrity of the review, approval, and dissemination
of reports and press releases based on its surveys? (2) What key
factors affected whether and how BJS followed available guidelines? In
addition, we provide information on scope and methodology changes in
the Police-Public Contact Surveys over time (see app. V).
To address the first objective, regarding the extent to which BJS
followed guidelines, we obtained quality guideline documents from BJS,
the Department of Justice's Office of Justice Programs, the Department
of Justice, and the National Research Council. The guidelines that we
obtained from these organizations covered the period between February
2001, when the first product based on the 1999 Police-Public Contact
Survey was issued, and June 2006, when the most recent product based on
the 2002 survey was issued. We included these federal organizations in
our review because BJS is a component of the Office of Justice
Programs, which in turn is a component of the Department of Justice,
and BJS considers itself to be "governed by" the information quality
guidelines of these organizations. We included the National Research
Council in our review because it is a widely recognized organization
that issued guidelines that were intended to be statements of best
practice and provide information on what constitutes an effective
statistical organization. We also reviewed guidance and directives
issued by the Office of Management and Budget (OMB) because OMB is
charged with issuing governmentwide policy and procedural guidance to
federal agencies, which are then encouraged to issue their own
implementation guidelines.
We took several steps to determine the extent to which BJS followed the
specific quality guidelines that it, the Office of Justice Programs,
the Department of Justice, and the National Research Council had
issued. From the documents provided by these four organizations, a GAO
analyst initially identified a total of 63 guidelines that pertained to
product review, approval, and dissemination processes. For verification
purposes, a GAO methodologist also reviewed the guideline documents.
The GAO methodologist agreed with the auditor that all 63 guidelines
were appropriate for inclusion in our review. Because many of the
guidelines issued by the four organizations were similar and
overlapping, the GAO auditor reduced the list to 24 nonduplicative
guidelines. The GAO methodologist again reviewed the work of the
auditor, and in all cases agreed with the auditor that similar
guidelines were being appropriately grouped.
We then developed a data collection instrument to determine whether BJS
was following guidelines for the 1999 and 2002 Police-Public Contact
Surveys, on which information could be recorded as to whether BJS fully
followed, partially followed, or did not at all follow each of the
guidelines. We defined "fully" as all aspects of the guideline being
followed; "partially" as some, but not all, aspects of the guideline
being followed; and "not at all" as no aspects of the guideline being
followed. We asked BJS to complete a separate data collection
instrument with respect to each of its 1999 and 2002 Police-Public
Contact Survey reports and one press release, and to support each
response by providing documentary evidence. To decrease the burden on
BJS, GAO analysts completed the data collection instrument for 9 of the
24 guidelines, for which we already had sufficient information (for
example, documents describing agency processes and procedures, and
interviews regarding the roles and responsibilities of noncareer
appointees).[Footnote 21] We provided our assessments regarding these
guidelines to BJS and asked officials to either confirm or not confirm
them. Two GAO analysts reviewed BJS's responses and all available
supporting documentary and testimonial evidence, and determined whether
BJS fully, partially, or did not at all meet each guideline. We
provided our findings to BJS for review and comment.
BJS's 1996 and 2005 Police-Public Contact Surveys were outside the
scope of our work. We excluded the 1996 survey because that was a
relatively small-scale pilot study; and we excluded the 2005 survey,
the most recent Police-Public Contact Survey conducted, because no
reports or press releases have yet been issued from this work.
To address the second objective, regarding key factors that affected
whether and how BJS followed guidelines, we reviewed processes and
procedures that described the review, approval, and dissemination
processes for BJS-generated reports and press releases, with particular
interest in identifying the roles of noncareer appointees involved in
each of these processes. We also reviewed pertinent statutory
provisions relating to the roles and responsibilities of officials with
respect to BJS. We conducted in-person interviews with, or obtained
written responses to our questions from, noncareer appointees in BJS,
the Office of Justice Programs, and the Department of Justice's Office
of Public Affairs. Specifically, we conducted in-person interviews with
the current BJS Director and the BJS Director who was involved in the
disagreement with the Acting Assistant Attorney General, as well as
with the current Assistant Attorney General and Deputy Assistant
Attorney General in the Office of Justice Programs. We obtained
detailed written responses to our questions from a former BJS Director,
the Acting Assistant Attorney General who was involved in the
disagreement with the BJS Director, and two former Assistant Attorneys
General from the Office of Justice Programs. We conducted a telephone
interview with the current Deputy Director of the Department of
Justice's Office of Public Affairs. Among other things, we asked these
noncareer appointees to provide us with information about BJS's process
for reviewing, approving, and disseminating reports and press releases;
the roles and responsibilities of noncareer appointees in that process;
changes, if any, that had occurred in the roles played by noncareer
appointees; procedures used to help ensure that BJS reports and press
releases were accurate, reliable, and unbiased; and any factors that
may have affected BJS's independence in the product issuance process.
Finally, we reviewed the guidelines of BJS, the Department of Justice's
Office of Justice Programs, the Department of Justice, and the National
Research Council to determine that they reflected the product issuance
processes and to consolidate them in order to eliminate duplication.
To determine what changes, if any, have occurred in the scope and
methodology of the Police-Public Contact Surveys between 1996 and 2006,
which we present in appendix V, we initially developed a matrix of key
scope and methodology dimensions, based on a review of the standard
social science literature. We then conducted interviews and reviewed
documents with respect to these dimensions, for all four Police-Public
Contact Surveys--the 1996 pilot survey and the surveys of 1999, 2002,
and 2005. We interviewed the current BJS Director and a former BJS
Director, and available report authors and the key statistician
participating in administrations of the survey, to ascertain their
views concerning the intended scope of the four surveys, the
methodologies used, scope and methodology changes that were made, and
reasons for any changes. We also obtained written responses to our
questions from these officials. We conducted a detailed documentary
review of the scoping and methodology sections of the issued Police-
Public Contact Survey reports and press releases, and extracted
information about changes in the data collection instruments used (for
example, the numbers and types of questions asked about searches and
the use of force). In addition, we reviewed documents prepared by the
American Statistical Association and the U.S. Bureau of the Census,
which conducted field tests to ensure that Police-Public Contact Survey
questions were appropriately devised. In cases where we noted that
changes had been made between surveys, we reviewed Census Bureau
documentation and interviewed staff and officials at BJS.
It was beyond the scope of this review to address any personnel issues
that may have arisen in connection with the disagreement over the
content of the 2005 draft press release based on the 2002 Police-Public
Contact Survey. We conducted our work between April 2006 and January
2007 in accordance with generally accepted government auditing
standards.
[End of section]
Appendix II: Applicable Quality Guidelines for Statistical Products
Issued by the Bureau of Justice Statistics and Other Agencies:
BJS followed numerous recommended data quality guidelines designed to
help ensure the accuracy and integrity of the Police-Public Contact
Survey products that it issued in 2001, 2002, 2005, and 2006 based on
its 1999 and 2002 surveys. The product issuance guidelines were used to
aid BJS's efforts to review, approve, and disseminate these statistical
products to the public and others.
The guidelines were issued at various points in time by the following
organizations:
* the National Research Council,
* the Bureau of Justice Statistics,
* the Department of Justice, and:
* the Office of Justice Programs.
In addition to reviewing the guidelines of these four organizations, we
also reviewed guidelines and directives issued by the Office of
Management and Budget (OMB). However, we did not specifically assess
BJS's practice with respect to following OMB's guidelines because OMB
issued governmentwide policy and procedural guidance to federal
agencies that called for agencies to develop their own implementing
guidelines.
Table 4 shows the guidelines that were available at the time BJS's 1999
and 2002 Police-Public Contact Survey products were issued, and which
guidelines BJS was in a position to follow.
Table 4: Extent to Which BJS Followed Available Quality Guidelines for
1999 and 2002 Police-Public Survey Products:
Guideline: A statistical agency should follow good practice, in reports
and other data releases, in documenting concepts, definitions, data
collection methodologies, and measures of uncertainty, and in
discussing possible sources of error;
Issuing entity: NRC;
Products issued from 1999 survey: Products issued in 2001-2002: Press
release[A]: Guideline was available and BJS followed it fully;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was available and BJS followed it fully;
Products issued from 1999 survey: Products issued in 2001-2002: Report
B[C]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
C[D]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was available and BJS followed it fully.
Guideline: An effective dissemination program includes policies for the
preservation of data that guide what data to retain and how they are to
be archived for future secondary analysis;
Issuing entity: NRC;
Products issued from 1999 survey: Products issued in 2001-2002: Press
release[A]: Guideline was available and BJS followed it fully;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was available and BJS followed it fully;
Products issued from 1999 survey: Products issued in 2001-2002: Report
B[C]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
C[D]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was available and BJS followed it fully.
Guideline: An agency should have an established publications policy
that describes, for a data collection program, the types of reports and
other data released to be made available, the audience served, and the
frequency of release;
Issuing entity: NRC;
Products issued from 1999 survey: Products issued in 2001-2002: Press
release[A]: Guideline was available and BJS followed it fully;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was available and BJS followed it fully;
Products issued from 1999 survey: Products issued in 2001-2002: Report
B[C]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
C[D]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was available and BJS followed it fully.
Guideline: Dissemination of data and information (basic series,
analytic reports, press releases, public-use tapes) should be timely
and public. Avenues of dissemination should be chosen to reach as broad
a public as reasonably possible;
Issuing entity: NRC;
Products issued from 1999 survey: Products issued in 2001-2002: Press
release[A]: Guideline was available and BJS followed it fully;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was available and BJS followed it fully;
Products issued from 1999 survey: Products issued in 2001- 2002: Report
B[C]: Guideline was unavailable during the product issuance stage;
Products issued from 2002 survey: Products issued in 2005-2006: Report
C[D]: Guideline was unavailable during the product issuance stage;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was unavailable during the product issuance stage.
Guideline: Statistical agency has dissemination policies that foster
regular, frequent release of major findings from an agency's
statistical programs to the public via the media, the Internet, and
other means;
Issuing entity: NRC;
Products issued from 1999 survey: Products issued in 2001-2002: Press
release[A]: Guideline was unavailable during the product issuance
stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was unavailable during the product issuance stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
B[C]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005- 2006: Report
C[D]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was available and BJS followed it fully.
Guideline: The statistical agency has maintenance of a clear
distinction between statistical information and policy interpretations
of such information by the President, the secretary of the department,
or others in the executive branch;
Issuing entity: NRC;
Products issued from 1999 survey: Products issued in 2001-2002: Press
release[A]: Guideline was available and BJS followed it fully;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was available and BJS followed it fully;
Products issued from 1999 survey: Products issued in 2001-2002: Report
B[C]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
C[D]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was available and BJS followed it fully.
Guideline: The statistical agency has authority for professional
decisions over the scope, content, and frequency of data compiled,
analyzed, or published. Most statistical agencies have broad authority,
limited by budgetary constraints, departmental requirements, OMB
review, and congressional mandates;
Issuing entity: NRC;
Products issued from 1999 survey: Products issued in 2001-2002: Press
release[A]: BJS was not in the position to follow this guideline fully;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was available and BJS followed it fully;
Products issued from 1999 survey: Products issued in 2001-2002: Report
B[C]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
C[D]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was available and BJS followed it fully.
Guideline: The statistical agency has recognition by policy officials
outside the statistical agency of its authority to release statistical
information without prior clearance;
Issuing entity: NRC;
Products issued from 1999 survey: Products issued in 2001-2002: Press
release[A]: BJS was not in the position to follow this guideline at
all;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was available and BJS followed it fully;
Products issued from 1999 survey: Products issued in 2001-2002: Report
B[C]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
C[D]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was available and BJS followed it fully.
Guideline: Release of information should not be subject to actual or
perceived political interference. In particular, the timing of the
public release of data should be the responsibility of the statistical
agency;
Issuing entity: NRC;
Products issued from 1999 survey: Products issued in 2001-2002: Press
release[A]: BJS was not in the position to follow this guideline fully;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was available and BJS followed it fully;
Products issued from 1999 survey: Products issued in 2001-2002: Report
B[C]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005- 2006: Report
C[D]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was available and BJS followed it fully.
Guideline: The objectivity of BJS statistics must be vigilantly
protected at all times by BJS staff;
Issuing entity: BJS;
Products issued from 1999 survey: Products issued in 2001-2002: Press
release[A]: Guideline was unavailable during the product issuance
stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was unavailable during the product issuance stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
B[C]: Guideline was unavailable during the product issuance stage;
Products issued from 2002 survey: Products issued in 2005-2006: Report
C[D]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was available and BJS followed it fully.
Guideline: All BJS reports and other statistical products must be
subject to an objective and appropriate verification process conducted
by qualified BJS staff other than the author of the report;
Issuing entity: BJS;
Products issued from 1999 survey: Products issued in 2001- 2002: Press
release[A]: Guideline was unavailable during the product issuance
stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was unavailable during the product issuance stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
B[C]: Guideline was unavailable during the product issuance stage;
Products issued from 2002 survey: Products issued in 2005-2006: Report
C[D]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was available and BJS followed it fully.
Guideline: BJS staff must archive all analysis and data sets as soon as
they become final in order to ensure the reproducibility of the
published statistics;
Issuing entity: BJS;
Products issued from 1999 survey: Products issued in 2001-2002: Press
release[A]: Guideline was unavailable during the product issuance
stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was unavailable during the product issuance stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
B[C]: Guideline was unavailable during the product issuance stage;
Products issued from 2002 survey: Products issued in 2005-2006: Report
C[D]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was available and BJS followed it fully.
Guideline: BJS statistical products must provide sufficient information
concerning instances where data have been imputed;
Issuing entity: BJS;
Products issued from 1999 survey: Products issued in 2001-2002: Press
release[A]: Guideline was unavailable during the product issuance
stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was unavailable during the product issuance stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
B[C]: Guideline was unavailable during the product issuance stage;
Products issued from 2002 survey: Products issued in 2005-2006: Report
C[D]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was available and BJS followed it fully.
Guideline: Effective dissemination program includes a variety of
avenues for data dissemination, chosen to reach as broad a public as
reasonably possible. Channels of dissemination include, but are not
limited to, an agency's Internet Web site, government depository
libraries, conference exhibits and programs, newsletters and journals,
e-mail address lists, and the media for regular communication of major
findings;
Issuing entity: BJS, NRC;
Products issued from 1999 survey: Products issued in 2001-2002: Press
release[A]: Guideline was unavailable during the product issuance
stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was unavailable during the product issuance stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
B[C]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005- 2006: Report
C[D]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was available and BJS followed it fully.
Guideline: A statistical agency should develop an understanding of the
validity and accuracy of its data and convey the resulting measures of
quality to users in ways that are comprehensible to nonexperts;
Issuing entity: NRC, BJS;
Products issued from 1999 survey: Products issued in 2001-2002: Press
release[A]: Guideline was available and BJS followed it fully;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was available and BJS followed it fully;
Products issued from 1999 survey: Products issued in 2001-2002: Report
B[C]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005- 2006: Report
C[D]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was available and BJS followed it fully.
Guideline: An effective dissemination program includes release of data
in a variety of formats so that the information can be accessed by
users with varying skills and needs for data retrieval and analysis;
Issuing entity: BJS, NRC;
Products issued from 1999 survey: Products issued in 2001-2002: Press
release[A]: Guideline was unavailable during the product issuance
stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was unavailable during the product issuance stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
B[C]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005- 2006: Report
C[D]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was available and BJS followed it fully.
Guideline: A statistical agency should use modern statistical theory
and sound statistical practice in all technical work;
Issuing entity: DOJ, NRC;
Products issued from 1999 survey: Products issued in 2001- 2002: Press
release[A]: Guideline was available and BJS followed it fully;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was available and BJS followed it fully;
Products issued from 1999 survey: Products issued in 2001-2002: Report
B[C]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
C[D]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was available and BJS followed it fully.
Guideline: OJP/DOJ will allow adequate time for reviews, consistent
with the level of standards required for the type of information to be
disseminated;
Issuing entity: DOJ, OJP;
Products issued from 1999 survey: Products issued in 2001-2002: Press
release[A]: Guideline was unavailable during the product issuance
stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was unavailable during the product issuance stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
B[C]: Guideline was unavailable during the product issuance stage;
Products issued from 2002 survey: Products issued in 2005-2006: Report
C[D]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was available and BJS followed it fully.
Guideline: OJP/DOJ will ensure compliance with relevant quality
guidelines (i.e., utility, objectivity, and integrity requirements);
Issuing entity: DOJ, OJP;
Products issued from 1999 survey: Products issued in 2001-2002: Press
release[A]: Guideline was unavailable during the product issuance
stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was unavailable during the product issuance stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
B[C]: Guideline was unavailable during the product issuance stage;
Products issued from 2002 survey: Products issued in 2005- 2006: Report
C[D]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was available and BJS followed it fully.
Guideline: OJP/DOJ will provide methodologies, origins of data,
limitations of the information, etc., whenever possible, as part of
information dissemination;
Issuing entity: DOJ, OJP, BJS;
Products issued from 1999 survey: Products issued in 2001-2002: Press
release[A]: Guideline was unavailable during the product issuance
stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was unavailable during the product issuance stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
B[C]: Guideline was unavailable during the product issuance stage;
Products issued from 2002 survey: Products issued in 2005-2006: Report
C[D]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was available and BJS followed it fully.
Guideline: OJP/DOJ components will ensure disseminated information, as
a matter of substance and presentation is accurate, reliable and
unbiased. Objectivity is achieved by using reliable data sources, sound
analytical techniques, and documenting methods and data sources;
Issuing entity: DOJ, OJP;
Products issued from 1999 survey: Products issued in 2001-2002: Press
release[A]: Guideline was unavailable during the product issuance
stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was unavailable during the product issuance stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
B[C]: Guideline was unavailable during the product issuance stage;
Products issued from 2002 survey: Products issued in 2005- 2006: Report
C[D]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was available and BJS followed it fully.
Guideline: OJP/DOJ components will review all information dissemination
products for their quality (including objectivity, utility, and
integrity) before they are disseminated;
Issuing entity: DOJ, OJP;
Products issued from 1999 survey: Products issued in 2001-2002: Press
release[A]: Guideline was unavailable during the product issuance
stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was unavailable during the product issuance stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
B[C]: Guideline was unavailable during the product issuance stage;
Products issued from 2002 survey: Products issued in 2005-2006: Report
C[D]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was available and BJS followed it fully.
Guideline: OJP/DOJ components will assess the usefulness of the
information to be disseminated to the public;
Issuing entity: DOJ, OJP;
Products issued from 1999 survey: Products issued in 2001-2002: Press
release[A]: Guideline was unavailable during the product issuance
stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was unavailable during the product issuance stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
B[C]: Guideline was unavailable during the product issuance stage;
Products issued from 2002 survey: Products issued in 2005-2006: Report
C[D]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was available and BJS followed it fully.
Guideline: OJP/DOJ will ensure that the information fulfills the
intentions stated and that the conclusions are consistent with the
evidence;
Issuing entity: DOJ, OJP;
Products issued from 1999 survey: Products issued in 2001-2002: Press
release[A]: Guideline was unavailable during the product issuance
stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
A[B]: Guideline was unavailable during the product issuance stage;
Products issued from 1999 survey: Products issued in 2001-2002: Report
B[C]: Guideline was unavailable during the product issuance stage;
Products issued from 2002 survey: Products issued in 2005- 2006: Report
C[D]: Guideline was available and BJS followed it fully;
Products issued from 2002 survey: Products issued in 2005-2006: Report
D[E]: Guideline was available and BJS followed it fully.
Source: GAO analysis of DOJ, OJP, and BJS data.
NRC: National Research Council:
DOJ: Department of Justice:
OJP Office of Justice Programs:
[A] Force or Threatened Force Used in Less Than 1 Percent of All Police-
Public Interactions (March 11, 2001).
[B] Contacts between Police and the Public, Findings from the 1999
National Survey (February 2001).
[C] Characteristics of Drivers Stopped by Police, 1999 (March 2002).
[D] Contacts between Police and the Public, Findings from the 2002
National Survey (April 2005).
[E] Characteristics of Drivers Stopped by Police, 2002 (June 2006).
[End of table]
[End of section]
Appendix III: Press Release Issued in 2001 Based on BJS's 1999 Police-
Public Contact Survey:
[See PDF for image]
[End of figure]
[End of section]
Appendix IV: BJS Processes for the Review, Approval, and Dissemination
of Police-Public Contact Survey Products:
Review, Approval, and Dissemination Process for Reports Issued from
Police-Public Contact Surveys:
Since the inception of the Police-Public Contact Survey in 1996, the
BJS Director has been the single noncareer appointee who has had a
decision-making role in BJS's review, approval, and dissemination
processes for reports. The BJS Director is a noncareer presidential
appointee subject to Senate confirmation. Figure 2 provides an overview
of the process followed by BJS in the review, approval, and
dissemination of Police-Public Contact Survey reports.
Figure 2: Review, Approval, and Dissemination Process for BJS Survey
Reports:
[See PDF for image]
Source: BJS.
[End of figure]
As indicated by the figure, the BJS report author and supervisor
prepare the draft report for review and approval. The BJS Director
reviews the draft, requests any changes, approves the final draft, and
transmits a memorandum of notification through the Office of Justice
Program's Assistant Attorney General up the chain of command to the
Attorney General. The memorandum contains an abstract of the report,
selected survey findings, and a projected release date for the report.
BJS sets the release date for 30 days from the date that the Assistant
Attorney General signs the memorandum of notification. The report is
posted to the Web site at that time, or sooner if the date and time are
specified in the notification memo.
Review, Approval, and Dissemination Process for Press Releases Issued
from Police-Public Contact Surveys:
In contrast to the process followed for survey reports, several
noncareer appointees are involved in the organizational review,
approval, and dissemination process, as shown in figure 3.
Figure 3: Review, Approval, and Dissemination Process for BJS Survey
Press Releases:
[See PDF for image]
Sources: BJS and Office of Justice Programs.
[End of figure]
As indicated in the figure, in addition to the Director of BJS, there
are three noncareer appointees within the Office of Justice Programs
who participate in the review and approval process--the Chief of Staff,
the Deputy Assistant Attorney General, and the Assistant Attorney
General, and at least one noncareer appointee within Department of
Justice headquarters: the Director of the Office of Public Affairs.
The BJS report author and supervisor jointly work with staff from the
Office of Justice Program's Office of Communications to prepare the
press release. The BJS director reviews the draft press release,
requests any changes, approves the final draft, and transmits the press
release up the chain of command to the Office of Justice Programs'
Assistant Attorney General for review and approval. Following approval
by the Assistant Attorney General, the Department of Justice's Office
of Public Affairs reviews the press release for clarity, and the BJS
Director then verifies that the information in the press release is
accurate. The Department of Justice's Office of Public Affairs is then
responsible for disseminating the press release to Congress, the media,
and executive department press offices, while BJS is responsible for
disseminating the press release through its Web site.
[End of section]
Appendix V: Changes in the Scope and Methodology of the Police-Public
Contact Surveys:
BJS has conducted four Police-Public Contact Surveys as supplements to
the National Crime Victimization Survey. The first Police-Public
Contact Survey was conducted as a pilot in 1996. Three subsequent, more
extensive surveys were conducted at 3-year intervals: 1999, 2002, and
2005. Although we do not discuss the 2005 survey in this report because
no reports or press releases have yet been issued from this survey, we
present information on the 2005 survey in this appendix because
information is available on this survey's scope and methodology. The
scope of the Police-Public Contact Surveys has consistently expanded
over time, while the methodology has remained generally consistent.
The Scope of the Police-Public Contact Survey Has Expanded with Each
Additional Survey:
The pilot survey was designed to test whether the survey could be
effectively used as a supplement to the National Crime Victimization
Survey to collect data on (a) the types of contacts the public have
with the police, and (b) police use of force. To conduct this test, BJS
employed a representative sample of 6,421 U.S. residents. The pilot
survey yielded useful information on the various types of contacts the
public had with the police, and whether force was used by the police.
However, the sample size of the pilot survey was not sufficiently large
for BJS to draw inferences about the extent to which the population at
large would report that they experienced "excessive" use of force by
police.
For its Police-Public Contact Survey in 1999, BJS increased the sample
size to a representative sample of 80,543 U.S. residents. The scope of
the survey was further enhanced by adding questions about traffic stops
(the most common form of public-police contact, as determined in the
1996 pilot survey), and including a question on whether the police used
excessive force during any contact with the public. BJS officials told
us that they added the traffic stop questions, at least in part, to
"address the growing public concern about racial profiling in
connection with traffic stops."
In its 2002 survey, BJS expanded and refined its survey questions
further. Specifically, according to BJS officials, they added questions
that would help BJS estimate the extent to which U.S. residents
nationwide would say that (1) they were stopped by the police while
driving, (2) they or their vehicle were searched by the police without
their permission during a traffic stop, and (3) they were arrested as a
result of the search. In addition, BJS officials said that they added
questions to estimate differences, if any, among racial groups in their
rates of traffic stops at various times of the day, and whether police
used force in situations where persons were engaged in such behaviors
as arguing with, cursing, or disobeying police.
In 2005, according to BJS officials, the scope of the Police-Public
Contact Survey was further extended in several ways, including the
following: (a) respondents were asked whether they had been arrested
for driving under the influence of alcohol during the year (in order to
make comparisons with Federal Bureau of Investigation (FBI) arrest
rates, so that potential undercounting rates could be determined); (b)
respondents were permitted to group themselves into any combination of
racial categories (rather than choosing a single category) to better
refine respondent demographic status; (c) respondents were asked
whether police used force during any of their police contacts during
the year, as opposed to the more limiting question in 2002, which was
directed only toward the most recent contact with police; and (d)
respondents were provided open-ended response fields on the survey
instrument to indicate any ways they believed that the police had acted
inappropriately toward them.
The Police-Public Contact Survey Methodology Has Remained Consistent
Following the 1996 Survey:
The 1999, 2002, and 2005 Police-Public Contact Surveys have
consistently maintained a similar methodology. The methodological
dimensions of the surveys that have remained consistent are geographic
coverage, target population, sampling design, data collection method,
sample size/response rate, survey administration, and sample
characteristics (as indicated in table 5). To illustrate, all three
surveys involved selecting nationally representative, stratified, multi-
cluster samples from the population of U.S. residents 16 years of age
or older.[Footnote 22] The data were collected either through face- to-
face or computer-assisted telephone interviews. The surveys were
administered during the last 6 months of the year, and the demographic
characteristics of the samples were similar across time periods.
The 1996 pilot survey differed in several ways from the subsequent
three surveys. Specifically, the pilot survey included residents
younger than 16, included far fewer people than the subsequent surveys,
and limited the sampling to individuals who had participated in the
last round of the National Crime Victimization Survey. In addition, the
percentage of face-to-face interviews was lower, and the survey
administration period was shorter, and during a different time of the
year, than in the other three surveys.
Table 3: Police-Public Contact Survey (PPCS) Methodologies for the Four
Surveys: 1996, 1999, 2002, and 2005:
Dimensions: Geographic Coverage;
1996 survey[A]: National survey;
1999 survey: National survey;
2002 survey: National survey;
2005 survey: National survey.
Dimensions: Target populations;
1996 survey[A]: Included: residents ages >12, living in the United
States, including persons living in group quarters (e.g., dormitories,
rooming houses, religious group dwellings); Excluded: crew members of
merchant vessels, armed forces personnel living in military barracks,
and institutionalized persons (e.g., correctional facility inmates);
1999 survey: Same as 1996, except the target population age range is
>16;
2002 survey: Same as 1996, except the target population age range is
>16;
2005 survey: Same as 1996, except the target population age range is
>16.
Dimensions: Sampling design;
1996 survey[A]: Nationally representative, stratified, multi-stage
cluster sample, limited to respondents participating in the outgoing
rotation panels of the National Crime Victimization Survey (i.e., the
seventh and last round of interviews);
1999 survey: Nationally representative, stratified, multi-stage cluster
sample, not limited to the out-rotation panels of the National Crime
Victimization Survey;
2002 survey: Nationally representative, stratified, multi-stage cluster
sample, not limited to the out-rotation panels of the National Crime
Victimization Survey;
2005 survey: Nationally representative, stratified, multi-stage cluster
sample, not limited to the out-rotation panels of the National Crime
Victimization Survey.
Dimensions: Data collection method;
1996 survey[A]: All survey interviews were collected using either the
CATI (computer-assisted telephone interview) or PAPI (paper-and-pencil
interview either in person or by telephone).
Interviews: In person: 19% Telephone: 81%;
1999 survey: All PPCS interviews were collected using either the CATI
or PAPI.
Interviews: In person: 31% Telephone: 69%;
2002 survey: All PPCS interviews were collected using either the CATI
or PAPI.
Interviews: In-person: 34% Telephone: 66%;
2005 survey: All PPCS interviews were collected using either the CATI
or PAPI.
Interviews: In-person: 37% Telephone: 63%.
Dimensions: Sample size/response rate;
1996 survey[A]: 6,421 (99%);
1999 survey: 80,543 (85%);
2002 survey: 76,910 (82%);
2005 survey: 63,943 (80%).
Dimensions: Survey administration period;
1996 survey[A]: 5/1/96 to 7/ 31/96;
1999 survey: 7/1/99 to 12/31/99;
2002 survey: 7/1/02 to 12/31/ 02;
2005 survey: 7/1/05 to 12/31/05.
Dimensions: Sample characteristics (unweighted percent);
1996 survey[A]:
Gender: Male: 50.7 Female: 49.3.
Race/ethnicity: White: 83.0 Black: 7.4 Hispanic: 5.7 Other: 3.9.
Age: 16-19: 11.1 20-39: 44.5 40-59: 32.8 > 60: 11.6;
1999 survey:
Gender: Male: 50.1 Female: 49.9.
Race/ethnicity: White: 79.3 Black: 9.1 Hispanic: 8.4 Other: 3.2.
Age: 16-19: 8.1 20-39: 45.6 40-59: 35.3 > 60: 11.1;
2002 survey:
Gender: Male: 50.2 Female: 49.8.
Race/ethnicity: White: 77.5 Black: 9.5 Hispanic: 9.8 Other: 3.2.
Age: 16-19: 7.5 20-39: 44.6 40-59: 36.3 > 60: 11.7;
2005 survey:
Gender: Male: 50.4 Female: 49.6.
Race/ethnicity: White: 76.5 Black: 9.1 Hispanic: 9.6 Other: 3.9.
Age: 16-19: 7.6 20-39: 41.3 40-59: 38.5 > 60: 12.6.
Source: BJS data.
Note: Percentages may not sum to 100 percent because of rounding.
[A] Pilot survey.
[End of table]
[End of section]
Appendix VI: Comments from the Department of Justice:
U.S. Department of Justice:
Office of Justice Programs:
Office of the Assistant Attorney General:
Washington, D.C. 20531:
Mr. Brian Lepore:
Acting Director:
Homeland Security and Justice Team:
Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Mar 12 2007:
Dear Mr. Lepore:
Thank you for the opportunity to review and comment on the Government
Accountability Office (GAO) draft report entitled "Bureau of Justice
Statistics, Quality Guidelines Generally Followed for Police-Public
Contact Surveys, but Opportunities Exist to Help Assure Agency
Independence" (GAO-07-340) (hereinafter "draft report").
The Office of Justice Programs (OJP) and its component, the Bureau of
Justice Statistics (BJS), share GAO's concern that the development and
dissemination of Federal statistics must be objective and of high
integrity. The leadership of OR is committed to ensuring that the
justice statistics produced by BJS adhere to the highest standards of
quality and remain free from any political interference.
We affirm the findings in the draft report that:
* GAO found no evidence of any political interference with the timing
of the 2001 survey press release or any other BJS document that GAO
examined.
* BJS successfully followed all applicable data quality guidelines, as
determined by GAO to be relevant and appropriate, for the Police-Public
Contact Survey (PPCS) statistical reports. BJS took proper steps to
ensure the accuracy and integrity of the review, approval, and
dissemination processes associated with issuing public reports based on
the PPCS surveys reviewed by GAO. The BJS Director is the sole non-
career appointee who has a decision-making role in these processes for
report issuance.
* Press releases issued in association with published reports are joint
releases of BJS, OJP, and the U.S. Department of Justice (DOJ), and non-
career appointees outside of BJS can become involved in the press
release process. The non-career appointees who played decision-making
roles in the PPCS press release process that GAO reviewed acted within
the scope of the roles and responsibilities accorded them under DOJ
policies.
GAO based its assessment on the guidelines as presented by the National
Research Council's (NRC's) Principles and Practices for a Federal
Statistical Agency and the data quality guidelines of the DOJ, OJP, and
BJS based on Office of Management and Budget (OMB) requirements. The
NRC's Principles and Practices is a valuable and influential document
for BJS and other Federal statistical agencies, and BJS has long
accepted these principles as guides for effective practice. However, it
is important to note that - by their own terms - the NRC guidelines are
explicitly not designed to be prescriptions, and as such are followed
by agencies solely in the exercise of their own discretion. It is
equally important to note that NRC expressly stated in its guidelines
that NRC does not expect every guideline to be followed.[Footnote 23]
In sharp contrast, the information quality guidelines as promulgated by
the OMB are mandatory and implement Pub. L. No. 106-554, providing
guidance to Federal agencies for ensuring and maximizing the quality,
objectivity, utility, and integrity of information disseminated by
Federal agencies. BJS is governed by the guidelines promulgated by OMB,
DOJ, OJP, and BJS; adherence to these guidelines is not discretionary.
GAO's draft report confirms OJP's declaration that, at all times, BJS
followed all guidelines promulgated by these executive branch agencies.
The findings in the draft report are predicated on GAO's assumption
that a press release is a statistical product. Based upon this
assumption, it is GAO's position that the NRC guidelines available in
2001 applied to press releases. We respectfully disagree with GAO's
assumption. A press release simply is not a statistical product and
thus should not be treated as a statistical product at all - let alone
one that is somehow covered by the NRC guidelines.
A press release, rather, is a public relations announcement issued to
encourage media coverage. The mere presence of statistics in a press
release does not transform a press release into a statistical product.
Considerable support for this proper understanding of a press release
may be found in the GAO draft report itself. The GAO draft report
describes in significant detail the review, approval, and dissemination
process for BJS reports. The draft report also describes a
significantly different review, approval, and dissemination process for
BJS press releases. The processes are different and the offices and
staff involved are different because a press release is distinctly
different from a statistical product.
GAO's assumption that a press release is a statistical product has
important ramifications that need to be kept in mind as one reads the
draft report. Of primary concern is that the draft report overstates
its findings relating to the potential threats to the independence of
BJS because, the draft report often utilizes the broad term,
"statistical products," when in many cases it is speaking only of press
releases. This lack of distinction between the two terms should be kept
in mind by the reader, especially in regard to the GAO finding that
"the potential exists for BJS's review, approval, and dissemination
process for statistical products to be subject to political
interference." This imprecise statement is a focused reference to one
press release, rather than a broader statement regarding BJS reports
and other data products.
As we state below, OMB, DOJ, OJP, and BJS all considered this exact
question - whether or not a press release falls under guidelines for
the dissemination of information - as these entities promulgated their
information quality guidelines that govern information disseminated to
the public. Each of these four executive branch agencies came to the
same conclusion: press releases are exempt from information quality
guidelines. See, e.g., Office of Management and Budget Information
Quality Guidelines, § I(A)(10)(October 1, 2002), "The guidelines focus
on procedures for the `dissemination' of `information,' as those terms
are defined herein. Accordingly, procedures specifically applicable to
forms of communication outside the scope of these guidelines, such as
those ,for correspondence or press releases, among others, are not
included." (emphasis added).
As the GAO draft report states, there was significant ambiguity on this
point in the NRC guidelines as they stood in 2001. For at least four
reasons, it is the view of OJP that it is not appropriate to apply the
NRC guidelines to BJS press releases:
1) The text of two of the three guidelines at issue refer specifically
to "data."[Footnote 24] For these guidelines to apply to press
releases, the premise must be accepted that a press release belongs
within the definition of "data." For the reasons discussed above, this
assumption is unwarranted and thus we do not accept it.
A fair reading of the purpose of these two guidelines is to ensure that
a federal statistical agency control the data that it collects,
analyzes, and releases to the public. OR affirms that BJS fully
controls the hundreds upon thousands of spreadsheets, data sets,
statistical reports, data briefs, and other statistical products that
it collects, analyzes, and releases to the public.
The text of the last guideline refers to the "release of statistical
information."[Footnote 25] Notwithstanding the fact that this guideline
is largely redundant to one referenced above, as OR has explained to
GAO, BJS does have full authority to release statistical information
without prior clearance, and BJS exercises this authority on a routine
basis to present the public with a tremendous volume of criminal
justice statistics in various formats without prior clearance.
2) When the press release for the 1999 PPCS was issued in 2001, there
was no written or other guidance from NRC supporting GAO's position
that the NRC guidelines covered press releases. Further, as of the date
of this draft report, there have been three editions of Principles and
Practices for a Federal Statistical Agency (1992, 2001, 2005), and not
one has stated that the documents were intended to apply to press
releases.
GAO's effort in the draft report to cite to NRC's current
interpretation of its guidelines is not persuasive. The relevant issue
here relates not to whatever current advisory opinions might be
provided to GAO by NRC staff, but whether the NRC written guidance
available in 2001 explicitly covered press releases, and GAO and OJP
both agree that it did not.
3) Despite OJP's efforts to explain this point to GAO in written and
verbal discussions, the draft GAO report mischaracterizes the BJS Data
Quality Guidelines as applying to press releases. OR reiterates that
the BJS Data Quality Guidelines apply only to the statistics contained
in BJS press releases, not to the entire document. This is because BJS
accepts, as it must under the law, the judgments of OMB, DOJ, and OJP
that press releases are outside the scope of these agencies'
information quality guidelines. The BJS Data Quality Guidelines, cited
by GAO, specifically state that they incorporate the exceptions found
in the DOJ information quality guidelines; therefore, press releases
are not included.
4) By statute, 42 U.S.C. § 3732(b), the Director of BJS "shall be
responsible for the integrity of data and statistics." In the exercise
of such authority, he may elect to follow the NRC guidelines, but he is
not and cannot be legally bound to do so, in the absence of some
supervening statute (in the nature of Pub. L. No. 106-554). We are
aware of no such statute relating to NRC; thus, even if the NRC written
guidelines did apply to press releases (and they do not), the Director
would and does decline, in the exercise of his statutory authority to
apply them to BJS press releases.
We are pleased that GAO has, through this effort, highlighted the
importance of safeguarding the integrity of Federal statistical data.
Because there are competing interpretations of what properly
constitutes a "statistical product," we believe that the Federal
statistical community would benefit from clarity in this area.
OR and BJS will continue their long-standing commitment to integrity
and objectivity in producing impartial and accurate statistics. Again,
OR appreciates the opportunity to comment on this draft report.
Sincerely,
Signed by:
Regina B. Schofield:
Assistant Attorney General:
[End of section]
Appendix VII: GAO Contact and Acknowledgments:
GAO Contacts:
Brian J. Lepore (202) 512-4523:
Acknowledgments:
In addition to the above, Evi. L. Rezmovic, Assistant Director; Ronald
S. Fecso, Chief Statistician; Jared A. Hermalin; Karen A. Jarzynka;
Amanda K. Miller; Amy L. Bernstein; Geoffrey R. Hamilton; Robert
Alarapon; and Tracy J. Harris made key contributions to this report.
FOOTNOTES
[1] At the time of our review, BJS had not issued any products from the
2005 survey.
[2] Of the consolidated list of 24 review, approval, and dissemination
guidelines issued by the National Research Council, Department of
Justice, Office of Justice Programs, and Bureau of Justice Statistics,
only 10 were available in 2001, when the press release based on the
1999 survey was issued.
[3] GAO, Data Quality: Expanded Use of Key Dissemination Practices
Would Further Safeguard the Integrity of Statistical Data, GAO-06-607
(Washington, D.C.: May 31, 2006).
[4] Pub. L. No. 96-157, 93 Stat. 1167 (1979).
[5] Office of Management and Budget, Statistical Programs of the United
States Government: Fiscal Year 2006, pp. 3-4 (Washington, D.C.: 2005).
[6] Pub. L. No. 103-322, 108 Stat. 1796, 2071 (1994) (codified at 42
U.S.C. 14142).
[7] For ease of presentation, in this report we use the term
"guidelines" to refer to the principles and practices described in the
National Research Council document, as well as to guidelines issued by
the Department of Justice, Office of Justice Programs, and BJS.
[8] The National Research Council's Committee on National Statistics
was established in 1972 at the recommendation of the President's
Commission on Federal Statistics in order to improve the statistical
methods and information on which public policy decisions are based.
[9] Margaret E. Martin, Miron L. Straf, and Constance F. Citro (eds.),
Principles and Practices for a Federal Statistical Agency, (3rd ed.)
(Washington, D.C.: The National Academies Press, 2005).
[10] Section 515 of the Treasury and General Government Appropriations
Act for Fiscal Year 2001 (Pub. L. No. 106-554, 114 Stat. 2763, 2763A-
125 (2000)), known as both the Information Quality Act and the Data
Quality Act, required OMB to issue governmentwide quality guidelines
including the requirement that such agencies issue their own quality
guidelines.
[11] See Notice of revisions to OMB's Standards and Guidelines for
Statistical Surveys, 71 Fed. Reg. 55522 (Sept. 22, 2006).
[12] After OMB issued policy and procedural guidance to federal
agencies in 2002, the Department of Justice, Office of Justice
Programs, and BJS developed their own guidelines.
[13] Information posted on the BJS Web site includes hypertext links to
information that is created and maintained by other public and private
organizations in the United States and by other nations.
[14] Guidelines developed by BJS, the Office of Justice Programs, and
the Department of Justice were not available at the time the press
release, based on the 1999 survey, was released.
[15] Our work focused exclusively on the Police-Public Contact Survey,
and we did not conduct a systematic review of other press releases that
have been issued based on BJS's work, to make an independent
determination of whether they constituted statistical products.
[16] See GAO-06-607.
[17] OMB's February 2002 guidelines for federal agencies exempted press
releases from the guidelines. These guidelines, however, were designed
to apply to a wide variety of government information dissemination
activities, and be generic enough to fit all types of media. As we
noted in our May 2006 report (GAO-06-607), the directive on which OMB
is currently working pertains directly to the release and dissemination
of statistical products.
[18] We did not review the extent to which BJS followed guidelines for
the 2005 draft press release (based on the 2002 survey) that was the
source of disagreement between the then-BJS Director and the then-
Acting Assistant Attorney General, because the press release was not
issued.
[19] U.S. Code, Title 42, Chapter 46, Subchapter I (42 U.S.C. Sections
3711 through 3715a).
[20] In contrast to the process followed for press releases, the only
noncareer appointee involved in the review, approval, and dissemination
of reports from Police-Public Contact Surveys is the BJS Director.
Appendix IV provides details on the roles of the Director and other
employees in this process.
[21] The number of guidelines that were applicable to the review,
approval, and dissemination of Police-Public Contact Surveys products
varied across the time periods covered. In general, the number of
guidelines increased between February 2001, when the first survey
report was issued, and June 2006, when the latest survey report was
issued. However, one National Research Council guideline, pertaining to
the dissemination of data and information to as broad a public as
reasonably possible, was dropped when the council issued the second
edition of its guidelines between the time that BJS issued its first
report (in February 2001) and second report (in March 2002) from the
1999 survey.
[22] This included persons living in group quarters, but excluded
institutionalized persons and some members of the armed services.
[23] Principles and Practices for a Federal Statistical Agency,
National Research Council, Committee on National Statistics, National
Academy Press, p. 4 (1992)("Some aspects of [agency] independence, not
all of which are required, are the following,").
[24] As stated in the draft report, the guidelines read (emphases
added):
"The statistical agency has authority for professional decisions over
the scope, content, and frequency of data compiled, analyzed, and
published."
"The release of information should not be subject to actual or
perceived political interference. In particular, the timing of the
public release of data should be the responsibility of the statistical
agency."
[25] As stated in the draft report, the guideline reads, "The
statistical agency has recognition by policy officials of its authority
to release statistical information without prior clearance."
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site (www.gao.gov). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site.
To have GAO e-mail you a list of newly posted products every afternoon,
go to www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125
Washington, D.C. 20548:
Public Affairs:
Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548:
