Health Information Technology
More Detailed Plans Needed for the Centers for Disease Control and Prevention's Redesigned BioSense Program Gao ID: GAO-09-100 November 20, 2008In 2003, the Centers for Disease Control and Prevention (CDC), an agency within the Department of Health and Human Services (HHS), developed an electronic syndromic surveillance system called BioSense that uses health-related data to identify patterns of disease symptoms prior to specific diagnoses. In late 2007, CDC began to redesign the program to improve collaboration with stakeholders and address identified management weaknesses. Pursuant to House Report 110-231, GAO evaluated the BioSense program, focusing on the cost and timeline estimates and performance measures and benchmarks for implementing the program, among other objectives. To accomplish this, GAO analyzed relevant program documentation and interviewed CDC officials responsible for developing and implementing BioSense.
While CDC identified annual and long-term cost and timeline estimates and performance measures for the initial design of BioSense, these estimates and measures did not reflect the implementation of its redesigned program. CDC subsequently developed a draft plan for the redesigned program that described high-level cost and timeline estimates; however, the estimates are not reliable, and the plan did not include performance measures. According to best practices, cost estimates should be well-documented, comprehensive and accurate, and must be credible before they can be considered to be reliable. However, CDC's cost estimates for the redesigned program are not reliable because they are only partially documented, are not comprehensive and accurate, and therefore are not credible. Best practices for reliable timeline estimates include the identification of resources to complete each task, establishment of a critical path, and analysis of risks to the schedule. However, the agency has not implemented these practices, resulting in timelines for the redesigned program that are not reliable. The Office of Management and Budget directs agencies to define outcome-based performance measures to gauge program results early enough for stakeholder review, and industry experts describe the need for stakeholder input in developing performance measures in order to monitor performance. While CDC established performance measures and benchmarks for the initial implementation of the BioSense program, it has not yet developed outcome-based performance measures to monitor the progress of the redesigned program and does not intend to complete their development until the end of 2009. Until program officials develop reliable cost and timeline estimates and outcome-based performance measures for the redesigned BioSense program, they will lack key components needed to effectively manage the program, increasing the risk that the agency will perpetuate weaknesses identified in its initial implementation of the program and related system.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-09-100, Health Information Technology: More Detailed Plans Needed for the Centers for Disease Control and Prevention's Redesigned BioSense Program
This is the accessible text file for GAO report number GAO-09-100
entitled 'Health Information Technology: More Detailed Plans Needed for
the Centers for Disease Control and Prevention's Redesigned BioSense
Program' which was released on November 20, 2008.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Committees:
United States Government Accountability Office:
GAO:
November 2008:
Health Information Technology:
More Detailed Plans Needed for the Centers for Disease Control and
Prevention's Redesigned BioSense Program:
GAO-09-100:
GAO Highlights:
Highlights of GAO-09-100, a report to congressional committees.
Why GAO Did This Study:
In 2003, the Centers for Disease Control and Prevention (CDC), an
agency within the Department of Health and Human Services (HHS),
developed an electronic syndromic surveillance system called BioSense
that uses health-related data to identify patterns of disease symptoms
prior to specific diagnoses. In late 2007, CDC began to redesign the
program to improve collaboration with stakeholders and address
identified management weaknesses. Pursuant to House Report 110-231, GAO
evaluated the BioSense program, focusing on the cost and timeline
estimates and performance measures and benchmarks for implementing the
program, among other objectives. To accomplish this, GAO analyzed
relevant program documentation and interviewed CDC officials
responsible for developing and implementing BioSense.
What GAO Found:
While CDC identified annual and long-term cost and timeline estimates
and performance measures for the initial design of BioSense, these
estimates and measures did not reflect the implementation of its
redesigned program. CDC subsequently developed a draft plan for the
redesigned program that described high-level cost and timeline
estimates; however, the estimates are not reliable, and the plan did
not include performance measures.
* According to best practices, cost estimates should be well-
documented, comprehensive and accurate, and must be credible before
they can be considered to be reliable. However, CDC‘s cost estimates
for the redesigned program are not reliable because they are only
partially documented, are not comprehensive and accurate, and therefore
are not credible.
* Best practices for reliable timeline estimates include the
identification of resources to complete each task, establishment of a
critical path, and analysis of risks to the schedule. However, the
agency has not implemented these practices, resulting in timelines for
the redesigned program that are not reliable.
* The Office of Management and Budget directs agencies to define
outcome-based performance measures to gauge program results early
enough for stakeholder review, and industry experts describe the need
for stakeholder input in developing performance measures in order to
monitor performance. While CDC established performance measures and
benchmarks for the initial implementation of the BioSense program, it
has not yet developed outcome-based performance measures to monitor the
progress of the redesigned program and does not intend to complete
their development until the end of 2009.
Until program officials develop reliable cost and timeline estimates
and outcome-based performance measures for the redesigned BioSense
program, they will lack key components needed to effectively manage the
program, increasing the risk that the agency will perpetuate weaknesses
identified in its initial implementation of the program and related
system.
What GAO Recommends:
GAO is recommending that CDC develop reliable cost and timeline
estimates and outcome-based performance measures for implementing the
redesigned BioSense program. In written comments on a draft of this
report, HHS stated it welcomed the conclusions and recommendations and
provided updated information about current efforts intended to address
the recommendations.
To view the full product, including the scope and methodology, click on
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-09-100]. For more
information, contact Valerie C. Melvin at (202) 512-6304 or
melvinv@gao.gov.
[End of section]
Contents:
Letter:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Briefing Slides:
Appendix II: Comments from the Department of Health and Human Services:
Appendix III: GAO Contact and Staff Acknowledgments:
Abbreviations:
CDC: Centers for Disease Control and Prevention:
DHS: Department of Homeland Security:
DOD: Department of Defense:
ESSENCE: Electronic Surveillance System for the Early Notification of
Community-based Epidemics:
HHS: Department of Health and Human Services:
NBIS: National Biosurveillance Integration System:
NHIN: Nationwide Health Information Network:
OMB: Office of Management and Budget:
RODS: Real-time Outbreak and Disease Surveillance:
VA: Department of Veterans Affairs:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
November 20, 2008:
The Honorable Tom Harkin:
Chairman:
The Honorable Arlen Specter:
Ranking Member:
Subcommittee on Labor, Health and Human Services, Education, and
Related Agencies:
Committee on Appropriations:
United States Senate:
The Honorable Dave Obey:
Chairman:
The Honorable James T. Walsh:
Ranking Member:
Subcommittee on Labor, Health and Human Services, Education, and
Related Agencies:
Committee on Appropriations:
House of Representatives:
In 2002, President Bush signed the Public Health Security and
Bioterrorism Preparedness and Response Act, which required specific
activities aimed at improving the nation's preparedness for
bioterrorism and other public health emergencies.[Footnote 1] In
response to the passage of this act, the Centers for Disease Control
and Prevention (CDC), an agency within the Department of Health and
Human Services (HHS), developed, in 2003, an electronic syndromic
surveillance information system called BioSense.[Footnote 2] In 2004,
CDC established the BioSense program office to improve the nation's
capabilities for conducting syndromic surveillance by providing
federal, state, and local public health officials access to existing
data from health care organizations across the country through use of
the BioSense system. In late 2007, the agency initiated efforts to
redesign the program to improve collaboration with public health
stakeholders and to address management weaknesses identified in its
initial implementation. CDC's National Center for Public Health
Informatics is responsible for managing the program.
Pursuant to House Report 110-231,[Footnote 3] we evaluated BioSense,
focusing on (1) the costs and benefits of operating the BioSense system
as compared to other state and local surveillance systems, (2) the
usefulness of data provided by the system to hospitals and state and
local public health officials, and (3) the cost and timeline estimates
and performance measures and benchmarks for implementing the program.
We provided subcommittee staff with preliminary results from our study
on April 1, 2008, and subsequently agreed with your offices to continue
our evaluation of CDC's plans for implementing BioSense, focusing on
the annual and long-term cost and timeline estimates and performance
indicators and benchmarks for implementing a redesigned program. On
August 26, 2008, we provided your offices with briefing slides that
outlined the final results of our study. The purpose of this report is
to issue the published briefing slides to you and to officially
transmit our recommendations to the Director of the Centers for Disease
Control and Prevention. The slides, which discuss our scope and
methodology and incorporate edits made since we initially provided the
briefing, are included in appendix I.
We conducted this performance audit from October 2007 to August 2008 at
CDC's headquarters in Atlanta, Georgia, in accordance with generally
accepted government auditing standards. Those standards require that we
plan and perform the audit to obtain sufficient, appropriate evidence
to provide a reasonable basis for our findings and conclusions based on
our audit objectives. We believe that the evidence obtained provides a
reasonable basis for our findings and conclusions based on our audit
objectives.
In summary, our study highlighted the following:
* The costs and benefits of using data from the current BioSense system
and other similar syndromic surveillance systems are difficult to
measure and compare. Further, public health and hospital officials with
whom we spoke said that because of the difficulty of doing so, they do
not track the costs and benefits of using syndromic surveillance
systems.
* Federal, state, and local public health officials expressed mixed
views on the usefulness of BioSense data as compared with the
usefulness of data from similar systems. While stating that they find
the data from the BioSense system useful to varying degrees, about half
of the officials with whom we spoke said that they rely on other
systems as their primary tool for syndromic surveillance. For example,
officials with the Department of Defense (DOD) stated that, rather than
BioSense, the department's public health personnel use DOD's Electronic
Surveillance System for the Early Notification of Community-based
Epidemics (ESSENCE). In contrast, Department of Veterans Affairs (VA)
public health personnel do not regularly rely on syndromic surveillance
system data; however, they use the civilian version of ESSENCE more
often than BioSense. Both DOD and VA officials stated that ESSENCE
allows more flexibility in evaluating and viewing data than BioSense.
[Footnote 4]
* Although CDC identified annual and long-term cost and timeline
estimates and established performance measures and benchmarks for the
initial implementation of the BioSense program and its related system,
these estimates and measures did not reflect the implementation of its
redesigned program. CDC subsequently developed a draft plan for the
redesigned program that described high-level cost and timeline
estimates; however, these estimates were not reliable, and the plan did
not include performance measures.
- According to best practices, cost estimates should be well-
documented, comprehensive and accurate, and must be credible before
they can be considered to be reliable. However, CDC's cost estimates
for the redesigned program are not reliable because they are only
partially documented, are not comprehensive and accurate, and therefore
are not credible.
- Best practices for reliable timeline estimates include the
identification of resources to complete each task, establishment of a
critical path, and analysis of risks to the schedule. However, the
agency has not implemented these practices, resulting in timelines for
the redesigned program that are not reliable.
- The Office of Management and Budget directs agencies to define
meaningful annual and long-term outcome-based performance measures to
gauge the results of a program or activity early enough to allow time
for stakeholder review, and industry experts describe the need for
stakeholder input in developing performance measures in order to
monitor project performance. CDC has not yet developed outcome-based
performance measures to monitor the progress of the redesigned program
and does not intend to complete their development until the end of
2009.
Until program officials develop reliable cost and timeline estimates
and outcome-based performance measures for the redesigned BioSense
program, they will lack key components needed to effectively manage the
program.
Conclusions:
The costs and benefits of using data from the current implementation of
the BioSense system and other similar systems are difficult to measure
and compare, and public health and hospital officials with whom we
spoke do not document or track the costs and benefits of using these
systems. These officials also expressed mixed views on the usefulness
of BioSense data as compared to data from other systems.
While BioSense program officials have developed cost and timeline
estimates for the initial implementation of the program, they have not
yet reliably estimated costs and timelines for the redesigned program.
Additionally, program officials do not expect to complete the
development of outcome-based performance measures for the redesigned
BioSense program before the end of 2009. Until CDC develops these key
components of effective information technology program management,
BioSense officials will lack the management and planning tools needed
to gauge the success of the redesigned BioSense program in improving
federal, state, and local partners' abilities to respond to public
health events and will risk perpetuating weaknesses identified in the
initial implementation of the program and its related system.
Recommendations for Executive Action:
To ensure that CDC defines reliable plans for effectively managing the
development and implementation of the redesigned BioSense program and
its related system, we are recommending that the Director of CDC
instruct the Director of the National Center for Public Health
Informatics to take the following three actions while in the planning
phase of the redesigned BioSense program:
* develop reliable cost estimates for the program that are well-
documented, comprehensive and accurate, and credible;
* develop reliable timeline estimates for implementing the program;
and:
* with stakeholder input, develop outcome-based performance measures
that address all phases of the program and that focus the success of
the program on CDC's federal, state, and local partners' use of the
system for responding to public health events.
Agency Comments and Our Evaluation:
The Department of Health and Human Services' Assistant Secretary for
Legislation provided written comments on a draft of this report. In the
comments, HHS welcomed the conclusions and recommendations discussed in
our report.
In its comments, HHS provided information about steps the department
has begun taking to address the conclusions and recommendations in our
report. In particular, the department stated that it has launched a
planning and implementation process to ensure that the program develops
reliable cost estimates and timelines and identifies appropriate
outcome-based performance measures. According to the department, among
other activities, the program has initiated working groups of internal
and external stakeholders to establish detailed tactical plans for
projects, tasks, deliverables, and resources. The department stated
that it intends to use the tactical plans to derive future cost and
timeline estimates and involve stakeholders in establishing outcome-
based performance measures. It added that a program plan for the
redesigned Biosense is scheduled for completion by February 2009. If
effectively implemented, HHS's stated actions toward developing
reliable cost and timeline estimates and defining outcome-based
performance measures should help improve the department's management of
the redesigned BioSense program.
Finally, with regard to the program, the department stressed the
importance of noting that BioSense is not intended to replace local
systems, but rather to add value by providing data, tools, and funding
intended to improve surveillance efforts. HHS's written comments are
reproduced in appendix II.
DOD and VA also reviewed the draft report and provided technical
comments, which we have incorporated as appropriate.
We are sending copies of this report to interested congressional
committees, the Director of the Office of Management and Budget, and
the Secretaries of the Departments of Health and Human Services,
Defense, and Veterans Affairs. In addition, the report will be
available at no charge on the GAO Web site at [hyperlink,
http://www.gao.gov].
Should you or your staffs have any questions concerning this report,
please contact me at (202) 512-6304 or melvinv@gao.gov. Contact points
for our Offices of Congressional Relations and Public Affairs may be
found on the last page of this report. Key contributors to this report
are listed in appendix III.
Signed by:
Valerie C. Melvin:
Director, Human Capital and Management Information Systems Issues:
[End of section]
Appendix I: Briefing Slides:
Health Information Technology: More Detailed Plans Needed for The
Centers for Disease Control and Prevention‘s Redesigned BioSense
Program:
Briefing for Staff Members of the House and Senate Subcommittees on
Labor, Health and Human Services, Education, and Related Agencies,
Committees on Appropriations:
August 26, 2008:
Overview:
Introduction:
Objectives:
Scope and Methodology:
Results in Brief:
Background:
Results:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
[End of Overview]
Introduction:
A catastrophic health event could cause hundreds of thousands of
casualties, weaken our economy, damage public morale and confidence,
and threaten our national security. Threats of bioterrorism and high-
profile disease outbreaks, such as a pandemic influenza outbreak, have
drawn attention to the need for surveillance systems for the early
detection of and response to public health emergencies.
In 2002, President Bush signed the Public Health Security and
Bioterrorism Preparedness and Response Act, which required specific
activities intended to improve the nation‘s preparedness for
bioterrorism and other public health emergencies.
* The act required the Secretary of Health and Human Services to
establish an integrated system of public health alert communication and
surveillance networks among federal, state, and local public health
officials, and laboratories, hospitals, and other health care
facilities.
In response to the passage of this act, in 2003 the Centers for Disease
Control and Prevention (CDC) developed an electronic syndromic
surveillance information system called BioSense.
In 2004, the agency established the BioSense program office to improve
the nation‘s capabilities for conducting surveillance by providing
federal, state, and local public health officials access to existing
data from health care organizations across the country through the use
of the BioSense system. In late 2007, CDC initiated efforts to redesign
the program in order to improve collaboration with federal, state, and
local public health stakeholders and to address management weaknesses
identified in the initial implementation of BioSense.The program is
managed by CDC‘s National Center for Public Health Informatics.
According to CDC, syndromic surveillance is a technique that uses
health-related data to identify patterns of disease symptoms prior to
specific disease diagnoses. Effective use of this technique can provide
both situational awareness and early event detection.
* Situational awareness is the knowledge of the size, location, and
rate of spread of an outbreak.
* Early event detection provides signals that a sufficient probability
of a disease outbreak exists to warrant public health response and lead
to the early identification of illnesses and other events of public
health importance.
In this regard, syndromic surveillance can facilitate preventing and
controlling the spread of diseases.
Pursuant to House Report 110-231,[Footnote 5] we evaluated BioSense,
focusing on the costs and benefits of operating the BioSense system as
compared to other state and local surveillance systems; the usefulness
of data provided by the system to hospitals and state and local public
health officials; and cost and timeline estimates and performance
measures and benchmarks for implementing the program. As agreed with
staff of the House Committee on Appropriations, Subcommittee on Labor,
HHS, Education, and Related Agencies, we provided preliminary results
from our study on April 1, 2008. Subsequently, we agreed with staff of
the House and Senate Committees on Appropriations, Subcommittees on
Labor,HHS, Education, and Related Agencies to continue our evaluation
of CDC‘s plans for implementing BioSense, focusing on the annual and
long-term cost and timeline estimates, and performance indicators and
benchmarks for implementing a redesigned program.
The specific objectives addressed in our study were to:
* Compare the cost and benefits of operating the BioSense system with
the cost and benefits of operating state, local, and private
surveillance systems.
* Identify the views of hospitals and federal, state, and local public
health officials on the usefulness of data provided by the BioSense
system versus data provided by local surveillance systems for
responding to emergencies.
* Identify annual and long-term cost and timeline estimates and
performance indicators and benchmarks for the initial implementation of
BioSense, and evaluate CDC‘s plans for implementing the redesigned
BioSense program, focusing on annual and long-term cost and timeline
estimates, and performance indicators and benchmarks.
[End of Introduction]
Scope and Methodology:
To compare the cost and benefits of operating the BioSense system with
the cost and benefits of operating other similar state, local, and
private surveillance systems, we:
* identified from CDC‘s fiscal year 2009 budget and planning
documentation the agency‘s expected cost and projected benefits of
operating the BioSense system;
* reviewed cost and benefits information related to the use of the
BioSense system and available information on cost and benefits reported
for other surveillance systems used by hospitals and state and local
public health organizations; and;
* discussed with the BioSense users selected for our second objective
the cost of and benefits realized from using data provided by the
BioSense system.
To determine the views of federal, state, and local public health
officials and hospitals on the usefulness of BioSense data versus local
surveillance data, we selected and interviewed officials of
organizations that conduct syndromic surveillance using BioSense, other
systems, or BioSense together with other systems. These organizations
include:
* the three federal agencies that have access to the BioSense
system”CDC, the Department of Defense (DOD), and the Department of
Veterans Affairs (VA); and;
* three state and five local public health departments and two
hospitals:
- state public health departments in Arizona, Georgia, and North
Carolina;
- local public health departments in Dallas, El Paso, and Tarrant
counties in Texas; Maricopa County in Arizona; and the East Metro
Health District in Georgia; and;
- Gwinnett Medical Center in Lawrenceville, Georgia, and a Tenet
Healthcare hospital in El Paso, Texas.
To select these organizations, we obtained from CDC the program
office‘s list of BioSense users from the most recent month at the time
of our request”October 2007.
* According to CDC, there are more than 500 users of the BioSense
system, and they are located throughout 150 hospital, federal, state,
and local facilities, including CDC.
We selected organizations from different geographic areas that use
BioSense and other syndromic surveillance systems.
* We sorted users by state to identify a subset of organizations from a
wide geographic dispersion.
* From that subset, we selected states with varying numbers of BioSense
users.
* From those states, we selected a final set of states with high,
midrange, and low numbers of BioSense users, based on the average
number of users across the selected states.
We then selected local health departments in municipalities with
populations that varied from large to small.
* From among the BioSense users we selected, we further identified
organizations that also use BioSense and other commercial or locally-
developed systems based on literature research and our knowledge of the
syndromic surveillance system user population.
* We also included in our selection BioSense facilities that have not
opted to use the data provided by the system.
* We discussed with the selected users the usefulness of BioSense data
as compared with data from other similar systems.
To supplement our discussions with selected users, we met with
officials from two public health associations to obtain their views on
the usefulness of BioSense and of syndromic surveillance systems in
general. We held discussions with officials from:
* the Association for State and Territorial Health Officials, and;
* the Council for State and Territorial Epidemiologists.
We also interviewed public health officials from four other state and
local jurisdictions at a meeting arranged for us by the National
Association of City and County Health Officials.
In addition, we asked the federal, state, local, and hospital
representatives with whom we spoke to identify any lessons they had
learned from developing and implementing other similar surveillance
systems.
Our selection of BioSense users does not provide a statistically-valid
sample; therefore, the information provided by the organizations we
selected cannot be generalized across the entire BioSense user
population.
To identify annual and long-term cost and timeline estimates and
performance indicators and benchmarks for implementing the initial
design of BioSense, we reviewed and assessed CDC‘s relevant program
management and budget justification documentation for the program.
To evaluate CDC‘s plans for implementing the redesigned BioSense
program and related system, we obtained CDC‘s draft plan for and other
documentation relevant to implementing the redesigned BioSense program.
We evaluated the costs, schedule, and performance components addressed
in this documentation against criteria established by federal guidance
to determine if CDC followed best practices to develop these components
of its plan.[Footnote 6]
* We analyzed CDC‘s documented cost estimates for implementing there
designed program, as well as documentation describing the practices
that cost estimators followed to estimate costs of the program.
* We assessed CDC‘s estimated time frames for completing the strategic
activities described in the plan.
* We reviewed CDC‘s documented approach for developing performance
measures described in the plan and compared it to the Office of
Management and Budget‘s (OMB) guidance and industry practices to
determine if the agency‘s documented approach was consistent with
federal and industry guidance.[Footnote 7]
We supplemented our documentation reviews with interviews of officials
from CDC‘s BioSense program office to obtain additional information
about the development of current and future plans for BioSense. We did
not independently verify CDC‘s cost or timeline estimates for the
program.
We conducted this performance audit from October 2007 to August 2008 at
CDC‘s headquarters in Atlanta, Georgia, in accordance with generally
accepted government auditing standards. Those standards require that we
plan and perform the audit to obtain sufficient, appropriate evidence
to provide a reasonable basis for our findings and conclusions based on
our audit objectives. We believe that the evidence obtained provides a
reasonable basis for our findings and conclusions based on our audit
objectives.
[End of section]
Results in Brief:
The cost and benefits of using data from the current BioSense system
and other similar syndromic surveillance systems are difficult to
measure and compare. The state and local public health and hospital
officials with whom we spoke stated that documenting the cost and
benefits of using syndromic surveillance systems such as BioSense is
difficult and that they do not develop or track cost or benefits of
using these systems.
Federal, state, and local public health and hospital officials
expressed mixed views on the usefulness of BioSense data as compared
with the usefulness of other similar systems‘ data. While they stated
that they find the data from the BioSense system useful in varying
degrees, about half of them stated that they rely on other systems as
their primary tool for syndromic surveillance. Officials from two
public health associations differed in their views on the usefulness of
nationwide surveillance overall and BioSense in particular. While an
official from one of the associations noted that other methods of
conducting syndromic surveillance were more useful than electronic
syndromic surveillance systems, officials from the other association
stated that establishing a national syndromic surveillance system is
essential. Further, preliminary results of a study on the use of
syndromic surveillance systems conducted for CDC reiterated public
health and hospital officials‘ views of the usefulness of BioSense
data.
Although CDC identified annual and long-term costs for the initial
implementation of the BioSense program, these estimates did not reflect
the cost of implementing the redesigned program and its related system.
CDC subsequently developed a draft plan for the redesigned program with
high-level cost estimates based on program officials‘ professional
judgment. Best practices require cost estimates to be well-documented,
comprehensive and accurate, and credible before they can be considered
to be reliable. However, CDC‘s cost estimates for the program are not
reliable because they are only partially documented, are not
comprehensive and accurate, and are not credible.
According to program officials, CDC is in the strategic planning phase
of the redesigned program and is only able to provide high-level cost
estimates during this phase. Program officials stated that they intend
to reflect more accurate and refined estimates as the program plans
evolve and the technical and performance requirements of the program
are better defined. Until program officials are able to define the
information needed to develop cost estimates that are comprehensive and
accurate, CDC will not be positioned to provide reliable cost estimates
for implementing the BioSense program.
CDC also identified annual and long-term timeline estimates for the
initial implementation of the BioSense program. However, the timelines
were not relevant to the goals of the redesigned program. While CDC has
developed timeline estimates for implementing the redesigned BioSense
program, they are not reliable. Best practices for reliable timeline
estimates include, among other things, the identification of resources
to complete each task, establishment of a critical path, and analysis
of risks to the schedule. However, the agency has not assigned
resources to activities, established a critical path, or analyzed risks
to the schedule to ensure that its estimates are reliable.
According to BioSense program officials, the redesigned program is in
the early stages of planning, and they have not yet completed tactical
plans that would identify resources needed to complete tasks and a
critical path for the schedule. Until program officials consider these
components, CDC will not be positioned to provide reliable timeline
estimates for its plans to fully implement the BioSense program.
CDC established performance measures and benchmarks for the initial
design of the BioSense program. However, the performance measures
defined in the business case were not focused on intended results of
the redesigned program. OMB directs agencies to define meaningful
annual and long-term performance measures to gauge the intended results
of carrying out a program or activity early in the program to allow
time for stakeholder review. Additionally, industry experts describe
the need for performance measures to be developed with
stakeholders‘input early in a project‘s planning process to provide a
central management and planning tool and to monitor the performance of
the project against plans and stakeholders‘needs.
CDC‘s plan for the redesigned BioSense program describes a need to
develop outcome-based performance measures.However, program officials
stated that they do not intend to complete development of performance
measures until the end of 2009. Until BioSense program officials
develop these measures, they will lack an effective management and
planning tool for gauging the success of the program against plans and
against federal, state, and local partners‘ use of the BioSense system
for responding to public health events.
We are recommending that the Director of the Centers for Disease
Control and Prevention instruct the Director of the National Center for
Public Health Informatics to take the following three actions while in
the planning phase of the redesigned BioSense program:
* develop reliable cost estimates for the program that are well-
documented, comprehensive and accurate, and credible;
* develop reliable timeline estimates for implementing the program;
and;
* develop performance measures that address all phases of the program
and focus the success of the program on CDC‘s federal and state
partners‘ use of the system for responding to public health events.
We received oral comments on a draft of this briefing from BioSense
program officials, including the Director of CDC‘s National Center for
Public Health Informatics. The officials generally agreed with the
information included in the draft and provided additional information
and technical comments which were incorporated into the briefing as
appropriate.
[End of Section]
Background:
Historically, disease outbreaks have been recognized based either on
accumulated case reports of diseases by local public health departments
or by clinicians and laboratorians who alert public health officials
about potential disease outbreaks.
Complete responsibility for disease surveillance is shared among health
care providers, including public health officials from 59 state and
territorial health departments; more than 3,000 county, city, and
tribal health departments; multiple federal agencies, including CDC;
and more than 180,000 public and private laboratories.
* States take the lead in conducting disease surveillance and
supporting local emergency response efforts through a public health
infrastructure that is made up of state and county, city, and tribal
health departments”i.e., local health departments. State health
departments are responsible for collecting surveillance information
from local health departments, coordinating investigations and response
activities, and sharing disease surveillance data with CDC.
Local health departments are responsible for conducting the initial
investigations into reports of diseases. Initial response to a public
health emergency is generally a local responsibility that could involve
multiple local health departments in a geographical region, with state
health departments providing additional support when needed.
The federal government‘s role in disease surveillance traditionally has
been to collect data from state and local health departments and
perform nationwide analyses of the aggregated state and local data
using surveillance systems. CDC uses information and communications
systems to share disease surveillance information with state and local
health departments and provides funding and technical expertise to
support surveillance at the state and local levels.
For nearly 10 years, federal, state, and local public health
organizations, private companies, and academic institutions have been
developing systems for collecting and analyzing electronic surveillance
data from sources such as hospital emergency departments, clinical
laboratories, and pharmacies. These systems are intended to better
support efforts to detect disease outbreaks through electronic
syndromic surveillance and to more efficiently communicate information
to public health officials. For example:
* The University of Pittsburgh and Carnegie Mellon University developed
the Real-time Outbreak and Disease Surveillance (RODS) system in 1999.
* DOD developed the Electronic Surveillance System for the Early
Notification of Community-based Epidemics (ESSENCE) in 1999. A civilian
version of the system is available, at no charge, to any health
department that wants to use the system locally.
* According to CDC, about 20 states have developed electronic
surveillance systems.
CDC officials stated that, as tools to share information more quickly
and efficiently become available, the roles of federal, state, and
local public health agencies in conducting disease surveillance are
likely to evolve.
According to BioSense program officials, the initial vision for the
program was to provide real-time health data to federal, state, and
local public health organizations and participating hospitals in order
to:
* provide health situational awareness by enabling public health
officials at the federal, state, and local levels to monitor the size,
location, and rate of an outbreak;
* support early event detection by enabling public health officials to
detect a potential public health emergency at the earliest possible
time; and;
* assist public health officials in responding to a disease outbreak or
other emergency.
A key component of the program is the BioSense system”an electronic
information system which CDC developed to provide data and analytical
tools for public health and other health organizations to use in
conducting syndromic surveillance.
The system is made up of (1) the hardware and software that enable
BioSense to collect relevant data from public health organizations and
hospitals and (2) software that analyzes the collected data and
provides the analyzed data and analytical tools via a Web-based
application for use by public health and hospital officials who conduct
syndromic surveillance activities.
The current system collects site-specific data directly from (1) state
and local public health departments that collect hospital data from
within their jurisdictions and (2) hospital, DOD and VA facility,
health department, and laboratory data sources. The collected data are
stored in a central CDC data repository.
To provide data to the system, facilities must electronically store and
maintain the data, and they must establish data-sharing agreements with
CDC. As part of these agreements, CDC provides to facilities, at
minimal or no charge,the hardware and software needed to collect and
translate the facility‘s data into a standard format to enable
transmitting the data to BioSense.
According to CDC, the agency has agreements with and the system
currently collects data from about 570 hospitals, 320 DOD military
treatment facilities, 860 VA hospitals and outpatient clinics, and
other data sources that provide public health officials with the
information needed to conduct syndromic surveillance, such as patients‘
reasons for seeking medical attention, over-the-counter drug sales,
prescription drug information from clinicians, and clinical laboratory
test requests that could indicate disease outbreaks.
The BioSense data analysis software aggregates and analyzes the
electronic data collected from these facilities to identify disease
syndromes within state and local jurisdictions. The system then
displays graphical views of the presence of syndromes across time and
geographical locations in the form of charts, graphs, and maps via a
Web-based application. BioSense users can access the application via
the Internet at no charge.
* The views displayed by the BioSense Web-based application can be
accessed by public health and health care professionals, such as public
health epidemiologists and hospital infectious disease clinicians, who
use the system to facilitate their ability to track the outbreak,
treatment, and spread of diseases within their own jurisdictions.
[Footnote 8]
* CDC reports that there are more than 500 users of the BioSense
system, and that they are located throughout 150 hospital, federal,
state, and local facilities, including CDC.
The following graphic provides an overview of the current BioSense
system.
[Refer to PDF for image]
This figure is an illustration of the information flow in the current
BioSense system, as follows:
Medical facilities: Hospital information systems: hospital data is
transmitted to:
State and local health departments:
* Health departments data collection and BioSense data collection:
State and local hospital data is transmitted to:
CDC:
BioSense data analysis and Web-based applications analyze data,
transmit it to the BioSense data repository.
Hospitals, DOD, and VA facilities, health departments, and commercial
laboratories transmit data to BioSense data collection, then transmit
it to:
CDC:
BioSense data analysis and Web-based applications analyze data,
transmit it to the BioSense data repository.
Source: GAO analysis of CDC data.
[End of figure]
According to program officials, the current BioSense system:
* collects data directly from medical facilities or local public health
departments, VA and DOD facilities, or from state public health
departments that have already collected the local health department and
hospital data within their jurisdictions; the collected data are
analyzed and stored in a central CDC data repository;
* is not integrated with other federal public health information
technology initiatives, such as CDC‘s electronic disease surveillance
system; and;
* was designed and developed with limited collaboration between CDC and
state and local public health stakeholders.
In June 2005, we reported on federal agencies‘ progress in implementing
public health information technology initiatives. We noted that public
health officials said they did not find the BioSense system useful
because of limitations in the data being collected.[Footnote 9]
In 2006, an independent contractor evaluated the BioSense program and
identified about 50 management weaknesses that increased program risks,
including the lack of a clear, consistent vision and supporting plans
to guide and constrain the scope of the program.
During the third quarter of fiscal year 2007, a new director that CDC
hired to lead the National Center for Public Health Informatics
initiated efforts to redesign the BioSense program.[Footnote 10]
* According to BioSense officials, the redesigned program includes a
new strategy that emphasizes increased collaboration with state and
regional public health officials by providing more opportunities for
stakeholder training and holding round-table meetings with stakeholders
to gain a better understanding of their needs.
* BioSense officials stated that, as part of the program‘s redesign,
they intend to address all of the weaknesses identified by the
contractor‘s evaluation that are relevant to the new strategy.
* The officials also told us that they plan to reengineer the BioSense
system to reflect the redesigned program‘s new mission to ’provide a
system that comprehensively monitors the health care system of the
United States for evidence of acute health threats to the public.“
Program officials intend for the redesigned BioSense system to:
* collect and analyze already-aggregated data from states and health
information exchanges, rather than collect site-specific data from
individual data sources as the current system does;
* store results of the analyzed data in a central data repository at
CDC (the data collected from state and health information exchanges
will remain with their original sources);
* include enhanced, user-customizable data analysis tools to allow
users to easily tailor the application to better detect disease
outbreaks and emergencies within their own jurisdictions;
* allow states to share data and views across jurisdictions by building
regional data-sharing collaboratives;
* incorporate case detection technology for use in catastrophic as well
as routine public health capacities; and;
* be integrated with other public health information technology
components, such as CDC‘s electronic notifiable disease surveillance
systems and electronic laboratory reporting, the Department of Homeland
Security‘s (DHS) National Biosurveillance Integration System (NBIS),
[Footnote 11] and with the planned Nationwide Health Information
Network (NHIN).[Footnote 12]
According to program officials, Web-based analytical tools and data
views will still be available to BioSense users via the Internet.
The following graphic provides an overview of CDC‘s initial proposal
for the redesigned BioSense system.
[Refer to PDF for image]
This figure is a schematic diagram of the CDC‘s initial proposal for
the redesigned BioSense system, as follows:
1) Medical facilities:
Hospital information system collects data; data is transmitted to:
State and local health departments; Using Syndromic surveillance,
health departments data collection is transmitted to BioSense data
collection, then transmitted to and from:
CDC, BioSense System, BioSense data repository, which feeds both:
* Federal component of the NHIN;
* DHS's NBIS.
2) Regional collaboratives and health information exchanges information
systems collect data for case detection, which is transmitted to:
State and local health departments; Using Disease surveillance, health
departments data collection is transmitted to BioSense data collection,
then transmitted to and from and from:
CDC, BioSense System, BioSense data repository, which feeds both:
* Federal component of the NHIN;
* DHS's NBIS.
3) State and local health departments also interact with Hospital, DOD
and VA facilities and commercial labs, which utilize their information
systems to collect data which is transmitted to BioSense data
collection, then transmitted to and from:
CDC, BioSense System, BioSense data repository, which feeds both:
* Federal component of the NHIN;
* DHS's NBIS.
Source: GAO analysis of CDC data.
[End of figure]
In May 2008, BioSense program officials developed a draft plan that
outlined a vision for implementing the redesigned BioSense program
within a four-year timeframe.[Footnote 13] The draft plan articulates
four strategic goals for the redesigned program along with strategic-
level activities for accomplishing the goals. The four strategic goals
are:
* sponsor regional collaboration and health information exchanges;
* pursue open collaborative development;
* •transition to a federated data model,[Footnote 14] and;
* expand case detection technology.
BioSense program officials intend for the four goals to be accomplished
by the end of 2012 and stated that they will continue to update the
plan and incorporate more tactical and operational planning details
throughout 2008 and 2009.
The plan describes 2008 as a transitional year during which CDC will
continue to support ongoing initiatives and work to expand
collaboration with its stakeholders.
* Program officials incorporated into the new strategy for BioSense
ongoing initiatives, such as proof-of-concept efforts for evaluating
technology solutions for a federated data model and case-detection, and
linked the outcomes of these initiatives to the new program goals.
BioSense program funding covers costs for the program office, including
staff, equipment, and contractor support to develop, implement,
maintain, and support the BioSense system. It also covers extramural
activities, such as stakeholder round-table meetings, and research
grants and cooperative agreements with universities and other public
health researchers to evaluate the usefulness of the system for
supporting early event detection, situational awareness, and emergency
response. CDC reported spending about $184 million on the program from
2003 through June 2008.
The following graphic illustrates the evolution of the BioSense program
and system since the initiation of the system‘s development.
[Refer to PDF for image]
This figure is a timeline illustrating the evolution of the BioSense
program as follows:
January 2002: Public Health Security and Bioterrorism Preparedness and
Response Act of 2002.
March, 2003: BioSense application including DOD, VA, and laboratory
data rolled out to state and local users.
May, 2004: BioSense application rolled out to state and local users.
May, 2004: CDC reorganization (creation of the National Center for
Public Health Informatics).
June, 2005: GAO reported limited use of BioSense system by state and
local users.
August, 2006: Independent evaluation of BioSense program.
June, 2007: First roundtable meeting held with stakeholders.
July, 2007: New Director of National Center for Public Health
Informatics initiates redesign of BioSense.
November, 2007: Second roundtable meeting held with stakeholders.
February, 2008: Kickoff meeting held with health information exchanges.
March, 2008: First three health information exchanges identified for
redesign program.
April, 2008: Third roundtable meeting held with stakeholders.
May, 2008: Draft strategic plan provided to Subcommittee staff.
Source: GAO based on CDC data.
[End of figure]
[End of section]
Results: BioSense Costs and Benefits:
Costs and benefits of using data from BioSense and other syndromic
surveillance systems that public health officials rely on are difficult
to measure and compare. The state and local public health and hospital
officials with whom we spoke stated that documenting costs and benefits
of using syndromic surveillance systems such as BioSense is difficult
and that they do not develop or track the costs or benefits of using
these systems. Factors limiting the ability to compare the costs and
benefits of operating BioSense with other surveillance systems include:
* Some of the systems in use, including BioSense, are provided to
state, local, and hospital users at very low or no cost and require
little investment by users.[Footnote 15] Therefore, any additional data
provided by these systems are beneficial. Without a need to justify a
substantial investment in the systems, these organizations have not
tracked costs or benefits.
* Costs of developing and implementing systems can vary widely
depending on the approach used to acquire, maintain, and monitor
surveillance data.
* Implementation of syndromic surveillance systems varies widely in
scope by the type and volume of data collected, size of the
jurisdiction,and number of data providers. BioSense is the only
national system that centrally supports federal, state, and local
public health and hospital users and that evaluates nationwide data to
provide a national view of the country‘s health. In contrast, DOD and
state and local public health departments‘ systems are intended
specifically for use within their jurisdictions, such as within
military bases or state and local public health districts.
* System costs and benefits are difficult to track within the wide
range of response activities that are undertaken during outbreaks.
Syndromic surveillance activities are conducted as a component of a
wide range of emergency response activities, including activities such
as identifying available hospital beds and identifying specific
patients, and it is difficult to segregate the costs of specific tasks
from the overall response effort.
Consistent with other public health officials‘ experiences in using
syndromic surveillance systems, BioSense program officials have not
widely reported actual quantifiable benefits of using the BioSense
system.
Views on the usefulness of data provided by the BioSense system as
compared with the usefulness of other similar systems‘ data are mixed.
The federal agencies that use the BioSense system for syndromic
surveillance reported mixed views on the usefulness of the data
provided by the system.
* BioSense program officials stated that CDC‘s epidemiologists and
infection control analysts find the data provided by the BioSense
system useful for detecting and monitoring potential events that could
have a nationwide impact on public health and for notifying local
public health officials about potential outbreaks in their
jurisdictions. Program officials reported that, in 2007, BioSense data
were useful in identifying ten local public health events.
- Among these events, one of them was detected by CDC personnel using
data from BioSense before it was detected by the local health
department.
* DOD and VA officials stated that their departments‘ public health
personnel do not rely on data provided by the BioSense system to
conduct syndromic surveillance activities.
* DOD officials told us that they use ESSENCE because it provides the
ability for users to monitor the local populations for which they are
responsible”i.e., populations within military installations”whereas the
current version of BioSense presents aggregated information at the
state level. Therefore, the results of the BioSense system do not
provide information that DOD can act on within its jurisdiction.
- VA officials stated that their public health personnel do not
regularly rely on data provided by BioSense or any other system to
conduct syndromic surveillance activities. They told us that they use
the civilian version of ESSENCE more often than BioSense, though both
systems are occasionally monitored.
* Both DOD and VA officials stated that ESSENCE allows them more
flexibility in evaluating and viewing data than BioSense does.
Epidemiologists and infectious disease personnel at state and local
public health organizations and hospitals use data from the BioSense
system, other similar systems, and manual techniques for conducting
syndromic surveillance. Many organizations operate multiple syndromic
surveillance systems simultaneously, allowing access to multiple
sources of data that help them identify and respond to emergencies.
* For example, one county public health department we visited used a
’dashboard“ that employs BioSense, ESSENCE, and RODS to conduct
syndromic surveillance.
Of the ten organizations and hospitals we contacted”five local public
health departments, three state health departments, and two
hospitals”eight reported that they use data provided by the BioSense
system.
All but one of the eight state and local public health departments and
hospitals that use the BioSense system reported that they also use
other electronic syndromic surveillance systems. The following table
identifies these users and the systems.
Local public health departments:
Organization: Maricopa County, Arizona;
BioSense: [Check];
ESSENSE[A]: [Empty];
RODS: [Check];
State-Developed system: [Empty];
Locally-developed system:
Manual analysis: [Empty].
Organization: Gwinnett County, Georgia;
BioSense: [Check];
ESSENSE[A]: [Empty];
RODS: [Empty];
State-Developed system: [Check];
Locally-developed system: [Empty];
Manual analysis: [Empty].
Organization: Tarrant County, Texas;
BioSense: [Check];
ESSENSE[A]: [Check];
RODS: [Check];
State-Developed system: [Empty];
Locally-developed system: [Empty];
Manual analysis: [Empty].
Organization: El Paso, Texas;
BioSense: [Check];
ESSENSE[A]: [Empty];
RODS: [Check];
State-Developed system: [Empty];
Locally-developed system: [Empty];
Manual analysis: [Empty].
State public health organizations:
Organization: Arizona;
BioSense: [Check];
ESSENSE[A]: [Empty];
RODS: [Check];
State-Developed system: [Check];
Locally-developed system: [Empty];
Manual analysis: [Check].
Organization: Georgia;
BioSense: [Check];
ESSENSE[A]: [Empty];
RODS: [Empty]; [Check];
State-Developed system: [Check];
Locally-developed system: [Empty];
Manual analysis: [Empty].
Organization: North Carolina;
BioSense: [Check];
ESSENSE[A]: [Empty];
RODS: [Check];
State-Developed system: [Check];
Locally-developed system: [Empty];
Manual analysis: [Empty].
Hospitals:
Organization: Tenet Healthcare, El Paso, Texas;
BioSense: [Check];
ESSENSE[A]: [Empty];
RODS: [Empty];
State-Developed system: [Empty];
Locally-developed system: [Empty];
Manual analysis: [Check].
Total:
BioSense: 8;
ESSENSE[A]: 1;
RODS: 4;
State-Developed system: 4;
Locally-developed system: 1;
Manual analysis: 2.
Source: GAO analysis based on data provided by state and local public
health organizations and hospitals.
[A] The civilian version of ESSENCE is used by state and local health
departments.
[End of table]
The users we interviewed at the public health departments and hospitals
expressed mixed views about the usefulness of the data provided by the
BioSense system.
* Seven of the eight users we interviewed stated that they use the data
for situational awareness and that the system was useful for this
activity. Although some users stated that they use BioSense for early
event detection, they found that the system was only somewhat useful
for this activity because it does not provide sufficient health-related
data for event detection within their jurisdiction.
* Six of the eight BioSense users we interviewed stated that they use
the data either weekly or daily. Three of these users stated that the
BioSense system is their most important electronic syndromic
surveillance tool; two of these users stated that BioSense is the only
tool that they use for syndromic surveillance.[Footnote 16]
* Three of the seven users who also use other electronic syndromic
surveillance systems stated that the BioSense system is less important
to their overall syndromic surveillance efforts because the other
systems they use better enable them to evaluate and view data based on
specific needs of their jurisdictions.
The users we interviewed stated that deficiencies in the data provided
by the BioSense system limited their ability to use the data to conduct
syndromic surveillance in their jurisdictions:
* Limited data about health within jurisdictions”Seven of the users we
interviewed indicated that the BioSense system would be more effective
if its analysis included data from more hospitals in the jurisdiction.
For example, one local health department stated that the BioSense
system analyzes and provides its users with data for only seven of the
27 hospitals within its jurisdiction.
* Limited flexibility to query data on multiple symptoms”Three of the
users we interviewed explained that the BioSense system does not
provide the ability to query and report on multiple symptoms or to
tailor the system to detect emerging threats specific to their
jurisdictions. Users stated that this type of system flexibility”which
is available in other systems such as ESSENCE and RODS”enables public
health officials to more accurately identify health conditions that
have more than one symptom, such as influenza, and to detect and
monitor new illnesses as they appear within their own jurisdictions.
* Inability to access patient-level data”Five users we interviewed
cited the inability to access patient-level data as a significant
limitation of the BioSense system. According to these users, the
BioSense system does not provide these data, while other systems, such
as ESSENCE, can be enabled to allow access to data about specific
patients. Without this ability, local health officials may be limited
in their ability to quickly identify an infected patient in their local
areas.[Footnote 17]
While these users expressed mixed views about the usefulness of the
data provided by the BioSense system and stated that the system does
not yet fully address their needs, when asked if BioSense helped them
to better respond to emergencies, six of them stated that the data
provided by the BioSense system was useful for this purpose. According
to four of these users, the system provides an additional data resource
for confirming data provided by the other syndromic surveillance
systems that they use.
Of four additional local public health officials with whom we spoke at
a meeting arranged by the National Association of County and City
Health Officials, three stated that their organizations use the
BioSense system on a very limited basis because their locally-developed
systems provide access to more, better, or more detailed data, and
allow more flexibility in using the data. The one other organization
does not use any electronic system for syndromic surveillance, but
instead relies on manual data analysis because the hospitals within its
jurisdiction do not provide electronic data to be processed by any
electronic surveillance system.
Additionally, the public health experts from the two associations with
whom we spoke offered differing views on the usefulness of the BioSense
system and other similar electronic syndromic surveillance systems.
* The Council for State and Territorial Epidemiologists official with
whom we spoke stated that surveillance for disease and outbreak
detection has been accomplished primarily by astute clinicians and
physicians within a jurisdiction, and that communication support is
more important for detecting outbreaks across jurisdictions than
syndromic surveillance systems. Some capabilities developed through the
BioSense program, such as the data-viewing tools and the collection of
real-time hospital data, have been very useful, and the BioSense system
has been shown to be helpful in monitoring cases of disease outbreaks,
such as influenza.
* On the other hand, the Association for State and Territorial Health
Officials representatives with whom we spoke believe that establishing
a national syndromic surveillance system is essential and there is no
question about the need to have such a system. They pointed out that a
national system such as BioSense should be made up of multiple systems
instead of a single system, build on existing local and state
capabilities, and include local and state health officials as integral
partners in its design. While these officials had concerns with the
initial design of BioSense, they stated that they are pleased with
CDC‘s new plans.
In October 2006, CDC funded four institutions to conduct evaluations of
syndromic surveillance systems. The results of the studies are intended
to be used together to evaluate the usefulness of the BioSense system
at the federal, state, and local levels of public health for supporting
early event and situational awareness activities.
Preliminary results for one of the studies were published in 2007.
[Footnote 18] Initial case studies for this evaluation focused on a
large salmonella outbreak and a tornado in Georgia, the 2006-2007
influenza season, and a wildfire in south Georgia and north Florida.
The study is to be completed in November 2008. The preliminary results
of the evaluation reiterated the views of the users we contacted and
found that:
* Among the four events, the utility of syndromic surveillance was
greatest for monitoring the influenza season.
* In most instances, BioSense was not used because of the lack of local
real-time hospital data from local hospitals and the inflexibility of
the system.
* Syndromic surveillance systems used first were those that were
developed locally to support state and local public health and health
care practices and relationships.
In addition to BioSense system users, officials at BioSense facilities
who do not use the system offered several explanations for not doing
so.
* An official we spoke with at one hospital said that they had never
used the system because they use the state‘s system; they stated that
they rely on relationships with state and local public health
departments for conducting surveillance and use the state-level system,
which was designed to meet specific needs of their jurisdiction.
* Officials at one local health department stated that they had
accessed the BioSense system only once during the past year. The
department uses ESSENCE, RODS, and manual analysis to conduct syndromic
surveillance. These officials stated that they do not use BioSense
because the system lacks adequate data for their jurisdiction and
because BioSense does not provide the flexibility available in ESSENCE
and RODS, such as allowing users to perform multiple queries on the
data or to customize views of the data analyzed by the systems.
State and local public health and hospital officials we interviewed
identified lessons they had learned from their experiences in
developing, implementing, and sustaining syndromic surveillance systems
to ensure the systems are useful. They reported learning that:
* Systems should be developed incrementally. Officials with experience
in developing or implementing state and local systems emphasized that
systems should be developed incrementally based on users‘ needs and
lessons learned from each increment.
* Systems must be sustainable at the local level. Initiatives to
develop and implement systems must build in support for IT
infrastructure, installation and maintenance, and staff training.
* Stakeholder communication and involvement in determining requirements
is critical. The development and implementation of syndromic
surveillance systems must involve frequent, effective communications
with stakeholders while defining, documenting, and sharing project
requirements. These experiences provide valuable lessons that BioSense
program officials could apply to their approach for reengineering the
system to help ensure that the data provided by the system are useful
for BioSense stakeholders.
Results: BioSense Plans: Cost Estimates:
OMB has made developing accurate life-cycle cost estimates a priority
for agencies in properly managing their portfolios of capital assets,
including information technology, that have an estimated life of two
years or more.[Footnote 19] In addition, the draft Cost Assessment
Guide, which we developed based on best practices, defines
characteristics of reliable cost estimates for federal
agencies‘information technology programs.[Footnote 20] The guide
describes reliable cost estimates as those that are well-documented,
comprehensive and accurate, and credible.
* Well-documented cost estimates are those that include source data and
their significance, clearly detailed calculations and results, and
explanations of why particular methods and references were chosen. The
data used to estimate the costs can be traced to their source
documents.
* Comprehensive and accurate cost estimates have enough detail to
ensure that cost elements are neither omitted nor double counted, and
all cost-influencing assumptions are detailed in the estimates‘
documentation.
* Cost estimates are credible when limitations of the analysis are
discussed, assumptions made about the costs are varied, and outcomes
are recomputed to determine how sensitive they are to changes in the
assumptions. Risk analyses are also performed to determine the level of
risk associated with the estimate. The estimate‘s results are cross-
checked, and an independent cost estimate is developed to determine
whether other estimating methods produce similar results.
By addressing these characteristics, program officials should be better
able to define reliable cost estimates that are comprehensive and
accurate and that can be easily and clearly traced, replicated, and
updated. Additionally, program officials should reliably estimate costs
before proceeding too far into development or production to reduce the
risk that the program will not be able to meet its established cost
targets.
CDC identified cost estimates for the initial BioSense in its fiscal
year 2009 business case that was submitted to OMB in September
2007.[Footnote 21] Program officials estimated the total cost for the
program through 2013 to be $369million. However, these estimates did
not reflect the cost of implementing the redesigned program and its
related system.
In May 2008, CDC included in its plans for the redesigned program cost
estimates for completing the strategic activities that are intended to
provide an operational program by 2012.[Footnote 22] Program officials
report the total estimated cost of implementing the redesigned program
from 2009 through 2012 to be about $314 million. The estimates cover
costs of completing each of the strategic activities, such as
federating existing data sources, developing case detection
methodologies, and supporting research and development for real time
surveillance. According to program officials, the estimates also cover
the costs of existing BioSense operations including the program
office‘s staff, travel, transportation, and contractor support.
CDC‘s cost estimates for implementing the redesigned BioSense program
are only partially documented. Specifically, the documentation for the
estimates consisted only of explanations of the methods and references
chosen to compute the estimates. According to an official with CDC‘s
Financial Management Office, the initial cost estimates for the
redesigned BioSense program were developed based on program officials‘
professional judgment.[Footnote 23] As professional judgment estimates,
the official did not believe that the agency was required to develop or
maintain supporting documentation.
Additionally, the cost estimates for BioSense are not comprehensive and
accurate because the estimates were developed without considering all
cost elements. According to program officials, the estimates do not
include the cost of technology required to support new methods for
measuring performance of the system, such as tools for simulating
emergency response scenarios to estimate the impact that use of the
BioSense system has on the time required to respond to health threats.
Finally, the cost estimates do not meet the criteria for credibility as
defined in our cost estimation guide. The BioSense cost estimators did
not vary major assumptions to determine whether and to what extent
outcomes are sensitive to changes in the assumptions. Program officials
did not conduct risk and uncertainty analyses to determine the level of
risk and the extent to which the actual costs may vary from the initial
estimates. Additionally, the cost estimates for BioSense have not been
independently reviewed and verified to determine if other estimating
methods would produce similar results.
Program officials stated that the cost estimation activities were
conducted in the early planning phases of the redesigned program and
that they intend to refine cost estimates for a more accurate
reflection of the cost to fully implement the BioSense program and its
related system as the program requirements are defined in more
detail.[Footnote 24] Until program officials develop well-documented,
comprehensive and accurate, and credible cost data, CDC will not be
able to reliably estimate the cost of fully implementing BioSense.
Results: BioSense Plans: Timeline Estimates:
Our draft Cost Assessment Guide identifies best practices for
estimating project timelines and schedules, which include:
* defining, sequencing, and estimating the duration for each activity;
* assigning resources (e.g., labor, material, and overhead) to all
activities;
* identifying the critical path for all activities;
* identifying float time”the amount of time a task can slip before
affecting the critical path”between activities; and;
* conducting a schedule risk analysis using statistical methods to
determine the amount of time to reserve for contingencies.
By following these practices, program officials should be better able
to define reliable timeline estimates that can be used to identify when
problems or changes may occur and the impact they may have on the
success of the program. Further, according to OMB guidance,[Footnote
25] reliable timeline estimates should be defined before program
officials proceed too far into development or production to reduce the
risk that the program will not be able to meet its time frames and to
reduce the risk of undesirable outcomes.
CDC‘s fiscal year 2009 OMB business case for the initial BioSense
program identified annual and long-term timelines and milestones for
meeting program goals through fiscal year 2013. CDC also reported the
goals and the status of the program‘s efforts to meet the goals for
fiscal years 2006 and 2007. For example, the business case identified a
milestone to support research related to BioSense algorithms,
bioterrorism, and pandemic flu in 2006 and reported that the program
had met that milestone.
In its draft plan for the redesigned program, CDC defined the timelines
for completing the program‘s strategic activities and goals within a
four-year time frame, as shown in the following table.[Footnote 26]
According to BioSense program officials, these timelines were defined
based on experiences and knowledge gained from implementing prior and
ongoing pilot projects.[Footnote 27]
Strategic Goals: Sponsor regional collaboration and health information
exchanges;
Strategic Activities: Expand state and local partnerships with the CDC
and develop regional collaboratives for real time surveillance;
Timeline: 2008-2010.
Strategic Goals: Sponsor regional collaboration and health information
exchanges;
Strategic Activities: Support linkages between public health and health
information exchanges for real time surveillance;
Timeline: 2009-2012.
Strategic Goals: Pursue open collaborative development;
Strategic Activities: Pursue open source collaborative development;
Timeline: 2009-2012.
Strategic Goals: Transition to a federated data model;
Strategic Activities: Federate existing state and local real time
surveillance data sources;
Timeline: 2009-2010.
Strategic Goals: Expand case detection technology;
Strategic Activities: Refine and deploy BioSense case detection
technologies;
Timeline: 2009-2012.
Strategic Goals: Expand case detection technology;
Strategic Activities: Enhance incentives for clinical providers to
transmit data to public health;
Timeline: 2009-2011.
Source: CDC‘s National Center for Public Health Informatics.
[End of table]
While program officials have defined timelines for the program‘s
strategic-level activities, they have not completed the detailed plans
needed to estimate reliable timelines for implementing the redesigned
program.
* While the strategic activities were defined and sequenced, resources
have not yet been assigned to the activities, a critical path for the
program‘s schedule has not been identified, float time between
activities has not been determined, and a schedule risk analysis has
not been performed to plan for contingencies.
BioSense program officials stated that the program is still in the
early stages of its redesign and that they have recently begun to
develop tactical-level plans that include the detailed information
needed to define more specific timelines for completing the program‘s
strategic activities. In this regard, they drafted a document that
reflects timelines for completing specific activities in fiscal years
2008 and 2009 and stated that they intend to finalize a detailed
tactical plan for the program by September 30, 2008.
Until the program completes detailed plans that define the steps and
resources needed to accomplish the plan‘s strategic activities,
identify the schedule‘s critical path, and perform a schedule risk
analysis, CDC will not be positioned to provide reliable timeline
estimates that can be used to identify when problems or changes may
occur and the impact they may have on CDC‘s plans to fully implement
BioSense by 2012.
Results: BioSense Plans: Performance Measures:
OMB directs agencies to define and select meaningful annual and long-
term outcome-based performance metrics that measure the intended result
of carrying out a program or activity.[Footnote 28] Additionally,
industry experts describe the need for performance measures to be
developed with stakeholders‘ input early in a project‘s planning
process to provide a central management and planning tool and to
monitor the performance of the project against plans and stakeholders‘
needs.[Footnote 29] According to the Software Engineering Institute,
performance measures are effective mechanisms for providing credible
evidence of a program‘s progress.[Footnote 30]
In the fiscal year 2009 business case for BioSense, program officials
identified performance measures for the initial implementation of the
program and reported the status of the program in meeting benchmarks
for 2006 and 2007. For example, a decrease in the time required to
update BioSense after receipt of data from data providers was
identified as a performance measure with a benchmark to reduce the time
from an average of 2.81 hours in 2006 to no more than 2 hours in 2007.
However, the performance measures defined in the business case were not
focused on intended results of the redesigned program.
CDC‘s draft plan for the redesigned BioSense program describes a need
to develop outcome-based performance measures that focus the success of
the program on federal, state, and local partners‘ use of the BioSense
system. They stated that new performance measures for the redesigned
system are to be included in the program‘s plan by the end of calendar
year 2009.[Footnote 31] However, it is important to complete the
development of performance measures early in the planning phase of the
program to provide an essential planning tool that can be utilized
throughout all phases of the development and implementation of
BioSense.
Unless BioSense program officials develop outcome-based performance
measures based on stakeholder input during the planning phase of the
program, they will lack an effective management and planning tool
needed to monitor the performance of the program against plans and
stakeholders‘needs throughout all phases of the program.
[End of section]
Conclusions:
The costs and benefits of using data from the current implementation of
the BioSense system and other similar systems are difficult to measure
and compare, and state and local public health and hospital officials
with whom we spoke do not document or track costs and benefits of using
these systems. Additionally, these officials expressed mixed views on
the usefulness of BioSense data as compared to the usefulness of data
from other systems.
BioSense program officials have developed initial high-level cost and
timeline estimates for completing the implementation of a redesigned
BioSense program and related system that are intended to address
weaknesses of the initial implementation of the program and improve the
usefulness of the system for CDC‘s stakeholders. However, program
officials have not yet reliably estimated costs and timelines for fully
implementing the redesigned BioSense and for managing the development
and implementation of the program. Additionally, program officials do
not expect to complete the development of performance measures before
the end of calendar year 2009, which increases the risk that they will
not be able to effectively monitor the performance of the program
against plans and stakeholders‘needs throughout all phases of
development and implementation.
Until BioSense officials develop reliable cost and timeline estimates
and outcome-based performance measures, CDC‘s plans for the redesigned
program and its related system will lack key components needed to
effectively manage the program throughout all phases of the program.
Further, the agency will lack management and planning tools needed to
effectively gauge the success of the BioSense system‘s performance
toward improving federal, state, and local partners‘ abilities to
respond to public health events. Moreover, the agency remains at risk
that it will perpetuate weaknesses identified in the agency‘s initial
implementation of the program and related system and will continue to
spend money on a program that does not provide intended results or meet
stakeholders‘ needs.
[End of section]
Recommendations for Executive Action:
To ensure that CDC defines reliable plans for effectively managing the
development and implementation of the redesigned BioSense program and
its related system throughout all phases of the program, we are
recommending that the Director of the Centers for Disease Control and
Prevention instruct the Director of the National Center for Public
Health Informatics to take the following three actions while in the
planning phase of the program:
* develop reliable cost estimates for fully implementing the program
that are well-documented, comprehensive and accurate, and credible;
* develop reliable timeline estimates for implementing the program;
and;
* with stakeholder input, develop outcome-based performance measures
that address all phases of the program and that focus the success of
the program on CDC‘s federal, state, and local partners‘ use of the
system for responding to public health events.
[End of section]
Agency Comments and Our Evaluation:
We received oral comments on a draft of this briefing from CDC‘s
BioSense program officials, including the Director of the National
Center for Public Health Informatics. The officials generally agreed
with the information included in our draft briefing and provided
additional information and technical comments related to the program,
which we have incorporated as appropriate.
BioSense program officials acknowledged areas of deficiencies in the
initial design of the BioSense system and stated that the current
managers of the program, with extensive user engagement through
BioSense roundtable meetings, had identified the shortcomings detailed
in our briefing. They stated that they are consequently taking steps to
reconfigure the system and to address these problems, as outlined in
the draft BioSense Strategic Plan and as described in this briefing.
[End of section]
[End of slide presentation]
Appendix II: Comments from the Department of Health and Human Services:
Department Of Health & Human Services:
Office Of The Secretary:
Assistant Secretary for Legislation:
Washington, DC 20201:
November 10, 2000:
Valerie C. Melvin:
Director, Human Capital and Management Information Systems:
U.S. Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Ms. Melvin:
Enclosed are the Departments comments on the U.S. Government
Accountability Office's (GAO) draft report entitled: "Health
Information Technology: More Detailed Plans Needed for the Centers for
Disease Control and Prevention's Redesigned BioSense Program" (GAO 09-
100).
The Department appreciates the opportunity to comment on this report
before its publication.
Sincerely,
Signed by:
Jennifer R. Luong, for:
Vincent J. Ventimiglia, Jr.
Assistant Secretary for Legislation:
Attachment:
General Comments Of The Department Of Health And Human Services (HHS)
On The Government Accountability Office's (GAO) Draft Report Entitled:
Health Information Technology: More Detailed Plans Needed For The
Centers For Disease Control And Prevention's Redesigned Biosense
Program (GAO-09-100):
The Centers for Disease Control and Prevention (CDC) wishes to thank
the Government Accountability Office (GAO) for the opportunity to
review and comment on this Draft Report. We appreciate the effort and
professionalism that went into the thorough evaluation of the BioSense
program and welcome the recommendations and conclusions therein. In
this regard we respectively submit the following general comments.
In the Report's opening page, under "What GAO found" the authors note
that "...cost estimates for the redesigned program are not reliable
because they are only partially documented, are not comprehensive and
accurate, and therefore are not credible", and "... the agency has not
implemented these practices, resulting in timelines for the redesigned
program that are not reliable."
A planning and implementation process has been launched to ensure that
the BioSense program develops reliable cost estimates and timelines,
and identifies appropriate outcome-based performance measures. This
process follows a phased approach that focuses on work products being
vetted through internal and external stakeholders to ensure maximum
input and impact to public health.
This first phase of this process focused on identifying the steps
necessary to implement the strategic objectives outlined in the
redesigned BioSense strategic plan. The program initiated a series of
working groups integrating stakeholders from the Association of State
and Territorial Health Officials, National Association of County and
City Health Officials, Council of State and Territorial
Epidemiologists, and Association of Public Health Laboratories, as well
as those internal to CDC. Each working group had representation by one
or more of the partner organizations. The main deliverables were a
series of work breakdown structures[Footnote 32] for each strategic
objective.
The second phase focused on synthesizing these work breakdown
structures and refining the steps identified by the working groups to
develop a tactical roadmap and define associated performance outcomes.
The draft tactical plan and comprehensive breakdown structure
deliverables are currently under review by CDC's National Center for
Public Health Informatics (NCPHI) leadership to be delivered to working
group members on 11/14/08.
To determine comprehensive and accurate cost estimates and performance
outcome measures, the BioSense program is preparing to implement two
complementary efforts: Program Planning and Performance Measurement.
The program planning effort expands the focus areas identified in the
tactical plan into projects, detailed tasks, deliverables, and
resources required. Cost estimates will be developed to implement these
tasks, and activities will be prioritized accordingly. This activity is
targeted for completion on 12/30/08. Once finalized, the program plan
will include all projects and their related tasks and costs. The
program plan is scheduled for completion in February 2009.
The program plan will be managed to ensure reliable, timely and
accurate activity and cost forecasting.
A performance measurement working group will engage stakeholders, state
and local representatives and subject matter experts such as members of
ASTHO, NACCHO, CSTE, APHL and the NCPHI Board of Scientific Counselors
to:
* Review and adjust the proposed performance outcomes;
* Identify baseline and target performance measures;
* Develop a tracking and reporting process.
CDC has preliminarily identified 24 milestones with accompanying target
completion dates. Of these 24 milestones, six have been completed, four
are in progress and fourteen are planned with estimated completion
dates ranging from two weeks to 12 months.
With regards to the statement made several times that BioSense is seen
by many state personnel as not as useful as the state systems (ESSENCE,
RODS, etc.), it is important to note that BioSense is not intended to
replace local systems. Rather, BioSense adds value to local systems by
1) aggregating data from local systems for regional or national view,
2) making national datasets available, 3) making software tools
available, and 4) funding regional collaboratives, health information
exchanges (HIEs) and research to improve the field overall. [Footnote
33]
In closing, CDC appreciates GAO's review and recommendations. CDC is
committed to continuing and enhancing its efforts to develop reliable
cost and timeline estimates and engaging stakeholders in developing
outcome-based performance measures for the redesigned BioSense program.
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
Valerie C. Melvin, (202) 512-6304 or melvinv@gao.gov:
Staff Acknowledgments:
In addition to the contact named above, key contributions to this
report were made by Teresa Tucker (Assistant Director), Heather A.
Collins, Neil J. Doherty, Amanda C. Gill, Nancy E. Glover, Franklin D.
Jackson, Mohammad S. Khan, and Lee A. McCracken.
[End of section]
Footnotes:
[1] 42 U.S.C. § 247d-4(b).
[2] According to CDC, syndromic surveillance is a technique that uses
health-related data to identify patterns of disease symptoms prior to
specific diagnoses. Effective use of this technique can provide both
situational awareness--knowledge of the size, location, and rate of
spread of an outbreak--and early event detection, signaling a
probability of an outbreak sufficient to warrant a public health
response and leading to early identification of illnesses and other
events of public health importance.
[3] H.R. Rep. No. 110-231, at 122 (2007).
[4] DOD developed the ESSENCE system in 1999. A civilian version of the
system is available, at no charge, to any health department that wants
to use the system locally.
[5] H.R Rep. No. 110-231, at 122 (2007).
[6] Executive Office of the President, Office of Management and
Budget,Planning, Budgeting, Acquisition, and Management of Capital
Assets,Circular No. A-11, Part 7 (Washington, D.C.: June 2008). See
also, U.S. Government Accountability Office, Cost Assessment Guide:
Best Practices for Estimating and Managing Program Costs, Exposure
Draft, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP]
(Washington, D.C.: July 2007).
[7] Executive Office of the President, Office of Management and Budget,
Guide to the Program Assessment Rating Tool, (Washington, D.C.: January
2008); Thomas Wettstein and Peter Kueng, "A Maturity Model for
Performance Measurement Systems," Management Information Systems 2002--
Incorporating GIS and Remote Sensing(Southampton: WIT Press, 2002),113-
122; and Karen J. Richter, Ph.D., Institute for Defense Analyses, CMMI®
for Acquisition (CMMI-ACQ) Primer, Version 1.2 (Software Engineering
Institute, May 2008).
[8] Users can access data for neighboring jurisdictions or other
locations only with additional approval from the other jurisdictions
involved and from CDC.
[9] GAO, Information Technology: Federal Agencies Face Challenges in
Implementing Initiatives to Improve Public Health Infrastructure,
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-05-308] (Washington,
D.C.: June 2005).
[10] The BioSense program is managed by the National Center for Public
Health Informatics within CDC.
[11] The National BioSurveillance Integration System is a federal
effort to combine multiple data streams from sector-specific
agencies”those with medical, environmental, agricultural, and
intelligence data”to give DHS situational awareness that is intended to
allow earlier detection of events and to assist response activities.
[12] The Nationwide Health Information Network is a critical portion of
the HHS Office of the National Coordinator for Health Information
Technology‘s agenda intended to provide a secure, nationwide,
interoperable health information infrastructure that will connect
providers, consumers, and others involved in supporting health and
health care. It is planned to be a ’network of networks“ built out of
state and regional health information exchanges and other networks to
support the exchange of health information by connecting these
networks.
[13] Centers for Disease Control and Prevention National Center for
Public Health Informatics, BioSense Strategic Plan, FY 2008–2012,
Version 6 (Draft); (Atlanta, Ga.).
[14] A federated data model supports data sharing across a large
community despite having data stored in different geographic locations.
Use of this model would be expected to enable the public health
community to store data locally, in state and local public health
departments, while enabling CDC and other organizations to access and
analyze that data.
[15] Three organizations in our review reported that the only cost for
using surveillance systems was staff time for using the systems, which
is performed in combination with other activities. Most state and local
users in our review used federal or state grant funding to cover system
costs.
[16] One of the two users stated that, although other tools were used
at the facility, they were not used specifically for syndromic
surveillance.
[17] In their comments on a draft of this briefing, BioSense program
officials stated that individual patient-level data are available from
application-level views provided by the BioSense system.
[18] James W. Buehler, M.D., Alexander P. Isakov, M.D, M.P.H., Michael
J. Prietula, Ph.D.,Donna J. Smith, M.A., Ellen A. Whitney, M.P.H.,
Preliminary Findings from the BioSense Evaluation Project(Rollins
School of Public Health, School of Medicine, and Goizueta Business
School, Emory University, Atlanta, Ga.; 2007).
[19] OMB Circular A-11, Part 7.
[20] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP].
[21] OMB requires agencies to submit business cases through OMB Exhibit
300‘s for major information technology programs to establish cost,
schedule, and measurable performance goals.
[22] According to program officials, CDC‘s cost estimates reflect the
agency‘s plan to complete the development and implementation of the
redesigned BioSense program and its related system by 2012 rather than
2013, as its previous plans reflected.
[23] CDC defines its professional judgment estimates as informal
estimates provided to Congress. According to CDC officials, the agency
provides professional judgment estimates without regard to the
competing priorities that the agency, the President, and their advisors
must consider as budget submissions to the Congress are developed.
[24] BioSense program officials stated that they intend to use the
refined estimates for the preparation of future congressional budget
submissions.
[25] Executive Office of the President, Office of Management and
Budget, Memorandum on Conducting Acquisition Assessments with
Guidelines for Assessing the Acquisition Function (Washington, D.C.:
May 21, 2008).
[26] According to program officials, CDC‘s timeline estimates were
developed to support completion of the development and implementation
of the redesigned BioSense program and its related system by 2012
rather than 2013, as its previous plans reflected.
[27] CDC officials described an additional activity, support for
research to develop innovative and promising technologies for real time
surveillance, as a cross-cutting activity that supports each goal from
2009 through 2012.
[28] OMB, Guide to the Performance Assessment Rating Tool.
[29] Thomas Wettstein and Peter Kueng, ’A Maturity Model for
Performance Measurement Systems,“ and Karen J. Richter, Ph.D.,
Institute for Defense Analyses, CMMI®for Acquisition (CMMI-ACQ) Primer,
Version 1.2.
[30] GAO, Veterans Benefits Administration: Progress Made in Long-Term
Effort to Replace Benefits Payment System, but Challenges Persist,
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-614] (Washington,
D.C.: April 2007).
[31] In commenting on a draft of this briefing, BioSense program
officials stated they intend for federal, state, and local partners to
be engaged in the development of outcome-based performance measures
early in the detailed planning process, beginning in November 2008.
[32] A work breakdown structure or WBS is a tree structure that permits
summing of subordinate costs for tasks, materials, etc., into their
successively higher level "parent" tasks, materials, etc. It is a
fundamental tool commonly used in project management and systems
engineering.
[33] With regards to progress on establishing regional collaboratives,
grants have been awarded to four sites: National Capital Region,
Missouri-Kansas, Pennsylvania-Ohio, Southeast Region (Alabama, Florida,
Georgia, Mississippi, North Carolina, South Carolina, Tennessee). In
addition, three funded HIEs have already begun work: Indiana, New York,
Eastern Washington/Western Idaho.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
