Medicare
CMS Needs Additional Authority to Adequately Oversee Patient Safety in Hospitals
Gao ID: GAO-04-850 July 20, 2004
Hospitals accredited by the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) are considered in compliance with Medicare participation requirements. GAO examined the extent to which JCAHO's pre-2004 hospital accreditation process identified hospitals not complying with Medicare requirements, the potential of JCAHO's new process for improving the detection of deficiencies in Medicare requirements, and the effectiveness of CMS's oversight of JCAHO's hospital accreditation program. GAO analyzed CMS data on hospitals state surveyors found to have deficiencies in Medicare requirements that JCAHO surveyors did not detect, analyzed CMS's measure of JCAHO's ability to detect noncompliance with Medicare requirements, and interviewed JCAHO officials.
JCAHO's pre-2004 hospital accreditation process did not identify most of the hospitals found by state survey agencies in CMS's annual validation survey sample to have deficiencies in Medicare requirements. In comparing the results of the two surveys, CMS considered whether it was reasonable to conclude that the deficiencies found by state survey agencies existed at the time JCAHO surveyed the hospital. In a sample of 500 JCAHO-accredited hospitals, state agency validation surveys conducted in fiscal years 2000 through 2002 identified 31 percent (157 hospitals) with deficiencies in Medicare requirements. Of these 157 hospitals, JCAHO did not identify 78 percent (123 hospitals) as having deficiencies in Medicare requirements. For the same validation survey sample, JCAHO also did not identify the majority--about 69 percent--of deficiencies in Medicare requirements found by state agencies. Importantly, the number of deficiencies found by validation surveys represents 2 percent of the 11,000 Medicare requirements surveyed by state agencies in the sample during this time period. At the same time, a single deficiency in a Medicare requirement can limit the hospital's capability to provide adequate care and ensure patient safety and health. Inadequacies in nursing practices or deficiencies in a hospital's physical environment, which includes fire safety, are examples of deficiencies in Medicare requirements that could endanger multiple patients. The potential of JCAHO's new hospital accreditation process to improve the detection of deficiencies in Medicare requirements is unknown because the process was just implemented in January 2004. JCAHO plans to move from using announced to unannounced surveys in 2006, which would afford JCAHO the opportunity to observe hospitals' operations when the hospitals have not prepared in advance to be surveyed. In addition, the pilot test of the new accreditation process was of limited value in predicting whether it will be an improvement over the pre-2004 process in detecting deficiencies. Limitations in the pilot test included that hospitals were not randomly selected to participate; that observers from JCAHO accompanied each surveyor, thus possibly affecting surveyors' actions; and that JCAHO evaluated the results instead of an independent entity. CMS has limited oversight authority over JCAHO's hospital accreditation program because the program's unique legal status effectively prevents CMS from taking actions that it has the authority to take with other health care accreditation programs to ensure satisfactory performance. For example, requiring JCAHO's hospital accreditation program to submit to a direct review process or placing the program on probation while monitoring its performance. Further, CMS relies on a measure to evaluate how well JCAHO's hospital accreditation program detects deficiencies in Medicare requirements that provides limited information and can mask problems with program performance, uses statistical methods that are insufficient to assess JCAHO's performance, and has reduced the number of validation surveys it conducts.
Recommendations
Our recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director:
Team:
Phone:
GAO-04-850, Medicare: CMS Needs Additional Authority to Adequately Oversee Patient Safety in Hospitals
This is the accessible text file for GAO report number GAO-04-850
entitled 'Medicare: CMS Needs Additional Authority to Adequately
Oversee Patient Safety in Hospitals' which was released on July 20,
2004.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
United States Government Accountability Office:
GAO:
July 2004:
Medicare:
CMS Needs Additional Authority to Adequately Oversee Patient Safety in
Hospitals:
GAO-04-850:
GAO Highlights:
Highlights of GAO-04-850, a report to congressional requesters:
Why GAO Did This Study:
Hospitals accredited by the Joint Commission on Accreditation of
Healthcare Organizations (JCAHO) are considered in compliance with
Medicare participation requirements. GAO examined the extent to which
JCAHO‘s pre-2004 hospital accreditation process identified hospitals
not complying with Medicare requirements, the potential of JCAHO‘s new
process for improving the detection of deficiencies in Medicare
requirements, and the effectiveness of CMS‘s oversight of JCAHO‘s
hospital accreditation program. GAO analyzed CMS data on hospitals
state surveyors found to have deficiencies in Medicare requirements
that JCAHO surveyors did not detect, analyzed CMS‘s measure of JCAHO‘s
ability to detect noncompliance with Medicare requirements, and
interviewed JCAHO officials.
What GAO Found:
JCAHO‘s pre-2004 hospital accreditation process did not identify most
of the hospitals found by state survey agencies in CMS‘s annual
validation survey sample to have deficiencies in Medicare requirements.
In comparing the results of the two surveys, CMS considered whether it
was reasonable to conclude that the deficiencies found by state survey
agencies existed at the time JCAHO surveyed the hospital. In a sample
of 500 JCAHO-accredited hospitals, state agency validation surveys
conducted in fiscal years 2000 through 2002 identified 31 percent (157
hospitals) with deficiencies in Medicare requirements. Of these 157
hospitals, JCAHO did not identify 78 percent (123 hospitals) as having
deficiencies in Medicare requirements. For the same validation survey
sample, JCAHO also did not identify the majority--about 69 percent--of
deficiencies in Medicare requirements found by state agencies.
Importantly, the number of deficiencies found by validation surveys
represents 2 percent of the 11,000 Medicare requirements surveyed by
state agencies in the sample during this time period. At the same
time, a single deficiency in a Medicare requirement can limit the
hospital‘s capability to provide adequate care and ensure patient
safety and health. Inadequacies in nursing practices or deficiencies in
a hospital‘s physical environment, which includes fire safety, are
examples of deficiencies in Medicare requirements that could endanger
multiple patients.
The potential of JCAHO‘s new hospital accreditation process to improve
the detection of deficiencies in Medicare requirements is unknown
because the process was just implemented in January 2004. JCAHO plans
to move from using announced to unannounced surveys in 2006, which
would afford JCAHO the opportunity to observe hospitals‘ operations
when the hospitals have not prepared in advance to be surveyed. In
addition, the pilot test of the new accreditation process was of
limited value in predicting whether it will be an improvement over the
pre-2004 process in detecting deficiencies. Limitations in the pilot
test included that hospitals were not randomly selected to participate;
that observers from JCAHO accompanied each surveyor, thus possibly
affecting surveyors‘ actions; and that JCAHO evaluated the results
instead of an independent entity.
CMS has limited oversight authority over JCAHO‘s hospital
accreditation program because the program‘s unique legal status
effectively prevents CMS from taking actions that it has the authority
to take with other health care accreditation programs to ensure
satisfactory performance. For example, requiring JCAHO‘s hospital
accreditation program to submit to a direct review process or placing
the program on probation while monitoring its performance. Further,
CMS relies on a measure to evaluate how well JCAHO‘s hospital
accreditation program detects deficiencies in Medicare requirements
that provides limited information and can mask problems with program
performance, uses statistical methods that are insufficient to assess
JCAHO‘s performance, and has reduced the number of validation surveys
it conducts.
What GAO Recommends:
GAO believes that Congress should consider giving CMS the authority
over JCAHO‘s hospital accreditation program that it has over other
accreditation programs and recommends that CMS modify its methods for
assessing JCAHO‘s performance. CMS agreed with GAO‘s recommendations.
JCAHO stated that GAO‘s methodology was incomplete and did not
comprehensively assess its overall performance. GAO emphasized that its
engagement was limited to one aspect of deficiency detection and was
not intended to reflect JCAHO‘s overall performance.
www.gao.gov/cgi-bin/getrpt?GAO-04-850.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Janet Heinrich at (202)
512-7119.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
JCAHO's Pre-2004 Hospital Accreditation Process Often Did Not Detect
Serious Deficiencies Found by State Survey Agencies:
Potential of JCAHO's New Hospital Accreditation Process Is Unknown, and
Testing Was Limited:
CMS Oversight Authority of JCAHO's Hospital Accreditation Program Is
Limited and Needs Improvement:
Conclusions:
Matter for Congressional Consideration:
Recommendations for Executive Action:
Agency and Other External Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Medicare Conditions of Participation:
Appendix III: Features of JCAHO's New Accreditation Process:
Appendix IV: Comments from the Centers for Medicare & Medicaid
Services:
Appendix V: Comments from the Joint Commission on Accreditation of
Healthcare Organizations:
Appendix VI: GAO Contact and Staff Acknowledgments:
GAO Contact:
Acknowledgments:
Related GAO Products:
Tables:
Table 1: Hospitals in CMS's Validation Survey Sample with Serious
Deficiencies that State Survey Agencies Identified but JCAHO Surveyors
Did Not, Fiscal Years 2000-2002:
Table 2: Percentage of Serious Deficiencies Identified by State Survey
Agencies but Not by JCAHO Surveyors in CMS's Validation Survey Sample,
Fiscal Years 2000-2002:
Table 3: Number of Serious Deficiencies, by COP, Identified by State
Survey Agencies but Not by JCAHO Surveyors in CMS's Validation Survey
Sample, Fiscal Years 2000-2002:
Table 4: Accreditation Decisions for Hospitals Surveyed Under JCAHO's
New Survey Process Pilot Test as Compared to Results from JCAHO's Pre-
2004 Survey Process:
Table 5: Hypothetical Examples of the Effect on the Rate of Disparity
of a Decrease in the Number of Hospitals with Serious Deficiencies in a
Sample of 200 Hospitals:
Table 6: Number of Hospitals Targeted for Validation Surveys Compared
with Usable Traditional Validation Surveys Completed:
Table 7: Medicare Conditions of Participation:
Table 8: JCAHO's Description of Features of Its New Hospital
Accreditation Process:
Abbreviations:
AOA: American Osteopathic Association:
CMS: Centers for Medicare & Medicaid Services:
COP: condition of participation:
HHS: Department of Health and Human Services:
JCAHO: Joint Commission on Accreditation of Healthcare Organizations:
OIG: Office of Inspector General:
PFP: priority focus process:
PPR: periodic performance review:
United States Government Accountability Office:
Washington, DC 20548:
July 20, 2004:
The Honorable Charles E. Grassley:
Chairman:
Committee on Finance:
United States Senate:
The Honorable Pete Stark:
Ranking Minority Member:
Subcommittee on Health:
Committee on Ways and Means:
House of Representatives:
In fiscal year 2002, nearly 7.4 million Medicare beneficiaries received
inpatient health care at hospitals that participated in Medicare.
Federal law establishes criteria for hospitals for purposes of
Medicare. The Centers for Medicare and Medicaid Services (CMS), the
agency responsible for administering Medicare, has established quality
and patient safety requirements called conditions of participation
(COP) that hospitals must meet in order to be eligible for Medicare
payment. Hospitals that are accredited by the Joint Commission on
Accreditation of Healthcare Organizations (JCAHO) are generally deemed
under federal law to be compliant with Medicare requirements for
patient safety and health and become eligible for payments from
Medicare.[Footnote 1],[Footnote 2] No other health care accreditation
program has this same statutory authority.
JCAHO is a private, not-for-profit organization that accredits most of
the hospitals that participate in Medicare. JCAHO sets standards that
accredited hospitals must meet and reports that these standards are
more comprehensive than the Medicare COPs.[Footnote 3] In January 2004,
JCAHO implemented a new hospital accreditation process with goals that
included further enhancing health care quality and safety.
CMS oversight of JCAHO's hospital accreditation program is limited
because it cannot restrict or remove JCAHO's accreditation authority if
the agency detects problems. To oversee the program, CMS conducts on-
site validation surveys of a sample of JCAHO-accredited hospitals and
reports annually to Congress on the results of these surveys. The
validation surveys, which are performed by agencies that CMS has
agreements with in each state, help CMS determine whether Medicare
quality and safety requirements are being met. CMS compares the results
of these state surveys against survey results obtained through JCAHO's
hospital accreditation program. CMS uses a measure called the rate of
disparity that summarizes the extent to which an accreditation program
has failed to cite deficiencies identified by state agency validation
surveys. We are using the term serious deficiency in this report to
indicate a deficiency in one or more Medicare COPs. Examples of serious
deficiencies include a hospital's inability to provide adequate nursing
services or failure to implement and enforce infection control
policies. According to CMS, serious deficiencies substantially limit a
hospital's capability to render adequate care and adversely affect the
safety and health of patients.
Questions have been raised by the Department of Health and Human
Services' (HHS) Office of Inspector General (OIG) and others regarding
whether accreditation by JCAHO ensures that hospitals provide adequate
care. Specifically, experts have questioned how well JCAHO's hospital
accreditation process identifies deficiencies in hospitals that could
jeopardize patient safety and health. A comprehensive study by the HHS
OIG found that JCAHO's surveys were not likely to identify patterns of
deficient care.[Footnote 4]
You asked that we examine the effectiveness of JCAHO's hospital
accreditation process in ensuring that hospitals comply with Medicare
COPs to ensure the safety and health of Medicare beneficiaries.
Specifically, we (1) examined the extent to which JCAHO's pre-2004
hospital accreditation process identified deficiencies in Medicare COPs
that were identified by state survey agencies, (2) determined whether
JCAHO's new hospital accreditation process has potential for improving
the detection of deficiencies in Medicare COPs and whether the process
was adequately tested, and (3) examined the effectiveness of CMS's
oversight of JCAHO's hospital accreditation program.
To determine the extent to which JCAHO's pre-2004 hospital
accreditation process identified deficiencies in Medicare COPs that
were identified by state survey agencies, we used data from a CMS
comparison of state validation survey findings with findings of JCAHO's
hospital accreditation surveys, which indicated whether JCAHO found
deficiencies in its standards. Of the four possible outcomes to this
comparison of survey findings--(1) JCAHO and state agencies both
identify no deficiencies, (2) JCAHO identifies deficiencies not found
by state agencies, (3) JCAHO and state agencies both identify the same
deficiencies, and (4) state agencies identify deficiencies that JCAHO
does not--we focused on the fourth because it highlights the need for
CMS oversight of the hospital accreditation program. For the second
outcome, there could be two reasons for the disparity between JCAHO's
and state survey agencies' findings: hospitals corrected deficiencies
identified by JCAHO prior to the state agency survey or the state
survey agency did not identify a deficiency that existed. In addition,
not all JCAHO findings are equivalent to noncompliance with a Medicare
COP. To determine whether JCAHO's findings on deficiencies in its
standards were comparable to the state agencies' findings, CMS staff
compared the two surveys and considered whether it was reasonable to
conclude that the deficiencies found by state survey agencies existed
at the time JCAHO surveyed the hospital. For deficiencies that CMS
determined that JCAHO failed to identify, CMS met with JCAHO to address
disputed findings and consider additional evidence on comparability
offered by JCAHO. CMS provided results for a sample of 500 JCAHO-
accredited hospitals from fiscal years 2000 through 2002. We determined
that the data CMS provided on serious deficiencies were adequate for
addressing the issues in this report. On the basis of this sample of
500 JCAHO-accredited hospitals, we determined, using CMS's data, both
the percentage of serious deficiencies and the percentage of hospitals
with serious deficiencies identified by the state survey agencies where
JCAHO surveyors did not find comparable deficiencies. The analysis we
performed on the results of the validation surveys was limited to the
hospitals included in the validation survey sample and cannot be
generalized to all JCAHO-accredited hospitals.
To determine whether JCAHO's new hospital accreditation process has
potential for improving the detection of serious deficiencies, we
identified changes in the accreditation process and analyzed
significant new features. To determine whether JCAHO's new hospital
accreditation process was adequately tested, we reviewed the testing
procedures and results that JCAHO used to determine the effectiveness
of its new survey process in identifying quality and safety
deficiencies. Because the new accreditation process was implemented
recently, we did not have information to compare JCAHO survey
performance in detecting serious deficiencies with state agency survey
performance.
To determine the adequacy of CMS's oversight of JCAHO's hospital
accreditation program, we reviewed relevant statutory and regulatory
provisions regarding oversight of health care accreditation programs
and how CMS had implemented this authority in order to provide
oversight. To supplement our review, we conducted interviews with
officials from CMS, state survey agencies, and JCAHO; representatives
from other organizations active in health care accreditation and the
hospital industry; and experts in quality of care. We conducted our
work from June 2003 through July 2004 in accordance with generally
accepted government auditing standards. (For a complete description of
our scope and methodology, see app. I.)
Results in Brief:
JCAHO's pre-2004 hospital accreditation process did not identify most
of the hospitals found by state survey agencies in CMS's annual
validation survey sample to have serious deficiencies in Medicare COPs.
In a sample of 500 JCAHO-accredited hospitals, state agency validation
surveys conducted in fiscal years 2000 through 2002 identified 31
percent (157 hospitals) with serious deficiencies; of these, JCAHO did
not identify 78 percent (123 hospitals) as having serious deficiencies.
For the same validation survey sample, JCAHO also did not identify the
majority-about 69 percent-of serious deficiencies found by state
agencies. Importantly, the number of deficiencies found by validation
surveys represents 2 percent of the 11,000 Medicare COPs surveyed by
state agencies in the sample during this time period. At the same time,
a single serious deficiency can limit a hospital's capability to
provide adequate care and ensure patient safety and health.
Inadequacies in nursing practices or deficiencies in a hospital's
physical environment, which includes fire safety, are examples of
serious deficiencies that could endanger multiple patients.
The potential of JCAHO's new hospital accreditation process to improve
the detection of serious deficiencies over the pre-2004 process is
unknown because the process was just implemented in January 2004. JCAHO
plans to move from announced to unannounced surveys in 2006, which
would afford JCAHO the opportunity to observe hospitals' operations
when the hospitals have not prepared in advance to be surveyed. In
addition, the pilot test of the new accreditation process was of
limited value in predicting whether it will be an improvement over the
pre-2004 process in detecting deficiencies. Limitations in the pilot
test included that hospitals participating in the pilot were not
randomly selected and that JCAHO evaluated the results instead of an
independent entity.
CMS has limited oversight authority over JCAHO's hospital accreditation
program because the program's unique legal status effectively prevents
CMS from taking actions, such as requiring JCAHO's hospital
accreditation program to submit to a direct review process or placing
the program on probation while monitoring its performance, that it has
the authority to take with other health care accreditation programs to
ensure satisfactory performance. Furthermore, CMS's existing oversight
of JCAHO's hospital accreditation program needs improvement. Although
CMS officials said that validation surveys are conducted to assure
Congress that JCAHO's accreditation process provides a reasonable
assurance that hospitals comply with Medicare requirements, there are
limitations to the agency's validation survey program. CMS has no
formal written protocol for selecting the hospitals to include in the
state agency validation survey sample; relies on a measure--the rate of
disparity--that provides limited information and could mask problems
with an accreditation program's performance in detecting serious
deficiencies; uses statistical methods that are insufficient to
accurately portray JCAHO's performance; and has reduced the percentage
of validation surveys from 5 percent to approximately 1 percent of
JCAHO-accredited hospitals, which provides less reliable information on
the performance of JCAHO's hospital accreditation program.
We suggest that Congress consider giving CMS the same oversight
authority over JCAHO's hospital accreditation program that CMS has for
all other health care accreditation programs. To improve CMS's
assessment of JCAHO's hospital accreditation process, we recommend that
CMS modify the measure it uses to indicate how well an accreditation
program detects serious deficiencies in Medicare COPs; maximize the
extent to which validation survey findings can be generalized to all
JCAHO-accredited hospitals and include its survey protocol in its
annual reports to Congress; and annually conduct validation surveys on
a sample of JCAHO-accredited hospitals that is equal to at least 5
percent of all JCAHO-accredited hospitals.
CMS and JCAHO commented on a draft of this report. In its comments, CMS
concurred with our findings and recommendations. JCAHO stated that it
did not object to our matter for congressional consideration that CMS
be given the same oversight authority over JCAHO's hospital
accreditation program that it has over other health care accreditation
programs. JCAHO took issue with our methodology, which it said was
incomplete and did not comprehensively assess the performance of
JCAHO's hospital accreditation program. Our review was not intended to
be a comprehensive evaluation of JCAHO's hospital accreditation
program. Rather, we focused on the ability of JCAHO's hospital
accreditation program to ensure that hospitals that accept Medicare
patients comply with Medicare COPs. In the same vein, JCAHO stated that
the report does not sufficiently recognize JCAHO's identification of
deficiencies in its surveys that may be corrected before state
surveyors arrive. We added language to the report to emphasize that our
focus was on the serious deficiencies state survey agencies found that
JCAHO did not because these serious deficiencies demonstrate the
importance of CMS oversight of the hospital accreditation process.
JCAHO also stated that we misrepresented the potential of its new
accreditation process to detect deficiencies in Medicare COPs and
provided new data for the first quarter of 2004 that indicate that 2004
JCAHO surveys may have detected a greater percentage of deficiencies
related to patient care compared with the pre-2004 accreditation
process. However, we maintain that until CMS validation surveys for
2004 are completed, there is no basis on which to determine whether the
new process improves the detection of noncompliance with Medicare COPs.
CMS and JCAHO also provided technical comments on the report, which we
incorporated as appropriate.
Background:
To participate in Medicare, hospitals must maintain standards of
patient safety and health that comply with Medicare COPs. For example,
the COP related to nursing services includes such requirements for
hospitals as providing a 24-hour nursing service that is supervised or
furnished by a registered nurse. There are currently 23 Medicare
COPs.[Footnote 5] (See app. II for a description of the 23 Medicare
COPs.) CMS proposed revisions to all of the COPs in 1997, but it did
not finalize them. Since then, CMS has revised several of the COPs,
including those concerning the life safety code; quality assessment and
performance improvement; organ, tissue, and eye donations; and nurse
anesthetist supervision.
Health care accreditation programs other than JCAHO's hospital
accreditation program may generally adopt their own requirements if CMS
determines that an accreditation program's requirements are at least
equivalent to Medicare COPs.[Footnote 6] If CMS also determines, among
other things, that the accreditation program's survey process is likely
to identify any serious deficiencies in COPs, it must generally grant
"deeming authority" to the accreditation program and treat entities
accredited by these organizations as meeting Medicare COPs. CMS has the
authority to review these programs, and it can impose a probationary
period while monitoring performance and remove deeming authority if
warranted.
JCAHO:
Most hospitals demonstrate compliance with standards equivalent to
Medicare COPs through accreditation by JCAHO.[Footnote 7] In 2002,
JCAHO accredited 4,211, or 82 percent, of Medicare-participating
hospitals.[Footnote 8] Hospitals accredited by JCAHO received payments
for Medicare-covered inpatient services of approximately $98 billion,
or 90 percent, of the $109 billion that was spent on hospital care in
2002. JCAHO, as part of its accreditation-related activities, also
develops survey procedures, trains its surveyors, and formulates
performance measures. JCAHO is governed by a 29-member board of
commissioners and has a staff of over 1,000.[Footnote 9]
JCAHO's deeming authority for hospitals is established in statute and
therefore can only be changed by Congress. As a result of this unique
statutory authority, hospitals accredited by JCAHO-because they meet
JCAHO standards-are deemed to meet Medicare COPs as well.[Footnote 10]
In contrast, the American Osteopathic Association (AOA)--a private,
not-for-profit professional organization that offers accreditation
services for hospitals and other health care organizations--holds
deeming authority that is subject to CMS's direct review and
approval.[Footnote 11] While hospital accreditation is its largest
program, JCAHO also has accreditation authority under Medicare for
certain other health care providers, including clinical laboratories,
hospices, ambulatory surgical centers, and home health care agencies.
All of these other JCAHO accreditation programs are subject to CMS's
direct review and approval.
To be accredited by JCAHO, a hospital must meet eligibility
requirements, satisfactorily complete a triennial on-site survey
process, and continue to maintain JCAHO's standards between surveys.
The accreditation surveys that JCAHO conducts every 3 years are
particularly important. For most hospitals, the triennial survey is the
only time that JCAHO conducts an on-site review of the hospital's
compliance with all quality standards and issues decisions on how well
the hospital has complied with JCAHO's standards. In 2004, JCAHO
implemented a new hospital accreditation survey process, which,
according to JCAHO, is intended to reduce the cost of accreditation to
health care organizations and JCAHO, enhance public confidence that
health care organizations are in continuous compliance with standards,
increase the real and perceived value of accredited organizations, meet
the requirements of deeming authorities and purchasers, and improve
satisfaction for hospitals participating in the accreditation program.
CMS Oversight of JCAHO:
CMS exercises oversight of JCAHO's hospital accreditation program
primarily through its validation surveys and annual reports to
Congress. Under federal law, CMS must continually study the operation
and administration of Medicare, including validating the JCAHO hospital
accreditation process, and submit annual reports to Congress.[Footnote
12] CMS has agreements with state agencies to conduct validation
surveys. There are different kinds of validation surveys, including
traditional validation surveys-surveys conducted on a sample of
hospitals within 60 days of their triennial JCAHO survey. [Footnote 13]
Traditional validation surveys provide the basis for assessing the
effectiveness of JCAHO's hospital accreditation process in detecting
deficiencies in Medicare COPs, which JCAHO-accredited hospitals are
treated as meeting. Validation surveys also include 18-month surveys,
which monitor how well JCAHO-accredited hospitals are complying with
Medicare COPs midway between their 3-year JCAHO surveys, and allegation
surveys, which are triggered by complaints or other reports of
situations that pose potential threats to patient health and safety in
JCAHO-accredited hospitals. CMS has the authority to remove the deemed
status of a JCAHO-accredited hospital where a state agency's validation
survey results in a finding that the hospital is out of compliance with
one or more Medicare COP.
CMS uses a rate of disparity measure to summarize the extent to which
an accreditation program, such as JCAHO's hospital accreditation
program, has not found serious deficiencies identified by CMS through
state agency validation surveys. For a hospital accreditation program,
using the results from validation surveys, the rate of disparity for
hospitals surveyed by the state survey agencies is calculated as the
difference between the number of hospitals found with serious
deficiencies by state agencies and the number of hospitals found with
comparable deficiencies by the accreditation program, divided by the
number of hospitals sampled. CMS regulations provide that if the
validation survey results for an accreditation organization with
deeming authority indicate a rate of disparity that reaches the
threshold level of 20 percent disparity or greater, CMS will notify the
organization that its deeming authority may be in jeopardy and that the
agency is initiating a deeming authority review.[Footnote 14] With
respect to JCAHO, CMS includes the rate of disparity in its annual
reports to Congress in which it reports the results of its validation
program for JCAHO's hospital accreditation program.
JCAHO's Pre-2004 Hospital Accreditation Process Often Did Not Detect
Serious Deficiencies Found by State Survey Agencies:
JCAHO's pre-2004 hospital accreditation process often did not identify
either hospitals with serious deficiencies or the individual serious
deficiencies found by state survey agencies through CMS's validation
program. In a sample of 500 JCAHO-accredited hospitals, state agency
validation surveys conducted in fiscal years 2000 through 2002
identified 31 percent (157 hospitals) with serious deficiencies; of
these, JCAHO did not identify 78 percent (123 hospitals) as having
serious deficiencies. For the same validation survey sample, the
majority of the serious deficiencies state survey agencies identified
but JCAHO did not were in the physical environment COP category, which
covers fire safety and prevention.
JCAHO Did Not Identify Three-Quarters of the Hospitals That State
Survey Agencies Found to Have Serious Deficiencies:
From fiscal years 2000 through 2002, JCAHO did not identify 123 of the
157 hospitals (78 percent) with serious deficiencies that CMS's
validation program identified out of a sample of 500 JCAHO-accredited
hospitals. Table 1 shows the hospitals with serious deficiencies that
state survey agencies identified and JCAHO did not during fiscal years
2000 through 2002. In 343 of the 500 hospital validation surveys, state
agency surveyors did not find serious deficiencies. Both state agency
surveyors and JCAHO surveyors identified 34 hospitals as having a
serious deficiency.
Table 1: Hospitals in CMS's Validation Survey Sample with Serious
Deficiencies that State Survey Agencies Identified but JCAHO Surveyors
Did Not, Fiscal Years 2000-2002:
Fiscal year: 2000;
Number of hospitals in CMS's validation sample: 184;
Hospitals state survey agencies found to have serious deficiencies:
Number: 61;
Hospitals state survey agencies found to have serious deficiencies:
Percent: 33;
Hospitals with serious deficiencies identified by state survey agencies
but not identified by JCAHO[A]: Number: 49;
Hospitals with serious deficiencies identified by state survey agencies
but not identified by JCAHO[A]: Percent: 80.
Fiscal year: 2001;
Number of hospitals in CMS's validation sample: 204;
Hospitals state survey agencies found to have serious deficiencies:
Number: 61;
Hospitals state survey agencies found to have serious deficiencies:
Percent: 30;
Hospitals with serious deficiencies identified by state survey agencies
but not identified by JCAHO[A]: Number: 49;
Hospitals with serious deficiencies identified by state survey agencies
but not identified by JCAHO[A]: Percent: 80.
Fiscal year: 2002;
Number of hospitals in CMS's validation sample: 112;
Hospitals state survey agencies found to have serious deficiencies:
Number: 35;
Hospitals state survey agencies found to have serious deficiencies:
Percent: 31;
Hospitals with serious deficiencies identified by state survey agencies
but not identified by JCAHO[A]: Number: 25;
Hospitals with serious deficiencies identified by state survey agencies
but not identified by JCAHO[A]: Percent: 71.
Total;
Number of hospitals in CMS's validation sample: 500;
Hospitals state survey agencies found to have serious deficiencies:
Number: 157;
Hospitals state survey agencies found to have serious deficiencies:
Percent: 31;
Hospitals with serious deficiencies identified by state survey agencies
but not identified by JCAHO[A]: Number: 123;
Hospitals with serious deficiencies identified by state survey agencies
but not identified by JCAHO[A]: Percent: 78.
Source: GAO analysis of CMS data.
Note: Hospitals with serious deficiencies are defined as those not
meeting one or more of the Medicare COPs. From fiscal year 2000 through
2002, JCAHO surveyed 4,666 hospitals for accreditation.
[A] Determined by CMS through its matching of deficient COPs found by
state agency surveyors to JCAHO surveyors' findings of JCAHO standards
out of compliance.
[End of table]
According to JCAHO, disparity between state agency and JCAHO findings
in the 123 hospitals in part may be attributed to the timing of the two
surveys, JCAHO's phasing in of new requirements, different
interpretations of the COPs by state surveyors, and inherent surveyor
bias. However, in its comparison to determine disparity between the two
surveys, CMS does consider whether it is reasonable to conclude that
the deficiencies found by state survey agencies existed at the time
JCAHO surveyed the hospital.
JCAHO Did Not Detect Two-Thirds of the Serious Deficiencies Identified
by State Survey Agencies:
From fiscal year 2000 through 2002, JCAHO did not detect 167 of the 241
serious deficiencies (69 percent) identified through CMS's validation
program from a sample of 500 JCAHO-accredited hospitals. The number of
serious deficiencies found by CMS's validation program represents 2
percent of the 11,000 Medicare COPs surveyed by state agencies in the
sample and were found in 157 hospitals. However, one serious deficiency
in any one of these hospitals could limit its ability to provide
adequate care to its patients. For example, a serious deficiency in the
nursing services COP at a hospital in Texas found by a state agency but
missed by JCAHO in 2000 included such problems as failure to prepare
and administer drugs in accordance with federal and state laws,
inadequate supervision and evaluation of the clinical activities of
nonemployee nursing personnel, and nursing care and procedures provided
to patients that were not within the scope of accepted standards of
practice. Among hospitals with serious deficiencies identified by CMS's
validation program but not by JCAHO, there were on average 1.1 serious
deficiencies per hospital, with a range from 1 to 6. Table 2 shows the
percentage of serious deficiencies identified by CMS's validation
program but not by JCAHO for fiscal years 2000 through 2002.
Table 2: Percentage of Serious Deficiencies Identified by State Survey
Agencies but Not by JCAHO Surveyors in CMS's Validation Survey Sample,
Fiscal Years 2000-2002:
Fiscal year: 2000;
Number of serious deficiencies identified by state survey agencies: 82;
Number of serious deficiencies identified by JCAHO: 12;
Serious deficiencies identified by state survey agencies but not by
JCAHO[A]: Number: 70;
Serious deficiencies identified by state survey agencies but not by
JCAHO[A]: Percent: 85%.
Fiscal year: 2001;
Number of serious deficiencies identified by state survey agencies:
103;
Number of serious deficiencies identified by JCAHO: 40;
Serious deficiencies identified by state survey agencies but not by
JCAHO[A]: Number: 63;
Serious deficiencies identified by state survey agencies but not by
JCAHO[A]: Percent: 61%.
Fiscal year: 2002;
Number of serious deficiencies identified by state survey agencies: 56;
Number of serious deficiencies identified by JCAHO: 22;
Serious deficiencies identified by state survey agencies but not by
JCAHO[A]: Number: 34;
Serious deficiencies identified by state survey agencies but not by
JCAHO[A]: Percent: 61%.
Fiscal year: Total;
Number of serious deficiencies identified by state survey agencies:
241;
Number of serious deficiencies identified by JCAHO: 74;
Serious deficiencies identified by state survey agencies but not by
JCAHO[A]: Number: 167;
Serious deficiencies identified by state survey agencies but not by
JCAHO[A]: Percent: 69%.
Source: GAO analysis of CMS data.
Note: Hospitals with serious deficiencies are defined as those not
meeting one or more of the Medicare COPs.
[A] Determined by CMS through its matching of deficient COPs found by
state agency surveyors to JCAHO surveyors' findings of JCAHO standards
out of compliance.
[End of table]
Of the 167 serious deficiencies identified by CMS's validation program
from fiscal year 2000 through 2002 but not detected by JCAHO, 87 were
related to a hospital's physical environment, which includes life
safety code standards on fire prevention and safety.[Footnote 15] For
these 3 years, JCAHO did not detect 81 percent of the serious physical
environment deficiencies identified by state agency surveyors. Table 3
shows the number of serious deficiencies, by category, identified by
state survey agencies in CMS's validation program but missed by JCAHO
surveyors. The larger number of deficiencies in physical environment
may be related to the difference in how state agencies generally survey
separately a hospital's compliance with the life safety code portion of
the physical environment COP. JCAHO surveys assess compliance with the
life safety code using a combination of the hospital's self-assessment,
a hospital building tour, and observations made by all surveyors during
the survey process. Examples of deficiencies in physical environment
that JCAHO did not identify but CMS's validation program found in a
hospital in Alabama in 2000 included the following: several exterior
exits lacked emergency exit lighting; several exterior exits were
illuminated only by single light bulbs; fire alarm system and fire
extinguishers had not been inspected annually as required; and an
automatic sprinkler system had not been inspected annually and
maintained by certified personnel as required. Serious deficiencies in
the COP on physical environment compromise patient safety and health.
Table 3: Number of Serious Deficiencies, by COP, Identified by State
Survey Agencies but Not by JCAHO Surveyors in CMS's Validation Survey
Sample, Fiscal Years 2000-2002:
COP: Physical environment;
Number of serious deficiencies identified by state survey agencies:
107;
Number of serious deficiencies identified by state survey agencies but
not by JCAHO[A]: 87.
COP: Quality of care: Anesthesia services;
Number of serious deficiencies identified by state survey agencies: 3;
Number of serious deficiencies identified by state survey agencies but
not by JCAHO[A]: 2.
COP: Quality of care: Discharge planning;
Number of serious deficiencies identified by state survey agencies: 2;
Number of serious deficiencies identified by state survey agencies but
not by JCAHO[A]: 2.
COP: Quality of care: Emergency services;
Number of serious deficiencies identified by state survey agencies: 2;
Number of serious deficiencies identified by state survey agencies but
not by JCAHO[A]: 2.
COP: Quality of care: Food and dietetic services;
Number of serious deficiencies identified by state survey agencies: 5;
Number of serious deficiencies identified by state survey agencies but
not by JCAHO[A]: 4.
COP: Quality of care: Governing body;
Number of serious deficiencies identified by state survey agencies: 16;
Number of serious deficiencies identified by state survey agencies but
not by JCAHO[A]: 7.
COP: Quality of care: Infection control;
Number of serious deficiencies identified by state survey agencies: 15;
Number of serious deficiencies identified by state survey agencies but
not by JCAHO[A]: 9.
COP: Quality of care: Laboratory services;
Number of serious deficiencies identified by state survey agencies: 1;
Number of serious deficiencies identified by state survey agencies but
not by JCAHO[A]: 1.
COP: Quality of care: Medical record services;
Number of serious deficiencies identified by state survey agencies: 7;
Number of serious deficiencies identified by state survey agencies but
not by JCAHO[A]: 4.
COP: Quality of care: Medical staff;
Number of serious deficiencies identified by state survey agencies: 10;
Number of serious deficiencies identified by state survey agencies but
not by JCAHO[A]: 1.
COP: Quality of care: Nursing services;
Number of serious deficiencies identified by state survey agencies: 17;
Number of serious deficiencies identified by state survey agencies but
not by JCAHO[A]: 10.
COP: Quality of care: Organ, tissue, and eye procurement;
Number of serious deficiencies identified by state survey agencies: 5;
Number of serious deficiencies identified by state survey agencies but
not by JCAHO[A]: 5.
COP: Quality of care: Outpatient services;
Number of serious deficiencies identified by state survey agencies: 1;
Number of serious deficiencies identified by state survey agencies but
not by JCAHO[A]: 1.
COP: Quality of care: Patients' rights;
Number of serious deficiencies identified by state survey agencies: 10;
Number of serious deficiencies identified by state survey agencies but
not by JCAHO[A]: 9.
COP: Quality of care: Pharmaceutical services;
Number of serious deficiencies identified by state survey agencies: 14;
Number of serious deficiencies identified by state survey agencies but
not by JCAHO[A]: 9.
COP: Quality of care: Quality assurance;
Number of serious deficiencies identified by state survey agencies: 18;
Number of serious deficiencies identified by state survey agencies but
not by JCAHO[A]: 8.
COP: Quality of care: Radiologic services;
Number of serious deficiencies identified by state survey agencies: 1;
Number of serious deficiencies identified by state survey agencies but
not by JCAHO[A]: 0.
COP: Quality of care: Rehabilitation services;
Number of serious deficiencies identified by state survey agencies: 1;
Number of serious deficiencies identified by state survey agencies but
not by JCAHO[A]: 1.
COP: Quality of care: Respiratory care services;
Number of serious deficiencies identified by state survey agencies: 1;
Number of serious deficiencies identified by state survey agencies but
not by JCAHO[A]: 1.
COP: Quality of care: Surgical services;
Number of serious deficiencies identified by state survey agencies: 5;
Number of serious deficiencies identified by state survey agencies but
not by JCAHO[A]: 4.
COP: Quality of care: Total quality-of-care COPs;
Number of serious deficiencies identified by state survey agencies: 134;
Number of serious deficiencies identified by state survey agencies but
not by JCAHO[A]: 80.
Source: GAO analysis of CMS data.
Note: Neither state survey agencies nor JCAHO identified serious
deficiencies in two of the categories-compliance with laws and nuclear
medicine services-which are not included in this table.
[A] Determined by CMS through its matching of deficient COPs found by
state agency surveyors to JCAHO surveyors' findings of JCAHO standards
out of compliance.
[End of table]
The total number of deficiencies not identified by JCAHO in the
quality-of-care COP categories--those COPs that involve the oversight
and delivery of patient care--is similar to the number not identified
by JCAHO in the physical environment COP. While the number of serious
deficiencies not found by JCAHO in individual quality-of-care COP
categories is smaller than the number not found in physical
environment, when these quality-of-care COPs are combined, the
proportion of serious deficiencies JCAHO missed is almost 60 percent of
the total number of serious deficiencies identified by state survey
agencies. The following are examples of hospitals found to be out of
compliance with multiple quality-of-care COPs:
* In 2000, CMS removed the deemed status as a Medicare provider of a
JCAHO-accredited hospital in California for failure to comply with two
COPs, one of which was infection control. The hospital failed to
provide a sanitary environment to avoid sources and transmission of
infections and communicable diseases and failed to develop a system for
ensuring the sterilization of medical instruments.
* Also in 2000, CMS notified a hospital in Texas that if it did not
implement a plan of correction the hospital's participation in the
Medicare program would be terminated. Serious deficiencies at this
hospital included lack of compliance with the pharmaceutical services
and nursing services COPs because medications were administered without
physician orders and a double dose of narcotics was given in the
emergency room, with no explanation for the excessive dosage, to a
patient who later died.
State surveyors in CMS's validation program also may miss serious
deficiencies. In related work on skilled nursing facilities and home
health agencies, we found that the number of serious deficiencies found
by state agencies was highly variable among states and may be
understated.[Footnote 16] State agencies' detection of serious
deficiencies in hospitals also varied widely among states for the 3
years we reviewed. For example, state survey agencies in California,
Illinois, and Ohio found serious deficiencies in over 45 percent of the
surveys they conducted between fiscal years 2000 through 2002. In
contrast, Florida and New York found serious deficiencies in less than
10 percent of the surveys they conducted, and Louisiana did not find
serious deficiencies in any of the surveys it conducted.[Footnote 17]
Potential of JCAHO's New Hospital Accreditation Process Is Unknown, and
Testing Was Limited:
The potential of JCAHO's new hospital accreditation process to improve
the identification of serious deficiencies is unknown because it is too
soon after its January 2004 implementation for a meaningful evaluation;
in addition, JCAHO's testing of the new process was limited. CMS has
not had the opportunity to complete its validation program for 2004 to
determine whether JCAHO surveyors using the new process are missing
serious deficiencies later identified by state agency validation
surveys. While unannounced surveys, which are planned for
implementation in 2006, have the potential to improve the detection of
serious deficiencies, other features of the new process that JCAHO did
not test before implementation may have limitations that could affect
the potential of the new process to identify problems with patient
care. JCAHO's pilot test of the new process had limitations, including
using a sample of hospitals that volunteered for the pilot instead of
using a random sample and self-evaluating the results instead of using
an independent entity.
Potential of New Process Is Unknown:
Because JCAHO's new accreditation process was implemented in January
2004, it is too soon to know whether the new process is better at
detecting serious deficiencies in Medicare COPs than the pre-2004
accreditation process. A JCAHO official told us the new process will
aid in the detection of deficiencies, but we found that some of the
features may have shortcomings that could limit their effectiveness.
New features of the accreditation process include the hospital's self-
assessment of compliance with accreditation standards midway through
the accreditation cycle, surveyor review of the care provided to
specific patients to determine the adequacy of the hospital's health
care delivery system, and performance of all accreditation surveys on
an unannounced basis beginning in 2006. (See app. III for a description
of selected new features of JCAHO's new hospital accreditation
process.)
Periodic Performance Review:
Periodic performance reviews assess hospital compliance with applicable
standards and are performed at the 18-month midpoint between 3-year on-
site accreditation surveys. According to JCAHO, the periodic
performance review will have several benefits. These include providing
hospitals with a process to assess their ongoing compliance and
requiring them to correct or plan to correct all deficiencies
identified. Periodic performance reviews must be conducted either by
the hospital as a self-assessment or, if the hospital chooses, by JCAHO
through an on-site review.
However, periodic performance reviews may not necessarily improve the
detection of deficiencies. JCAHO did not pilot test these reviews for
the potential to detect deficiencies and did not test whether hospitals
that conducted reviews do a better job of continuing to comply with
standards. In addition, for hospitals performing self-assessments,
JCAHO will not check these self-assessments to determine whether
hospitals fully and accurately identified quality problems and
developed adequate corrective action plans to address the problems
identified.
Priority Focus Process and Patient Tracer Methodology:
According to JCAHO, the priority focus process and patient tracer
methodology together have the potential to enhance the ability of
surveys to detect deficiencies by directing the attention of surveyors
to key patient care areas. The priority focus process uses a data-based
formula to identify a limited number of areas in each hospital that are
particularly important to patient health and safety. Priority focus
areas might include infection control, medication management, or
patient safety. Surveyors use the priority focus process combined with
the patient tracer methodology to focus their surveys to specific areas
for review. The patient tracer methodology guides their choice of
current patients to "trace" through the experience of care within an
organization. For example, if the hospital's priority focus process
data suggest that a patient with an orthopedic-related diagnosis such
as a hip fracture should be traced, the JCAHO surveyor would review the
patient's medical record, noting where the patient had entered into the
hospital and any services and transfers that occurred. Then the
surveyor would retrace the steps in the patient's care process by
observing and talking to staff in some of the areas in which the
patient received care. If the patient entered through the emergency
department, was transferred to a medical/surgical unit, and then went
to the operating room, the surveyor would go to these areas to
interview staff about the care given to this specific patient. With
information from patient tracers, the surveyor will assess whether any
compliance issues exist with JCAHO standards. If the surveyor
identifies a compliance issue while tracing one patient, the surveyor
may review the records of similar patients to determine whether the
problem is isolated or represents a pattern of care.
However, JCAHO did not test the extent to which the priority focus
process and the patient tracer methodology could help surveyors detect
deficiencies. A JCAHO official told us these new features of the
accreditation process were intended to help surveyors trace patients in
a consistent way and not necessarily to improve the detection of
deficiencies.
Unannounced Surveys:
JCAHO plans to conduct all hospital accreditation surveys on an
unannounced basis beginning in 2006.[Footnote 18] JCAHO stated that
unannounced surveys will ensure that hospital performance is based on
the observation of hospitals' routine operations rather than on how
they operate after they have the opportunity to prepare to be surveyed.
A JCAHO official also indicated that unannounced surveys will be more
likely to detect deficiencies. The OIG and other organizations share
JCAHO's position on the value of unannounced surveys of hospitals and
other health care organizations. The value of unannounced surveys also
has been recognized for nursing homes, which state agencies survey on
an unannounced basis.
JCAHO's Pilot Test of New Process Was Limited:
JCAHO's pilot test of its new hospital accreditation process was
limited and therefore unable to help determine the potential of the new
process to detect deficiencies in Medicare COPs. According to JCAHO,
the pilot test suggests that the new process was more likely than the
former process to find quality problems. However, the pilot test sample
included hospitals that volunteered or were selected by JCAHO and were
not randomly selected, pilot test surveyors were accompanied by
observers from JCAHO's central office, and pilot test results were not
independently evaluated. In addition, CMS has not completed its fiscal
year 2004 validation program, which will include hospitals surveyed by
JCAHO using the new process and thus does not yet have sufficient data
on which to base a meaningful evaluation.
According to JCAHO's analysis of the pilot test, the new hospital
accreditation process is more likely to identify quality problems since
proportionately more hospitals under the new process received
unfavorable accreditation decisions. JCAHO based its conclusion on a
comparison of survey outcomes, called accreditation decisions, between
18 hospitals in the pilot test conducted in 2002 and 2003 and the 1,524
hospitals that had been surveyed under the pre-2004 accreditation
process during 2003. Table 4 presents the data JCAHO used to make the
comparison. As shown, proportionately fewer hospitals under the new
process were accredited without having to make corrections. Although
JCAHO provided the accreditation decision outcomes for these 18 pilot
tests, it stated it preferred to use the number of "requirements for
improvement" as the basis for analysis.
Table 4: Accreditation Decisions for Hospitals Surveyed Under JCAHO's
New Survey Process Pilot Test as Compared to Results from JCAHO's Pre-
2004 Survey Process:
Accreditation decision: Accreditation;
Pilot test of new survey process: Number of hospitals surveyed: 0;
Pilot test of new survey process: Percentage of hospitals surveyed: 0%;
Pre-2004 survey process: Number of hospitals surveyed: 320;
Pre-2004 survey process: Percentage of hospitals surveyed: 21%.
Accreditation decision: Survey findings with requirements for
improvement[A];
Pilot test of new survey process: Number of hospitals surveyed: 13;
Pilot test of new survey process: Percentage of hospitals surveyed:
72%;
Pre-2004 survey process: Number of hospitals surveyed: 1,191;
Pre-2004 survey process: Percentage of hospitals surveyed: 78%.
Accreditation decision: Conditional accreditation;
Pilot test of new survey process: Number of hospitals surveyed: 3;
Pilot test of new survey process: Percentage of hospitals surveyed:
17%;
Pre-2004 survey process: Number of hospitals surveyed: 13;
Pre-2004 survey process: Percentage of hospitals surveyed: 1%.
Accreditation decision: Preliminary denial of accreditation;
Pilot test of new survey process: Number of hospitals surveyed: 2;
Pilot test of new survey process: Percentage of hospitals surveyed: 11;
Pre- 2004 survey process: Number of hospitals surveyed: 0;
Pre-2004 survey process: Percentage of hospitals surveyed: 0%.
Accreditation decision: Total;
Pilot test of new survey process: Number of hospitals surveyed: 18;
Pilot test of new survey process: Percentage of hospitals surveyed:
100%;
Pre-2004 survey process: Number of hospitals surveyed: 1,524[B];
Pre-2004 survey process: Percentage of hospitals surveyed: 100%.
Source: JCAHO.
Note: JCAHO reported that it conducted pilot tests of the new
accreditation process in an additional 12 hospitals in 2001. However,
JCAHO was unable to provide the accreditation decisions for these 12
pilot site hospitals.
[A] Hospitals in the pilot test with deficiencies were accredited
contingent upon evidence of correcting deficiencies. The hospitals in
the comparison group with deficiencies received accreditation with
requirements for improvement.
[B] These 1,524 hospitals represent all those surveyed for
accreditation by JCAHO during 2003.
[End of table]
However, JCAHO's pilot test analysis was limited in three respects,
which may have accounted for the smaller number of favorable
accreditation decisions hospitals received under the new process.
* The hospitals participating in the pilot test were not randomly
selected by JCAHO. As a result, these hospitals may not be
representative of all JCAHO-accredited hospitals and therefore results
cannot be generalized.
* During the pilot test, an observer from JCAHO's central office
accompanied each surveyor, and the knowledge that they were being
observed may have influenced the surveyors' actions.[Footnote 19] Under
the pre-2004 process, observers only rarely accompanied JCAHO
surveyors.
* JCAHO conducted its own evaluation of pilot test results. Evaluation
of the pilot test by an entity independent of either JCAHO or the
hospitals tested could help to ensure that survey outcomes were
impartially interpreted. For example, CMS used an independent group to
evaluate its redesign of the nursing home survey process.
CMS Oversight Authority of JCAHO's Hospital Accreditation Program Is
Limited and Needs Improvement:
CMS has limited oversight authority over JCAHO's hospital accreditation
program, and its existing oversight activities need improvement. The
unique status of JCAHO's hospital accreditation program, which is
specified in statute, does not permit CMS to take corrective action,
such as restricting or removing its deeming authority. Additionally,
CMS uses a measure that provides limited information to evaluate the
performance of JCAHO's hospital accreditation program, has
significantly reduced the number of surveys conducted as part of CMS's
validation program, and does not use measures that are based on sound
statistical methods to assess the performance of JCAHO's hospital
accreditation program.
CMS Oversight Authority of JCAHO Is Limited:
Because of JCAHO's unique legal status, CMS's oversight of JCAHO's
hospital accreditation program is limited in two major ways: Unlike
other accreditation programs with deeming authority, JCAHO does not
have to reapply to CMS to reauthorize its deeming authority, and CMS
cannot take action to address performance problems with JCAHO's
hospital accreditation program.
JCAHO's hospital accreditation program is the only Medicare
accreditation program for which CMS does not have to conduct an
evaluation of the accreditation standards and the processes used to
conduct surveys. Without this evaluation, CMS is deprived of key
oversight tools it is authorized to use with other accreditation
programs: detailed information about any proposed changes to the
accreditation process and public input. CMS cannot require JCAHO to
provide information about proposed changes to its accreditation
requirements and hospital survey processes. Also, because it is not
required to reapply to CMS for deeming authority, JCAHO does not have
to provide CMS information that other accreditation programs must
provide, such as a detailed description of its survey processes, a
comparison of its standards to Medicare requirements, and the
qualifications of its surveyors, which CMS reviews to ensure that the
programs comply with Medicare requirements. For example, when JCAHO's
hospice accreditation program applied for deeming status in 1999, CMS
required changes to JCAHO's hospice accreditation process, including
requiring JCAHO to make unannounced surveys of Medicare-certified
hospices. According to a CMS official, JCAHO's hospital accreditation
program has provided much of the information required of other
accreditation organizations; however, CMS has no authority to require
JCAHO to make changes to the hospital accreditation program as it does
with other health care accreditation programs. Statutory provisions
regarding public notice and comment do not apply to JCAHO's hospital
accreditation program as they do to other accreditation programs. The
reapplication process for other accreditation programs requires
affording the public an opportunity to provide input to CMS on an
accreditation program's request for deeming authority. Because JCAHO
does not have to reapply for deeming authority, the public does not
have the opportunity to review and comment on JCAHO's hospital
accreditation program.[Footnote 20]
A second limitation is CMS's inability to address any performance
issues with JCAHO's hospital accreditation program. Although the rate
of disparity for JCAHO's hospital accreditation program exceeded 20
percent in fiscal years 2000, 2001, and 2002 -a rate that would have
triggered a deeming authority review for any other Medicare
accreditation program-CMS was unable to take enforcement action to
address JCAHO's performance. When other Medicare accreditation programs
have a rate of disparity of 20 percent or more, CMS can take steps such
as imposing a year-long probationary period and removing deeming
authority at the end of the probationary period if the rate of
disparity remains at 20 percent or more. For JCAHO, however, CMS's
actions toward correcting the program's deficiencies are limited to
including recommendations for improvement in its annual reports to
Congress and negotiating with JCAHO to voluntarily adopt CMS's
recommendations.
In its annual report to Congress, CMS made recommendations in fiscal
year 2002 aimed at improving JCAHO's ability to detect serious
deficiencies in the life safety code, part of the COP on physical
environment. CMS noted that JCAHO permits hospitals to self-assess
compliance with life safety code requirements.[Footnote 21] While CMS
stated that it did not object to the concept of hospital self-
assessment of life safety code requirements, it made five
recommendations to JCAHO for improving implementation:
1. Require hospitals to use qualified personnel, such as fire marshals
and architects, to conduct self-assessments of compliance with the life
safety code requirements.
2. Set minimum standards for identifying and improving life safety code
deficiencies identified by hospital self-assessments.
3. Require hospitals to submit their self-assessments on life safety
code issues prior to JCAHO conducting accreditation surveys to provide
surveyors and personnel in JCAHO's central office time to review the
material prior to the accreditation surveys.
4. Increase the use of JCAHO experts in the life safety code
requirements in its central office.
5. Address the issue of hospitals that do not make improvement within
self-determined time frames.
JCAHO did not adopt all of these recommendations. It disagreed with the
first recommendation. Its response indicated that its requirement to
use qualified personnel to complete the self-assessment, while more
general, was sufficient. It further indicated that policies were in
place for CMS's second and fifth recommendations. CMS later agreed that
JCAHO's policies do satisfactorily address the fifth recommendation.
JCAHO planned to examine ways to adopt CMS's third and fourth
recommendations. CMS however, had no authority to compel JCAHO to
comply with the remaining recommendations. According to CMS, it
continues to discuss implementation of its recommendations with JCAHO.
JCAHO stated that while its initial response to CMS's recommendations
in 2003 reflected then current JCAHO policies, subsequent policy
evolutions are addressing CMS's recommendations. Specifically, JCAHO is
working with the American Society of Hospital Engineers to develop a
process for review by experts of hospital self-assessments on life
safety code issues prior to JCAHO's conducting on-site accreditation
surveys and to identify those hospitals for which engineering expertise
should be added to on-site surveys.
CMS's Validation Program Needs Improvement:
CMS states that the goal of its validation program is to provide
reasonable assurance to Congress that the JCAHO accreditation process
ensures hospital compliance with Medicare COPs. However, the measure
CMS uses to evaluate the performance of JCAHO's hospital accreditation
program provides limited information and could mask problems with an
accreditation program's performance in detecting serious deficiencies,
and it is based on a target sample size of 1 percent of JCAHO-
accredited hospitals. In addition, CMS does not report the extent to
which its sample reflects the performance of the larger population of
JCAHO-accredited hospitals.
Rate of Disparity:
The rate of disparity between JCAHO's hospital accreditation survey
findings and state survey agency findings, as currently calculated by
CMS, does not fully explain the performance of JCAHO's hospital
accreditation program in detecting serious deficiencies. CMS uses this
measure in its reports to Congress to assess JCAHO's hospital
accreditation program and as the basis for making recommendations for
improvement. CMS calculates the rate of disparity as the difference
between the number of hospitals found with serious deficiencies by
state survey agencies and the number of hospitals found with serious
deficiencies by the accreditation survey, divided by the number of
hospitals in the sample. For example, if state survey agencies
conducted 200 surveys as part of CMS's validation program and found 60
hospitals out of compliance with at least one COP, but JCAHO's survey
found that only 22 of the hospitals were out of compliance, the rate of
disparity would be 19 percent ((60 - 22)/200).
CMS has established in regulation a rate of disparity of 20 percent or
greater as the threshold for taking action against an accreditation
program. According to a CMS official, the use of 20 percent as the
threshold is not based on empirical evidence but rather on what CMS
believed Congress would find acceptable. Consequently, the threshold
may not be appropriately placed to indicate unacceptable performance by
a hospital accreditation program. For example, if JCAHO failed to
identify serious deficiencies in all 14 hospitals that the state
agencies identified with serious deficiencies from a sample of 79
hospitals, the rate of disparity would be a satisfactory 18 percent
((14-0)/79).[Footnote 22]
CMS's rate of disparity measure used in isolation does not consistently
reflect an accreditation program's ability to detect serious
deficiencies. As the number of hospitals with serious deficiencies
detected by state survey agencies decreases, regardless of JCAHO's
performance in detecting them, it is more likely that the rate of
disparity will be less than CMS's 20 percent threshold. As a result,
the performance of JCAHO's hospital accreditation program is difficult
to judge based on this measure alone. For example, if state survey
agencies performed 200 validation surveys and found 100 hospitals or 50
percent with serious deficiencies and JCAHO found 30 hospitals or 30
percent of the hospitals found by state agencies, the rate of disparity
would be 35 percent ((100-30)/200). However, if the state agencies
found 50 hospitals, or 25 percent, of the 200 hospitals with serious
deficiencies and JCAHO found 15 hospitals, or 30 percent of the
hospitals that the state agencies identified, the rate of disparity
would be almost 18 percent ((50-15)/200). The percentage of serious
deficiencies found by state survey agencies and also by JCAHO remained
the same in both examples, but the rate of disparity was improved
significantly by the larger number of hospitals without serious
deficiencies in the second example. This indicates that the rate of
disparity does not consistently measure the accreditation program's
ability to detect serious deficiencies found by state survey agencies.
(See table 5.) In addition to the rate of disparity, other components,
such as the proportion of hospitals with serious deficiencies and the
total number of serious deficiencies found by state agencies but missed
by the accreditation program, are important indicators of an
accreditation program's overall performance.
Table 5: Hypothetical Examples of the Effect on the Rate of Disparity
of a Decrease in the Number of Hospitals with Serious Deficiencies in a
Sample of 200 Hospitals:
Number of hospitals with serious deficiencies;
Example 1: State agencies: 100;
Example 1: JCAHO: 30;
Example 2: State agencies: 50;
Example 2: JCAHO: 15.
Percentage of hospitals state agencies found with serious deficiencies
that were also found by JCAHO;
Example 1: JCAHO: 30%;
Example 2: JCAHO: 30%.
Percentage of hospitals without serious deficiencies identified by
state agencies;
Example 1: JCAHO: 50%;
Example 2: JCAHO: 75%.
Rate of disparity;
Example 1: State agencies: 35% ((100-30)/200);
Example 2: State agencies: 18% ((50-15)/200).
Performance level;
Example 1: State agencies: Above threshold;
Example 2: State agencies: Below threshold.
Source: GAO.
Note: CMS's rate of disparity threshold is 20 percent.
[End of table]
Statistical Analysis of Validation Survey Sample:
CMS does not analyze the statistical results of its validation survey
samples in ways that would allow it to better assess JCAHO's ability to
detect serious deficiencies. CMS has not documented the methods it uses
to select hospitals for validation surveys and did not supply us with
clear technical justification for the methods used. Further, CMS's
validation sample includes hospitals that, because of its sampling
method, have varying chances of selection, but it does not take this
into account when calculating statistics based on the sample. According
to CMS's sampling method, the selection of hospitals is influenced by
factors such as the month in the fiscal year that JCAHO performed the
accreditation survey and how many hospitals were targeted for
completion that year in the state in which the hospital was located.
Thus, some hospitals have a greater chance of selection than others.
CMS also does not take these different chances of selection into
account when calculating statistics for its annual reports to Congress,
which prevents CMS from accurately assessing JCAHO's performance.
Moreover, CMS does not measure and report in its annual reports the
extent to which its estimates based on the validation survey sample are
likely to reflect how well JCAHO detects deficiencies in the larger
population of hospitals it accredits.[Footnote 23]
In addition, the number of usable traditional validation surveys
completed is smaller than the number of hospitals CMS samples for
validation surveys. This difference may affect the accuracy of the data
that CMS presents to Congress if the hospitals where the traditional
surveys were completed produce different results than those where
surveys are not completed or are not usable. During its sampling
process, CMS selects a sample size close to the targeted number of
hospitals each year. Some hospitals from this sample may be excluded
because CMS chose to perform another type of survey for them that
cannot be used to validate a JCAHO accreditation survey. In addition,
state agencies are not always able to complete the requested
traditional validation surveys within 60 days from the JCAHO
accreditation survey, as required, or a hospital may be excluded
because it lost its deemed status or closed. The size of the difference
between the number of hospitals sampled and the number of usable
traditional validation surveys completed therefore varies, as it did
during the 3-year review period (see table 6).
Table 6: Number of Hospitals Targeted for Validation Surveys Compared
with Usable Traditional Validation Surveys Completed:
Fiscal Year: 2000;
Hospitals targeted for validation surveys[A]: 236;
Hospitals sampled for validation surveys[B]: 236;
Usable traditional validation surveys completed[C]: 184.
Fiscal Year: 2001;
Hospitals targeted for validation surveys[A]: 227;
Hospitals sampled for validation surveys[B]: 217;
Usable traditional validation surveys completed[C]: 204.
Fiscal Year: 2002;
Hospitals targeted for validation surveys[A]: 227;
Hospitals sampled for validation surveys[B]: 235;
Usable traditional validation surveys completed[C]: 112.
Source: CMS.
[A] The targeted number is set at the beginning of the fiscal year and
is used for planning and resource allocation by CMS and the state
survey agencies.
[B] The sampled hospitals are the hospitals selected for validation
surveys during the year.
[C] Usable surveys exclude those not completed, those completed after
the required 60-day time frame, and other types of surveys that can not
be used to validate a JCAHO accreditation survey.
[End of table]
Annual Number of Validation Surveys:
CMS reduced the number of validation surveys conducted by state
agencies from a target of approximately 5 percent of the total number
of hospitals that JCAHO accredits to a target of approximately 1
percent, with at least one survey in each state. Reducing the target of
validation surveys from 5 percent to 1 percent results in the number of
validation surveys being reduced from 227 in fiscal year 2002 to a
target of 75 validation surveys in fiscal year 2003 and 72 in fiscal
year 2004.
Reducing the targeted number of validation surveys to 1 percent
provides less reliable information on how well JCAHO's hospital
accreditation program ensures compliance with Medicare COPs. For
example, for a 5-percent target, the estimate of the proportion of
JCAHO-accredited hospitals with a particular deficiency that is derived
from the validation survey could be as much as 6.0 percentage points
higher or lower, for a range of 12.0 percentage points. If the 5-
percent target produced an estimate that 50 percent of JCAHO-accredited
hospitals had a particular deficiency, the percentage of JCAHO-
accredited hospitals not complying could range from 44.0 to 56.0
percent. However, for a 1-percent target the estimate could be 11.4
percentage points higher or lower, for a range of about 22.8 percentage
points. For example, if the 1-percent target produced an estimate that
50 percent of JCAHO-accredited hospitals had a particular deficiency,
the percentage of JCAHO-accredited hospitals not complying with a
Medicare COP could range from 38.6 to 61.4 percent.[Footnote 24]
This reduction in the number of validation surveys is of additional
concern because it coincides with the implementation of JCAHO's new
accreditation process, which has an unproven capacity to detect
deficiencies. CMS's target sample size for traditional validation
surveys for fiscal year 2004 will be further reduced because the sample
also includes 18-month validation surveys. In 2004, CMS is planning to
conduct 17 of these 18-month surveys as part of its overall validation
survey target of 72. Thus, CMS could be using as few as 55 validation
surveys to determine JCAHO's performance.
Conclusions:
For 3 consecutive years, JCAHO's hospital accreditation program, which
accredits most of the hospitals participating in Medicare, exceeded
CMS's threshold for unacceptable performance. CMS validation surveys
during that time period confirmed that JCAHO missed the majority of
serious deficiencies found by state survey agencies. Yet, CMS was
unable to take action against JCAHO's hospital accreditation program as
it can with other accreditation programs because it lacked the
authority to do so. Although CMS has recommended in its annual reports
to Congress that JCAHO make changes in its hospital accreditation
program to improve its ability to detect serious deficiencies, some of
these recommendations have not been implemented. Thus, it is vital for
patient safety that JCAHO hospital accreditation surveys detect
existing serious deficiencies and deny accreditation to hospitals that
do not comply with Medicare COPs.
CMS is unable to present to Congress an adequate assessment of JCAHO's
performance because of limitations in its process for selecting
hospitals for validation surveys and analysis of the survey results.
CMS does not consistently portray the extent to which serious
deficiencies are missed and does not identify the limitations in
reporting the estimates it makes from its survey sample. CMS cannot
assure Congress that JCAHO-accredited hospitals meet Medicare COPs
because the measure for the rate of disparity, which determines poor
performance, allows JCAHO to miss the majority of serious deficiencies
and still be in an acceptable range of performance. Further, CMS's
reduction in the number of validation surveys it uses to determine the
performance of JCAHO's hospital accreditation program will provide less
reliable information at a time when JCAHO is implementing a new
hospital accreditation process that is unproven in its ability to
detect serious deficiencies. In light of these limitations in CMS's
validation of JCAHO's hospital accreditation program, we believe that
CMS must improve its oversight so it can provide Congress with more
accurate information regarding JCAHO's performance.
Matter for Congressional Consideration:
Given the serious limitations in JCAHO's hospital accreditation program
and that efforts to improve this program through informal action by CMS
have not led to necessary improvements, Congress should consider giving
CMS the same kind of authority over JCAHO's hospital accreditation
program that it has over all other Medicare accreditation programs.
Recommendations for Executive Action:
To strengthen the ability of CMS to identify and report to Congress on
JCAHO's ability to ensure that the hospitals it accredits protect the
safety and health of patients through compliance with the Medicare
COPs, we recommend that the Administrator of CMS take the following
three actions:
* modify the method used to measure the rate of disparity between
validation survey findings and accreditation program findings to
provide a reasonable assurance that Medicare COPs are being met and
consider whether additional measures are needed to accurately reflect
an accreditation program's ability to detect deficiencies in Medicare
COPs;
* provide in the annual report to Congress an estimate, based on the
validation survey sample, of the performance of all JCAHO-accredited
hospitals, including the limitations and protocols for these estimates
based on generally accepted sampling and statistical methodologies; and
develop a written protocol for these calculations; and:
* annually conduct traditional validation surveys on a sample of JCAHO-
accredited hospitals that is equal to at least 5 percent of all JCAHO-
accredited hospitals.
Agency and Other External Comments and Our Evaluation:
CMS and JCAHO commented on a draft of this report. In its comments, CMS
concurred with our recommendations. JCAHO stated it had no objection to
our suggestion that Congress give CMS the same authority over its
hospital accreditation program as it does over other Medicare
accreditation programs. However, JCAHO took issue with the methodology
we used for evaluating the performance of its hospital accreditation
program. CMS's and JCAHO's specific comments and our response follow.
CMS's comments are reprinted in appendix IV and JCAHO's comments are
reprinted in appendix V. CMS and JCAHO also provided technical
comments, which we incorporated as appropriate.
CMS stated that it has begun to examine the need for additional or
alternative measures for the rate of disparity calculation. CMS also
stated it will seek additional resources to further develop and
implement new sampling and statistical methodologies that may allow
results to be projected to all JCAHO-accredited hospitals, and to
increase the validation sample size. CMS specifically noted that it
considers life-safety code compliance, on the part of all provider
types, to be critically important. In the past 8 years, in its annual
reports to Congress and its dialogues with JCAHO regarding its hospital
accreditation program, it has identified physical environment as an
important area where JCAHO needs to focus attention, and CMS noted that
68 percent of facilities that had a deficiency finding not identified
by JCAHO had them in the physical environment area.
JCAHO stated that our methodology for evaluating the performance of its
hospital accreditation program was incomplete and did not provide a
comprehensive assessment of its program's performance. We did not
intend to do a comprehensive evaluation of JCAHO's overall hospital
accreditation program. Rather, we focused our evaluation on how well
JCAHO's hospital accreditation program ensures hospitals' compliance
with Medicare participation requirements. There are four possible
outcomes to a comparison between JCAHO's accreditation survey and a
state validation survey: (1) both JCAHO and state agencies identify no
deficiencies, (2) JCAHO identifies deficiencies not found by state
agencies, (3) both JCAHO and state agencies identify the same
deficiencies, and (4) state agencies identify deficiencies that JCAHO
does not. We limited our evaluation to the fourth outcome because it
illustrates the need for CMS oversight of the hospital accreditation
process. We have clarified the scope of our evaluation to emphasize our
focus on this outcome.
JCAHO raised a concern that our characterization of JCAHO's missed
deficiencies that state survey agencies found misleads readers to
believe that JCAHO misses hospitals with deficiencies 78 percent of the
time. We have revised language in the report to further emphasize that
the missed deficiency rate applies to hospitals in the validation
survey sample in which the state survey agencies found deficiencies and
cannot be generalized to all JCAHO-accredited hospitals. JCAHO further
stated that our report does not take into account that JCAHO's hospital
accreditation program detects deficiencies in hospitals that CMS does
not find. However, it is to be expected that state survey agencies will
not find all deficiencies found by JCAHO because hospitals may have
corrected the deficiencies prior to the state agency surveys.
JCAHO stated that we misrepresented the potential of the new
accreditation process in detecting deficiencies in Medicare COPs and
provided new data regarding its first quarter 2004 performance that
indicate that JCAHO surveys may have detected a greater percentage of
deficiencies related to patient care compared with the pre-2004
accreditation process. However, we maintain that until CMS validation
surveys for 2004 are completed, there is no basis on which to determine
whether the new process improves the detection of deficiencies in
Medicare COPs. In addition, JCAHO stated and we agree that evaluating
and improving the quality of care in hospitals is not about counting
deficiencies, it is about finding those deficiencies that, if not
fixed, will generate poor results for patients and making sure that
these deficiencies are remedied in a timely fashion.
JCAHO stated that we mischaracterized its response to the five
recommendations that CMS made in 2002 to improve JCAHO's ability to
detect deficiencies in the life safety code and that it is involved in
frequent and ongoing dialogue with CMS regarding the recommendations
and other life safety code issues. We have clarified language in the
report regarding JCAHO's response to CMS's recommendations.
As agreed with your offices, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days
after its date. We will then send copies of this report to the
Secretary of Health and Human Services and other interested parties. We
will also make copies available to others upon request. In addition,
the report will be available at no charge at the GAO Web site at http:/
/www.gao.gov.
If you or your staffs have any questions about this report, please call
me at (202) 512-7119. Another contact and key contributors are listed
in appendix VI.
Signed by:
Janet Heinrich:
Director, Health Care-Public Health Issues:
[End of section]
Appendix I: Scope and Methodology:
We examined the extent to which JCAHO's pre-2004 survey process
identified hospitals with deficiencies and individual deficiencies in
Medicare COPs that were identified by state survey agencies. We chose
these measures because they reflect performance in detecting and
correcting serious deficiencies, which according to CMS, substantially
limit a hospital's capability to render adequate care and adversely
affect the health and safety of patients. We reviewed data, provided by
CMS, on 500 traditional validation surveys conducted by state survey
agencies during fiscal years 2000 through 2002. In these validation
surveys, state survey agencies documented whether they found serious
deficiencies in Medicare COPs. CMS compared state survey agency
findings with JCAHO's accreditation surveys that identified
deficiencies in JCAHO's standards. CMS then determined whether the
state survey agencies' findings on serious deficiencies in the 22
Medicare COPs that can be deemed were comparable to JCAHO's findings on
deficiencies in JCAHO's standards in the following way. Two CMS experts
such as nurses reviewed the comparability of serious deficiencies in
the quality-of-care conditions identified in validation surveys to
deficiencies in JCAHO's accreditation standards identified in JCAHO's
hospital accreditation surveys. Two experts, such as building
engineers, reviewed the comparability of serious deficiencies
identified in the validation surveys on the condition on physical
environment. Where there was disagreement, the two experts met to
resolve their differences. CMS does not have written protocols for
determining comparability. Experts are expected to use their best
professional judgment. CMS experts also had to consider whether it is
reasonable to conclude that the deficiencies existed at the time that
JCAHO surveyed the hospital. For those deficiencies that CMS determines
that JCAHO has failed to identify, it met with JCAHO to address
disputed findings and to consider additional evidence on comparability
offered by JCAHO. There are four possible outcomes to this comparison
of survey findings--(1) JCAHO and state agencies both identify no
deficiencies, (2) JCAHO identifies deficiencies not found by state
agencies, (3) JCAHO and state agencies both identify the same
deficiencies, and (4) state agencies identify deficiencies that JCAHO
does not--we focused on the fourth because it highlights the need for
CMS oversight of the hospital accreditation program. For the second
outcome, there could be two reasons for the disparity between JCAHO's
and state survey agencies' findings: hospitals corrected deficiencies
identified by JCAHO prior to the state agency survey or the state
survey agency did not identify a deficiency that existed. In addition,
not all JCAHO findings are equivalent to noncompliance with a Medicare
COP.
From these 500 surveys, we determined the number of hospitals with
serious deficiencies and the total number of serious deficiencies
identified by state agencies but that CMS determined were not
identified by JCAHO. These data include 123 hospitals in which state
survey agencies identified one or more serious deficiencies and JCAHO
did not make comparable findings according to CMS. These data also
include 167 serious deficiencies identified by state agencies but that
CMS determined comparable findings were not identified by JCAHO.
For fiscal years 2001 and 2002, we obtained from CMS a comparison
between the validation surveys conducted by the state survey agencies
and the accreditation surveys conducted by JCAHO, which identified
serious deficiencies identified by the state agencies but not by JCAHO
as determined by CMS. For fiscal year 2000, CMS did not supply its
determinations of the comparability of findings in validation and
accreditation surveys for 31 of 82 serious deficiencies. We followed a
protocol similar to the one used by CMS to determine the comparability
of the remaining 31 serious deficiencies, which included 29 quality-of-
care serious deficiencies and 2 physical environment serious
deficiencies. Two analysts with nursing backgrounds compared the
findings and made determinations on their comparability based on their
professional judgment. In cases of disagreement, a third analyst with a
background in nursing made the determination.
We did not include 1998 and 1999 data in our analysis because CMS used
a method that undercounted the number of deficiencies identified by
state survey agencies but not identified by JCAHO. CMS did not count as
deficient those cases in which state survey agencies determined that a
hospital was not meeting the COP on physical environment but JCAHO
determined that the hospital was in compliance because the hospital was
following correction plans approved by JCAHO.
To determine the potential of JCAHO's new accreditation process in
improving the detection of deficiencies in Medicare COPs, we reviewed
material supplied by JCAHO on development and testing of its new
process and interviewed JCAHO officials about the steps taken to test
the new process and to analyze results. We also examined the features
of the new accreditation process by reviewing descriptive material
obtained from JCAHO and interviewing experts in health care quality.
Because the new accreditation process was implemented in January 2004,
we were limited in our ability to determine the effectiveness of the
new accreditation process because we were not able to perform a
comparative analysis of validation survey and JCAHO survey results
under the new process.
To examine the effectiveness of CMS's oversight of JCAHO's
accreditation process, we analyzed the laws and regulations that define
CMS's authority and JCAHO's authority. We reviewed the annual reports
submitted to Congress on JCAHO's performance in identifying serious
deficiencies and reviewed correspondence between CMS and JCAHO and
interviewed officials in both organizations. We analyzed the rate of
disparity that CMS uses to determine the performance of JCAHO's
hospital accreditation process in identifying deficiencies in Medicare
COPs.
To evaluate CMS's statistical methodology for the validation surveys,
we interviewed CMS officials about the sampling and statistical
methods. In the absence of written methodological documentation, we
relied on information provided by CMS officials to evaluate the
methodology. They gave us the following information about their
sampling method. At the beginning of each year, CMS determines a target
for the number of hospitals that will be sampled for validation surveys
in each state. Each month, CMS receives a list of hospitals scheduled
for a JCAHO accreditation survey in that month. Prior to sampling, CMS
removes from the list those hospitals that have received a validation
survey in the last 3-year accreditation cycle and hospitals that do not
participate in Medicare. In the first month of the year, CMS selects a
random sample of hospitals to be surveyed from JCAHO's list. In
subsequent months, CMS removes hospitals in states in which the state
target has been met and then selects a random sample of hospitals.
Prior to sending the list to state survey agencies, CMS determines
which hospitals will receive traditional validation surveys and which
will receive other types of surveys that cannot be used to assess the
performance of JCAHO's hospital accreditation program. State survey
agencies must then complete traditional validation surveys within 60
days of the completion of JCAHO's accreditation survey for the results
to be used by CMS to measure the performance of JCAHO's hospital
accreditation program. According to CMS officials, the sampling
procedures CMS uses are necessary because they are not informed more
than 1 month in advance which hospitals JCAHO will survey for
accreditation.
In reviewing the sampling procedures they described, we determined that
CMS initially selects a probability sample of hospitals for its state
agency validation surveys.[Footnote 25] However, hospitals have varying
chances of selection in the sample depending on the month in the fiscal
year that JCAHO performs the accreditation survey and the number of
hospitals targeted for completion that year in the state in which the
hospital was located. Additionally, the way that CMS determines which
type of survey the sampled hospital receives is not random. Therefore,
the analysis we performed is limited to those hospitals included in the
validation survey sample and cannot be projected to all JCAHO-
accredited hospitals.
[End of section]
Appendix II: Medicare Conditions of Participation:
To participate in Medicare, hospitals must maintain standards of
patient safety and health that comply with Medicare requirements. There
are currently 23 Medicare COPs. Table 7 provides a description of each
Medicare COP.
Table 7: Medicare Conditions of Participation:
Medicare COP: Anesthesia services[A];
Description: Anesthesia services must be well organized and directed by
a qualified doctor of medicine or osteopathy. The service is
responsible for all anesthesia administered.
Medicare COP: Compliance with federal, state, and local laws;
Description: A hospital must comply with applicable federal laws on
patient health and safety and state and local laws on hospital and
personnel licensing.
Medicare COP: Discharge planning;
Description: A hospital must have a discharge planning process
applicable to all patients. Policies and procedures must be in writing.
Medicare COP: Emergency services[A];
Description: If emergency services are provided they must be organized
under the direction of a qualified member of the medical staff and have
adequate medical and nursing personnel qualified in emergency care to
meet the needs anticipated by the facility.
Medicare COP: Food and dietetic services;
Description: Dietary services must be organized, directed, and staffed
by qualified personnel. Contracted services must meet certain
requirements.
Medicare COP: Governing body;
Description: A hospital must have a legally responsible governing body
or persons charged with the responsibilities of a governing body.
Medicare COP: Infection control;
Description: A hospital's sanitary environment must avoid sources and
transmission of infections and communicable diseases. It must have an
active program to prevent, control, and investigate infections and
communicable diseases.
Medicare COP: Laboratory services;
Description: The hospital must maintain, or have available, adequate
laboratory services.
Medicare COP: Medical record services;
Description: A hospital must have a medical record service that has
administrative responsibility for medical records.
Medicare COP: Medical staff;
Description: A hospital must have an organized medical staff that
abides by bylaws approved by the governing body and is responsible for
the quality of patient medical care.
Medicare COP: Nuclear medicine services[A];
Description: Nuclear medicine services must meet the needs of the
patients in accordance with acceptable standards of practice.
Medicare COP: Nursing services;
Description: An organized nursing service must provide 24-hour nursing
services that are supervised or furnished by registered nurses.
Medicare COP: Organ, tissue, and eye procurement;
Description: The hospital must have and implement written protocols on
procurement and have adequate organ transplant policies.
Medicare COP: Outpatient services[A];
Description: Outpatient services must meet patient needs consistent
with acceptable standards of practice.
Medicare COP: Patients' rights;
Description: A hospital must protect and promote patients' rights.
Medicare COP: Pharmaceutical services;
Description: The hospital must have pharmaceutical services that meet
patient needs.
Medicare COP: Physical environment;
Description: Hospital construction, arrangements, and maintenance must
ensure patient safety and provide diagnostic and treatment facilities
and special hospital services appropriate to community needs.
Medicare COP: Quality assessment and performance improvement;
Description: A hospital must have an effective, hospitalwide quality
assurance program.
Medicare COP: Radiologic services;
Description: The hospital must maintain, or have available, diagnostic
radiologic services. Therapeutic services provided must meet
professionally approved standards for safety and personnel
qualifications.
Medicare COP: Rehabilitation services[A];
Description: Rehabilitation, physical therapy, occupational therapy,
audiology, or speech pathology services must be organized and staffed
to ensure the health and safety of patients.
Medicare COP: Respiratory services[A];
Description: Respiratory services must meet patient needs in accordance
with acceptable standards of practice.
Medicare COP: Surgical services[A];
Description: Surgical services must be well organized and provided in
accordance with acceptable standards of practice. Outpatient services
must be consistent with inpatient care quality in accordance with the
complexity of services offered.
Medicare COP: Utilization review;
Description: Utilization review plans must provide for review of the
services that a hospital and its medical staff provide to Medicare and
Medicaid patients.
Source: GAO summary of Medicare COPs.
[A] Optional services not required by Medicare.
[End of table]
[End of section]
Appendix III: Features of JCAHO's New Accreditation Process:
In January 2004, JCAHO introduced a new hospital accreditation process
that includes several new features. Table 8 includes a description of
selected new features of JCAHO's hospital accreditation process.
Table 8: JCAHO's Description of Features of Its New Hospital
Accreditation Process:
Feature of the new accreditation process: Periodic performance review;
Description: The periodic performance review (PPR) is a new form of
evaluation that is conducted by the organization and focuses on
patient safety and quality of care issues. The organization self-
evaluates its compliance with all standards that are applicable to the
services that the organization provides, and develops a plan of action
for all areas of performance identified as needing improvement. JCAHO
will work with the organization to refine its plan of action to assure
that its corrective efforts are on target. The organization will also
identify measures of success for validating resolution of the
identified problem areas when the organization undergoes its complete
on-site survey 18 months later;
Three options to the full PPR are available to organizations. The
options and their requirements are:
Option 1;
The organization performs the mid-cycle self-assessment, develops the
plan of action and measures of success but does not submit PPR data to
JCAHO. The organization attests that it has completed the foregoing
activities but has, for substantive reasons, been advised not to submit
its self-assessment or plan of action to JCAHO;
The organization may discuss standards-related issues with JCAHO staff
without identifying its specific levels of standards compliance;
At the time of the complete on-site survey, the organization provides
its measures of success to JCAHO for assessment;
Option 2;
The organization need not conduct a mid-cycle self-assessment or
develop a plan of action;
The organization undergoes an on-site survey at the mid-point of its
accreditation cycle. The survey will be approximately one-third the
length of a typical full on-site survey and the organization will be
charged a fee to cover survey costs;
The organization develops and submits to JCAHO a plan of action to
address any areas of non- compliance found during the on-site survey.
JCAHO will work with the organization to refine its plan of action. At
the time of the complete on-site survey, the organization provides its
measures of success to JCAHO for assessment;
Option 3;
The mid-cycle survey would be performed, as in Option 2, but, if the
organization chooses, no written documentation or report of the survey
would be left with the organization. Findings would be conveyed orally.
This would eliminate the availability of a survey report for possible
discovery from the organization, and would permit the organization, as
is the case with Option 1, to control the language and documentation of
the mid-cycle assessment activity. At the subsequent full survey,
surveyors would not discuss with the organization, unless asked to do
so, the fact that any particular standard had been found out of
compliance at the mid-cycle assessment. Rather, they would focus on
compliance with those standards at the time of the full survey;
If the plan of action is approved, the organization's accreditation
decision will remain the same. However, if the plan of action is not
approved, the organization's accreditation decision will be changed to
reflect the appropriate status. At the triennial on-site survey,
implementation of the plan of action will be validated.
Feature of the new accreditation process: Priority focus process;
Description: The priority focus process (PFP) is a data-driven tool
that focuses survey activity on issues most relevant to patient safety
and quality of care at the specific health care organization being
surveyed. The PFP uses automation to gather pre-survey data from
multiple sources including JCAHO, the hospital and other public
sources. The PFP then applies rules to 1) identify relevant standards
and appropriate survey activities, and 2) guide the selection of
patient tracers. As part of the priority focus process, surveyors will
track patients through their experience of care within an
organization, assessing the quality and safety of care provided. The
PFP does not imply that priority areas are out of compliance or
deficient in any way. Rather, it lends consistency to the surveyor's
on-site sampling process. The PFP also helps to focus the surveyor's
assessment on quality and safety issues specific to an individual
health care organization;
The output of the PFP process will include: the top four to five
priority focus areas-the processes, systems, or structures within a
health care organization known to significantly impact the safety and
quality of care specific to the health care organization being
surveyed.
Feature of the new accreditation process: Tracer methodology;
Description: An evaluation method in which surveyors select a patient
and use that individual's record as a roadmap or "tracer" to assess
and evaluate an organization's compliance with selected standards and
the organization's systems of providing care and services. Using
tracers, JCAHO surveyors will look at the care provided by each
department within an organization, and how departments work together.
Surveyors retrace the specific care processes that the individual
experienced by observing and talking to staff in areas that the
individual received care. As the individual's case is examined, the
surveyor may identify performance issues in one or more steps of the
process-or the interfaces between steps-that affect the care of the
patient. Surveyors will look for commonalities that might point to
potential system-level issues in the organization. The tracer activity
also provides several opportunities for surveyors to provide education
to organization staff and leaders, as well as to share best practices
from other similar health care organizations;
Tracer patients will primarily be selected from an active patient list.
Typically, individuals selected for the tracer activity are those who
have received multiple or complex services.
Source: JCAHO.
[End of table]
[End of section]
Appendix IV: Comments from the Centers for Medicare & Medicaid
Services:
DEPARTMENT OF HEALTH & HUMAN SERVICES:
Centers for Medicare & Medicaid Services:
Administrator:
Washington, DC 20201:
DATE: JUL 14 2004:
TO: Janet Heinrich:
Director, Health Care-Public Health Issues:
General Accounting Office:
FROM: Mark B. McClellan, M.D., Ph.D.:
Administrator:
SUBJECT: General Accounting Office Draft Report: MEDICARE: CMS Needs
Additional Authority and Better Measures to Adequately Oversee the
Hospital Accreditation Program (GAO-04-850)
Thank you for the opportunity to comment on the above-referenced draft
report from the General Accounting Office (GAO). The Centers for
Medicare & Medicaid Services (CMS), state survey agencies, and the
Joint Commission on the Accreditation of Healthcare Organizations
(JCAHO) conduct frequent reviews of hospital performance. In the three
years under consideration in the GAO study (2000-2002), there were
17,780 surveys in the approximately 4,500 accredited hospitals in the
United States, including the following:
* 4,616 JCAHO full triennial surveys;
* 1,290 JCAHO complaint or "for cause" special investigations:
* 500 CMS-sponsored validation surveys to assess the adequacy of JCAHO
surveys: 7,542 complaint investigations by CMS or state agencies; and
* 3,832 revisits or other investigations by CMS or states to follow-up
on prior complaint investigations, assess whether hospitals took action
to fix significant problems identified on earlier surveys, or similar
purposes.
These 17,780 reviews reflect a considerable national investment in
external quality assurance for hospitals. They are complemented by the
external assistance provided by Quality Improvement Organizations
(QIOs) and, of course, the internal quality efforts of hospitals
themselves.
Each year CMS conducts or arranges with states for a full survey of a
sample ofaccredited hospitals that have undergone a JCAHO accreditation
survey. We call these "CMS validation surveys" because their purpose is
to assess the extent to which any disparity exists between the JCAHO
findings and those of CMS. The GAO study focuses on the disparity
information made possible by the 500 CMS validation surveys.
In 343 out of the 500 hospitals in which full CMS validation surveys
were conducted during the three-year time period of the GAO study, no
serious deficiencies were identified in the hospital. In the 157
hospitals in which a serious deficiency was found through the CMS
validation surveys, the number of deficiencies averaged about 1.6 per
hospital. "Physical environment" issues (principally fire-safety)
represented the single most frequent issue, one we discuss in more
detail later in this letter. The GAO report observes that serious
deficiencies identified in hospitals represent only 2 percent of all
the Medicare requirements surveyed during the three-year period under
study.
While we regard all deficiencies as serious matters, the overall low
rate of identified deficiencies relative to the total number of
hospitals is an encouraging sign that suggests that the overall
accreditation process has merit.
However, additional improvements can and should. be made, no matter how
much progress has already been accomplished or how low the rate of
deficiency might be. Our own on-going, two-year study indicates that
there may be further actions we can take under existing law to
strengthen both CMS oversight and JCAHO's efforts. Examples include the
following:
CMS Validation Surveys: Within the President's 2005 budget request we
are working to increase the number of CMS validation surveys beyond the
current level.
More Sensitive Performance Indicators: We developed the conceptual
framework for more sensitive indicators to improve our ability to
discern the nature and extent of any JCAHO performance issues. We will
soon initiate testing of specific options for such indicators.
Complaint Data: We will explore the extent to which the approximately
2700 complaint investigations conducted each year by CMS or states may
be used as a valuable database to assess JCAHO accreditation practice.
We are gratified by observations in the report that the CMS has
properly executed its statutory responsibility to continually study the
operation and administration of the JCAHO hospital accreditation
process and to submit an annual report to Congress.
With regard to JCAHO's performance, it is worth noting that "physical
environment" represents the area of greatest discrepancy between the
JCAHO findings and the CMS-sponsored validation surveys. Compliance
with life-safety codes is the most common issue in the area of
"physical environment," typically involving fire-safety precautions.
Of all facilities in which JCAHO missed a deficiency finding, about 68
percent were accounted for by "physical environment" issues. This
compared with a facility discrepancy rate of about 29 percent for
health care only, and 3 percent where there was a finding of deficiency
for both health care and physical environment.
Facilities with Findings Missed by JCAHO FY 2000-2002:
[See PDF for image]
[End of figure]
In the past eight years, in its annual reports to Congress and in its
dialogue with JCAHO, CMS has identified the issue of physical
environment as an important area that needs further attention. For
example, the 1996 report to Congress observed that "validation surveys
show that the CoP of Physical Environment continues to be the most
frequently cited condition based on noncompliance with LSC." More
recently, CMS' 2002 report to Congress continued to emphasize that "we
have identified inconsistencies in its [LSC] implementation that we
believe contributes to the differences in the validation findings."
CMS has always considered life-safety code compliance, on the part of
all provider types, to be of critical importance. For this reason the
2002 CMS report to Congress summarized five specific remedial steps
that we recommended to JCAHO, and the 2003 report reported on the
extent of progress being made. While JCAHO implementation of these
recommendations has not been as expeditious as we desired, we are
pleased to see some significant progress. The most recent example of
such progress is the JCAHO agreement with the American Society for
Healthcare Engineering (ASHE) to construct an electronic assessment
tool, train JCAHO staff, and to support JCAHO's efforts to recruit
health care facility engineers.
We will continue to emphasize with JCAHO the need to improve both
health care and life-safety code compliance. Improvement in the area of
life-safety code compliance would, by itself, bring JCAHO accreditation
considerably closer to the findings rendered in the CMS-sponsored
validation surveys.
With respect to the role of CMS and the CMS validation program, the
report contends that the "rate of disparity" measure, as codified in
federal regulations, is not sufficiently sensitive. The "rate of
disparity" measure is used to gauge an accreditation organization's
performance. In addition, the GAO report contends that the methodology
for sample selection is not conducted in a statistically sophisticated
manner, nor are the results presented in a way that extends the
findings to all JCAHO hospitals through the application of an
algorithm. Finally, the report conveys concern about the previous CMS
decision to reduce the validation sample size from 5 percent to a
state-stratified 1 percent sample.
We concur with GAO's view that the sampling methodology, sample size,
and the formula for calculating the rate of disparity should be
reevaluated. As GAO staff are aware, CMS has been actively studying
these issues in the past two years. We have reached conclusions similar
to those in the GAO study. as articulated in the attached document.
Thank you again for the opportunity to review and comment on the draft
report. We look forward to seeing the final report and to working
together to improve hospital care for the nation's Medicare
beneficiaries. Attached are CMS' specific comments to GAO's
recommendations.
Attachment:
Centers for Medicare & Medicaid Services' Comments to the GAO Draft
Report: MEDICARE. CMS Needs Additional Authority and Better Measures to
Adequately Oversee the Hospital Accreditation Program (GAO-04-850).
GAO Recommendation:
Modify the method used to measure the rate of disparity between
validation survey findings and accreditation program findings to
provide a reasonable assurance that Medicare COPS are being met and
consider whether additional measures are needed to accurately reflect
an accreditation program's ability to detect deficiencies in Medicare
COPS.
CMS Response:
The rate disparity calculation is specified in Federal regulations at
42 CFR section 488.8. However, it is quite appropriate to reexamine the
rule and to consider additional or alternative measures to assess the
performance of the accreditation organizations. The CMS has already
begun to examine this issue as part of the agency's hospital quality
improvement activities. We are working to refine existing measures and
develop new ones.
It will be necessary to undertake rulemaking to revise the formula for
calculating the rate of disparity measure, as well as to validate the
threshold for acceptable performance or reasonable assurance. We
believe that the notice and comment procedures inherent in the
rulemaking process will provide an appropriate forum for this
discussion of this significant public policy and will allow all of the
stakeholders to participate. It will also provide for exposure to new
perspectives and may yield innovative approaches to these problems that
may have eluded us in the past.
In addition, we will explore regulatory strategies to address the long-
standing JCAHO performance issues with respect to the Life Safety Code.
We will propose that this initiative be added to the Department's
Regulatory Plan for FY 2005. These approaches will require additional
CMS resources in terms of FTEs and additional funding.
GAO Recommendation:
Provide in the annual report to Congress an estimate, based on the
validation survey sample, of the performance of all JCAHO-accredited
hospitals, including the limitations and protocols_for these estimates
based on generally accepted sampling and statistical methodologies, and
develop a written protocol for these methodologies.
CMS Response:
It is appropriate to explore the possibility of developing and
implementing new sampling and statistical methodologies within
generally accepted statistical practices. We will examine whether
alternate measures can more appropriately be generalized to the
universe of all JCAHO hospitals. We will attempt to secure the
additional resources necessary to undertake a thorough
examination of these issues, to propose alternative sampling
methodologies and develop more robust statistical analyses.
GAO Recommendation:
Annually conduct traditional validation surveys on a sample of JCAHO-
accredited hospitals that is equal to at least 5% of all JCAHO-
accredited hospitals.
CMS Response:
We will seek to increase the validation sample size as we formulate
future budget requests. However, rather than simply increasing the
sample rate to 5 percent, there may be more cost-effective approaches
to enhancing our survey activities.
We note that a return to the 5 percent validation sample would require
additional survey and certification funding that ranges from about $2.6
million annually to almost $4.8 million per year, depending on the
sampling methodology. Thus, additional cost-effective methods to assess
JCAHO performance that would offset the need for major additional
investments in full, traditional CMS validation surveys are likely to
be valuable.
One such approach may be to make use of the database represented by the
approximately 2700 complaint investigations conducted in accredited
hospitals by CMS and states. We will undertake an initiative to analyze
the extent to which this database may be useful in assessing JCAHO
accreditation practice, and then develop analytic tools to put relevant
findings into an improvement plan with JCAHO. In addition, to the
extent that we can increase surveyor time in accredited hospitals, we
will explore risk-based approaches that valuable surveyor time on those
areas of JCAHO accreditation in which problems are most likely.
Finally, we will also seek regulatory changes that would provide CMS
with additional and more substantial information on the JCAHO processes
and findings so as to improve both overall CMS oversight and the
effectiveness of CMS validation surveys.
[End of section]
Appendix V: Comments from the Joint Commission on Accreditation of
Healthcare Organizations:
Joint Commission on Accreditation of Healthcare Organizations:
July 12, 2004:
Mr. David Walker:
Comptroller General:
Government Accountability Office
441 G Street, N.W.
Washington, DC:
Dear Mr. Walker:
We would like to thank the Government Accountability Office for the
opportunity to review the draft report entitled Medicare: C MS needs
Additional Authority and Better Measures to Adequately Oversee the
Hospital Accreditation Program. Because evaluating the performance of
any organization is a complex undertaking, solicitation of the views of
the entity under scrutiny helps to improve the accuracy of the analysis
and provides context for the assessment.
The GAO's key recommendation is that "Congress should consider giving
CMS the same kind of authority over the Joint Commission's hospital
accreditation program that it has over all the other Medicare
accreditation programs." The Joint Commission interposes no objection
to this suggested statutory change, but takes great exception to the
fact that the GAO arrives at this conclusion based upon a flawed study
methodology and erroneous, alarming statistics that seriously mislead
the public and do a great disservice to the Joint Commission.
When the deeming provision respecting hospital oversight was
incorporated into the Medicare statute in 1965, the Congress did so on
its own cognizance. The Joint Commission never sought this deemed
status relationship nor was it even aware of the framing of the
statutory provision respecting hospital accreditation that, to this
date, limits the Executive Branch's oversight of the Joint Commission.
Nevertheless, the Joint Commission has always worked with CMS as if CMS
had the same oversight authority for hospitals that it exercises for
the other newer federal deemed status relationships with the Joint
Commission (e.g., for home health care, for ambulatory surgery
centers). This long-standing, positive working relationship has
provided the nation enormous benefits through assuring continuous
access to state-of-the-art evaluation of health care quality and
patient safety in hospitals that are unparalleled in the world, and are
looked to internationally as the gold standard for assessing hospital
services. Further, the fact that only 2% of Medicare Conditions of
Participation were found out of compliance by the CMS in hospitals
during the three years of this subject GAO study is testimonial to the
positive impacts of the effective working relationship between the
Joint Commission and the CMS (and its predecessor) over the past four
decades. These efforts to continuously improve health care quality and
patient safety not only give beneficiaries confidence in their
providers, but also ensure that the government is getting value for its
spending. The Joint Commission leadership role in this area is evident
in the fact that many private insurers and employers insist that
hospitals serving their plan members be accredited by the Joint
Commission.
The Joint Commission launched a new accreditation process in January of
this year after three years of careful design and field testing. Each
of the evaluation techniques incorporated into the new evaluation
approach had previously been validated by other established evaluators
both in health care and in other venues. We undertook this set of
sweeping changes because of our commitment to continuous quality
improvement and because we believe that a patient-centered approach to
evaluation provides the most meaningful assessment of hospital
performance. It also provides a strengthened vehicle for assuring
continuous hospital attention to our standards requirements. While
technically not authorized to approve the new accreditation process,
CMS staff was briefed on its design on several occasions. Were CMS to
have had such authority, this would have clearly made the Joint
Commission even more comfortable in implementing the new accreditation
process.
Thus, the Joint Commission takes no exception to the GAO's
recommendation that the Congress consider giving CMS the same
authority, over the Joint Commission's hospital accreditation program
that it has over the other deemed Joint Commission accreditation
programs. However, such a chance would make sense irrespective of the
performance of the Joint Commission-and certainly should not be colored
by the inflammatory, grossly inaccurate portrayal of the Joint
Commission that is set forth in this seriously flawed GAO study. On
July 7, 2004, the Joint Commission submitted to the GAO a 26-page
technical corrections document which details the serious errors in and
omissions from the draft GAO study. These are summarized below:
Methods and Use of Statistics:
Evaluating performance is a complex task. Like so many issues involving
assessment, if one asks the wrong question, one gets the wrong answer.
The GAO methodology seeks to assess the ability of the Joint Commission
to evaluate hospital compliance with the Medicare CoPs by conducting
"missed deficiency rate" analyses. However, the calculations performed
by the GAO are at best incomplete, for the GAO has not included in the
calculus the number of deficiencies found by the Joint Commission but
not found by the State Survey Agency (SSAs.) The GAO has continued to
ignore this key point, and has been undeterred in its focus on how many
times the Joint Commission agrees with the SSAs. Essentially, what the
GAO is providing to the reader is an incomplete ratio of "non-
agreement" between the Joint Commission and the State Survey Agencies.
This ratio, especially when used in isolation from other information,
is neither a true indicator of Joint Commission effectiveness, nor an
adequate exploration of whether the Join Commission's hospital
accreditation program ensures that Medicare beneficiaries receive high
quality care in keeping with the Medicare program's expectations.
Specifically, the GAO does not acknowledge that non-agreement between
the SSA and the Joint Commission is influenced by a number of factors,
including differences in interpretation of standards compliance; the
disproportionate rigidity of the CoPs requirements and their related
scoring mechanisms compared to the Joint Commission's accreditation
process; variations in the timing of the Joint Commission and SSA
surveys; and other artifacts inherent in the validation program. Even
with these imponderables, it is extraordinary that accredited hospitals
are found to be in compliance with 98 percent of the CoPs in SSA
validation surveys. It is even more extraordinary that this significant
finding is omitted from the highlights page and is buried in the text
under a finding that misleads the Congress and policymakers into
believing that the Joint Commission does not identify serious
deficiencies.
We further note that while the GAO claims that the Joint Commission
misses 69 percent of-out-compliance CoPs, the GAO has neither the
complete files of survey information nor the expertise necessary to
make such a calculation. This allegation is therefore no more than
conjectural and cannot be defended.
A paradox inherent in the GAO's statistical applications is that the
GAO has chosen an even more inappropriate measure of evaluating whether
COPS are in compliance than the one that they criticize the CMS for
using over the past few years. Using the GAO metric, the greater the
degree of Joint Commission success in ensuring that hospitals are, or
become, in compliance with the Cops following its on-site accreditation
surveys, the greater the likelihood that any SSA findings later in time
will be considered "new" deficiencies. Even if these new deficiencies
are exquisitely small in number, they will then represent a 100 percent
"missed deficiency rate," thereby further misleading and potentially
alarming the Congress and the public.
Furthermore, to provide an adequate, more accurate assessment of the
Joint Commission's performance in assuring that safe, high quality care
is available to Medicare beneficiaries; the GAO should be advising the
CMS to take into account: (1) the actual number of serious deficiencies
that existed in hospitals to be surveyed by SSAs before the Joint
Commission and SSA surveys and how many of these were identified by the
Joint Commission and corrected before the validation surveys; (2) the
percentage of allegation (complaint) surveys that resulted in a finding
that the hospital had at least one serious deficiency (an astounding
low number of hospitals during the GAO study period, as previously
reported to the Congress); and (3) the multiple value-added
requirements that the Joint Commission requires of accredited
organizations, such as the public reporting of clinical performance
data, special requirements related to national patient safety goals,
and over 100 standards that do not have corresponding Medicare
requirements and reflect expectations relating to the state-of-the-art
provision of care in hospitals. While Medicare's CoPs have not been
fully updated since 1986, the Joint Commission has annually updated its
standards - with public sector input - to push hospitals to continually
improve the quality and safety of the care they provide.
The New Survey Process:
The GAO misunderstands and misrepresents the Joint Commission's
evolution to a new accreditation process and in so doing appears to
lack a basic understanding of the tenets of quality improvement. The
Joint Commission's goal is to leverage hospitals to become better at
what they do and to give the public confidence that their care is
meeting contemporary standards. The GAO has been unrelenting in its
focus on whether the pilot test findings of the new accreditation
process would have led to a different distribution of hospitals with
full accreditation status versus lower levels of accreditation, such as
conditional or provisional. Further, the GAO has ignored the compelling
data provided by the Joint Commission which show that the new survey
accreditation process results in better discernment of the types of
deficiencies that are directly related to patient care than the old
process. We cannot over-emphasize this important fact. Evaluating and
improving the quality of care in hospitals is not about counting
deficiencies, it is about finding those deficiencies which, if not
fixed, will generate poor results for patients, and making sure that
these deficiencies are remedied in a timely fashion.
Life Safety Code (LSC):
The GAO has mischaracterized the Joint Commission's response to the
five 2002 CMS recommendations for improving the LSC disparity rate. The
Joint Commission has taken significant steps to address each of the CMS
recommendations, and that information was provided to the GAO. Evidence
of our commitment to the CMS recommendations is reflected by the fact
that there was an approximately 50 percent decline in the number of
hospitals found to be out of compliance with the LSC in the validation
surveys during the study period. While the "disparity rate" declined
only slightly over the study period, the number and percentage of
hospitals that were found to be out of compliance with the LSC by the
SSAs during this time period declined from 43 hospitals (23 percent) in
the 2000 Medicare validation report to the Congress to 25 hospitals (12
percent) in the 2002 Congressional report. This point underscores the
inadequacy of the "missed deficiency rate" metric suggested by the GAO.
By its nature, this metric does not account for improvements in
hospital standards compliance.
An additional important point not mentioned in the GAO report is that
the Joint Commission lobbied strongly and eventually successfully to
have the CMS adopt the 2000 version of the LSC rather than the 1985
version in use throughout the GAO study period. This difference in
requirements contributed significantly to the identified disparities
between Joint Commission and SSA surveys.
Finally, the GAO report fails to put hospital physical safety issues
into perspective for the Congress, thus leaving the reader with the
impression that accredited hospitals are not safe. Hospitals arc in
fact one of the safest health care occupancies in the nation, owing in
large part to the attention that the Joint Commission has placed on the
safety of the physical environment.
Conclusion:
In closing, the Joint Commission is deeply concerned that the GAO has
provided the public with a report that neither uses credible metrics
nor includes highly relevant information about the Joint Commission's
performance. In our view, it is irresponsible to alarm the public using
statistics that have little meaning, and that do not reflect the true
oversight of America's hospitals through Medicare's public-private
sector partnership with the Joint Commission.
Sincerely,
Signed by:
Dennis S. O'Leary, M.D.
President:
[End of section]
Appendix VI: GAO Contact and Staff Acknowledgments:
GAO Contact:
Marcia A. Mann, (202) 512-9526:
Acknowledgments:
In addition to the contact named above, Elaine Swift, Linda Kohn, Behn
Kelly, Elizabeth T. Morrison, Roseanne Price, and Marie Stetser made
key contributions to this report.
[End of section]
Related GAO Products:
Medicare Home Health Agencies: Weaknesses in Federal and State
Oversight Mask Potential Quality Issues. GAO-02-382. Washington, D.C.:
July 19, 2002.
Medicare: HCFA's Approval and Oversight of Private Accreditation
Organizations. GAO/ HEHS-99-197R. Washington, D.C.: September 30, 1999.
Home Health Care: HCFA Properly Evaluated JCAHO's Ability to Survey
Home Health Agencies. GAO/HRD-93-33. Washington, D.C.: October 26,
1992.
Health Care: Criteria Used to Evaluate Hospital Accreditation Process
Need Reevaluation. GAO/HRD-90-89. Washington, D.C.: June 11, 1990.
FOOTNOTES
[1] See 42 U.S.C. § 1395bb(a) (2000).
[2] JCAHO is referred to in statute under its former name, the Joint
Commission on Accreditation of Hospitals.
[3] JCAHO develops its standards with a committee of experts and
stakeholders, such as the government, hospitals, and consumers.
[4] HHS OIG, The External Review of Hospital Quality: A Call for
Greater Accountability, OEI-01-97-00050 (Washington, D.C.: July 1999).
[5] One of the 23 COPs cannot be deemed by an accreditation
organization. CMS relies on organizations other than the accreditation
organizations to certify that hospitals comply with the COP that
requires hospitals to establish a utilization review plan for services
provided to Medicare beneficiaries.
[6] Specifically, the agency's regulations require the accreditation
organization's standards to be at least as stringent as the Medicare
COPs, when taken as a whole. See 42 C.F.R. § 488.6(a) (2003).
[7] Forty-nine states allow JCAHO hospital accreditation as a full or
partial substitute for meeting health care quality standards and other
requirements for state licensure.
[8] The remaining 18 percent of hospitals choose to be accredited by
the American Osteopathic Association (AOA) or to be certified by state
survey and certification agencies.
[9] The board includes seven members chosen by the American Hospital
Association, seven chosen by the American Medical Association, three
chosen by the American College of Physicians--American Society of
Internal Medicine, three chosen by the American College of Surgeons,
and one chosen by the American Dental Association. In addition, the
board consists of a nurse-at-large and six public members. The
president of JCAHO is an ex officio member of the board.
[10] When Congress first established JCAHO's deeming authority in 1965,
it prohibited federal authorities from issuing standards on patient
health and safety for hospitals higher than comparable requirements for
hospital accreditation by JCAHO in deference to the expertise of
professional accreditation organizations sponsored by medical and
hospital associations. See Pub. L. No. 89-97, § 102(a), 79 Stat. 286,
315 (1965). Subsequent legislation removed the prohibition and required
JCAHO to demonstrate that its standards were at least equivalent to any
such higher standards issued by the Secretary in order to have deeming
authority in that area. See Pub. L. No. 92-603, § 244(c), 86 Stat.
1329, 1423 (1972).
[11] AOA solely accredits approximately 2 percent of hospitals and
JCAHO and AOA jointly accredit less than 1 percent of hospitals. While
JCAHO and AOA are currently the only hospital accrediting
organizations, federal law permits CMS to approve any other national
accreditation body that demonstrates that Medicare requirements will be
met by hospitals it accredits.
[12] See 42 U.S.C. § 1395ll(b).
[13] For this report, we will refer to traditional validation surveys
as validation surveys.
[14] 42 C.F.R. 488.8(e).
[15] Between fiscal years 2000 and 2002, JCAHO used more recent life
safety code standards than state survey agencies performing validation
surveys. CMS stated that these differences could account for some of
the disparate findings between JCAHO's surveys and state agency
validation surveys. However, CMS considered these different standards
in determining whether JCAHO had not detected serious deficiencies in
the life safety code.
[16] U.S. General Accounting Office, Medicare Home Health Agencies:
Weaknesses in Federal and State Oversight Mask Potential Quality
Issues, GAO-02-382 (Washington, D.C.: July 19, 2002) and U.S. General
Accounting Office, Nursing Home Quality: Prevalence of Serious
Problems, While Declining, Reinforces Importance of Enhanced Oversight,
GAO-03-561 (Washington, D.C.: July 15, 2003).
[17] All six states conducted at least 15 validation surveys from
fiscal year 2000 through 2002.
[18] In 2004 and 2005, JCAHO will continue to conduct its accreditation
surveys on an announced basis.
[19] For example, we found in our nursing home survey work in 1999 that
state surveyors may perform their tasks more attentively when they are
being observed by federal surveyors than they would if performing their
surveys unobserved, thus masking a state surveyor's typical
performance. U.S. General Accounting Office, Nursing Home Care:
Enhanced HCFA Oversight of State Programs Would Better Ensure Quality,
GAO/HEHS-00-6 (Washington, D.C.: Nov. 4, 1999).
[20] Whenever CMS considers, approves or removes an accreditation
organization's deeming authority, the agency is required to publish
detailed notices in the Federal Register, and consider public comment.
See 42 U.S.C. § 1395bb(b)(3); 42 C.F.R. § 488.8(b) and (f)(7).
[21] Beginning in 1995, JCAHO-accredited hospitals have assessed their
own compliance with the life safety code and developed correction
plans, which JCAHO must approve. If hospitals are in compliance with
their correction plans, JCAHO's surveyors do not record outstanding
life safety code deficiencies.
[22] The example is based on the analysis of the rate of disparity in
American Institutes of Research, Measurement and Evaluation of Revised
Accredited Hospital Validation and Oversight (Washington, D.C.: Nov. 6,
2002).
[23] For example, CMS does not measure and report the precision of the
estimates from the sample of validation surveys through the use of
confidence intervals or margins of error, which define the range of
estimates that sample results would yield given different random
samples for a specified level of certainty.
[24] These estimates were developed assuming that the validation
surveys are conducted on a simple random sample of JCAHO-accredited
hospitals and a 95 percent confidence level.
[25] In a probability sample, each eligible hospital accredited in a
given year would have to have a known, nonzero chance for selection in
the sample.
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548: