VA Health Care
Adequacy of Resident Supervision Is Not Assured, but Plans Could Improve Oversight Gao ID: GAO-03-625 July 2, 2003The Department of Veterans Affairs (VA) provides graduate medical education (GME) to as many as one-third of U.S. resident physicians, but oversight responsibilities spread across VA's organizational components and multiple affiliated hospitals and medical schools could allow supervision problems to go undetected or uncorrected. GAO was asked to examine VA's procedures for (1) monitoring VA medical centers' adherence to VA's requirements for resident supervision, (2) using evaluations of supervision by GME accrediting bodies and residents, and (3) using information about resident supervision drawn from VA's programs for monitoring the quality and outcomes of patient care.
VA cannot assure that the resident physicians who provide care in its facilities receive adequate supervision because its procedures for monitoring supervision are insufficient. VA does not know whether medical centers have adopted VA's national requirements for supervision of residents' diagnosis, treatment, or discharge of patients. VA officials require a review of only one specific requirement that is intended to ensure availability of supervision when a supervising physician does not need to be in the operating or procedural suite while a resident performs a diagnostic or therapeutic procedure. Four of 11 network officials we interviewed had not conducted this review, and the requirement at one medical center in one of these four networks was less stringent than VA's national requirement. To obtain more complete information about adherence to its national supervision requirements, VA plans to have external peer reviewers examine documentation of supervision in patients' medical records. VA's plans for this review have not been finalized. For example, as of May 2003, VA had not decided whether reviewers would examine records from VA's new outpatients. Without records from new patients, reviewers will not be able to assess documentation of residents' supervision during a veteran's first outpatient visit. To improve its oversight, VA is making efforts to obtain information from accrediting bodies and residents about the quality of resident supervision. For example, VA has taken steps to obtain direct access to letters from accrediting bodies that contain evaluations of the GME programs in which its medical centers participate. To solicit feedback from residents, VA implemented a national survey, but was unable to send this survey to a representative sample of residents from each VA medical center because it does not have a complete central list of its residents. VA is taking action to obtain this information. In addition, VA uses information from its broader programs for monitoring the quality and outcomes of patient care, such as its patient safety and surgical quality improvement programs, to identify and correct problems with resident supervision. Information from these programs has served as the basis for corrective actions by VA officials.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-03-625, VA Health Care: Adequacy of Resident Supervision Is Not Assured, but Plans Could Improve Oversight
This is the accessible text file for GAO report number GAO-03-625
entitled 'VA Health Care: Adequacy of Resident Supervision Is Not
Assured, but Plans Could Improve Oversight' which was released on July
02, 2003.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
United States General Accounting Office:
GAO:
July 2003:
VA HEALTH CARE:
Adequacy of Resident Supervision Is Not Assured, but Plans Could
Improve Oversight:
GAO-03-625:
GAO Highlights:
Highlights of GAO-03-625, a report to congressional requesters
Why GAO Did This Study:
The Department of Veterans Affairs (VA) provides graduate
medical education (GME) to as many as one-third of U.S.
resident physicians, but oversight responsibilities spread
across VA‘s organizational components and multiple affiliated
hospitals and medical schools could allow supervision problems
to go undetected or uncorrected. GAO was asked to examine VA‘s
procedures for (1) monitoring VA medical centers‘ adherence to
VA‘s requirements for resident supervision, (2) using
evaluations of supervision by GME accrediting bodies and
residents, and (3) using information about resident supervision
drawn from VA‘s programs for monitoring the quality and
outcomes of patient care.
What GAO Found:
VA cannot assure that the resident physicians who provide care in its
facilities receive adequate supervision because its procedures for
monitoring supervision are insufficient. VA does not know whether
medical centers have adopted VA‘s national requirements for supervision
of residents‘ diagnosis, treatment, or discharge of patients. VA
officials require a review of only one specific requirement that is
intended to ensure availability of supervision when a supervising
physician does not need to be in the operating or procedural suite
while a resident performs a diagnostic or therapeutic procedure. Four
of 11 network officials we interviewed had not conducted this review,
and the requirement at one medical center in one of these four networks
was less stringent than VA‘s national requirement. To obtain more
complete information about adherence to its national supervision
requirements, VA plans to have external peer reviewers examine
documentation of supervision in patients‘ medical records. VA‘s plans
for this review have not been finalized. For example, as of May 2003,
VA had not decided whether reviewers would examine records from VA‘s
new outpatients. Without records from new patients, reviewers will not
be able to assess documentation of residents‘ supervision during a
veteran‘s first outpatient visit.
To improve its oversight, VA is making efforts to obtain information
from accrediting bodies and residents about the quality of resident
supervision. For example, VA has taken steps to obtain direct access to
letters from accrediting bodies that contain evaluations of the GME
programs in which its medical centers participate. To solicit feedback
from residents, VA implemented a national survey, but was unable to
send this survey to a representative sample of residents from each VA
medical center because it does not have a complete central list of its
residents. VA is taking action to obtain this information.
In addition, VA uses information from its broader programs for
monitoring the quality and outcomes of patient care, such as
its patient safety and surgical quality improvement programs,
to identify and correct problems with resident supervision.
Information from these programs has served as the basis for
corrective actions by VA officials.
What GAO Recommends:
GAO recommends that VA
* ensure that VA medical centers that provide GME adopt and adhere to
VA‘s national requirements for resident supervision and
* ensure that external peer review of documentation of resident
supervision includes records from VA‘s new outpatients.
VA concurred with the recommendations.
www.gao.gov/cgi-bin/getrpt?GAO-03-625
To view the full product, including the scope and methodology, click on the link
above. For more information, contact Cynthia A. Bascetta at
(202) 512-7101.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
VA Lacks Adequate Procedures to Monitor Implementation of Its
Supervision Requirements:
VA Is Acting to Obtain Information about Supervision from Accrediting
Bodies and Residents:
VA Uses Its Programs for Monitoring Patient Care to Identify and
Correct Problems with Resident Supervision:
Conclusions:
Recommendations for Executive Action:
Agency Comments:
Appendix I: Scope and Methodology:
Appendix II: Comments from the Department of Veterans Affairs:
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
Staff Acknowledgments:
Tables:
Table 1: Examples of Requirements from VA's Resident Supervision
Handbook Issued on October 25, 2001, by Domain of Residents' Health
Care Activities:
Table 2: Examples of Questions about Monitoring Processes from the
Annual Report on Residency Training Programs Completed by Medical
Centers and Networks:
Table 3: Number of VA Medical Centers That Reported Monitoring Some
Aspect of the Documentation of Resident Supervision, by Domain of
Residents' Health Care Activities:
Table 4: VA Networks Included in Our Sample:
Table 5: VA Medical Centers Included in Our Sample:
Abbreviations:
ACGME: Accreditation Council for Graduate Medical Education:
GME: graduate medical education:
NSQIP: National Surgical Quality Improvement Program:
OAA: Office of Academic Affiliations:
VA: Department of Veterans Affairs:
United States General Accounting Office:
Washington, DC 20548:
July 2, 2003:
The Honorable Bob Graham
Ranking Minority Member
Committee on Veterans' Affairs
United States Senate:
The Honorable John D. Rockefeller IV
United States Senate:
The Veterans Health Administration of the Department of Veterans
Affairs (VA) is the largest single provider of graduate medical
education (GME) training sites in the United States, with as many as
one-third of the nation's resident physicians receiving part or all of
their training in VA health care facilities. Residents are medical
school graduates who receive supervised training in a medical specialty
(such as internal medicine or surgery) prior to providing care without
supervision.[Footnote 1] As a provider of GME, VA faces the dual
challenges of ensuring the safety and quality of the health care its
patients receive from residents while simultaneously providing
residents with appropriate educational opportunities. Supervision of
residents by qualified physicians is central to balancing these patient
care and educational goals, and responsibility for the care provided by
a resident to any patient belongs to the licensed physician who
supervises that resident.[Footnote 2] Through observation and
direction, supervising physicians are to impart knowledge and skills to
residents while making sure that patients receive appropriate, timely,
and effective care.
Effective oversight is necessary if VA is to assure the adequacy of
resident supervision. Key components of oversight include procedures to
assess the supervision residents receive and to initiate corrective
action when there is a problem. Information from multiple,
complementary sources can be used to assess supervision; such
information includes evidence of whether residents receive required
supervision, evaluations of the adequacy of supervision by
organizations that accredit GME programs and by residents, and analyses
of the quality and outcomes of care provided by residents. In 1986 and
1992, we reported that VA headquarters officials had not adequately
overseen resident supervision and that the documentation of resident
supervision at some medical centers was inadequate.[Footnote 3]
Although documentation does not fully communicate the extent or quality
of supervision, it is an important record of whether a supervising
physician was involved in a patient's care.
Responsibilities for resident supervision and its oversight are
distributed across multiple VA organizational components and are shared
by VA's affiliated medical schools and teaching hospitals. VA
headquarters established national requirements for supervision of
residents' health care activities--including diagnosis, treatment, and
discharge of patients--and for oversight of supervision.
Responsibilities for implementing these requirements are assigned to
the administrators of its regional networks[Footnote 4] of medical
facilities, medical center managers, and supervising physicians. Most
residency training within VA medical centers is conducted through GME
programs run by medical schools or other teaching hospitals, which are
known as sponsoring institutions. GME accrediting bodies hold the
sponsoring institutions responsible for the quality of the GME program
in each medical specialty. As a result, VA medical centers share
responsibility for ensuring the adequacy of residents' supervision with
these affiliated sponsoring institutions.
Concerned that overlapping authority for residents' activities could
allow problems with resident supervision to go undetected or
uncorrected, you asked us to examine the adequacy of VA's oversight of
resident supervision. In response to your request, we examined VA's
procedures for (1) monitoring VA medical centers' adherence to VA's
requirements for resident supervision, (2) using evaluations of
supervision by GME accrediting bodies and residents, and (3) using
information about resident supervision drawn from VA's programs for
monitoring the quality and outcomes of patient care.
To address these objectives, we examined VA's policy for resident
supervision and reviewed relevant documents from VA headquarters
offices, networks, and medical centers. We analyzed annual reports on
residency training submitted to VA headquarters for the 2000/2001
academic year by VA's regional networks and 114 of the approximately
130 VA medical centers that provide GME training.[Footnote 5] We
interviewed VA officials, as well as GME experts and officials of
accrediting bodies, medical associations, and other stakeholder groups.
We also interviewed GME managers from 11 of VA's 21 regional networks
and 11 medical centers. The sample included one medical center from
each sampled network and was designed to cover a range in total number
of residency positions and number of medical specialties in which
training occurred. We reviewed information from three additional
medical centers involved in GME programs that, as of May 2002, had been
placed on probationary accreditation or for which accreditation was to
be withdrawn.[Footnote 6] We visited two of those medical centers. Our
work covered VA's oversight of resident supervision and did not include
an evaluation of the quality of care provided by residents or the
quality of the supervision provided to residents. We conducted our work
from September 2001 through June 2003 in accordance with generally
accepted government auditing standards. See appendix I for a more
detailed discussion of our scope and methodology.
Results in Brief:
VA cannot assure that the residents who provide care in its facilities
receive adequate supervision because its procedures for monitoring
supervision are insufficient. VA does not know whether medical centers
have adopted resident supervision policies that are consistent with
VA's national requirements for supervision of residents' health care
activities, such as diagnosis, treatment, and discharge of patients. VA
officials require a review of only the requirement that is intended to
ensure availability of supervision when a supervising physician does
not need to be in the operating or procedural suite while a resident
performs a diagnostic or therapeutic procedure. Network GME officials
are to review this requirement in medical centers' policies, and 4 of
11 network GME managers we interviewed had not conducted this review.
Moreover, we found that the requirement at one medical center in one of
these four networks was less stringent than VA's national requirement.
To learn which aspects of resident supervision medical centers and
networks monitor, VA requires medical centers and networks to submit
annual reports on residency training. Medical centers' annual reports
for the 2000/2001 academic year indicate that most medical centers
monitor some documentation of supervision, but few conduct
comprehensive reviews. About half of the 91 medical centers that
reported having a review process also reported finding inadequate
documentation of supervision and then taking steps to improve it. To
obtain more complete information about adherence to its national
requirements for supervision, VA plans to have external peer reviewers
examine the documentation of supervision in patients' medical records.
External peer review could allow assessment of whether most of VA's key
documentation requirements are being met. VA's plans for this review,
however, have not been finalized, and as of May 2003, VA had not
decided whether reviewers would examine records from VA's new
outpatients. Without a sample of records from new patients, reviewers
would not be able to assess the required documentation of supervisory
involvement during a veteran's first outpatient visit.
VA is making efforts to obtain information from accrediting bodies and
residents about the quality of supervision provided to its residents.
For example, VA has taken steps to gain copies of accreditation letters
directly from GME accrediting bodies. These letters contain evaluations
of the GME programs in which VA medical centers participate and are
sent to the institutions that sponsor GME programs, but not to medical
centers that participate in those programs. As a result, VA medical
centers must generally rely on affiliated sponsoring institutions to
inform them of problems identified by the accrediting body. We found
that most VA medical center managers indicated that their affiliated
sponsors had shared this information, and when problems with VA were
identified, VA managers reported taking action to solve them. We also
found, however, that one sponsoring institution did not provide a
participating VA medical center with timely information about an
impending withdrawal of accreditation for a GME program. To obtain
standardized feedback from residents about their educational
experiences, including the quality of their supervision, VA implemented
a national survey in 2001. In 2001 and 2002, VA could not send the
survey to a representative sample of residents from each VA medical
center because it lacked a complete list of its residents. VA is taking
action to obtain this information so that it can send the survey to a
sample of residents at each medical center. Medical centers' annual
reports provide network and headquarters officials with additional
information about concerns expressed by residents and steps taken to
address those concerns.
VA also uses information from its broader programs for monitoring the
quality and outcomes of patient care, such as its patient safety and
surgical quality improvement programs, to identify and correct problems
with resident supervision. Although too new to evaluate, the patient
safety program VA implemented in January 2002 established a process for
determining the causes of events that led to or could have led to
patient harm and for taking steps to eliminate or minimize identified
risks, such as inadequate resident supervision. In addition, VA's
program for monitoring and improving surgical outcomes allows VA to
examine residents' performance of surgical procedures. Information
generated by this program has prompted medical center and network
officials to take steps to improve the supervision of surgical
residents. For example, a team of experts from this program noted
inadequate supervision of surgeries performed by urology residents at
one medical center they visited in 2002. The medical center responded
by arranging for urologists to spend more time at the medical center
and ensuring that they understood VA's supervision requirements. Tort
claim review is another way VA monitors the possible role of resident
supervision in problems with patient care. Review of paid tort claims
led VA in 2001 to clarify its requirements for supervision in inpatient
settings by adding an explicit reference to weekends and holidays to
the requirement that each inpatient must be seen by the supervising
physician within 24 hours of admission.
To improve VA's oversight of resident supervision and help ensure the
quality of both health care and GME, we are making recommendations to
the Secretary of Veterans Affairs to ensure that VA medical centers
adopt and adhere to VA's national requirements for resident supervision
and to ensure that external peer review of documentation of resident
supervision includes records from VA's new outpatients. VA concurred
with our recommendations.
Background:
Education is one of VA's four core missions,[Footnote 7] and in fiscal
year 2002, VA paid approximately $383 million to residents training at
about 130 VA health care facilities.[Footnote 8] For the 2002/2003
academic year, VA supported almost 8,800 residency slots, about 9
percent of all residency training positions in the United States.
Moreover, because several residents typically rotate through each slot,
VA estimates that it provides graduate medical training to more than
28,000 residents each year, or as many as one-third of the nation's
residents. The number of residency slots VA allocates to individual
medical centers involved in GME ranges from less than 1 to more than
200.[Footnote 9] Although about half of VA's residency positions are in
primary care, VA supports GME in 45 recognized medical specialties and
subspecialties; individual medical centers provide training in from 1
to more than 30 specialties.
VA headquarters officials have ultimate oversight responsibility for
the activities of residents within VA medical centers, and several
different headquarters offices have monitoring functions that relate to
resident supervision. VA's Office of Academic Affiliations (OAA) has
responsibility for developing and overseeing policies for resident
supervision, monitoring VA's GME activities, and allocating residency
slots. Under the Patient Safety Program VA implemented in January 2002,
VA's National Center for Patient Safety collects and analyzes
information from VA medical centers about patient risk events and their
causes. Medical centers are required to report all patient safety
events--including adverse events and close calls[Footnote 10]--to the
National Center for Patient Safety. In addition, medical centers are
required to determine the root causes of patient safety incidents with
severe or potentially severe outcomes and develop plans to prevent them
in the future. The success of this program will depend on the extent to
which VA is able to establish a culture in which employees feel safe to
make these reports.[Footnote 11] VA's Office of Patient Care Services
establishes and monitors health care programs. For example, its
National Surgical Quality Improvement Program (NSQIP) examines
postoperative outcomes.[Footnote 12] Additional oversight of resident
supervision is provided by VA's Office of Inspector General.[Footnote
13]
Because VA's health care system is decentralized, responsibilities for
implementing VA's national policy for resident supervision are assigned
to networks and medical centers. Network officials are to provide
medical centers with the resources necessary to ensure that residents
are supervised in accordance with VA's national policy and are to
evaluate the strengths and weaknesses of medical centers' GME
activities. Medical center directors are responsible for establishing
facility policies for resident supervision that fulfill the
requirements of VA's national policy,[Footnote 14] and medical center
chiefs of staff are responsible for the educational and patient care
activities of all residents within the facility. In addition, a
physician in each medical specialty is responsible for ensuring that
the residents training in that specialty are supervised as required.
VA medical centers typically also share responsibility for the
oversight of residents with affiliated institutions that sponsor GME
programs. VA participates in more than 1,900 distinct GME programs, 29
of which are sponsored by VA medical centers.[Footnote 15] The rest are
sponsored by about 120 medical schools and teaching hospitals with
which VA medical centers are affiliated. The majority of VA medical
centers work with one GME sponsoring institution, but individual VA
medical centers participate in the GME programs of up to four different
sponsors. When a VA medical center serves as a training site for
residents, but is not the sponsoring institution, it is known as a
participating institution. GME accrediting bodies hold sponsoring
institutions responsible for all aspects of their educational programs,
including aspects conducted within participating institutions. GME
accrediting bodies do not separately accredit participating
institutions and do not evaluate the extent to which supervision that
occurs within participating institutions, such as VA medical centers,
meets requirements set by those participating institutions.
VA requires accreditation of each GME program through which its
residents obtain training. More than 98 percent of VA's residency slots
are filled by residents in GME programs that are subject to
accreditation review by the Accreditation Council for Graduate Medical
Education (ACGME); the remaining slots are filled by residents in
osteopathic programs that are subject to accreditation review by the
American Osteopathic Association. GME accreditation status indicates an
overall assessment of the quality of an educational program in a
particular medical specialty. Accrediting bodies evaluate several
aspects of each GME program, including provisions for the supervision
and safety of residents, the adequacy of institutional resources,
educational curriculum, and the extent to which the program meets that
specialty's specific training requirements. A program can be fully
accredited, or a program can be granted accreditation with notification
of problems that must be corrected. Accreditation can also be
withdrawn. A program's accreditation status is made public, but to
safeguard confidential information,[Footnote 16] specific problems
with the program or its training sites are described in letters sent
only to the sponsoring institution. Accrediting bodies have not been
sending these letters to participating institutions.
Accrediting bodies state that the quality of patient care must remain
the highest priority of GME programs. Health care organizations that
provide GME must ensure that qualified staff physicians supervise
residents and that the same standards for the quality and safety of
patient care apply when residents are involved in health care delivery
as when they are not. GME accrediting bodies require that supervising
physicians adjust the level of supervision to meet the educational goal
of increasing residents' competence by giving them appropriate
opportunities to assume greater independence in their patient-care
activities, that is, allowing residents to assume graduated
responsibilities. The supervising physician relies on his or her
professional judgment and knowledge of the patient's medical condition
and the resident's level of mastery to determine the degree of
independence of the resident's patient-care responsibilities.
VA's national policy on resident supervision is detailed in a handbook
that establishes specific requirements for (1) the involvement of
supervising physicians in the care provided by residents who diagnose,
treat, or discharge patients and (2) the documentation of that
involvement.[Footnote 17] These specific requirements apply to four
domains of residents' clinical activity--inpatient care, outpatient
care, diagnostic and therapeutic procedures, and consultations--and
provide guidelines for putting into practice GME accrediting bodies'
principles of resident supervision and graduated levels of
responsibility. (See table 1 for an example of VA's requirements for
supervision in each of the four domains.) Experts on GME told us that
the requirements in VA's handbook are reasonable and appropriately
consider the role of supervision in ensuring the quality of patient
care and of resident education. Some of these experts described it as a
best practice model.
Table 1: Examples of Requirements from VA's Resident Supervision
Handbook Issued on October 25, 2001, by Domain of Residents' Health
Care Activities:
Domain: Inpatient care; Examples of requirements for supervision and
its documentation: The supervising physician must meet each new
inpatient within 24 hours of admission (including weekends and
holidays) and personally document that encounter in the patient's
medical record. Concurrence with, or modifications to, the resident's
diagnosis and treatment plan must be documented in the supervising
physician's progress note.
Domain: Outpatient care; Examples of requirements for supervision and
its documentation: The supervising physician must supervise the initial
visit of each new patient to an outpatient clinic, either by seeing the
patient or discussing the patient with the resident at that initial
visit. Involvement of the supervising physician must be documented in
the medical record.
Domain: Diagnostic and therapeutic procedures; Examples of requirements
for supervision and its documentation: When a resident is involved in
the care of a patient who is to undergo an elective or scheduled
procedure, the supervising physician is to write a preprocedural note
that indicates the diagnosis and treatment plan.
Domain: Consultations; Examples of requirements for supervision and its
documentation: The supervising physician must meet with each patient
who was seen by a resident for a consultation and document his or her
personal evaluation in the patient's medical record.
[End of table]
Source: VA.
VA Lacks Adequate Procedures to Monitor Implementation of Its
Supervision Requirements:
VA does not have adequate procedures to determine whether residents at
VA medical centers are supervised in accordance with its national
requirements. For example, VA does not check whether each medical
center involved in GME has adopted policies that are consistent with
VA's requirements for resident supervision. To learn what medical
centers and networks do to monitor whether supervision is consistent
with VA's national requirements, VA requires that medical centers and
networks submit annual reports on residency training. Medical centers'
reports filed for the 2000/2001 academic year indicate that most
medical centers review some documentation of resident supervision, but
few conduct comprehensive reviews. To obtain more complete information
about the supervision residents receive, VA is planning to use external
peer review to assess adherence to its requirements for documenting
resident supervision. These plans have not been finalized. For example,
as of May 2003, VA had not decided whether reviewers would examine
records from VA's new outpatients.
VA Does Not Determine Whether VA Medical Centers' Policies Are
Consistent with Its National Requirements for Resident Supervision:
VA does not know whether all its medical centers have adopted policies
that are consistent with the specific requirements in its resident
supervision handbook for the supervision of residents' diagnosis,
treatment, and discharge of patients. The director of each medical
center involved in GME is to establish facility policies for resident
supervision that fulfill the requirements in VA's handbook, but VA
requires a review of only one requirement involving the supervision of
diagnostic and therapeutic procedures--the medical centers'
requirements for the minimal acceptable level of supervision for
diagnostic and therapeutic procedures. Specifically, in situations in
which the supervising physician is not in the operating or procedural
suite, VA requires that the supervisor must, at a minimum, be
immediately available in the facility or campus to provide direct
supervision of the procedure if necessary.[Footnote 18] Network GME
managers[Footnote 19] are supposed to review and approve this
requirement; they are not required to report the results of their
reviews to OAA. There is no separate OAA review of any of the
requirements in medical centers' supervision policies.
We found that not all networks have completed the one required review
and that medical centers' policies are not always consistent with VA's
national policy. Of the 11 network GME managers we interviewed, 7 told
us that they had completed this required review of the minimal
requirements for supervision of procedures in medical center policies,
but 4 told us that they had not. We found that the requirement of a
medical center in one of the four networks that had not conducted this
review was less stringent than the requirement in VA's handbook for
supervision of diagnostic and therapeutic procedures. The written
policy at this medical center stated that the supervising physician can
be immediately available by telephone rather than requiring him or her
to be immediately available in the facility or on campus.[Footnote 20]
One network GME manager who did review this requirement for supervision
of diagnostic and therapeutic procedures told us that in 2002, he
identified three medical centers that had written requirements for
supervision of these procedures that were less stringent than the
requirement in VA's handbook and that he instructed each of these
facilities to change its policy to be consistent with VA's national
requirement.
VA Headquarters Monitors Medical Center and Network Oversight of
Resident Supervision through Annual Reports:
To learn what medical centers and networks do to monitor whether
supervision is consistent with VA's resident supervision handbook, VA
has required annual reports on residency training programs beginning
with the 1999/2000 academic year. Medical center managers are to
provide narrative answers to specific open-ended questions about their
monitoring processes as well as about the problems they identified and
actions they took to address them for each of three areas of oversight.
(See table 2.) These medical center reports are channeled through VA's
networks to OAA. Network officials are to review them and summarize the
strengths and weaknesses of the medical centers' GME programs in
network-level annual reports, which are also submitted to OAA.
Table 2: Examples of Questions about Monitoring Processes from the
Annual Report on Residency Training Programs Completed by Medical
Centers and Networks:
Area of oversight: Supervision requirements; Examples of questions to
be completed by medical center managers: Describe your process for
reviewing and monitoring medical center data collected for assessing
resident supervision in the following areas: (1) inpatient admission,
continuing care, and discharge supervision; (2) outpatient visit
supervision; and (3) supervision of diagnostic and therapeutic
procedures and consultations.[A]; Examples of questions to be completed
by network managers: Describe any network-level process for review of
medical center data collected for assessing adherence to VA's
educational supervision requirements and the results of such review in
the following areas: (1) inpatient admission, continuing care, and
discharge supervision; (2) outpatient visit supervision; and (3)
supervision of diagnostic and therapeutic procedures and
consultations.[A]
Area of oversight: Evaluations of resident supervision; Examples of
questions to be completed by medical center managers: Describe concerns
of the accrediting bodies specific to VA clinical rotations.[B];
Describe your process for obtaining and reviewing resident comments
related to their VA clinical training experience.[A]; Examples of
questions to be completed by network managers: Describe any network-
level process for review of residents' comments related to their VA
clinical training experience and the results of such review.
Area of oversight: Patient care; Examples of questions to be completed
by medical center managers: Describe your process for reviewing and
monitoring all incidents and risk events[C] with complications to
ensure that the appropriate level of resident supervision occurred.[A]
Examples of questions to be completed by network managers: Describe any
network-level review process for assessing incidents and risk events[C]
to ensure that the appropriate level of resident supervision occurred
and the results of that review.
Source: VA.
[A] Medical centers are also asked to describe results of their reviews
and action plans for correction or remediation of problems found.
[B] Medical centers are also asked to note each program's accreditation
status, summarize affiliate and VA responses to accrediting body
concerns, and describe any corrective actions.
[C] Risk events include events that did result, or could have resulted,
in an adverse outcome.
[End of table]
These annual reports can provide managers with limited, but useful,
information about the extent and quality of monitoring performed by
medical centers, including whether medical centers monitor
documentation or some other indication of supervision. Some medical
centers and networks provided little detail in response to the annual
reports' open-ended questions. For example, not all medical centers
described which specific aspects of resident supervision they
monitored. OAA used open-ended questions in part to accommodate
differences among medical centers in the number and type of residents
they train.
VA officials have used information from annual reports to monitor
medical center oversight of resident supervision. For example, one
network GME manager followed up on a problem identified through a
medical center annual report by requiring the medical center to submit
an action plan for improving supervision of ophthalmology residents by
the beginning of the 2002/2003 academic year. An OAA official told us
that analysis of these annual reports not only helped identify areas of
vulnerability with residency programs, but also pointed to possible
best practices.
Most VA Medical Centers Monitor Some Documentation of Resident
Supervision, but Few Conduct Comprehensive Reviews:
VA does not require its medical centers or networks to conduct
systematic reviews of the documentation of resident
supervision,[Footnote 21] and medical centers differ in the extent to
which they monitor adherence to VA's requirements for supervision. More
than three-fourths of medical centers' annual reports included a
description of an independent review of the documentation of
supervision of at least one aspect of care provided by residents, but
most medical centers did not describe reviews of all four domains of
residents' health care activities.[Footnote 22] For each of three
domains--inpatient care, outpatient care, and diagnostic and
therapeutic procedures--over half the medical centers described a
process for an independent review of at least one element of the
documentation of resident supervision, that is, a review by someone
other than a physician with related supervisory responsibilities (see
table 3). For example, the quality management office at one medical
center reviews medical records each month to determine whether
documentation indicates that inpatients were seen by supervising
physicians within 24 hours of admission. As shown in table 3, few
medical centers, however, described such a process for review of
supervisory documentation when residents provide consultations to
patients' primary physicians.
Table 3: Number of VA Medical Centers That Reported Monitoring Some
Aspect of the Documentation of Resident Supervision, by Domain of
Residents' Health Care Activities:
Explicit independent review process described[A]; Inpatient care: 77;
Outpatient care: 58; Diagnostic and therapeutic procedures: 65;
Consultations: 21.
No explicit independent review process described[B]; Inpatient care:
26; Outpatient care: 47; Diagnostic and therapeutic procedures: 42;
Consultations: 86.
Blank, missing, or reported to be not applicable; Inpatient care: 11;
Outpatient care: 9; Diagnostic and therapeutic procedures: 7;
Consultations: 7.
Total; Inpatient care: 114; Outpatient care: 114; Diagnostic and
therapeutic procedures: 114; Consultations: 114.
Source: VA.
Notes: GAO analysis of VA medical center 2000/2001 academic year annual
reports on resident supervision submitted to VA by June 18, 2002. We
considered the review process to be independent if the description
indicated that documentation is reviewed by someone other than a
physician with related supervisory responsibilities.
[A] Includes all descriptions of systematic independent review
processes of one or more aspects of documentation, as well as less
systematic review processes and reviews of only some services provided
by residents.
[B] Includes medical centers that stated that they had no process for
that domain or for which the description included insufficient
information to determine whether the process was independent and
systematic.
[End of table]
In addition, medical centers' annual reports did not always include
clear, detailed descriptions of the documentation requirements they
monitor. Few specifically mentioned monitoring particular VA-wide
requirements, such as the requirement for documentation of supervisory
involvement at the time of each new outpatient's first visit. In some
instances, medical centers described a less systematic review process
or one that was used for only some services provided by residents. For
diagnostic and therapeutic procedures, for example, some medical
centers described processes for reviewing only selected procedures,
such as endoscopies or major surgeries.[Footnote 23]
About half of the 91 medical centers that reported having an
independent review process indicated they found deficiencies with the
documentation of resident supervision, and all but one discussed
actions they took to correct these problems.[Footnote 24] For example,
officials from one medical center told us that they implemented a
program to discipline individual physicians who consistently do not
meet the medical center's requirements for documenting supervision. The
acting chief of staff there told us that during the 2001/2002 academic
year, three physicians had each been suspended without pay for 1 day
for not consistently meeting documentation requirements and that there
had been significant improvement in the documentation of resident
supervision since this disciplinary program went into effect. This
medical center has also developed a strategy for linking contract
physicians' pay to their provision and documentation of
supervision.[Footnote 25]
Documentation reviews have proven useful in identifying inadequate
supervision. We identified three medical centers that described in
their annual reports finding evidence of inadequate resident
supervision through their documentation reviews. In their annual
reports, two of these three medical centers stated that there were no
adverse patient events involving resident supervision. The third did
not state whether there had been any adverse patient outcomes. In the
first instance, the medical center reported that its review of
documentation indicated that some staff physicians provided a "low
level" of supervision to residents in the inpatient surgical setting.
Medical center officials responded by meeting with those physicians and
conducting a follow-up review to monitor the level of supervision. In
the second instance, the medical center reported that its supervision
of residents was generally satisfactory, but that it had found through
its documentation review one episode in which the attending surgeon had
left the city during a procedure that he was supposed to be
supervising. This medical center reported that the surgeon was formally
reprimanded. In the third instance, a medical center reported that
through its documentation review, it identified two specialties--
urology and plastic surgery--for which it wanted to increase the number
of procedures performed with the staff physician physically present and
directly involved in the surgery. The medical center reported that its
management was working with the surgery service chief to achieve this
goal.
We also identified a few medical centers that described independent
processes for monitoring resident supervision that went beyond
reviewing documentation. One medical center, for example, reported that
staff in its intensive care unit are required to report to the nurse
manager any situation they observe in which the supervision of a
resident was inappropriate.
In addition to monitoring processes established by medical centers,
five of VA's networks indicated in their 2000/2001 annual reports that
they had a networkwide process for assessing adherence to one or more
VA requirements for documentation of resident supervision. For example,
two networks stated that they monitor the documentation of supervising
physicians' involvement in the care of inpatients within 24 hours of
admission and another network assesses documentation of the supervision
of high-risk procedures. Two other networks reported they are
developing networkwide monitoring processes.
VA's Plans to Use External Peer Review to Monitor Documentation of
Supervision Have Not Been Finalized:
To obtain more complete information about the extent to which its
requirements for supervision are being followed, VA has begun to test
its plans to monitor adherence through external peer review of the
documentation of supervision. External peer reviewers would examine a
sample of medical records from each medical center involved in GME to
determine whether they include required documentation of
supervision.[Footnote 26] Although documentation does not provide full
information about the extent or quality of supervision, it can provide
VA oversight officials with important information about whether
supervisors were involved in patient care. We compared the instructions
that external reviewers would follow with the requirements for
supervision in VA's handbook and found that the instructions would
allow reviewers to assess adherence to most of VA's key documentation
requirements in the four domains of residents' health care activities.
For example, if a resident participated in the care of an inpatient or
an outpatient during the current academic year, the external reviewer
is to determine whether documentation of supervision in the patient's
medical record met the requirements in VA's national handbook.
Reviewers are also to assess documentation of the supervision of
residents who performed diagnostic or therapeutic procedures or
provided consultations to other physicians. Results from each medical
center are to be provided to that medical center, as well as to
headquarters managers.
External peer review of documentation of supervision in medical records
will be facilitated by features of VA's computerized patient record
system.[Footnote 27] For example, the system automatically records the
date and time of notes; it also has the capacity to require that notes
written by a resident be co-signed by the supervising physician, in
which case the note is not considered complete until the required
co-signature has been entered. In addition, supervising physicians with
whom we spoke noted that immediate and easy access to legible
information facilitates supervisors' review of residents'
activities.[Footnote 28]
VA is in the early stages of testing its procedures for external peer
review of the documentation of resident supervision, and a VA official
told us that this effort is a high priority. A pilot test of portions
of the inpatient assessment methodology was conducted from October 2001
through June 2002 on a sample of almost 10,000 medical records. That
pilot test indicated that the central database used to select the
sample of medical records does not include information about which
patients were seen by residents. As a result, reviewers were unable to
select an appropriate sample of medical records. Until this problem is
resolved, VA cannot implement its plans for external peer review of
resident supervision. OAA has worked with other headquarters offices to
revise VA's information technology software to ensure that this
database contains information about whether patients' physicians were
residents. VA expects to implement this revision to its software by
July 2003. The pilot test did not indicate any other obstacles to
implementing the portion of the plan for reviewing documentation of
resident supervision in inpatient settings. Pilot tests of methods for
assessing documentation of outpatient care, diagnostic and therapeutic
procedures, and consultations will not begin until patients seen by
residents can be clearly identified through the central database.
One unresolved issue that will affect the usefulness of the external
review of supervision documentation in the outpatient setting involves
selection of the sample of medical records. The two options under
consideration are relying on the main outpatient sample used for VA's
other external peer reviews or developing a sample specifically for
review of the documentation of supervision. The main outpatient sample
in any given year includes only patients who have received primary
health care from VA in the past and excludes most new patients who
began obtaining health care through VA within the preceding
year[Footnote 29]--a group that has greatly expanded in recent
years.[Footnote 30] Without a sample of records from new patients, it
will not be possible to assess adherence to VA's requirement for
supervisory involvement during a veteran's first outpatient visit. An
OAA official told us that developing an additional sample of outpatient
records for review of documentation of supervision, distinct from the
main outpatient sample used for other purposes, would add to the
expense of the review. As of May 2003, VA had not made a decision about
which sample to use.
VA Is Acting to Obtain Information about Supervision from Accrediting
Bodies and Residents:
VA is making efforts to obtain consistent access to information
provided by accrediting bodies and residents about the quality of
resident supervision in VA medical centers. VA has taken steps to gain
direct access to the letters accrediting bodies send to sponsoring
institutions to describe concerns about GME programs. VA headquarters
also developed a survey to obtain feedback from residents, but cannot
send it to a random sample of residents because VA does not have a
complete list of its residents. VA is improving its ability to obtain
that information. According to their annual reports for the 2000/2001
academic year, most VA medical centers that provide GME have some
procedure for obtaining feedback from residents.
VA Is Taking Steps to Gain Access to Accreditation Reviews of Its
Affiliates' GME Programs:
VA does not currently have direct access to accreditation letters that
contain reviews of the GME programs sponsored by VA medical centers'
affiliates. These letters document concerns about residents' education
or clinical experience that the GME program must address to retain
accreditation. Timely access to the information in these letters can
allow medical centers to take corrective actions. Until early 2000,
ACGME sent copies of its accreditation letters to OAA,[Footnote 31] and
OAA made VA support for residency slots contingent on VA medical
centers' taking action to correct identified problems. In 2000,
however, ACGME adopted new policies to safeguard confidential
accreditation information. As a result, ACGME stopped sending the
letters to VA, instead sending these letters only to the institution
that sponsors the GME program.
Without direct access to ACGME accreditation letters, VA medical
centers are dependent on sponsoring institutions to inform them of
concerns about the GME programs in which VA participates, and we
learned of one instance in which a sponsoring institution did not do so
when ACGME notified it of problems. Officials from a medical center
told us that the sponsoring institution of a thoracic surgery program
did not tell them that ACGME had previously identified multiple
problems with the program until ACGME decided, in September 2002, to
withdraw the program's accreditation. ACGME did not cite any problems
with the VA rotation. Nonetheless, unanticipated withdrawal of a
program's accreditation can affect a medical center's educational and
patient care missions. In this case, the VA medical center will lose
one full-time advanced surgical resident in July 2003 and had to hire a
physician's assistant to provide some of the services that had been
provided by the resident.
Most medical centers indicated in their 2000/2001 annual reports that
their GME sponsors had shared information from accreditation letters,
and these annual reports provided network and headquarters officials
with information about accrediting bodies' concerns and medical
centers' corrective actions. Fifty-six medical centers stated that
accrediting bodies had identified concerns about VA rotations in 145 of
the more than 1,900 GME programs in which VA is involved. Concerns
about 17 of these programs related to resident supervision.[Footnote
32] For example, according to one medical center's annual report, ACGME
concluded that residents required more direct supervision during
certain oncology rotations. Medical centers reported that they had
taken corrective action in response in all but one instance. In this
case, the accrediting body expressed concern that the VA medical center
had provided inadequate supervision and teaching in its physical
medicine and rehabilitation rotation, but the medical center did not
describe a corrective action in its annual report.
We found that when OAA had direct access to ACGME accreditation
letters--through early 2000--it took action to ensure that VA medical
centers knew of and responded to ACGME concerns about VA rotations. Our
review of OAA's correspondence about accreditation issues covering a
period from late 1998 through early 2000 indicated that ACGME mentioned
concerns that were specific to VA rotations in its letters about 17 GME
programs. In 6 of these cases, ACGME cited a concern about the adequacy
of resident supervision. For example, ACGME determined that
ophthalmology residents at one VA medical center had not been given
clear information about lines of supervisory responsibility. On receipt
of these letters, OAA contacted the participating VA medical center.
Three of the medical centers submitted documents to substantiate a
resolution to the problem within 2 months of hearing from OAA. In the
other three cases, OAA asked VA's chief consultant for the relevant
medical specialty (such as the Chief Consultant for Ophthalmology) to
assess the situation. In each case, the consultant reported to OAA that
a resolution had been achieved. For example, the consultant reported
that the ophthalmology program cited for unclear lines of supervision
was preparing a written document to clarify supervisory
responsibilities.
OAA has taken steps to arrange for renewed direct access to ACGME
accreditation letters. As part of that effort, VA issued a revised
policy on confidential documents in July 2002 to make sure that
accreditation reviews would be treated confidentially. In February
2003, VA signed a memorandum of understanding with ACGME that lays the
foundation for OAA to receive copies of accreditation letters.
According to this memorandum, VA must now obtain revised affiliation
agreements between VA medical centers and GME sponsors that authorize
ACGME to provide OAA with its accreditation letters. VA is taking steps
to ensure that these revised agreements will be in place by July 2004.
OAA has come to a similar agreement with the American Osteopathic
Association.
As a further step to obtain information about, and monitor responses
to, GME issues--including accreditation concerns--OAA reissued a policy
requiring VA medical centers to establish an affiliation partnership
council and submit minutes of council meetings to OAA.[Footnote 33] The
council is to include representatives of the medical center and its
academic affiliate or affiliates and is to advise VA managers as they
work to meet educational accreditation requirements and correct
deficiencies or resolve problems.
VA Is Improving Its Ability to Obtain Feedback from a Representative
Group of Residents:
A mechanism OAA uses to obtain standardized information about
residents' views on the quality of their supervision and other aspects
of their training is its Learners' Perceptions Survey, which was first
distributed in March 2001.[Footnote 34] The survey asks residents to
indicate their satisfaction with the supervision they received from VA
faculty by rating supervising physicians' teaching ability,
accessibility/availability, and approachability/openness, as well as
overall satisfaction with VA clinical faculty. Residents are also asked
to evaluate their satisfaction with the degree of supervision and
degree of autonomy they experienced.
In 2001 and 2002, VA headquarters could not send the survey to a
random, representative sample of residents from each of its medical
centers involved in GME because it did not have a complete list of its
trainees. OAA was able to obtain feedback from many residents who did
receive the survey[Footnote 35] and gave those results to medical
centers and networks. OAA is taking steps to capture each trainee's
name and address in its automated and centrally accessible information
system and expects to implement this procedure in July 2003. Once VA
has a full registry of its trainees, OAA plans to send the survey to a
representative sample of residents in different medical specialties
that will include residents from all VA medical centers involved in
GME.
Medical centers' annual reports can provide network and headquarters
officials with additional information about concerns expressed by
residents and steps taken to address those concerns. According to the
annual reports for the 2000/2001 academic year, most VA medical centers
used VA's nationwide Learners' Perceptions Survey or another mechanism,
such as residents' confidential evaluations obtained by sponsoring
institutions, to obtain feedback about supervision. About half of the
109 medical centers whose annual reports indicate that they had a
process for obtaining residents' feedback said that residents had
concerns about their VA rotations. None of these concerns, however,
involved the adequacy of supervision.
VA Uses Its Programs for Monitoring Patient Care to Identify and
Correct Problems with Resident Supervision:
VA headquarters, network, and medical center officials use information
from VA's programs for monitoring the quality and outcomes of patient
care to identify and correct problems with resident supervision. VA's
monitoring programs include its new Patient Safety Program and NSQIP.
Reviews of paid tort claims by VA's Chief Patient Care Officer provide
another mechanism for identifying problems with resident supervision.
OAA monitors medical centers' use of these programs through the annual
reports on residency training. In their annual reports for the 2000/
2001 academic year, most medical centers indicated that they monitor
patient care information to determine whether resident supervision
affected the quality or outcomes of patient care.
The system for reporting adverse events and close calls established by
VA's Patient Safety Program has the potential to capture information
about instances in which inadequate resident supervision contributed to
heightened risk of adverse health care outcomes. Based on analysis of
the 17,000 reports of adverse events and close calls filed with VA's
National Center for Patient Safety as of April 2002, its director
estimated that resident supervision was mentioned--in any context--in
less than 0.1 percent of the incidents reported by VA medical centers
and that inadequate supervision was a causal factor in very few of
those cases.[Footnote 36]
Analyses of postoperative outcomes recorded in the NSQIP database,
including mortality and morbidity, provide VA with a way to study the
effects of residents' involvement in surgical procedures. NSQIP
personnel analyze nationwide data from major surgeries, provide site-
specific reports to medical centers and networks, and conduct site
visits at medical centers.[Footnote 37] A NSQIP official told us that
these data are routinely examined for signs that supervision of
residents might be inadequate. For example, NSQIP analysts review the
data to ensure that residents are not performing surgeries that are
more advanced than would be appropriate for their level of training. In
addition to reviewing NSQIP reports, headquarters officials who oversee
VA's surgical services monitor the frequency with which supervising
physicians are in the operating or procedural suite when residents
perform surgeries.[Footnote 38]
Medical center and network officials have used NSQIP reports to help
monitor resident supervision. For example, a team of experts selected
by NSQIP visited one medical center at its request in February 2002 to
help it evaluate the efficiency of its operating rooms. During its
visit, the team noted inadequate supervision of surgeries performed by
urology residents.[Footnote 39] The medical center corrected this
problem by arranging for urologists to spend more time at the medical
center and ensuring that they understood VA's requirements for
supervision. In another instance, a network GME manager observed that
NSQIP data indicated that orthopedic surgery outcomes at a particular
medical center were less favorable than expected. After a site visit,
network officials concluded that the medical center could not support
complex surgeries and determined that continued training of orthopedic
residents at that medical center would require a decrease in the
complexity of cases and greater involvement by supervising physicians.
When the sponsoring institution decided that the medical center would
not meet its training needs under those conditions, VA officials chose
to transfer patients with complex surgical needs to VA's tertiary
hospitals in the network and shift its two VA-funded residency slots in
orthopedic surgery to a different VA medical center.
Researchers using the NSQIP database have studied ways in which
participation in GME affects postoperative outcomes. To determine
whether residency training places surgical patients at risk for worse
outcomes, researchers using the NSQIP database[Footnote 40] compared
risk-adjusted mortality rates in VA's teaching and nonteaching
hospitals and found that they did not differ, although the patients who
underwent surgeries at teaching hospitals had a higher prevalence of
risk factors, underwent more complex operations, and had longer
operation times. Morbidity rates were higher in teaching than
nonteaching hospitals for some surgical specialties that were
studied.[Footnote 41] On the basis of their analyses, the authors
suggested that differences in morbidity rates could reflect incomplete
adjustment for risks, such as severity of illness, or the more complex
systems of managing and coordinating care that characterize teaching
hospitals, and not necessarily the involvement of residents. Another
study begun in September 2001 is designed to use the NSQIP database to
clarify the relationship between residents' working conditions and
surgical outcomes, with data from 90 VA hospitals and 3 nonfederal
hospitals in which surgical residents are trained.
Tort claims also provide information that VA uses to identify problems
with resident supervision that affected patient care. Review of paid
tort claims by VA's Chief Patient Care Services Officer resulted in
clarification of VA's written requirements for resident supervision
when patients are admitted to inpatient units. In the specific case
that led to this change, a supervising physician did not come to the
hospital during a weekend to see a patient who had been admitted by a
resident; the patient died on Monday. At that time, the resident
supervision policy of the VA hospital in which the incident occurred
did not specifically require supervising physicians to come in on
weekends. As a result of this case, in October 2001 an explicit
reference to weekends and holidays was added to the handbook's
requirement that each new inpatient be seen by the supervising
physician within 24 hours of admission.
OAA monitors incidents in which resident supervision contributed to
adverse events or patient risks through the annual reports it requires
from medical centers. In their 2000/2001 annual reports on residency
training, all but 11 of 114 medical centers indicated that they
monitored patient safety events associated with residents.[Footnote 42]
They used a variety of processes to collect this information, including
root cause analyses and tort claim reviews, as well as additional
processes such as mortality and morbidity conferences and reviews
triggered by unexpected events, such as readmission within 10 days of
discharge from the medical center. Annual reports indicated that
reviews of at least 18 actual or potential adverse patient outcomes at
a total of 14 medical centers identified resident supervision as a
possible contributing factor or led medical center officials to
strengthen supervision to minimize the chance of future problems. For
example, one medical center established a requirement for greater
involvement by supervising physicians before a resident initiates
chemotherapy orders. Medical centers described taking corrective
actions in response to these reviews.
Conclusions:
VA cannot assure that the residents who provide care in its facilities
receive adequate supervision because its current procedures for
monitoring supervision are insufficient. To oversee the supervision of
its residents, VA needs various types of information, including
information regarding supervising physicians' adherence to VA's
requirements for resident supervision, accrediting bodies' and
residents' concerns about supervision, and whether the quality or
outcomes of patient care indicate problems with supervision. Systematic
monitoring of each of these types of information would help ensure that
problems with resident supervision are detected and corrected by the
various officials of VA medical centers and affiliated institutions who
have responsibilities for residents' activities.
Although VA issued a handbook that established specific standards for
resident supervision, VA does not know what its medical centers'
supervision requirements are and does not ensure that its national
requirements are adopted at each medical center where residents train.
Moreover, VA does not know whether the supervision its residents
receive adheres to its national requirements. VA's current plans for
external peer review of documentation have the potential to enhance its
oversight capability, but these plans have not been finalized. For
example, as of May 2003, VA had not decided whether external reviewers
would examine documentation of supervision for VA's new outpatients,
who make up a significant and growing number of VA's patients.
Including these new outpatients in the external review could help
ensure adequate supervision of residents during a patient's first visit
to VA.
To further improve its oversight of resident supervision, VA will need
to complete its initiatives to obtain timely access to evaluations by
accrediting bodies and residents. VA will also need to continue to take
advantage of its programs for monitoring the quality and outcomes of
patient care. VA officials have generally acted to improve supervision
when faced with evidence of problems, and better access to information
will enhance their ability to monitor and improve resident supervision.
By strengthening its oversight capabilities, VA could help promote both
the quality of the health care in its facilities and the education its
residents receive. As the largest provider of residency training sites
in the United States, VA's actions to enhance the quality of resident
supervision and its oversight will have benefits beyond the VA health
care system.
Recommendations for Executive Action:
We recommend that the Secretary of Veterans Affairs direct the Under
Secretary for Health to take steps to improve VA's oversight of the
supervision of residents by:
* ensuring that all VA medical centers that provide GME adopt and
adhere to the requirements for resident supervision established in VA's
handbook and:
* ensuring that external peer review of documentation of resident
supervision includes examination of records from VA's new outpatients.
Agency Comments:
In written comments on a draft of this report, VA agreed with our
findings and our recommendations. VA said our report described many
steps it has already taken that would help assure systematic
implementation of its national resident supervision policies and
adequate headquarters oversight of resident supervision. In concurring
with our recommendation to ensure that all VA medical centers that
provide GME adopt and adhere to requirements for resident supervision
established in its handbook, VA indicated its intention to monitor
compliance with policy requirements and highlight those requirements in
a memorandum to network officials. In concurring with our
recommendation to ensure that external peer review of documentation of
resident supervision includes examination of records from its new
outpatients, VA indicated that it would develop a strategy to identify
new outpatients who were seen by a resident. It stated that it expects
to draw its first sample of records from outpatients, including new
outpatients, in the second quarter of fiscal year 2004. VA also
reported that it completed a revision of its centralized patient
information database. This revision was necessary to allow selection of
an appropriate sample of inpatient records for external peer review.
VA's comments are in appendix II.
We are sending copies of this report to the Secretary of Veterans
Affairs, appropriate congressional committees, and other interested
parties. We will also make copies available to others who are
interested upon request. In addition, the report will be available at
no charge on the GAO Web site at http://www.gao.gov.
If you or your staff have any questions, please contact me at (202)
512-7101. An additional contact and the names of other staff members
who made contributions to this report are listed in appendix III.
Cynthia A. Bascetta
Director, Health Care--Veterans' Health and Benefits Issues:
Signed by Cynthia A. Bascetta:
[End of section]
Appendix I: Scope and Methodology:
To do our work, we examined oversight of resident supervision at each
of the Department of Veterans Affairs (VA) Veterans Health
Administration's three organization levels--headquarters, networks,
and medical centers. Our work covered VA's oversight of resident
supervision and did not include an evaluation of the quality of care
provided by residents or the quality of the supervision provided to
residents. To assess oversight by VA's headquarters officials, we
reviewed documents and interviewed officials from VA's Office of
Academic Affiliations (OAA), Office of Patient Care Services, National
Center for Patient Safety, Office of Quality and Performance, and
Office of Information. We analyzed VA's plans to have external peer
reviewers examine documentation of supervision and compared the
instructions the reviewers are to be given with VA's requirements for
supervision.
To assess oversight of resident supervision by network officials, we
analyzed each network's annual report to OAA on resident supervision
covering the 2000/2001 academic year.[Footnote 43] These were the most
recent annual reports available at the time. We did not assess the
accuracy of information provided in these reports. We also interviewed
network GME managers (known as network academic affiliations officers)
from a sample of 11 of VA's 21 regional networks of health care
facilities and analyzed documents they provided (see table 4). We used
a stratified random sampling strategy to ensure variation in the number
of VA-funded residency slots among the selected networks.[Footnote 44]
Network 19 was included in our sample prior to randomization because it
is the only network that did not summarize the information in its
medical centers' reports. Another network was excluded from our sample
because it had been formed by the merger of two former networks in
January 2002. Our results from these 11 networks cannot be generalized
to other networks.
Table 4: VA Networks Included in Our Sample:
Network: 1 (Boston); Number of VA-funded residency slots[A]: 501.43.
Network: 3 (Bronx); Number of VA-funded residency slots[A]: 603.00.
Network: 6 (Durham); Number of VA-funded residency slots[A]: 348.30.
Network: 10 (Cincinnati); Number of VA-funded residency slots[A]:
255.90.
Network: 11 (Ann Arbor); Number of VA-funded residency slots[A]: 314.00.
Network: 15 (Kansas City); Number of VA-funded residency slots[A]:
339.00.
Network: 16 (Jackson); Number of VA-funded residency slots[A]: 671.95.
Network: 18 (Phoenix); Number of VA-funded residency slots[A]: 305.70.
Network: 19 (Denver); Number of VA-funded residency slots[A]: 230.00.
Network: 21 (San Francisco); Number of VA-funded residency slots[A]:
383.02.
Network: 22 (Long Beach); Number of VA-funded residency slots[A]: 729.50.
Source: VA.
[A] The number of VA-funded residency slots allocated to networks
during the 2001/2002 academic year ranged from 195.00 to 729.50.
[End of table]
To assess oversight of resident supervision by medical center
officials, we reviewed and analyzed 2000/2001 academic year annual
reports to OAA on resident supervision. OAA provided us with 114 annual
reports from the approximately 130 VA medical centers that were
involved in GME during the 2000/2001 academic year after it removed
identifying information, such as the names of medical centers,
affiliates, and specific individuals. These were all the medical center
annual reports for the 2000/2001 academic year that OAA had received as
of June 18, 2002. We did not assess the accuracy of information in the
annual reports. We also interviewed GME managers at 11 VA medical
centers (see table 5) and analyzed their 2000/2001 academic year annual
reports on resident supervision (without redaction) and other
documents. We used a stratified random sampling strategy to ensure that
the medical centers we selected varied in the number of VA-funded
residency slots they were allocated for the 2001/2002 academic
year.[Footnote 45] We also ensured that our sample included one medical
center from each of the networks we had sampled and that the medical
centers differed in the number of medical specialties in which their
residents train. We did not review a systematically selected sample of
medical centers' resident supervision policies. Our results from these
11 medical centers cannot be generalized to other medical centers.
Table 5: VA Medical Centers Included in Our Sample:
VA medical center: White River Junction, Vt.; Network: 1 (Boston);
Number of VA-funded residency slots[A]: 39.70; Number of medical
specialties[B]: 14.
VA medical center: New York, N.Y.; Network: 3 (Bronx); Number of
VA-funded residency slots[A]: 135.00; Number of medical specialties[B]:
25.
VA medical center: Hampton, Va.; Network: 6 (Durham); Number of
VA-funded residency slots[A]: 45.00; Number of medical specialties[B]:
7.
VA medical center: Cleveland, Ohio; Network: 10 (Cincinnati); Number of
VA-funded residency slots[A]: 112.40; Number of medical specialties[B]:
23.
VA medical center: Detroit, Mich.; Network: 11 (Ann Arbor); Number of
VA-funded residency slots[A]: 79.00; Number of medical specialties[B]:
26.
VA medical center: St. Louis, Mo.; Network: 15 (Kansas City); Number of
VA-funded residency slots[A]: 120.00; Number of medical specialties[B]:
24.
VA medical center: Biloxi, Miss.; Network: 16 (Jackson); Number of
VA-funded residency slots[A]: 10.40; Number of medical specialties[B]:
6.
VA medical center: Tucson, Ariz.; Network: 18 (Phoenix); Number of
VA-funded residency slots[A]: 93.01; Number of medical specialties[B]:
21.
VA medical center: Salt Lake City, Utah; Network: 19 (Denver); Number
of VA-funded residency slots[A]: 110.50; Number of medical
specialties[B]: 25.
VA medical center: Fresno, Calif.; Network: 21 (San Francisco); Number
of VA-funded residency slots[A]: 42.00; Number of medical
specialties[B]: 4.
VA medical center: Long Beach, Calif.; Network: 22 (Long Beach); Number
of VA-funded residency slots[A]: 158.50; Number of medical
specialties[B]: 28.
Source: VA.
[A] The number of VA-funded residency slots allocated to medical
centers involved in GME for the 2001/2002 academic year ranged from
0.60 to 218.00.
[B] The number of distinct medical specialties in which VA medical
centers had residency slots during the 2001/2002 academic year ranged
from 1 to 32.
[End of table]
We also reviewed documentary and testimonial evidence from four medical
centers that participate in internal medicine or general surgery GME
programs that had received adverse accreditation decisions as of May
2002.[Footnote 46] One of these--the Fresno VA Medical Center--was part
of our sample of medical centers. Of the others, we visited the medical
centers in West Haven, Connecticut and Gainesville, Florida and
interviewed officials of the medical center in Albuquerque, New Mexico.
We also spoke to officials of the institutions that sponsor these three
GME programs.
To obtain additional information about GME and VA's residency training,
we analyzed accreditation requirements of the Accreditation Council for
Graduate Medical Education, American Osteopathic Association, and Joint
Commission on Accreditation of Healthcare Organizations and interviewed
officials of those bodies. We also interviewed representatives of
professional associations that are involved in GME, including the
American Board of Medical Specialties, American College of Surgeons,
American Hospital Association, American Medical Association, American
Medical Student Association, Association of American Medical Colleges
and its Council of Deans, Association of Professors of Medicine,
Committee of Interns and Residents, and Council on Graduate Medical
Education, and we reviewed relevant documents issued by these groups.
We interviewed representatives of physicians who teach internal
medicine, ophthalmology, psychiatry, general surgery, orthopedic
surgery, and urology--specialties for which a large number of VA
medical centers provide residency slots. We also interviewed
representatives of veterans' service organizations. We reviewed
published literature regarding the quality of care provided by
residents.
We conducted our work from September 2001 through June 2003 in
accordance with generally accepted government auditing standards.
[End of section]
Appendix II: Comments from the Department of Veterans Affairs:
THE SECRETARY OF VETERANS AFFAIRS WASHINGTON:
June 11, 2003:
Ms. Cynthia A. Bascetta Director, Health Care-Veterans' Health and
Benefits Issues:
U. S. General Accounting Office 441 G Street, NW Washington, DC 20548:
Dear Ms. Bascetta:
The Department of Veterans Affairs (VA) has reviewed your draft report,
VA HEALTH CARE: Adequacy of Resident Supervision Is Not Assured, but
Plans Could Improve Oversight (GAO-03-625) and agrees with your
findings and concurs with your recommendations.
The Veterans Health Administration (VHA) has taken steps prior to the
conclusion of your evaluation to assure systematic implementation of
national resident policies and adequate levels of centralized
oversight. The body of the report reflects numerous examples of actions
VA has taken to correct identified deficiencies.
VHA has made additional progress since the time the General Accounting
Office's (GAO) evaluators completed their data collection. For example,
VHA has just completed a major revision of its computerized patient
information database, VISTA (Veterans Health Information Systems and
Technology Architecture). VISTA now includes enhancements that support
new sampling methods to assess inpatient evaluations that residents
conduct.
As the attached action plan details, VHA continues to build upon
existing strengths through ongoing improvements. Issuance of the VHA
Handbook on Resident Supervision, creation of an annual reporting
mechanism, and the introduction of central oversight through the
External Peer Review Program (ERRP) are major recent improvements in
resident supervision. In fact, VHA's Resident Supervision Handbook has
been used as a model by many academic affiliates because of its
attention to the educational process and the insistence on the goal of
quality patient care. VHA's new process of external peer review of
resident supervision is also likely to drive national improvements in
education of physicians and other health professionals. VHA has also
developed an outpatient module for the EPRP review of ongoing resident
care, and is now developing a sampling strategy to identify new
outpatients.
The enclosure outlines specific actions VHA has already taken as well
as future plans to improve management of its resident supervision.
Thank you for the opportunity to comment on your draft report.
Sincerely yours,
Signed by: Anthony J. Principi:
Enclosure:
Enclosure:
Department of Veterans Affairs Comments to GAO Draft Report, VA HEALTH
CARE: Adequacy of Resident Supervision Is Not Assured, but Plans Could
Improve Oversight (GAO-03-625):
GAO recommends that the Secretary of Veterans Affairs direct the Under
Secretary for Health to take steps to improve VA's oversight of the
supervision of residents by:
* ensuring that all VA medical centers that provide GME adopt and adhere to the requirements for resident supervision established in
VA's handbook and:
Concur-The Veterans Health Administration (VHA) has established a goal
to assure systemwide implementation of resident supervision policies.
To achieve this goal, VHA conducts centralized data collection that
provides a nationwide assessment of policy implementation. These
assessments monitor compliance with policy requirements, enable regular
data reviews, and provide feedback. Also, in July 2003, VHA will
communicate its national policy requirements to facility leadership
through network directors/chief medical officers via a senior level
memo transmitting GAO's report. Also, VHA will emphasize the importance
of local leadership in managing its Graduate Medical Education program.
* Ensuring that external peer review of documentation of resident
supervision includes examination of records from VA's new outpatients:
Concur-To implement this recommendation, the Department will initiate
required changes in VISTA. VA is developing information technology (IT)
requirements necessary to identify resident-provided care for new
outpatients. It is anticipated that any necessary IT software patches
to enable sampling of new outpatients will be released by the second
quarter, fiscal year 2004. Further, VHA is developing a sampling
strategy to identify new patient records for review and anticipates
drawing its first sample in the second quarter, fiscal year 2004.
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
Helene F. Toiv, (202) 512-7162:
Staff Acknowledgments:
In addition to the person named above, key contributors to this report
were Kristen J. Anderson, William D. Hadley, Martha Fisher, Krister
Friday, and Donald Morrison.
FOOTNOTES
[1] In this report, the term "residents" also refers to fellows,
physicians who have already completed a residency and are obtaining
additional training in an advanced specialty or subspecialty.
[2] VA requires that the supervisor be a licensed physician who has
been credentialed and privileged as a member of the staff of the
medical facility in which the care is provided.
[3] See U.S. General Accounting Office, VA Hospitals: Surgical
Residents Need Closer Supervision, GAO/HRD-86-15 (Washington, D.C.:
Jan. 13, 1986) and VA Health Care: Medical Centers Are Not Correcting
Identified Quality Assurance Problems, GAO/HRD-93-20 (Washington,
D.C.: Dec. 30, 1992).
[4] VA health care facilities are organized into 21 regional networks,
known as Veterans Integrated Service Networks, which are to coordinate
the activities of and allocate funds to VA health care facilities. VA
had 22 networks until January 2002, when it merged two of them.
[5] The number of VA medical centers that provide GME varies slightly
over time, in part because at facilities with only a few allocated
residency slots, it is possible that no slot might be filled at a
particular time.
[6] In addition to these three programs, one program at 1 of the 11
medical centers in our sample was on probationary accreditation. Of the
four programs that had received adverse accreditation decisions, two
are no longer in an adverse accreditation status. Reevaluation of the
other two was not complete as of May 2003.
[7] VA's four core missions are patient care, education, research, and
medical backup to the Department of Defense in the event of a national
security emergency.
[8] These expenditures included stipends and benefits for residents
training in accredited medical specialties and subspecialties and an
additional 150 special fellows training in emerging, as yet
nonaccredited fields of medicine, such as geriatric neurology and
palliative care.
[9] Allocations of fractions of slots are possible because residents
might obtain only a part of their training at a VA medical center.
[10] Adverse events include adverse drug events and procedural errors
or complications that are associated with care. Close calls are events
or situations that could have resulted in an adverse event but did not,
either by chance or through timely intervention. VA specifies that
alternative procedures are to be used for reporting intentionally
unsafe acts.
[11] VA arranged for the National Aeronautics and Space Administration
to provide an independent external system for reporting patient safety
concerns. This system allows anyone who feels uncomfortable reporting
an event to VA's internal patient safety managers to file a voluntary,
confidential report to an outside agency. Reports entered in this
database are anonymous. Nationwide implementation of this second
reporting system began in March 2002.
[12] NSQIP is housed administratively in VA's Office of Patient Care
Services. It exercises its monitoring and advisory functions through
the chief medical officers of VA's networks.
[13] In April 2003, VA's Office of Inspector General reported that
part-time physicians were not always present in the clinics where the
residents they supervised provided care. See VA Office of Inspector
General, Audit of the Veterans Health Administration's Part-Time
Physician Time and Attendance, 02-01339-85 (Washington, D.C.: April
2003).
[14] Medical centers must adopt their own policies to ensure that local
requirements, such as those established by affiliated GME sponsors, are
included.
[15] These 29 GME programs are sponsored by seven VA medical centers,
each of which also participates in GME programs that are sponsored by
affiliated institutions.
[16] ACGME classifies certain records as confidential to foster candor
by residency programs, residents, and others as they submit information
during the accreditation process.
[17] The most recent revision of the handbook was issued on October 25,
2001.
[18] VA's requirements for the minimum level of supervision for
diagnostic and therapeutic procedures do not apply to procedures
performed in emergency situations, in which immediate action is
necessary to save a patient's life or prevent serious impairment of the
patient's health, or to procedures that are elements of routine and
standard patient care, such as drainage of superficial abscesses.
[19] These managers are known as network academic affiliations
officers.
[20] An official of this medical center told us in September 2002 that
there had been no adverse patient outcomes associated with resident
supervision during the preceding 2 years.
[21] An OAA official told us that OAA does not require medical centers
or networks to conduct comprehensive documentation reviews to avoid
duplicating the cost and effort VA headquarters is expending to develop
a plan for systemwide external peer review of supervision
documentation. This plan will be addressed in the next section of this
report.
[22] OAA provided us with annual reports from 114 of the approximately
130 medical centers that were allocated VA-funded residency slots
during the 2000/2001 academic year. These were all the medical center
annual reports for the 2000/2001 academic year OAA had received as of
June 18, 2002. Before giving these reports to us, OAA redacted them to
remove identifying information such as the names of medical centers and
sponsoring institutions. We analyzed these reports to determine whether
the medical centers described a systematic, independent review of the
documentation of resident supervision in a sample of medical records.
[23] Seven of the 11 medical center GME managers we interviewed told us
that since preparing their 2000/2001 annual reports, their medical
centers have implemented or are developing additional reviews of the
documentation of supervision. For example, one medical center that had
not reviewed documentation of resident supervision in inpatient
settings during the 2000/2001 academic year began reviewing that
documentation on a quarterly basis during the 2001/2002 academic year.
[24] Insufficient documentation does not necessarily indicate a lack of
supervision. For example, some medical centers reported that
supervision was documented, but not in a way that met VA's
requirements, and others reported that interviews with staff indicated
that appropriate supervision had occurred, although documentation was
lacking.
[25] In addition to employing salaried physicians, VA medical centers
sometimes use contracts to obtain the services of medical specialists.
[26] As part of its broader quality management process, VA began its
External Peer Review Program in 1995. Through this program, trained
reviewers from outside VA examine documentation from a sample of
medical records from each medical center to determine whether specific
health care activities, such as influenza immunization, have occurred.
These data have allowed VA to monitor its progress in meeting specific
health care objectives.
[27] The core features of VA's computerized patient record system,
which was developed to support its health care mission, have been
installed at all VA medical centers, although medical centers differ in
the extent to which it is used. External reviewers will review either
electronic or paper records, whichever are available.
[28] The Association of American Medical College's Joint Committee of
the Group on Resident Affairs and Organization of Resident
Representatives has reported that computerized medical records can
enhance the safety of patient care in teaching hospitals.
[29] VA told us that it excludes new patients from its main outpatient
sample to facilitate comparison of its performance measures to those
from the National Committee for Quality Assurance's Health Plan
Employer Data and Information Set, which collects data from private-
sector patients who have been enrolled in a health plan for two
consecutive years.
[30] From fiscal year 1996 to fiscal year 2002, the number of patients
who received health care from VA increased from about 2.9 million to
4.7 million.
[31] During the time when OAA received copies of ACGME's accreditation
letters, OAA did not have direct access to accreditation letters from
the American Osteopathic Association, which accredits a small number of
the GME programs in which VA medical centers participate.
[32] The annual reports indicated that most concerns noted by GME
accrediting bodies did not involve resident supervision, but instead
involved other problems, such as insufficient ancillary staff or
inadequate rooms where residents can rest while they are on-call in the
medical center.
[33] By reissuing this policy, OAA reasserted its requirement for
submission of minutes, which it had not consistently enforced in recent
years.
[34] VA's Learners' Perceptions Survey is designed to obtain
information about the perceptions of all trainees who work within the
VA system, including residents, student nurses, and psychology interns.
Data from this survey are used to assess VA's systemwide performance
measure involving trainees' ratings of their VA educational experience.
In addition to GME, VA provides training in more than 40 associated
health disciplines.
[35] During 2001, surveys were sent to 3,338 residents and returned by
1,775. During 2002, surveys were sent to 6,084 residents and returned
by 2,622.
[36] We did not independently verify this estimate.
[37] Each VA medical center that performs major surgeries receives an
annual report that reports its mortality and morbidity outcomes,
adjusted for risk factors, in comparison to VA's other medical centers,
along with suggestions for improvement. Networks also receive these
reports. In addition, a team of experts visits medical centers with
mortality rates that are consistently higher than expected to identify
problems and recommend improvements.
[38] The computer software used in VA medical centers for recording
information about surgical procedures allows the generation of hospital
reports that indicate the level of supervision provided for surgical
procedures. Quarterly reports submitted to the Surgical Service at VA
headquarters also include this information.
[39] There was no evidence that any adverse patient safety events
resulted from inadequate supervision of urology residents.
[40] Shukri F. Khuri and others, "Comparison of Surgical Outcomes
Between Teaching and Nonteaching Hospitals in the Department of
Veterans Affairs," Annals of Surgery, vol. 234, no. 3 (2001).
[41] NSQIP defines morbidity as the occurrence of any one or more of 20
specific postoperative adverse events such as deep wound infection,
pneumonia, or stroke within 30 days of the operation. Morbidity rates
were higher in teaching than nonteaching hospitals for general surgery,
orthopedics, urology, and vascular surgery, but did not differ
significantly for otolaryngology, neurosurgery, or thoracic surgery.
[42] Medical centers that did not describe a process for monitoring
patient safety events that involve residents either left the section on
patient safety events blank or did not describe systematic review
processes that are specific to incidents involving residents.
[43] These annual reports included separate reports from two networks
that were merged in 2002.
[44] Numbers of VA-funded residency slots were based on allocations for
the 2001/2002 academic year.
[45] We excluded medical centers that received an allocation of 10 or
fewer VA-funded residency slots or with fewer than three separate GME
programs during the 2001/2002 academic year from our sampling set,
resulting in a possible set of 97 medical centers.
[46] Two of these programs are no longer under an adverse accreditation
status. Reevaluation of the other two programs was not complete as of
May 2003.
GAO's Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office
441 G Street NW,
Room LM Washington,
D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.
General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.
20548:
