Child Welfare
Most States Are Developing Statewide Information Systems, but the Reliability of Child Welfare Data Could Be Improved Gao ID: GAO-03-809 July 31, 2003To better monitor children and families served by state child welfare agencies, Congress authorized matching funds for the development of statewide automated child welfare information systems (SACWIS) and required that the Department of Health and Human Services (HHS) compile information on the children served by state agencies. This report reviews (1) states' experiences in developing child welfare information systems and HHS's role in assisting in their development, (2) factors that affect the reliability of data that states collect and report on children served by their child welfare agencies and HHS's role in ensuring the reliability of those data, and (3) practices that child welfare agencies use to overcome challenges associated with SACWIS development and data reliability.
HHS reported that 47 states are developing or operating a SACWIS, but many continue to face challenges developing their systems. Most state officials said they recognize the benefit their state will achieve by developing SACWIS, such as contributing to the timeliness of child abuse and neglect investigations; however, despite the availability of federal funds since 1994, states reported a median delay of 2 and a half years beyond the timeframes they set for completion. States reported that they encountered some difficulties during SACWIS development, such as challenges receiving state funding and creating a system that reflected their work processes. In response to some of these challenges, HHS has provided technical assistance to help states develop their systems and conducted on-site reviews of SACWIS to verify that the systems meet federal requirements. Despite efforts to implement comprehensive information systems, several factors affect the states' ability to collect and report reliable adoption, foster care, and child abuse and neglect data. States responding to GAO's survey and officials in the 5 states GAO visited reported that insufficient caseworker training and inaccurate and incomplete data entry affect the quality of the data reported to HHS. In addition, states reported technical challenges reporting data. Despite HHS's assistance, many states report ongoing challenges, such as the lack of clear and documented guidance on how to report child welfare data. In addition, although states were mandated to begin reporting data to the Adoption and Foster Care Analysis and Reporting System (AFCARS) in 1995, few reviews of states' AFCARS reporting capabilities have been conducted to assist states in resolving some of their reporting challenges. Some states are using a variety of practices to address the challenges associated with developing SACWIS and improving data reliability. For example, 44 states included caseworkers and other system users in the design and testing of SACWIS, and 28 states reported using approaches to help caseworkers identify and better understand the data elements that are required for federal reporting.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-03-809, Child Welfare: Most States Are Developing Statewide Information Systems, but the Reliability of Child Welfare Data Could Be Improved
This is the accessible text file for GAO report number GAO-03-809
entitled 'Child Welfare: Most States Are Developing Statewide
Information Systems, but the Reliability of Child Welfare Data Could Be
Improved' which was released on August 12, 2003.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
On November 16, 2003, this document was revised to add various
footnote references missing in the text of the body of the document.
Report to Congressional Requesters:
United States General Accounting Office:
GAO:
July 2003:
Child Welfare:
Most States Are Developing Statewide Information Systems, but the
Reliability of Child Welfare Data Could Be Improved:
GAO-03-809:
GAO Highlights:
Highlights of GAO-03-809, a report to Congressional Requesters
Why GAO Did This Study:
To better monitor children and families served by state child welfare
agencies, Congress authorized matching funds for the development of
statewide automated child welfare information systems (SACWIS) and
required that the Department of Health and Human Services (HHS)
compile information on the children served by state agencies.
This report reviews (1) states‘ experiences in developing child
welfare information systems and HHS‘s role in assisting in their
development, (2) factors that affect the reliability of data that
states collect and report on children served by their child welfare
agencies and HHS‘s role in ensuring the reliability of those data, and
(3) practices that child welfare agencies use to overcome challenges
associated with SACWIS development and data reliability.
What GAO Found:
HHS reported that 47 states are developing or operating a SACWIS, but
many continue to face challenges developing their systems. Most state
officials said they recognize the benefit their state will achieve by
developing SACWIS, such as contributing to the timeliness of child
abuse and neglect investigations; however, despite the availability of
federal funds since 1994, states reported a median delay of 2-½ years
beyond the timeframes they set for completion. States reported that
they encountered some difficulties during SACWIS development, such as
challenges receiving state funding and creating a system that
reflected their work processes. In response to some of these
challenges, HHS has provided technical assistance to help states
develop their systems and conducted on-site reviews of SACWIS to
verify that the systems meet federal requirements.
Despite efforts to implement comprehensive information systems,
several factors affect the states‘ ability to collect and report
reliable adoption, foster care, and child abuse and neglect data.
States responding to GAO‘s survey and officials in the 5 states GAO
visited reported that insufficient caseworker training and inaccurate
and incomplete data entry affect the quality of the data reported to
HHS. In addition, states reported technical challenges reporting data.
Despite HHS‘s assistance, many states report ongoing challenges, such
as the lack of clear and documented guidance on how to report child
welfare data. In addition, although states were mandated to begin
reporting data to the Adoption and Foster Care Analysis and Reporting
System (AFCARS) in 1995, few reviews of states‘ AFCARS reporting
capabilities have been conducted to assist states in resolving some of
their reporting challenges.
Some states are using a variety of practices to address the challenges
associated with developing SACWIS and improving data reliability. For
example, 44 states included caseworkers and other system users in the
design and testing of SACWIS, and 28 states reported using approaches
to help caseworkers identify and better understand the data elements
that are required for federal reporting.
What GAO Recommends:
In order to improve the reliability of state-reported child welfare
data, GAO recommends that the Secretary of HHS consider ways to
enhance the guidance and assistance offered to states to help them
overcome the key challenges in collecting and reporting child welfare
data. In commenting on this report, HHS generally agreed with GAO‘s
findings and commented that the report provides a useful perspective
of the problems states face in collecting data and of HHS‘s effort to
provide ongoing technical assistance to improve child welfare data.
www.gao.gov/cgi-bin/getrpt?GAO-03-809.
To view the full product, including the scope and methodology, click
on the link above. For more information, contact Cornelia M. Ashby at
(202) 512-8403 or AshbyC@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
Most States Are Developing SACWIS, but Challenges Remain Despite HHS's
Oversight and Technical Assistance:
Several Factors Affect the States' Ability to Ensure Reliable Data on
Children's Experiences, and Some of HHS's Oversight and Assistance Is
Problematic:
States Are Using Various Practices to Overcome System Development
Challenges and Improve Data on Children's Experiences:
Conclusion:
Recommendation to the Secretary of Health and Human Services:
Agency Comments:
Appendix I: Scope and Methodology:
Appendix II: State SACWIS Stages of Development:
Appendix III: Comments from the Department of Health and Human
Services:
Appendix IV: GAO Contacts and Acknowledgments:
GAO Contacts:
Staff Acknowledgments:
Bibliography:
Related GAO Products:
Tables:
Table 1: Significant Child Welfare Information and Data Systems:
Table 2: Number of States in Various Stages of SACWIS Development:
Table 3: Selected SACWIS Child Welfare and Administrative Services:
Table 4: Commonly Used SACWIS Development Participants and Their Level
of Helpfulness:
Table 5: State-Reported Benefits of SACWIS Development:
Table 6: Number of Months States Delayed in SACWIS Development:
Table 7: Common SACWIS Development Challenges:
Table 8: Selected AFCARS Elements and Six States' Levels of Compliance
in Meeting Reporting Requirements:
Figures:
Figure 1: Most Common Caseworker Issues That Affect Data Quality:
Figure 2: Federal Practices That Affect Data Quality:
Abbreviations:
ACF: Administration for Children and Families:
ADP: advance planning document:
AFCARS: Adoption and Foster Care Analysis and Reporting System:
CAPTA: Child Abuse Prevention and Treatment Act:
CRSR: Child and Family Services Reviews:
FFP: federal financial participation:
HHS: Department of Health and Human Services:
IT: information technology:
NCANDS: National Child Abuse and Neglect Data System:
NYTD: National Youth in Transition Database:
SACWIS: Statewide Automated Child Welfare Information System:
VCIS: Voluntary Cooperative Information System:
United States General Accounting Office:
Washington, DC 20548:
July 31, 2003:
The Honorable Charles E. Grassley
Chairman, Committee on Finance United States Senate:
The Honorable Tom DeLay
Majority Leader
House of Representatives:
Recent news reports of tragedies involving child welfare agencies
across the country highlight the long-standing problems states have had
monitoring children in their care. Given that hundreds of thousands of
children are found to be victims of abuse and neglect and are estimated
to spend some time in foster care each year, the Congress required that
the Department of Health and Human Services (HHS) compile information
on the children served by state agencies and authorized federal funds
to match those of states for use in the development of state child
welfare information systems. Since 1994, designated federal matching
funds have been available to states to develop and implement
comprehensive case management systems--Statewide Automated Child
Welfare Information Systems (SACWIS)--to manage their child welfare
cases as well as to report child abuse and neglect, foster care, and
adoption information to the federal government. States have the option
to implement a SACWIS or develop different information systems without
using SACWIS funds to support their child welfare agencies and collect
information on their child welfare cases. Regardless of the type of
system a state develops, child welfare caseworkers at the county or
local level are the key personnel who collect and document information
on children and families served by child welfare agencies, in addition
to performing a wide range of services to protect children--such as
investigating child abuse or neglect reports or providing support
services to maintain the children in their homes. Currently, HHS
compiles state-reported child welfare data in two databases: the
Adoption and Foster Care Analysis and Reporting System (AFCARS) and the
National Child Abuse and Neglect Data System (NCANDS). HHS relies on
the information available in its databases to analyze and track
children's experiences in the child welfare system, to determine
states' performance on federal child welfare outcome measures, and to
report to Congress on children's well being and child welfare
experiences.
Because of your interest in how states have developed systems to
collect and report information on the children they serve and the
reliability of that information, you asked us to determine (1) states'
experiences in developing child welfare information systems and HHS's
role in assisting in their development; (2) factors that affect the
reliability of data that states collect and report on children served
by their child welfare agencies, and HHS's role in ensuring the
reliability of those data; and (3) practices that child welfare
agencies use to overcome challenges associated with SACWIS development
and data reliability.
To conduct our work, we surveyed all 50 states and the District of
Columbia regarding their experiences in developing and using
information systems and their ability to report data to HHS. We
received responses from 49 states and the District of
Columbia,[Footnote 1] although all states did not respond to every
question. We also reviewed a variety of HHS documents, including the
protocol and reports for its reviews of SACWIS systems and states'
AFCARS reporting capabilities. In addition, we visited 5 states--
Colorado, Iowa, New York, North Carolina, and Oklahoma--to obtain
first-hand information on their experiences developing SACWIS and
reporting data to HHS. We selected these states to represent geographic
diversity and different stages of SACWIS implementation. Finally, we
interviewed HHS officials and child welfare and data experts and
reviewed relevant literature. We conducted our work between June 2002
and June 2003 in accordance with generally accepted government auditing
standards. A more detailed discussion of our scope and methodology
appears in appendix I.
Results in Brief:
HHS reported that 47 states are developing or operating a SACWIS, but
many continue to face challenges despite HHS's oversight and technical
assistance. Most states are using federal SACWIS funds and are in
various stages of development. States reported in our survey that they
have spent approximately $2.4 billion in federal, state, and local
funding on SACWIS. Most state officials said they recognize the benefit
their state will achieve by developing SACWIS, such as contributing to
the timeliness of child abuse and neglect investigations; however,
despite the availability of federal funds since 1994, many child
welfare agencies lag behind the timeframes they set for completion,
with delays ranging from 2 months to 8 years. Forty-two of the 46
states responding to our survey that they are developing SACWIS
reported at least some challenge obtaining state funding. In Iowa, for
example, state officials reported that insufficient state funds delayed
them in making the necessary modifications to meet federal requirements
for system completion. Some states had difficulties developing a system
that met the state child welfare agency's needs statewide. For example,
state officials in New York--a state where the counties are responsible
for administering child welfare services--said that building a uniform
system was stalled when significant frustration with the system's
design led some county officials to request that the state stop SACWIS
development. In addition, 32 states reported at least some challenge
securing information technology contractors with knowledge of child
welfare practice to develop their SACWIS. In response to some of these
challenges, HHS has provided technical assistance to help states
develop their systems and conducted on-site reviews of SACWIS to verify
that the systems meet all federal requirements. For example, at the
time of our review, HHS had conducted on-site reviews in 26 states with
operational SACWIS to ensure that the systems met all federal
requirements and to offer assistance to states that faced challenges
completing the development of their SACWIS.
Several factors affect states' ability to collect and report reliable
data on children served by state child welfare agencies, and some
problems exist, such as a lack of clear and documented guidance, with
HHS's oversight and technical assistance. Almost all of the states
responding to our survey reported that insufficient caseworker training
and inaccurate and incomplete data entry into their information system
affect the quality of the data reported to HHS. Although most states
reported these as separate factors, HHS and the states we visited found
that insufficient training and inaccurate and incomplete data entry are
often linked. In addition, 36 of the 50 states that responded to our
survey reported that technical challenges, such as matching their state
data element definitions to HHS's data categories, affected the quality
of the data that they report to the federal government. Similarly,
during assessments of 6 states' compliance with AFCARS reporting
standards, HHS found that these issues affect data reliability. Despite
HHS's assistance in helping states improve their data, such as testing
state data quality and providing the results to the states to aid them
in resubmitting data, states report ongoing challenges. For example, 41
of the 50 states responding to our survey reported that a lack of clear
and documented guidance from HHS affects the quality of the data they
report to AFCARS, and 25 states said the lack of clear, documented
guidance also affected the data reported to NCANDS. In addition,
although states were mandated to begin reporting AFCARS in 1995, few
reviews of states AFCARS reporting capabilities have been conducted to
assist states in resolving some of their challenges.
Some states are using a variety of practices to address the challenges
associated with developing SACWIS and improving data reliability,
although no formal evaluations are available on their effectiveness. To
overcome the challenge of developing a system to meet statewide needs,
many states relied on caseworkers and other system users for input on
design and testing of SACWIS. Few states reported that they devised
strategies to overcome the other challenges, such as limited funding
and finding contractors with knowledge of child welfare. However,
Oklahoma child welfare officials--in order to maximize the limited
state funding for maintaining their SACWIS--reported saving $1 million
each year by hiring some of the contractors who developed their SACWIS
as permanent staff. To improve data reliability, the 5 states we
visited routinely review their data to identify data entry errors so
that managers can ensure that the missing data are entered
appropriately. In addition, some states reported that frequent use of
the data, such as publishing periodic management reports detailing
local offices' performance on outcome measures, helps caseworkers
understand the importance of entering timely information.
In order to improve the reliability of state-reported child welfare
data, we are recommending that the Secretary of HHS consider ways to
enhance the guidance and assistance offered to states to help them
overcome the key challenges in collecting and reporting child welfare
data. In commenting on this report, HHS's Administration for Children
and Families (ACF) generally agreed with our findings and commented
that the report provides a useful perspective of the problems states
face in collecting data and of ACF's effort to provide ongoing
technical assistance to improve the quality of child welfare data. In
response to our recommendation, ACF said that we did not recognize the
long-term efforts it has taken to provide AFCARS and NCANDS guidance.
ACF also noted that the data definitions need to be updated and revised
and said it is currently in the process of revising the AFCARS
regulations to further standardize the information states are to
report--which we acknowledge in our report. Further, ACF added that
although staff turnover in state child welfare agencies is a
significant contributor to data quality issues, we did not focus on
this as a significant factor. However, because we recently issued a
detailed report on a variety of caseworker issues, we primarily focused
in this report on the key data entry challenges caseworkers face and
refer readers to our previous work for additional information on
challenges related to caseworker recruitment and retention and their
affect on child welfare agencies. ACF commented that it is firmly
committed to continue to support the states and to provide technical
assistance and other guidance as its resources will permit. We believe
that the recent activities to formally obtain, document, and
incorporate feedback from the states with regard to collection and
reporting adoption and foster care data represent are a step towards
improving states' data. Our recommendation encourages HHS to consider
ways to enhance the guidance and assistance already offered to states
as a step to helping them better comply with the reporting
requirements.
Background:
ACF's Children's Bureau is responsible for the administration and
oversight of federal funding to states for child welfare services under
Titles IV-B and IV-E of the Social Security Act. However, the
monitoring of children served by state child welfare agencies is the
responsibility of the state agencies that provide the services to these
children and their families. Child welfare caseworkers at the county or
local level are the key personnel responsible for documenting the wide
range of services offered to children and families, such as
investigations of abuse and neglect; treatment services offered to
families to keep them intact and prevent the need for foster care; and
arrangements made for permanent or adoptive placements when children
must be removed from their homes. Caseworkers are supported by
supervisors who typically assign new cases to workers and monitor
caseworkers' progress in achieving desired outcomes, analyzing and
addressing problems, and making decisions about cases.
A number of efforts at the national level have been taken to implement
comprehensive data systems that capture, report, and analyze the child
welfare information collected by the states (see table 1 for
information on national data systems as well as information on state
systems).
Table 1: Significant Child Welfare Information and Data Systems:
System: State information systems:
System: Statewide Automated Child Welfare Information System (SACWIS);
History: The Omnibus Budget Reconciliation Act (OBRA) of 1993
authorized the use of an enhanced federal financial participation (FFP)
rate of 75 percent to assist states develop uniform automated
information systems--SACWIS--that support the administration of
services offered under their child welfare programs.[A] The enhanced
FFP was available initially from federal fiscal years 1994 through 1996
and subsequently extended through federal fiscal year 1997. After 1997,
states receive a 50 percent match for SACWIS-related activities.
Funding approval for SACWIS is based on states' estimated costs for
development and operation, and no time limits are placed on the receipt
of federal funding; Use: A SACWIS is designed and developed for use by
states' caseworkers and other personnel for the purposes of
establishing an electronic case file for children and families served
by the state child welfare agency. Some of the data captured in SACWIS
are reported to HHS.
System: National data systems:
System: Voluntary Cooperative Information System (VCIS); History: With
funding from HHS, the American Public Welfare Association--now known as
the American Public Human Services Association--established VCIS in an
effort to compile state-specific child welfare data. States voluntarily
reported aggregate-level data on the characteristics of children in
foster care and those adopted from state child welfare agencies. VCIS
compiled data between 1982 and 1994; Use: The data available in VCIS
was used to produce reports at the national level on the
characteristics of children in foster care and those adopted from state
child welfare agencies. However, the data reported to VCIS were
inconsistent.[B] Following the implementation of AFCARS, states no
longer reported to VCIS.
System: Adoption and Foster Care Analysis and Reporting System
(AFCARS); History: In 1986, Congress added Section 479 to Title IV-E of
the Social Security Act, which required HHS to establish and implement
a mandatory foster care and adoption data collection system. Despite
the legislative requirement that HHS complete final regulations for the
system by the end of calendar year 1988, regulations were not issued
until December 1993. States began reporting to AFCARS in 1995 on the
demographic characteristics of adoptive and foster children and their
parents as well as foster children's type of placement and permanency
goals. States are required to report 66 foster care and 37 adoption
data elements for each child in care during 6-month periods. Between
1998 and 2001, states not meeting certain reporting standards were
subject to penalties. The penalties were rescinded in January 2002
following an appeal filed by 12 states; Use: Some of the key uses of
AFCARS data include: Calculating the national standard for five
federal outcome measures--incidence of child abuse and/or neglect in
foster care, foster care re-entries, stability of foster care
placements, length of time to achieve reunification, and length of time
to achieve adoption; The publication of the annual Child Welfare
Outcomes Report; Determining the allocation of funds in the adoption
incentive program and Chafee Foster Care Independence Program.[C].
System: National Child Abuse and Neglect Data System (NCANDS); History:
In 1988, Congress amended the Child Abuse Prevention and Treatment Act
(CAPTA) by directing HHS to establish a national data collection and
analysis system consisting of state child abuse and neglect
information. Walter R. McDonald & Associates was awarded the contract
to compile and analyze the state-reported data. States began
voluntarily reporting annually in 1990; however, the 1996 CAPTA
amendments directed states to report certain data to receive CAPTA
grant funding. States submit either child-specific records or
aggregate-level data. In 2001--the most recently available data from
HHS--39 states and the District of Columbia submitted child-specific
data, which includes the demographics of the children and their
perpetrators, the types of maltreatment, investigation or assessment
findings, risk factors, and services provided as a result of the
investigation or assessment. The remaining 11 states submitted
aggregate data; Use: Some of the key uses of NCANDS data include:
Calculating the national standard for two federal outcome measures on
child safety--recurrence of maltreatment and incidence of child abuse
and/or neglect in foster care; The publication of the annual Child
Maltreatment Report; The publication of the annual Child Welfare
Outcomes Report.
System: National Youth in Transition Database (NYTD); History: In
response to requirements included in the Foster Care Independence Act
of 1999, HHS has proposed to develop the NYTD to capture more detailed
data than is currently reported to AFCARS on older foster children who
receive independent living services. Such services include life-skills
training in financial management or career planning. HHS reported to
Congress in September 2001 that nationwide implementation of NYTD was
planned for October 2003, but recent HHS information indicates that the
actual implementation date will likely be 2005 or 2006. Once
implemented, states will be required to report to NYTD; Use: In
response to the legislation, data available in the NYTD will be used,
in part, to track (1) the number and characteristics of older youth
receiving independent living services, (2) the type and quantity of
services states provided, and (3) state performance on the outcome
measures required by the legislation.
Source: GAO analysis based on federal legislation, regulations, and
other documents.
[A] The allowable costs under the 75 percent FFP included planning,
design, development, and installation of a SACWIS. Other costs, such as
operation of the SACWIS, were covered at 50 percent.
[B] See www.acf.hhs.gov/programs/cb/dis/vcis/maintoc.htm for more
complete details on the shortcomings of the data available in VCIS.
[C] The total number of finalized adoptions reported to AFCARS are used
to determine the amount, if any, of the adoption incentive payments
awarded to states. In addition, AFCARS data are used to determine the
ratio of the number of children in foster care in each state to the
total number of children in foster care in all states to calculate
state allotments under the Chafee Foster Care Independence Program.
[End of table]
To qualify for federal funding for SACWIS, states must prepare and
submit an advance planning document (APD) to ACF's Children's Bureau,
in which they describe the state's plan for managing the design,
development, implementation, and operation of a SACWIS that meets
federal requirements and state needs in an efficient, comprehensive,
and cost-effective manner.[Footnote 2] In addition, the state must
establish SACWIS and program performance goals in terms of projected
costs and benefits in the APD. States are required to submit separate
APDs for the planning and development phases, in addition to periodic
updates.
Since the administration and structure of state child welfare agencies
vary across the nation, states can design their SACWIS to meet their
state needs, as long as states meet certain federal requirements.
Federal funding is available to states for SACWIS that:
* meet the requirements for reporting AFCARS data to HHS;
* to the extent practicable, are capable of linking with the state data
collection system that collects information on child abuse and neglect;
* to the extent practicable, are capable of linking with, and
retrieving information from, the state data collection system that
collects information on the eligibility of individuals under Title IV-
A--Temporary Assistance for Needy Families; and:
* provides for more efficient, economical, and effective administration
of the programs carried out under a state's plans approved under Titles
IV-B and IV-E of the Social Security Act.
A SACWIS must operate uniformly as a single system in each state and
must encompass all entities that administer programs provided under
Titles IV-B and IV-E. In some cases, HHS will allow the statewide
system to link to another state system to perform required functions,
such as linking to financial systems to issue and reconcile payments to
child welfare service providers. The state's APD must describe how its
SACWIS will link to other systems to meet the requirements in the
SACWIS regulations.
In addition to monitoring the APDs of the states that are developing
SACWIS, HHS reviews state information systems through formal SACWIS
assessment reviews and the Child and Family Services Reviews (CFSR)--a
federal review process to monitor states' compliance with child welfare
laws and federal outcome measures. The formal SACWIS reviews are
conducted by ACF's Children's Bureau to determine if a state has
developed and implemented all components detailed in the state's APD
and if the system adheres to federal requirements. The CFSR assesses
statewide information systems, along with other systemic factors, to
determine if the state is operating a system that can readily identify
the status, demographic characteristics, location, and goals for
placement of every child who is in foster care. This systemic factor is
reviewed in all states, regardless of whether the state is developing a
SACWIS or the stage of system development. According to results from
the fiscal years 2001 and 2002 CFSRs, 4 of the 32 states in which HHS
reviewed were not in substantial conformity on the statewide
information system indicator. These 4 states must address how they will
come into conformity with this factor in a program improvement plan.
HHS has also conducted SACWIS reviews in 2 of these states.
Most States Are Developing SACWIS, but Challenges Remain Despite HHS's
Oversight and Technical Assistance:
While 47 states are developing or operating a SACWIS, many challenges
remain despite HHS's oversight and technical assistance. Since 1994,
states reported that they have spent approximately $2.4 billion in
federal, state, and local funding on SACWIS. While most state officials
we interviewed and those responding to our survey said that they
recognize the benefits their state will achieve by developing a
statewide system, many states reported that the development of their
SACWIS is delayed between 2 months and 8 years beyond the time frames
the states set for completion, with a median delay of 2-½ years. Most
states responding to our survey faced challenges, such as obtaining
state funding and developing a system that met the child welfare
agency's needs statewide. In response to some of these challenges, HHS
has provided technical assistance to help states develop their systems
and conducted on-site SACWIS reviews to verify that the systems meet
all federal requirements.
States Are Using Federal and State Funds and Various Participants to
Develop MultiComponent SACWIS:
Currently, 47 states are developing or operating a SACWIS and are in
various stages of development--ranging from planning to complete. The
states responding to our survey reported using approximately $1.3
billion in federal funds[Footnote 3] and approximately $1.1 billion in
state and local funds[Footnote 4] for their SACWIS. However, HHS
estimated that it allocated approximately $821 million between fiscal
years 1994 and 2001 in SACWIS developmental funds[Footnote 5] and $173
million between fiscal years 1999 and 2001 in SACWIS operational
funds.[Footnote 6] The total amount of federal funding provided to
states for SACWIS is unknown because states claimed operational costs
as a part of their Title IV-E administrative expenses prior to
1999.[Footnote 7] Although the federal government matched state funding
at an enhanced rate of 75 percent beginning in 1994, many states did
not apply for federal funding or begin SACWIS development until 1996
and 1997 when more than $467 million--the bulk of federal funds--were
allocated. Most states were still developing their SACWIS by the time
enhanced funding expired in 1997, after which states could receive a 50
percent FFP for SACWIS development and operation. Although 47 states
are currently developing or operating a SACWIS, all states except
Hawaii received some federal SACWIS funds. For example, according to
figures provided by HHS, North Carolina received approximately $9.6
million in developmental funds and North Dakota received approximately
$2.4 million in developmental funds and $245,000 in operational funds
for SACWIS, but both states encountered difficulties that prevented
them from completing their systems. In these situations, HHS entered
into negotiations with the states about the amount of money that the
states must return to the federal government.
In order to track states' SACWIS development, HHS places them in six
categories that identify their stage of development (see table 2).
States are required to submit APD updates periodically, which inform
HHS of their progress in developing SACWIS. See appendix II for a
complete list of states' phases of development. Although most states
continue to advance in the development of their systems, some encounter
problems that cause HHS to recategorize them into a lower stage of
development. In Pennsylvania, for example, the child welfare agency
encountered difficulties, such as inadequate computer software to
support a comprehensive SACWIS, after attempting to implement its
SACWIS in 2000. Due to these problems, the state is in the process of
shutting down the system and has resubmitted an APD for a new system to
HHS for review and approval for further federal funding. According to
figures provided by HHS, Pennsylvania has received approximately $9.7
million in federal funding thus far. In addition, while HHS may
classify a state system as complete following an assessment of their
SACWIS, a state may make additional changes to the system since SACWIS,
like other computer systems, continually evolve as technology and child
welfare practices change. States can claim federal funding for these
changes as operational expenses. For example, Oklahoma's SACWIS was the
first system to be determined complete, but it has made enhancements to
its system since HHS found the system in compliance with federal
requirements in 1998. In addition, Oklahoma is currently considering
moving to a Web-based system. An HHS official reported that such
changes do not need prior approval unless they are in excess of $5
million.
Table 2: Number of States in Various Stages of SACWIS Development:
Stage: Complete[A]; Number of states: 5.
Stage: Operational[B]; Number of states: 24.
Stage: Partially operational[C]; Number of states: 9.
Stage: Implementation[D]; Number of states: 2.
Stage: Planning[E]; Number of states: 7.
Stage: No SACWIS[F]; Number of states: 4.
Source: HHS.
Note: Status is as of May 22, 2003.
[A] SACWIS assessment process is completed, and all functional
requirements and specifications set forth in the APD are either
included in the system or in an accepted corrective action plan.
[B] All functional requirements and specifications in APD are included
in system, and system is functional statewide, but state has not
completed SACWIS assessment or is working on other issues.
[C] State is still rolling out system to field sites or still adding
functions to systems that are operational statewide.
[D] In active design and development, even if delayed while waiting to
resolve problems such as funding.
[E] Working through options for a SACWIS.
[F] Have never pursued SACWIS funding or have abandoned plans to
develop a system.
[End of table]
In developing a system, states have considerable flexibility in the
design of their SACWIS. According to HHS officials, a state should be
using its SACWIS as a case management tool that uses automation to
support the various aspects of state child welfare programs, such as
recording child protection, out of home care, and foster care and
adoption services. To further assist child welfare practice, states
have designed their systems to follow the natural flow of child welfare
practice in their state and have added design features to help track
key events during a case. For example, in Iowa child welfare work is
divided between child abuse and neglect investigations and ongoing case
management for children brought into the care of the child welfare
agency. As a result, Iowa designed a SACWIS to reflect this work
process by linking two databases--one to record child abuse and neglect
information and one to record ongoing case records--that share
information with one another.[Footnote 8] In Rhode Island, the SACWIS
was designed to alert caseworkers if an alleged perpetrator has been
the subject of three reports of abuse or neglect within 1 year.
Regardless of the findings of each report, this alert notifies the
caseworker to initiate an investigation when a third report is
received.
Since many states are in different phases of SACWIS development, their
systems currently support to varying degrees a variety of child welfare
and administrative components (see table 3). According to HHS, while
the components represented in table 3 are required for a state's SACWIS
to be considered compliant with federal guidance--either through an
interface or built within the system--some of the subcomponents, such
as a function that helps caseworkers manage their caseloads, are
optional. HHS has encouraged states to automate as many functions as
possible in the SACWIS in an effort to cut down on the additional
paperwork or duplicative steps inherent in manual data collection. One
of these services, tracking independent living, is becoming more
important for states as HHS decides how to implement the Foster Care
Independence Act of 1999 and considers the development of the
NYTD.[Footnote 9] Some states have already started collecting data on
older youth and the services they receive. Currently, 27 states
reported in our survey that they are at some stage of using their
SACWIS to track independent living services, and an additional 14
states plan to include this component in their system in preparation
for the requirements. However, 21 of the 46 states reporting to our
survey that they are developing or operating a SACWIS reported that
they would have to make substantial changes to their SACWIS in order to
capture this information.
Table 3: Selected SACWIS Child Welfare and Administrative Services:
Service: Child Welfare Services:
Service: Child protection[A]; Fully or partially operational in SACWIS:
38; Planned for SACWIS: 5.
Service: Out of home care[B]; Fully or partially operational in SACWIS:
35; Planned for SACWIS: 8.
Service: Adoption; Fully or partially operational in SACWIS: 34;
Planned for SACWIS: 9.
Service: Independent living; Fully or partially operational in SACWIS:
27; Planned for SACWIS: 14.
Service: Intensive home-based services[C]; Fully or partially
operational in SACWIS: 27; Planned for SACWIS: 13.
Service: Administrative Services:
Service: Workload management; Fully or partially operational in SACWIS:
32; Planned for SACWIS: 8.
Service: IV-E eligibility[D]; Fully or partially operational in SACWIS:
29; Planned for SACWIS: 14.
Service: Foster care maintenance payments; Fully or partially
operational in SACWIS: 28; Planned for SACWIS: 14.
Service: Adoption assistance payments; Fully or partially operational
in SACWIS: 25; Planned for SACWIS: 17.
Service: Contract provider payment; Fully or partially operational in
SACWIS: 24; Planned for SACWIS: 15.
Source: GAO survey.
Note: Based on responses from 46 states developing or operating a
SACWIS. The rows for the columns "fully or partially operational" and
"planned" do not add to 46 because the respondents may have answered
"not supported," "don't know," or "no answer.":
[A] Child protection includes services such as intake and screening,
investigation, and disposition.
[B] Out of home care includes things such as foster care, group homes,
and residential placement.
[C] Intensive home-based services include efforts to avoid placing a
child in foster care.
[D] IV-E funding is available for foster care, adoption, and
independent living services.:
[End of table]
To assist with the design of their SACWIS, states relied on a number of
different participants including internal users, such as caseworkers
and managers, information technology (IT) staff, and contractors. Most
states found these participants to be extremely or very helpful in the
process (see table 4). In Oklahoma, for example, 150 child welfare
staff from the field worked closely with the contractor in intensive
work group sessions to design and test the system. To complement the
caseworkers' knowledge of child welfare practice, 43 states relied on
IT staff. In Colorado, for example, IT staff said that during SACWIS
design and development, they shared office space with program staff
that had been assigned to help with SACWIS development. This co-
location of staff aided in the exchange of information pertaining to
the development of the system. Finally, 42 states reported that they
hired private contractors to conduct a large part of SACWIS design and
development. The contractors helped states meet federal requirements,
designed the system with state specific options, wrote the necessary
software, tested and implemented the system, and trained users.
Table 4: Commonly Used SACWIS Development Participants and Their Level
of Helpfulness:
SACWIS development participant: Internal users; Number of states using
participant: 44; Number of states rating participant as extremely or
very helpful: 41.
SACWIS development participant: IT staff; Number of states using
participant: 43; Number of states rating participant as extremely or
very helpful: 31.
SACWIS development participant: Private contractors; Number of states
using participant: 42; Number of states rating participant as extremely
or very helpful: 37.
Source: GAO survey.
Note: Based on responses from 46 states developing or operating a
SACWIS.:
[End of table]
At the time of our review, HHS reported that 4 states were not pursuing
SACWIS development and most of these states reported various reasons in
our survey for not developing a system. In Hawaii, the child welfare
agency chose not to pursue SACWIS because it already had a statewide
system in place that it believed was adequately meeting its needs and
which was collecting and reporting federal child welfare data. After an
attempt to develop a system, North Carolina cancelled its efforts
because it could not build consensus across its 100 counties on the
design of a uniform system. On our site visit to North Carolina, child
welfare officials reported that they are currently working on a
statewide information system that will encompass a number of social
services, such as food stamps and mental health services, but an HHS
official reported that North Carolina is not seeking federal SACWIS
funding to support the development of this system. Vermont officials
reported that they did not pursue SACWIS because the legislature
declined to provide the matching state funds. In retrospect, they
believe that the choice not to develop SACWIS was best for the state
because they found the SACWIS requirements too restrictive to enable
the state to design a system to meet its needs. Officials said that the
state would not use a number of the required SACWIS components, such as
developing all the required electronic links to other agencies'
systems, especially since the state has a small child welfare
population. Another state--North Dakota--did not report in our survey
the reason for stopping SACWIS development; however, HHS officials
reported that the state had attempted to develop a SACWIS, but faced a
variety of problems, such as receiving state funding.:
States Accrue Benefits from Using SACWIS, but Several Issues Create
Delays in Completing States' Systems:
While most state child welfare agency officials said they recognize the
benefits the state will achieve by developing SACWIS, such as enhancing
their ability to track the whereabouts of foster children, 31 state
agencies lag behind the time frames they set for completion, with 26
states reporting delays ranging from 2 months to 8 years. State
officials reported in our survey and during site visits that SACWIS has
contributed to more efficient and effective agency functioning, which
can improve states' capabilities to manage their child welfare cases,
including keeping track of where the children are living and the
services they are receiving. Child welfare officials in Colorado
reported that automation has improved agency functioning by making
child welfare case information available statewide, which is especially
helpful when families move from one county to another. In Oklahoma,
caseworkers and state officials noted that they believe their children
are safer since the implementation of SACWIS simply because the
information on the children is easily accessible to the caseworkers and
their supervisors. According to survey results, automated systems
provided easier access to data and allowed caseworkers to better
monitor children in their care, which may contribute to additional
child welfare and administrative benefits, such as decreased incidences
of child abuse and neglect, shortened length of time to achieve
adoption, timeliness of payments to foster families, and timeliness of
payments to foster facilities (see table 5). New Jersey reported in our
survey that its goal in developing a SACWIS is to integrate the more
than 40 stand-alone systems that currently capture information on the
children served by their child welfare agency. By pulling all of these
systems together into a uniform SACWIS, the state hopes to improve the
recording of casework activities in a timely manner and to develop a
tool to better target resources and services. Effectively integrating
these systems will require the state to use a disciplined IT management
approach that includes (1) detailed analyses of users' needs and
requirements, (2) a clearly defined strategy for addressing information
needs, and (3) sufficient technical expertise and resources to support
the effort.
Table 5: State-Reported Benefits of SACWIS Development:
Child welfare benefits: Decreased incidence of child abuse and neglect;
Number of states identifying SACWIS as extremely or very effective in
achieving measure: 25.
Child welfare benefits: Shortened length of time to achieve adoption;
Number of states identifying SACWIS as extremely or very effective in
achieving measure: 24.
Child welfare benefits: Decreased recurrence of child maltreatment;
Number of states identifying SACWIS as extremely or very effective in
achieving measure: 23.
Child welfare benefits: Shortened length of time to achieve
reunification; Number of states identifying SACWIS as extremely or very
effective in achieving measure: 22.
Management benefits:
Management benefits: Timeliness of payment to foster families;
Number of states identifying SACWIS as extremely or very effective in
achieving measure: 36.
Management benefits: Timeliness of payment to foster facilities;
Number of states identifying SACWIS as extremely or very effective in
achieving measure: 34.
Management benefits: Overall case management for children and
families; Number of states identifying SACWIS as extremely or very
effective in achieving measure: 33.
Management benefits: Timeliness of child abuse and neglect
investigations; Number of states identifying SACWIS as extremely or
very effective in achieving measure: 33.
Source: GAO survey.
Note: Based on responses from 46 states developing or operating SACWIS.
States not included answered "moderately effective," "somewhat
effective," "not at all effective," "not a state goal," "system doesn't
support," "don't know," or "no answer.":
[End of table]
Despite the benefits that many states have accrued with SACWIS, 31
states reported in our survey that they have been delayed in system
completion beyond their initial deadline and identified a number of
challenges that have led to the delay (see table 6).[Footnote 10] Some
of the common difficulties states reported in developing SACWIS
included receiving state funding approval, reaching internal agreement
on system development, and creating a system that reflects child
welfare work processes and is user friendly (see table 7).
Table 6: Number of Months States Delayed in SACWIS Development:
State: Alabama; Length of delay in months[A]: 36.
State: Arkansas; Length of delay in months[A]: 6.
State: California; Length of delay in months[A]: 36.
State: Colorado; Length of delay in months[A]: 26.
State: Connecticut; Length of delay in months[A]: 96.
State: District of Columbia; Length of delay in months[A]: 36.
State: Georgia; Length of delay in months[A]: 25.
State: Idaho; Length of delay in months[A]: 21.
State: Illinois; Length of delay in months[A]: 79.
State: Indiana; Length of delay in months[A]: 6.
State: Kansas; Length of delay in months[A]: 72.
State: Louisiana; Length of delay in months[A]: 12.
State: Maryland; Length of delay in months[A]: 12.
State: Michigan; Length of delay in months[A]: 26.
State: Minnesota; Length of delay in months[A]: 12.
State: Mississippi; Length of delay in months[A]: 12.
State: New Jersey; Length of delay in months[A]: 42.
State: New Mexico; Length of delay in months[A]: 3.
State: Ohio; Length of delay in months[A]: 36.
State: Oregon; Length of delay in months[A]: 70.
State: Rhode Island; Length of delay in months[A]: 14.
State: South Carolina; Length of delay in months[A]: 47.
State: Tennessee; Length of delay in months[A]: 36.
State: Utah; Length of delay in months[A]: 48.
State: Virginia; Length of delay in months[A]: 2.
State: Washington; Length of delay in months[A]: 36.
Source: GAO survey.
Note: While 31 states reported in the survey that they have experienced
a delay in SACWIS development, only 26 states reported the length of
their delay. The survey was issued in October 2002 and completed by
states as late as December 2002.
[A] States were asked to report the number of months the delays
exceeded the time line outlined in their APD.
[End of table]
Table 7: Common SACWIS Development Challenges:
SACWIS development challenges: Receiving state funding approval; Number
of states experiencing at least some challenge[A]: 42.
SACWIS development challenges: Reaching internal agreement on system
development; Number of states experiencing at least some challenge[A]:
41.
SACWIS development challenges: Creating a system that reflects work
processes; Number of states experiencing at least some challenge[A]:
40.
SACWIS development challenges: Creating a user friendly system; Number
of states experiencing at least some challenge[A]: 39.
SACWIS development challenges: Insufficient state funding allocation;
Number of states experiencing at least some challenge[A]: 32.
SACWIS development challenges: Securing contractors with knowledge of
child welfare; Number of states experiencing at least some
challenge[A]: 32.
Source: GAO survey.
Note: Based on responses from 46 states developing or operating a
SACWIS.:
[A] States were asked the extent to which certain factors were a
challenge in developing SACWIS using the following scale: very great,
great, moderate, some, or no challenge. The number of states reported
in our analysis of challenges represents the total number of states
reporting any extent the factor posed a challenge. States not included
answered "no challenge," "have not yet reached this stage," "don't
know," or "no answer.":
[End of table]
Forty-two states reported challenges receiving funding approval, and 32
states reported that insufficient state funding allocations for SACWIS
development were a challenge in developing a comprehensive system. For
example, Vermont officials reported that the state legislature declined
to provide the matching state funds needed to secure federal funding
for SACWIS. As a result, the state could not pursue development. In
addition to the development challenges reported in our survey, 2 of the
5 states we visited reported that insufficient funding affected ongoing
SACWIS maintenance. In Colorado, state agencies have received a series
of budget cuts, which child welfare officials report have impacted
their ability to respond to child welfare caseworkers' needs for system
improvements. In Iowa, child welfare officials reported that the state
legislature appropriated $17,000 for state fiscal year 2002 for all
child welfare automated systems activity, which they noted was an
insufficient amount to maintain and upgrade systems as well as to pay
staff. They reported that, as a result, the child welfare agency
provided the information systems department with approximately $1
million from other parts of the agency's budget.
Despite user involvement in system design, some states still faced
challenges trying to reach internal agreement among agency officials
and caseworkers on the design of a system, resulting in a delay in
development. In New York--a state where the counties are responsible
for administering child welfare services--the development of SACWIS was
stalled when significant frustration with the system's design led
Commissioners from five large counties and New York City to request
that the state stop SACWIS development until a reassessment of the
design and plans for the implementation of the system was completed.
After a thorough evaluation of the project, the state made changes to
the project plan and developed statewide work groups to ensure all
counties were in agreement with the system design. In addition, they
hired a contractor to monitor system development and ensure that all
users' requirements are seriously considered.
Similarly, despite states' heavy reliance on contractors, many reported
that securing contractors with knowledge of child welfare practice was
a challenge for timely SACWIS development. Contractors are hired by the
state for their system development knowledge but often are unfamiliar
with child welfare policies and practices, especially since they vary
from state to state. Officials in Colorado, for example, said they
encountered difficulties with their contractors because of high
turnover among the contractor staff and their lack of knowledge of
child welfare policies. A contractor who has worked with 7 states to
develop their SACWIS reported that contractors are asked to learn the
child welfare business practices of a state in a short amount of time
and that states cannot devote many resources, such as caseworkers, to
help in the design process because caseworkers need to devote their
time to providing services to children and families. Therefore,
contractors often have to acquire knowledge on their own.
Many states reported that creating a system that reflects child welfare
work processes and is user friendly was a challenge in developing
SACWIS. These issues were also identified in the federal reviews of
states' SACWIS. For example, one state explained in the SACWIS review
that it had designed a system to meet the caseworkers' needs and
reflect the nature of the child welfare work processes by developing a
system that required events to be documented as they occurred. However,
this design limited the SACWIS's functionality because it did not allow
the caseworkers to go back and enter information after an event
happened. The state explained that caseworkers do not use the system in
real time, but provide services to the children and families and then
record the information in the system. The state had to redesign the
system to correct for this design flaw. In addition, the 14 states
reporting that they have adapted a system from another state have
experienced some challenge modifying the systems to reflect their work
processes. While HHS advises states to consider adapting another
state's system if it requires few changes, states report that they and
their contractors were not always aware of the changes that would need
to be made to adapt a system. Although Colorado and New York decided to
modify another state's SACWIS instead of designing a new system, child
welfare officials in these states reported that adapting a SACWIS from
another state has created more problems than anticipated. Colorado and
New York adapted systems from state-administered child welfare
agencies, which required extensive modifications to meet their needs as
county-administered states. For example, Colorado needed a system that
supported its administrative structure and could handle a larger number
of cases.
HHS Provides Some Assistance to Help States Meet SACWIS Requirements:
HHS has assisted states in a variety of ways in developing and
completing their SACWIS.[Footnote 11] As a part of its regulatory
responsibilities, HHS must review, assess, and inspect the planning,
design, development, installation, and operation of SACWIS. In addition
to reviewing and monitoring states' APDs, HHS conducts on-site SACWIS
reviews to comply with these responsibilities. HHS officials told us
that these reviews are a detailed and thorough assessment of state
systems to ensure the systems' compliance with SACWIS requirements. In
addition, officials reported that they provide verbal technical
assistance during the on-site review to help states that do not fully
conform with the applicable regulations and policies. At the time of
our review, HHS had conducted 26 SACWIS reviews--5 of which were
determined as meeting all the requirements and classified as complete.
HHS officials told us that since states have the flexibility to build a
SACWIS that meets their needs, a large portion of the formal reviews
concentrate on ensuring that the systems conform to state business
practices. For example, while SACWIS regulations require that a state
report all AFCARS data from their SACWIS, one state HHS reviewed relied
on a separate state system to report data on the children served by the
juvenile justice agency who are eligible for IV-E foster care funds.
The state proved it had developed an automated process to merge data
from both systems to compile a single AFCARS report that included
children captured in both their SACWIS and juvenile justice systems.
Therefore, HHS recognized that this process best met the state's needs
and determined the SACWIS to be complete and meeting all requirements.
Few systems have been determined complete after an on-site review
because of unresolved issues, such as not being able to build links to
other state information systems or not implementing certain eligibility
determination functions. To help states address some of these
development challenges, the SACWIS review team provides the state with
recommendations for complying with SACWIS requirements. For example,
HHS observed during a review in one state that the SACWIS was available
statewide, but information collected in one county was not available to
caseworkers in other counties. The federal officials offered
recommendations to the state to meet the SACWIS requirement that all
information be available statewide. In addition, HHS officials reported
that once the draft report with the results of the SACWIS review are
completed, federal staff schedule a conference call with the state
officials to walk through the system's deficiencies and offer guidance
on how the state can move forward.
HHS facilitates the sharing of information between states developing
SACWIS through an automated system users group that allows state and
federal officials to exchange information, ideas, and concerns.
According to some state and HHS officials, the trust level at these
meetings is very high, which promotes open discussions and also creates
an atmosphere for informal dialogue with HHS. The systems users group
developed out of another active group--the child welfare users group--
when HHS solicited state representatives to help HHS define a model
child welfare information system, which was later used as the basis for
the SACWIS functional requirements after the passage of the 1993
legislation authorizing enhanced federal funding. State officials in
Iowa and New York reported that the systems users group continues to
play an important role in providing a forum for the honest exchange of
information on SACWIS development. For example, child welfare and
technical officials in New York said that the systems users group has
been very beneficial because they have learned from other states'
positive and negative experiences in developing SACWIS, as well as the
experiences unique to states with county-administered agencies. In
addition to the users group, HHS officials also sponsor a listserv--an
electronic mailing list--that allows state officials to exchange
information, and a monthly conference call with state information
technology directors. Iowa child welfare information technology
officials said that they find the monthly SACWIS telephone conference
call helpful because project managers discuss issues such as promising
practices and new regulations.[Footnote 12]
Technical assistance for SACWIS development is also available to states
through the National Resource Center for Information Technology in
Child Welfare (Resource Center). According to survey results, 9 states
said they used the Resource Center for assistance in developing SACWIS
and 14 states reported using it for help with SACWIS maintenance and
improvements. According to Resource Center officials, they assist
states with SACWIS development by helping states understand the
technology that is available for use, providing information on the
automation of child welfare work and converting data, and reviewing the
APD documentation. For example, the Resource Center offered technical
assistance to Pennsylvania to help the state decide if it should
continue development of its current SACWIS, abandon the SACWIS project
and allow the counties to operate individual systems, or design a
different SACWIS. The Resource Center evaluated the current SACWIS to
determine if it could capture information based on the SACWIS
regulations and if it was user friendly for the caseworker. Following
the Resource Center's analysis, Pennsylvania decided to discontinue the
existing SACWIS and develop a new SACWIS. When the Resource Center
opened in 1999--5 years after many states started developing SACWIS--
staff were not very familiar with many of the efforts states made
during development. In an attempt to remedy this lack of knowledge on
states' issues developing SACWIS, Resource Center staff participated in
some of the on-site SACWIS reviews conducted by HHS. Both HHS and
Resource Center officials believe this exposure to the SACWIS systems
enhanced the availability of technical assistance resources and
knowledge available to the states.
Several Factors Affect the States' Ability to Ensure Reliable Data on
Children's Experiences, and Some of HHS's Oversight and Assistance Is
Problematic:
Several factors affect states' ability to collect and report
reliable[Footnote 13] data on children served by state child welfare
agencies, and some problems exist, such as a lack of clear and
documented guidance, with HHS's oversight and technical assistance.
Almost all of the states responding to our survey reported that
insufficient caseworker training and inaccurate and incomplete data
entry affect the quality of the data reported to HHS.[Footnote 14] In
addition, 36 of the 50[Footnote 15] states that responded to our survey
reported that technical challenges, such as matching their state data
element definitions to HHS's data categories, affected the quality of
the data that they report to the federal government. For example, North
Carolina officials told us that while state policy mandates that they
count every location in which a child resides, including hospital
stays, AFCARS regulations say that hospital stays and other short-term
placements should not be included in the count of foster care
placements. In cases where state policy differs from federal policy,
state officials must carefully re-format their data in order to meet
federal reporting requirements. Similarly, during assessments of 6
states' compliance with AFCARS reporting standards, HHS found that
these issues affect data reliability. Despite the assistance that HHS
offers to states, such as testing state data quality and providing the
results to states to aid them in resubmitting data, states report
ongoing challenges receiving clear and documented guidance and
accessing technical assistance.
Insufficient Caseworker Training and Inaccurate and Incomplete Data
Entry Are the Most Common Factors That Affect Data Reliability:
Almost every state responding to our survey and all the states we
visited reported that insufficient training for caseworkers and
inaccurate and incomplete data entry affect the quality of the data
reported to AFCARS and NCANDS (see fig. 1). Although most states
reported these as separate factors, HHS and the states we visited found
that insufficient training and inaccurate and incomplete data entry are
often linked. For example, in official reviews of states' information
systems capability to capture data and report them to AFCARS, HHS
advised states to offer additional training to caseworkers on several
AFCARS data elements, such as recording the reasons for a child leaving
foster care, to improve the accuracy of the data submitted. Similarly,
Oklahoma reported that the state found that caseworkers were
misinterpreting reports of policy violations by foster parents and
inaccurately recording them as abuse or neglect allegations. However,
state officials told us that training is typically one of the first
programs cut when states face tight budget restrictions. For example,
Iowa officials told us that training has been significantly reduced in
recent years because of budget cuts and new workers may wait 2 to 3
months before being trained how to enter data appropriately into their
SACWIS.
Figure 1: Most Common Caseworker Issues That Affect Data Quality:
[See PDF for image]
Notes: Based on responses from 50 states.
[End of figure]
The results reported in the figure are a sum of the states that
reported the issue had a very great affect, great affect, moderate
affect, or some affect on the quality of state data submitted to HHS.
Very great and great affect responses are represented in the top
section of each bar. Moderate and some affect responses are represented
in the bottom section of each bar. States not included answered "no
affect," "don't know," or "no answer.":
Inaccurate and incomplete data entry can also result from a number of
other factors, such as caseworkers' hesitation to ask families for
sensitive information. For example, caseworkers in Oklahoma reported
that they did not feel comfortable asking if a child's mother was
married at the time of birth or if a child is of Hispanic origin--both
of which are required AFCARS data elements. In commenting on a draft of
this report, Oklahoma added that caseworkers did not understand why the
data elements were required and how the federal government used the
information. In addition, Iowa state officials said that caseworkers
may guess the racial backgrounds of children in their care or record
them as unknown, especially when children come from mixed racial
backgrounds, rather than asking the family for the information. HHS
noted similar issues in 5 states that have had an AFCARS
review.[Footnote 16] Caseworkers were inaccurately recording a child's
race as "unable to determine" even though this option should be
selected only if the child's parents or relatives cannot provide the
information, such as when a child is abandoned.[Footnote 17]
Caseworkers, supervisors, and managers in the 5 states we visited
reported that additional factors, such as difficulties balancing data
entry with the time that they spend with the families and children,
contributed to inaccurate or incomplete data entry. In addition, our
recent work on caseworker recruitment and retention found that
caseworkers struggle to balance the time they spend with children and
data entry, and reportedly spend at least 50 percent of their time
documenting case records.[Footnote 18] Supervisors in Iowa explained
that since caseworkers are responsible for ensuring that children and
their families receive the services they need, the caseworkers tend to
initially limit data entry to the information that is necessary to
ensure timely payment to foster care providers, and complete all other
data elements when the caseworkers have time. In addition, caseworkers
in Colorado said that they are between 30 and 60 days behind in their
data entry, so the information in the automated system may not
accurately reflect the current circumstances of children in care. The
caseworkers reported that they tend to concentrate only on entering
data that will allow them to open a case in their SACWIS. HHS's
Inspector General recently issued a report in which more than two-
thirds of the states reported that caseworkers' workloads, turnover, a
lack of training, and untimely and incomplete data entry affected the
reporting of AFCARS data.[Footnote 19]
Technical Challenges, such as Matching State Definitions to Federal
Definitions, Affect Data Reliability:
In addition to data quality being affected by caseworker issues, many
states experienced technical challenges reporting their data to HHS.
The problems reported by states are typically a result of challenges
associated with data "mapping"--matching state data elements to the
federal data elements. For example, 36 states reported in our survey
that matching their state-defined data to HHS's definitions affected
the quality of the data reported to NCANDS and AFCARS. Similarly, 24
states reported that matching the more detailed data options available
in their states' information systems to the federal data elements
affected the quality of the data reported to NCANDS. Twenty-nine states
reported that this issue created challenges in reporting data to
AFCARS. For example, following an AFCARS assessment, HHS instructed a
state that collects detailed information on children's disabilities,
such as Downs Syndrome, Attention Deficit Disorder, and eating
disorders, to map the information to the more limited options in
AFCARS, such as mental retardation and emotionally disturbed. The
Inspector General's report found that states faced similar challenges
mapping their data to meet the AFCARS reporting requirements.
In many cases, states have to balance state policy with federal
requirements to ensure that they are reporting accurate data to AFCARS
and NCANDS, but are not contradicting their state policies. For
example, Texas officials reported that although the findings of their
AFCARS review instructed them to modify their SACWIS to collect, map,
and extract data on guardianship placements, the state does not support
guardianship arrangements.[Footnote 20] In addition, a recent report
from the Child Welfare League of America (CWLA) found that when
reporting the number of times children move from one foster care
placement to another, states varied in the type of placements included
in that count.[Footnote 21] For example, 29 percent of the states
responding to CWLA's survey included respite,[Footnote 22] 25 percent
included runaways, and 16 percent included trial home visits when
reporting the number of placements a child had during the AFCARS report
period. According to federal guidance, the "number of placements"
element is meant to gather information on the number of times the child
welfare agency found it necessary to move a child while in foster care
and that by including runaways or trial home visits, a state is
inflating the number of moves a child experienced. However, North
Carolina officials told us that although the federal definition for
placements instructs states not to include such stays when counting the
number of children's foster care placements, the state instructs them
to count each time a child is sleeping in a different place as a new
placement. The Inspector General reported that the placement
definitions were the most commonly cited source of confusion among the
states surveyed.[Footnote 23]
In addition to the challenges reported in our survey, HHS reported that
transferring data from older data systems into SACWIS affects the
quality of the data reported to AFCARS and NCANDS. HHS officials
reported that they have observed that states experience the biggest
change in data quality when they begin reporting from their SACWIS. In
general, the first data submissions are of low quality because of the
time it takes states to transfer data or the system re-sets the
information for data elements. For example, in 1 state, 65 percent of
the records reviewed by HHS during an AFCARS assessment recorded the
date the children were removed from their homes as July 28, 1997--the
date the SACWIS came on-line; however, the actual dates of removal for
these children ranged from 1988 to 1997.
Although HHS Has Taken Steps to Help States Improve Their Data, Some
Problems with Its Efforts Exist:
HHS provides technical assistance for AFCARS and NCANDS reporting
through a number of resources. HHS officials in the central office and
NCANDS contractor staff serve as the points of contact for states to
ask questions and seek guidance on reporting child welfare data. HHS
officials reported that assistance is offered in a number of ways,
including telephone and e-mail communication. The officials in 3 of the
5 states that we visited said that the one-on-one focused technical
assistance was useful when provided in a timely fashion. Most state
officials found the NCANDS data easier to report, in part because more
people were available for consultation and they were more accessible
and responsive. For example, states have access to four NCANDS
specialists and staff in the contractor's central office when they need
assistance reporting child abuse and neglect information. However, some
of the states we visited reported that only one or two staff in HHS's
central office are available to assist with AFCARS reporting.
In addition, the Resource Center offers states assistance with
improving data quality; however, Resource Center staff reported that
the assistance is geared more towards improving the limited data used
in the federal review process to monitor states' compliance with child
welfare laws and federal outcome measures--CFSR--rather than all the
data reported to HHS. The Resource Center also sponsors an annual
information technology conference during which sessions covering all
data-related issues are held, including practices for ensuring data
quality and outcome evaluation in child welfare.
In conjunction with the national data conference, the HHS officials and
the contractors that operate NCANDS hold an annual technical assistance
meeting for states to share ideas with one another, discuss data
elements that pose difficulties, and explore ways to address these
problems. For example, at a recent technical assistance meeting,
approximately 43 state representatives attended sessions on preparing
the calendar year 2002 NCANDS data submissions and received a detailed
explanation of how the NCANDS staff test states' data submissions for
quality. In addition, an NCANDS state advisory group meets annually to
talk with HHS officials about NCANDS data and their experiences
reporting data. From these meetings, the state advisory group proposes
changes or improvements to NCANDS. HHS and state officials reported
that this partnership has helped ease some of the challenges in
reporting child abuse and neglect data.
In addition to the direct assistance through consultation with HHS
officials and the Resource Center, HHS has made available to states the
software it uses to examine states' AFCARS and NCANDS submissions for
inconsistencies and invalid data. Officials in all the states we
visited said that they regularly use this software, and an HHS official
said that nearly every state has used the software at least once. When
the data are submitted to HHS, they are run through the same software,
and HHS notifies the states of areas where data are missing or
inconsistent and allows the states to resubmit the data after errors
are corrected. For example, HHS officials said that they worked with
one state that was trying to determine the source of data errors in
reporting to AFCARS the race or ethnicity of children in their care.
The state was not able to determine the source of the problem, so an
HHS official examined the state's submissions and helped correct the
data errors. The officials reported that these tests help them to
identify some data quality errors, such as missing data, and said that
they believe that, in general, data have improved in recent years.
However, they indicated that the tests cannot pinpoint the underlying
problems contributing to these errors. Furthermore, one official
reported that no specific efforts have been conducted to track the
individual data elements and, therefore, HHS cannot report on how data
quality has changed over time. The results of these quality tests had
been the basis for penalties levied against states that submitted low
quality AFCARS data before the penalties were rescinded. HHS officials
reported that the penalties served as an effective motivation to states
to correct their data. Although HHS was not able to report how the lack
of penalties might be affecting recent data quality, an official
reported that the agency plans to conduct this analysis in the future.
In an attempt to help states comply with the reporting standards and
address some of the factors that contribute to data quality problems,
HHS performs comprehensive reviews of state information systems'
ability to capture AFCARS data to identify problems associated with
data collection and reporting, and to ensure that the information in
the automated system correctly reflects children's experiences in care.
The assessments include a technical review of the states' computer
code, a comparison of the data from selected cases available in the
information system to the case files, and an improvement plan to
resolve any errors. In addition, HHS officials offer guidance to the
states on improvements that can be made to the information system and
changes to program code used to report the AFCARS data. HHS conducted
pilot reviews in eight states between 1996 and 2000. By March 2003, HHS
had conducted eight official reviews--even though states began
reporting to AFCARS in 1995. According to results from six of the eight
official AFCARS assessments we reviewed, no state met the reporting
requirements for all AFCARS data elements. Table 8 shows a selection of
the data elements and the states' ratings. The problems noted in the
reviews are similar to those we heard from states responding to our
survey and those we visited. For example, most states received ratings
of 2 or 3, indicating technical and/or data entry errors that affect
the AFCARS data quality.[Footnote 24]
Table 8: Selected AFCARS Elements and Six States' Levels of Compliance
in Meeting Reporting Requirements:
Data Elements: Foster Care:
Data Elements: Child race (American Indian or Alaska native, Asian,
Black or African American, Native Hawaiian or other Pacific Islander,
White, unable to determine); Rating factor[A]: (# of states); 2 (4
states); 3 (1 state); 4 (1 state).
Data Elements: Has the child been diagnosed with a disability? (yes,
no, not yet determined); Rating factor[A]: (# of states); 2 (6 states).
Data Elements: Mental retardation, visually/hearing impaired,
physically disabled, emotionally disturbed, other diagnosed
condition[B[(DOES NOT APPLY, APPLIES)] es); Rating factor[A]: (# of
states); 2 (5 states); 3 (1 state).
Data Elements: Has child ever been adopted? (yes, no, unable to
determine); Rating factor[A]: (# of states); 1 (1 state); 2 (3 states); 3
(2 states).
Data Elements: Total number of removals from home[C]; Rating factor[A]:
(# of states); 2 (3 states); 3 (2 states); 4 (1 state).
Data Elements: Date of discharge from previous episode[C]; Rating
factor[A]: (# of states); 2 (3 states); 3 (2 states); 4 (1 state).
Data Elements: Date of latest removal[C]; Rating factor[A]: (# of
states); 2 (3 states); 3 (3 states).
Data Elements: Date of placement in current setting; Rating factor[A]:
(# of states); 2 (5 states); 4 (1 state).
Data Elements: Number of previous placement settings in this
episode[C]; Rating factor[A]: (# of states); 2 (6 states).
Data Elements: Current placement setting (pre-adoptive home, foster
family home-relative, foster family home-nonrelative, group home,
institution, supervised independent living, runaway, trial home visit);
Rating factor[A]: (# of states); 2 (4 states); 3 (1 state); 4 (1 state);.
Data Elements: Out of state placement (yes, no); Rating factor[A]: (#
of states); 2 (4 states); 3 (1 state); 4 (1 state).
Data Elements: Most recent case plan goal (reunify with parents or
principal care takers, live with other relative(s), adoption, long term
foster care, emancipation, guardianship, case plan goal not yet
established); Rating factor[A]: (# of states); 1 (1 state); 2 (5
states).
Data Elements: Date of discharge[C]; Rating factor[A]: (# of states); 2
(2 states); 3 (2 states); 4 (2 states).
Data Elements: Reason for discharge[C] (NOT APPLICABLE, REUNIFICATION
WITH PARENT(S) OR PRIMARY CARETAKER(S), LIVING WITH OTHER
relative(s), adoption, emancipation, guardianship, transfer to another
agency, runaway, death of child); Rating factor[A]: (# of states); 2 (4
states); 3 (1 state); 4 (1 state).
Data Elements: Adoption:
Data Elements: Child race (American Indian or Alaska native, Asian,
Black or African American, Native Hawaiian or other Pacific Islander,
White, unable to determine); Rating factor[A]: (# of states); 2 (4
states); 3 (1 state); 4 (1 state).
Data Elements: Primary basis for determining special needs (not
applicable; racial/original background; age; membership in a sibling
group; medical conditions or mental, physical, or emotional
disabilities; other); Rating factor[A]: (# of states); 2 (6 states).
Data Elements: Mental retardation, physically disabled, emotionally
disturbed[B[(DOES NOT APPLY, APPLIES)] es); Rating factor[A]: (# of
states); 2 (5 states); 3 (1 state).
Data Elements: Visually/hearing impaired (does not apply, applies);
Rating factor[A]: (# of states); 2 (4 states); 3 (2 states).
Data Elements: Other diagnosed condition (does not apply, applies);
Rating factor[A]: (# of states); 2 (6 states).
Source: HHS.
Note: Analysis based on results of AFCARS reviews in 6 states. States
are rated on 66 foster care elements and 37 adoption elements.
[A] HHS rates each data element using a four-point scale: (1) the
AFCARS requirement(s) has not been implemented in the information
system; (2) the technical system requirements for AFCARS reporting do
not fully meet the standards; (3) the technical system requirements for
AFCARS reporting are in place, but there are data entry problems
affecting the quality of the data; (4) all of the AFCARS requirements
have been met.
[B] States are rated on each category. The data elements were combined
for purposes of this analysis.
[C] AFCARS data element used in the CFSR.:
[End of table]
State officials in these 6 states reported that they found the reviews
useful for improving their AFCARS data submissions. In particular, they
valued the thorough review by HHS officials of the computer code states
use to report the data. Some of these officials reported that if all
states were reviewed, the quality of data available in AFCARS would
improve tremendously. However, HHS officials reported that they are not
mandated to conduct the AFCARS reviews and that priority is placed on
other reviews, such as the CFSR and SACWIS reviews. In addition,
officials explained that the AFCARS reviews are not conducted in states
developing SACWIS until the systems are operational. HHS expects to
complete approximately four reviews each year depending on available
resources and has scheduled states through 2006. Similar to the SACWIS
reviews, HHS officials offer recommendations and technical assistance
to states during the review on how they can improve the quality of the
data reported to AFCARS.
Although the states we visited appreciated some of HHS's efforts to
assist with improving state data quality, they and most states
responding to our survey agreed that the assistance is not always
consistent or easily accessible (see fig. 2). States reported similar
information to the Inspector General--AFCARS data elements were not
clearly and consistently defined and technical assistance is effective
but difficult to access.
Figure 2: Federal Practices That Affect Data Quality:
[See PDF for image]
Notes: Based on responses from 50 states.
[End of figure]
The results reported in the figure are a sum of the states that
reported the issue had a very great affect, great affect, moderate
affect, or some affect on the quality of state data submitted to HHS.
Very great and great affect responses are represented in the top
section of each bar. Moderate and some affect responses are represented
in the bottom section of each bar. States not included answered "no
affect," "don't know," or "no answer.":
The primary concerns reported by the states we visited were delays in
receiving clear, written guidance on defining and reporting certain
data elements and the lack of state input in suggesting changes to
AFCARS. Despite the written guidance available to states in the form of
regulations and an on-line policy manual, states reported that the
variation in state policies and practices make it difficult to
interpret how to apply the general guidance. As a result, states
consult with HHS to ensure they are applying the regulations
appropriately. However, in commenting on a draft of this report,
officials in Oklahoma told us that a common concern among the states is
the lack of timely response from HHS when seeking guidance on how to
report data. In addition, officials in New York explained they have
made it a practice to check the HHS Web site on a regular basis for
current guidance, but have not found it a useful tool, and may turn to
other states for guidance on AFCARS reporting. In commenting on a draft
of this report, HHS explained that it first refers states to its Web
site for information and believes that the available guidance addresses
states' concerns in most instances. In addition, the states that have
had an AFCARS review experienced delays in obtaining guidance on how to
proceed following the on-site review. Although they found the review to
be very helpful, some states reported that HHS officials are delayed in
responding to their questions. For example, Texas officials reported
that the state sought clarification on its improvement plan and
submitted additional questions to HHS following the review; however,
when we spoke with the state officials, they said that they had been
waiting 3 months for a response on how to proceed. An HHS official told
us that since the review process is relatively new, the agency is still
developing a process to respond to the states and recognizes that it
has not been responsive to the states already reviewed. In addition,
HHS is taking steps to gather feedback from states and other users of
AFCARS data to determine how to improve the system to make the data
more accurate and useable. As a part of these efforts, HHS has
published a Federal Register notice soliciting comments and held focus
group meetings at national conferences.
The difficulties states face in receiving federal guidance and
assistance, as well as the other challenges they face in reporting
data, may negatively impact the reliability of the data available in
AFCARS and NCANDS. As a result, states are concerned that the national
standards used in the CFSR are based on unreliable data and should not
be used as a basis of comparison and potential financial penalty. The
variation in states' reporting practices may affect the validity of the
measures and may place some states at a disadvantage. For example, the
CWLA and Inspector General studies found that approximately half the
states include the juvenile justice population in their AFCARS reports,
while the other states do not.[Footnote 25] Child welfare experts and
some state officials believe that the states that include children
served by the juvenile justice agency in their AFCARS report may report
a higher number of re-entries into the child welfare system or a higher
number of moves within the system when compared to states that do not
have IV-E agreements[Footnote 26] with their juvenile justice
systems.[Footnote 27] As a result, a state that includes such children
in their AFCARS report are likely to fare less favorably when compared
to the national standard than other states on two outcome measures--
foster care re-entries and stability of foster care placements--and may
face financial penalties associated with the CFSR.
States Are Using Various Practices to Overcome System Development
Challenges and Improve Data on Children's Experiences:
Some states are using a variety of practices to address the challenges
associated with developing SACWIS and improving data reliability,
although no formal evaluations are available on their effectiveness. To
address the challenge of developing a system to meet statewide needs,
states relied on caseworkers and supervisors from local offices to
assist in the design and testing of the system. Few states reported in
our survey strategies to overcome the other key challenges, such as
limited funding and securing knowledgeable contractors, but some states
we visited have devised some useful approaches. For example, Oklahoma
child welfare officials--in order to maximize the limited state funding
for maintaining their SACWIS--reported saving $1 million each year by
hiring some of the contractors who developed their SACWIS as permanent
staff. To improve data reliability, the 5 states we visited routinely
review their data to identify data entry errors so that managers can
ensure that the missing data are entered appropriately. In addition,
some states reported that frequent use of the data, such as publishing
periodic management reports detailing local offices' performance on
outcome measures, helps caseworkers understand the importance of
entering timely information.
States Are Primarily Relying on SACWIS Users to Overcome Some of the
Challenges to Completing Their Systems:
To overcome development challenges, survey respondents emphasized the
importance of including system users in the various phases of
completing SACWIS--planning, design, development, testing, and
implementation. Past GAO work and other research efforts have
determined similar approaches as best practices in building information
systems.[Footnote 28] Forty-four of the 46 states responding to our
survey that they are developing or operating a SACWIS indicated that
they relied on internal users, such as caseworkers and supervisors, in
the development of their systems and 34 of these states said that they
were extremely helpful participants. The extent to which the users were
involved in development differed across the states. For example, in
Texas, caseworkers from all of their child welfare regions were
recruited to provide input on design and development, as well as during
initial testing, pilot testing, and implementation of the system.
Arkansas reported establishing a committee made up of users to review
the work plan and sign off on recommended changes. In addition, states
reported that their system users served a number of purposes, including
serving as experts on the different specialties within child welfare,
such as child abuse, foster care, or adoption, and as representatives
from local or county offices to assist in identifying the diverse
approaches to capturing information across the state. For example,
Indiana reported that caseworkers involved in SACWIS development
represented the unique needs of the different geographical areas of the
state and helped design a uniform statewide system to meet the diverse
needs of large, intermediate, and small local offices.
Ten states noted that user input should not be limited to frontline
workers, such as caseworkers, but should include representatives from
other areas of the agency, such as the financial staff, and other
agencies that serve children, such as child support
enforcement.[Footnote 29] Since many SACWIS link with other state
information systems, states advised that developing a collaborative
relationship with other state agencies will help the development of the
system. While not one of the most common challenges reported in our
survey, New Hampshire reported that one of its challenges with meeting
its SACWIS timeframe was not working collaboratively with other
agencies, such as Temporary Assistance for Needy Families
(TANF)[Footnote 30] and child support enforcement, to develop the
payment component of SACWIS. Similarly, we previously reported that the
difficulty developing linkages between social services agencies limits
the effectiveness of all the programs to serve families.[Footnote 31]
To attempt to overcome this challenge, 26 of the 46 states responding
to our survey that they are developing or operating a SACWIS indicated
that they included external public agency users and 23 reported using
representatives from other state agencies that serve children in
developing their SACWIS. Indiana said that a task force made up of
representatives from the TANF and child support enforcement agencies
was developed to design the linkages between the systems. In addition,
Colorado officials reported that they are working with the Department
of Youth Corrections--an agency that shares the SACWIS with child
welfare--to ensure that the shared screens use the same definitions.
In addition to seeking input from caseworkers and other system users
while developing SACWIS, many states continue to include users as a
part of the implementation teams, to serve as contacts in the field and
provide ongoing assistance, and to provide input on system
enhancements. Alabama responded in our survey that the state had
"mentors" in each county to help caseworkers adjust to the new system.
These mentors continue to provide ongoing support now that the system
is implemented. Similarly, Oklahoma developed Field Implementation
Teams consisting of one contractor and one child welfare staff person.
During system implementation, the teams went to field offices to
provide on-site assistance with using SACWIS and becoming accustomed
with the new method of recording child welfare information.
Furthermore, Oklahoma recruits experienced child welfare field staff
for its SACWIS help desk because of their knowledge of the system and
child welfare policy and practice.
Although states faced other challenges in completing their SACWIS, few
reported implementing approaches to overcome the barriers. According to
survey results, a common problem states faced in developing SACWIS was
receiving insufficient state funding for development. However, in our
previous work on managing information technology, we found that the IT
products can become obsolete in a matter of months rather than years,
calling for more frequent investments in upgrades and
enhancements.[Footnote 32] In addition, officials in Iowa told us that
maintaining systems takes just as much money as building them. States
did not report in our survey approaches for obtaining more funding for
developing SACWIS, and few states reported developing strategies in an
attempt to overcome the challenges associated with tight budgets for
maintaining their systems. For example, Iowa officials engaged in
careful planning with system users to ensure that they addressed the
highest priorities when enhancing the system. In particular, the
officials reported that maintaining tight control over the development
and maintenance processes helps them avoid investing inordinate amounts
of resources to make corrections to the system. In Oklahoma, child
welfare officials reported that they relied on the contractors who
developed their SACWIS to conduct ongoing maintenance activities until
the contract expired in 2001. At that time, the agency hired some of
the contract staff as full-time state employees to continue with the
maintenance activities. State officials explained that this approach
ensured continuity of service, in addition to saving the agency
approximately $1 million each year. Similarly, few states reported on
approaches to overcome the challenge of finding contractors with
knowledge of child welfare practice. However, Iowa officials explained
that once the contract staff are hired, they are required to attend the
same training as new caseworkers to ensure that they are familiar with
the state's child welfare policies and to familiarize themselves with
casework practices.
States Use Strategies, such as Producing Reports That Identify Missing
Data, in an Attempt to Improve the Reliability of the Data Reported to
HHS:
Twenty-eight states reported using approaches to help caseworkers
identify the data elements that are required for federal reporting and
to help them better understand the importance of entering timely and
accurate data. Ten states responding to our survey reported reviewing
the federal reporting requirements in training sessions as a promising
approach they use to improve data quality or as a lesson learned. For
example, Tennessee reported that the state added a component about
AFCARS to the initial and ongoing training workers receive about using
SACWIS. The curriculum addresses the AFCARS report in general and the
individual data elements to help the caseworkers better understand the
purpose of collecting the information. In Nebraska, a "desk aid" that
explains the data elements and where and why to enter them in the
system is available on the caseworkers' computer desktops. In addition,
New York has developed a step-by-step guide explaining to workers how
NCANDS data should be entered, with references to the policy or statute
requiring the information.
To improve data reliability, some states have designed their
information systems with special features to encourage caseworkers to
enter the information. Four states responding to our survey and 3
states we visited designed their SACWIS with color-coded fields to draw
attention to the data elements that caseworker are required to enter.
For example, the AFCARS data fields in Oklahoma's system are coded red
until the data are entered, after which the fields change to blue. In
addition, workers can look at a single screen in the Oklahoma system to
see what AFCARS data elements need to be completed without having to
scroll through the entire case record. Colorado, Iowa, New York, and
Oklahoma have built into their systems alerts--also known as
"ticklers"--to remind caseworkers and supervisors of tasks that they
need to complete. For example, in Iowa, alerts are sent to supervisors
if a caseworker fails to enter the data necessary to complete a payment
to a foster care provider. Whereas, in Oklahoma, a stoplight icon on
the caseworker's computer desktop reminds the worker when tasks are
due. A green light indicates that nothing is due within 5 days; a
yellow light means that something is due within 5 days; and a red light
means that something is overdue. Caseworkers and supervisors in the
states we visited had mixed responses about the usefulness and
effectiveness of the alerts. Some caseworkers found them to be a
nuisance, while other caseworkers and supervisors found them to be
useful tools in managing workloads and prioritizing daily tasks.
Six states reported that the best way to improve data quality was to
use the data in published reports and hold the caseworkers and
supervisors accountable for the outcomes of the children in their care.
In addition, 6 states responding to our survey reported using the data
available in their information systems to measure state outcomes
similar to the CFSR. State officials reported that this approach is an
effective way to get local offices invested in the quality of the data.
For example, North Carolina publishes monthly reports for each county
comparing their performance on state data indicators, such as the
length of time children spend in care, to counties of similar size and
the state as a whole. County officials reported that these reports
encourage workers to improve the quality of the data collected and
entered into the state system since their performance is being widely
published and compared to other counties.
In addition, all the states we visited reported that frequent review of
their data, such as using software from HHS to test their AFCARS and
NCANDS data to pin-point data entry errors prior to submitting them to
HHS, has helped improve data quality. When the states identify poor
data, they alert the caseworkers and supervisors of needed corrections
and data entry improvements. For example, Colorado runs these reports
about 4 to 5 times a year, with one run occurring approximately 6 weeks
before each AFCARS submission. When the data specialists find errors,
they notify the caseworker to clean up the data. New York officials
told us that they incorporate the results from these tests in training
if a consistent pattern of errors is identified.
Conclusion:
While most states are developing statewide information systems,
challenges with data reliability remain. Although SACWIS development is
delayed in many states, state officials recognize the benefits of
having a uniform system that enhances the states' ability to monitor
the services provided and the outcomes for children in their care.
Although states began reporting to NCANDS in1990 and were mandated to
begin reporting to AFCARS in 1995, most states continue to face
challenges providing complete, accurate, and consistent data to HHS. In
addition, the results of more recent HHS efforts, such as conducting
AFCARS-related focus groups, are unknown. Reliable data are essential
to the federal government's development of policies that address the
needs of the children served by state child welfare agencies and its
ability to assist states in improving child welfare system
deficiencies. Without well-documented, clearer guidance and the
completion of more comprehensive reviews of states' AFCARS reporting
capabilities, states are limited in overcoming challenges that affect
data reliability. Because these challenges still remain, HHS may be
using some questionable data as the foundation for national reports and
national standards for the CFSR and may not have a clear picture of how
states meet the needs of children in their care.
Recommendation to the Secretary of Health and Human Services:
To improve the reliability of state-reported child welfare data, we are
recommending that the Secretary of HHS consider, in addition to HHS's
recent efforts to improve AFCARS data, ways to enhance the guidance and
assistance offered to states to help them overcome the key challenges
in collecting and reporting child welfare data. These efforts could
include a stronger emphasis placed on conducting AFCARS reviews and
more timely follow-up to help states implement their improvement plans
or identifying a useful method to provide clear and consistent guidance
on AFCARS and NCANDS reporting.
Agency Comments:
We obtained comments on a draft of this report from the Department of
Health and Human Services' Administration for Children and Families
(ACF). These comments are reproduced in appendix III. ACF also provided
technical clarifications, which we incorporated when appropriate.
ACF generally agreed with our findings and commented that the report
provides a useful perspective of the problems states face in collecting
data and of ACF's effort to provide ongoing technical assistance to
improve the quality of child welfare data. In response to our
recommendation, ACF said that we categorized its efforts as "recent"
and did not recognize the long-term efforts to provide AFCARS and
NCANDS related guidance to the states. Although we did not discuss each
effort in depth, we do mention the agency's ongoing efforts in our
report. However, we refer to the recent efforts in the recommendation
in recognition of the agency's current activities to formally obtain,
document, and incorporate feedback from the states with regard to
collecting and reporting adoption and foster care data. ACF also noted
in its comments that the data definitions need to be updated and
revised and said it is currently in the process of revising the AFCARS
regulations to further standardize the information states are to
report--which we acknowledge in our report. In addition to the steps
HHS is taking to further improve the AFCARS data, our recommendation
encourages HHS to consider ways to enhance the ongoing guidance and
assistance offered to states to help them overcome the key challenges
in collecting and reporting child welfare data. ACF requested specific
recommendations on approaches to overcome the difficulty of collecting
and merging information from multiple state and county programs into a
single national database. While there may be additional methodologies
that the agency could use to overcome such challenges, our
recommendation focuses on improving the guidance already offered to the
states as a step to helping them better comply with the reporting
requirements.
In addition, ACF added that although staff turnover in state child
welfare agencies is a significant contributor to data quality issues,
we did not focus on this as a significant factor. ACF also commented
that it is firmly committed to continue to support the states and to
provide technical assistance and other guidance as its resources will
permit. However, because we recently issued a detailed report on a
variety of caseworker issues,[Footnote 33] we primarily focused in this
report on the key data entry challenges caseworkers face and refer
readers to our previous work for additional information on challenges
related to caseworker recruitment and retention and their affect on
child welfare agencies. In commenting on our previous work prior to its
release, HHS indicated that it does not have the authority to require
states to address factors that contribute to staff turnover, such as
high caseloads and said that it has limited resources to assist the
states in the area of staff recruitment and retention.
ACF commented that it provided increased funding to the National
Resource Centers in fiscal year 2003, which they believe will improve
ACF's ability to provide assistance to the states. After receiving the
draft report for comment, HHS separately provided information on an
additional service the National Resource Center for Information
Technology in Child Welfare provides to states. In an effort to assist
states with improving the quality of their AFCARS data, the Resource
Center will review states' programming code used for AFCARS data. As of
June 2003, HHS reported that the Resource Center provided this
assistance to Arkansas, Louisiana, Mississippi, North Carolina, Nevada,
New Jersey, and Rhode Island, and 3 states--Maryland, Michigan, and
Wisconsin--and the District of Columbia have requested the assistance.
In response to our survey methodology, ACF requested that we explain
why the territory of Puerto Rico was not included in the state survey.
Although Puerto Rico receives federal child welfare funds, we
traditionally focus on the states and therefore do not include the U.S.
territories, including American Samoa, the Commonwealth of the Northern
Mariana Islands, Guam, Puerto Rico, and the Virgin Islands, in the
scope of our reviews.
Finally, in response to our discussion of the AFCARS review process,
ACF provided a few clarifications. During the course of our review, an
HHS official characterized the AFCARS review process as relatively new
and explained that the agency is still developing a process to respond
to the states following the completion of the on-site review. When
responding to a draft of this report, ACF disagreed with this
characterization. ACF commented that the review process has been in
place since 1996, pointing to the pilot reviews as evidence that the
agency has a defined process. However, when we requested AFCARS reports
for review, HHS explained that the states undergoing pilot reviews
would be re-reviewed and that the official process was formalized in
2001 with the release of an AFCARS review guide and the start of the
official reviews. In addition, ACF commented that SACWIS reviews do not
take priority over AFCARS reviews. However, officials had previously
explained that although SACWIS and AFCARS reviews can happen at the
same time, in practice, the AFCARS reviews are scheduled to occur in
the states that are developing SACWIS after they have participated in a
SACWIS review. Furthermore, ACF explained that states do not develop
their improvement plan following the conclusion of the AFCARS review.
Instead, ACF officials draft the plan for the state. Although state
representatives had described a challenge in receiving timely feedback
on their improvement plan, we have changed the language in the report
to reflect ACF's comment.
We also provided a copy of our draft to child welfare officials in the
5 states we visited--Colorado, Iowa, North Carolina, New York, and
Oklahoma. Iowa and New York had technical clarifications, which we in
incorporated when appropriate. Oklahoma provided additional
information, which was incorporated. Colorado had no suggested
corrections or edits. North Carolina did not provide any comments.
As agreed with your offices, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days
after its issue date. At that time, we will send copies of this report
to the Secretary of Health and Human Services, state child welfare
directors, and other interested parties. We will make copies available
to others on request. In addition, the report will be available at no
charge on GAO's Web site at http://www.gao.gov. If you or your staff
have any questions or wish to discuss this material further, please
call me at (202) 512-8403 or Diana Pietrowiak at (202) 512-6239. Key
contributors to this report are listed in appendix IV.
Cornelia M. Ashby:
Director, Education, Workforce, and Income Security Issues:
Signed by Cornelia M. Ashby:
[End of section]
Appendix I: Scope and Methodology:
To determine the progress states have made in developing Statewide
Automated Child Welfare Information Systems (SACWIS), we surveyed all
50 states and the District of Columbia through a Web-based survey. We
pretested the survey instrument in Maryland and the District of
Columbia. We received responses from 49 states and the District of
Columbia. The state of Nevada did not respond to the survey. We
discarded a question that asked states to report the date their advance
planning document (APD) was approved by the Department of Health and
Human Services (HHS). Due to a technical error, the date was truncated
and a valid answer was not stored in the responses. Of the 50 survey
responses, 46 were from states that are pursuing SACWIS development.
The 4 states not developing SACWIS were asked to skip sections of the
survey that asked about SACWIS development, system modifications, and
supported services and links. We did not independently verify the
survey responses. In addition, we visited 5 states to obtain more
detailed and qualitative information regarding states' experiences
developing SACWIS. We conducted site visits in Colorado, Iowa, New
York, North Carolina, and Oklahoma. We selected these states to
represent a range of SACWIS development stages, sizes of foster care
populations, and geographic locations. During our site visits, we
interviewed state and local child welfare staff, state and local staff
that regularly exchange information with the child welfare agency, and
private contractors. We also spoke with HHS staff in the central and
regional offices, National Resource Center officials, contractors
involved in SACWIS development, and child welfare experts from the
Child Welfare League of America and the American Public Human Services
Association.
To determine how states and HHS ensure reliable data exist on children
served by child welfare agencies we surveyed states using the above-
mentioned survey instrument. In addition, we interviewed state and HHS
officials on their efforts to analyze and compile data and HHS's role
in providing technical assistance to states. We spoke with state
officials during our site visits and HHS officials in the central and
regional offices and attended the 6th National Child Welfare Data
Conference. We obtained and reviewed available SACWIS and Adoption and
Foster Care Analysis and Reporting System (AFCARS) reports. At the time
of our review, HHS had conducted 26 SACWIS reviews. We obtained and
reviewed 23 reports. The remaining reports were not available for
review because HHS has not yet completed the report or shared the
results with the state. Most of the SACWIS reports were considered
drafts since many states are in the process of resolving issues with
completing their systems. We reviewed AFCARS assessment reports from 6
of the 8 states assessed by HHS--Arkansas, Connecticut, New Mexico,
Texas, Vermont, and Wyoming. HHS conducted reviews in Delaware and West
Virginia after we completed our analysis. We did not review any of the
eight pilot review reports since these were not final reports and HHS
plans to conduct official reviews in these states. These AFCARS
assessment reports were analyzed to understand the breadth of on-site
assistance HHS provides to states during the review and to identify
common data collection and reporting difficulties among states.
Finally, we talked with officials in 6 of the 8 states that had an
AFCARS review about their experiences during the review and child
welfare experts.
To identify practices state and local child welfare agencies are using
to help ensure the accuracy, timeliness, and completeness of child
welfare data we interviewed state and local child welfare officials on
our site visits and inquired about the practices they are employing. We
also included questions on practices and lessons learned in our survey.
In addition, we spoke with numerous child welfare experts, including
individuals from the National Resource Center for Information
Technology in Child Welfare, the Child Welfare League of America, and
the American Public Human Services Association[Footnote 34].:
[End of section]
Appendix II: State SACWIS Stages of Development:
[See PDF for image]
Source: HHS.
Note: Status is as of May 22, 2003.
[End of table]
[End of section]
Appendix III: Comments from the Department of Health and Human Services:
DEPARTMENT OF HEALTH & HUMAN SERVICES:
ADMINISTRATION FOR CHILDREN AND FAMILIES Office of the Assistant
Secretary, Suite 600:
370 L'Enfant Promenade, S.W. Washington, D.C. 20447:
DATE: JUL 2 2003:
TO: Cornelia M. Ashby:
Director, Education, Workforce, and Income Security Issues:
FROM: Wade F. Horn, Ph.D. /,4Z 72 Assistant Secretary:
for Children and Families:
[See PDF for image]
[End of figure]
SUBJECT: Comments on the GAO Draft Report: "Most States are Developing
Statewide Information Systems, but the Reliability of Child Welfare
Data Could be Enhanced" (GAO-03-809):
Attached are the Administration for Children and Families' comments on
the GAO Draft Report: "Most States are Developing Statewide Information
Systems, but the Reliability of Child Welfare Data could be Enhanced"
(GAO-03-809).
We appreciate the opportunity to comment on this report. Should you
have any questions or need additional information, please contact Susan
Orr, Associate Commissioner of the Children's Bureau at (202) 205-8618.
Attachment:
COMMENTS OF THE ADMINISTRATION FOR CHILDREN AND FAMILIES (ACF) ON THE
GENERAL ACCOUNTING OFFICE'S (GAO) DRAFT REPORT: "MOST STATES ARE
DEVELOPING STATEWIDE INFORMATION SYSTEMS, BUT THE RELIABILITY OF CHILD
WELFARE DATA COULD BE ENHANCED," GAO-03-809:
The Administration for Children and Families (ACF) appreciates the
opportunity to comment on the GAO's draft report.
GAO Recommendation:
To improve the reliability of state-reported child welfare data, GAO
recommends that the Secretary of HHS consider, in addition to HHS's
recent efforts to improve AFCARS data, ways to enhance the guidance and
assistance offered to states to help them overcome the key challenges
in collecting and reporting child welfare data. These efforts could
include a stronger emphasis placed on conducting the Adoption and
Foster Care Analysis and Reporting System (AFCARS) reviews and more
timely follow-up to help states implement their improvement plans or
identifying a useful method to provide clear and consistent guidance on
AFCARS and the National Child Abuse and Neglect Data System (NCANDS)
reporting.
ACF Comments:
The background provided in the report was useful in giving a
perspective of the problem states face in collecting data and ACF's
effort to provide ongoing technical assistance to improve the quality
of that data. However, the GAO categorizes ACF's efforts as "recent"
and does not recognize the long-term efforts to provide AFCARS and
NCANDS related guidance to the states. We recommend that the term
"ongoing" be substituted for the term "recent" in the "Recommendation"
section of the report.
ACF has provided extensive guidance on how states can improve the
quality of their AFCARS data since before the first AFCARS report was
submitted. To categorize this effort as "recent" might be misconstrued
by the reader. What follows is a short list of some of the guidance
provided by this agency:
* State Technical Advisory Group:
March 1994
May 1994
March 1995:
* Statewide Automated Child Welfare Information Systems (SACWIS)
Conferences: AFCARS sessions were held during these conferences to
provide assistance and updates to states:
September 1994
September 1995:
* ACF User Group Meetings: Held two times a year since 1994. AFCARS
workshops were conducted as well as one-on-one meetings with states to
provide technical assistance.
* AFCARS Conferences/Meetings:
State Technical Assistance Workshops: San Francisco and Washington D.C.
- July & September 1994:
State Technical Assistance Workshops: San Francisco and Washington D.C.
- July & September 1995:
SACWIS/AFCARS Conference - 1996 AFCARS Data Conference -1997 AFCARS
Data Conference -1998 AFCARS Data Conference -1999 AFCARS Data
Conference -2000 AFCARS Data Conference -2001 AFCARS Data Conference -
2002 AFCARS Data Conference -2003:
* Technical Assistance Documents:
AFCARS: Part I - Strategic Planning Guide: A guide for state systems
planning and development; Part II - Data Dictionary.
AFCARS Guide to the Elements - September 1995 (Rescinded):
NRC-ITCW AFCARS Toolkit (Available in draft form, final version to be
released summer of 2003):
Disability Code Table - Last updated 10/30/01:
State Data Compliance Software - First issued in 1995 State Data
Quality Software - First issued in 1996 Statistical package of
programming logic for the State Data Profile - 1998:
* Policy Guidance:
Policy Interpretation Questions (PIQ) March 1994 Policy Interpretation
Questions (PIQ March 1995 ACYF-PI-CB-95-09 (REVISED), issued 5/23/95:
Methods of Extracting Data for Submission to AFCARS:
ACYF-CB-IM-00-04, issued March 30, 2000: Adoption and Foster Care
Analysis and Reporting System Re-issuance of Technical Bulletins ACYF-
CB-PI-01-07, issued 5/4/2001: Change in the State AFCARS File Name
Child Welfare Policy Manual: September 24, 2001 (Replaces the two PIQs
for AFCARS):
ACYF-CB-IM-02-03, issued April 8, 2002: Decision regarding AFCARS
Penalties, AFCARS File Name:
ACYF-CB-11\1-02-05, issued April 22, 2002: AFCARS Assessment Reviews:
ACF has issued several Technical Bulletins since 1994. These have been
updated and modified as necessary.
[See PDF for image]
[End of table]
As noted in the GAO report, we concur with the general finding that
states face significant challenges as they strive to improve the
quality of the data that they report to the federal government.
However, we are concerned that the report does not focus more attention
on the continuing problem that the turnover of state and contractor
staff has on the quality of data reported to the federal government.
The GAO report briefly mentions the impact that the loss of social
workers and contractor technical staff has on the quality of data, but
the report does not focus on this as a significant factor. ACF has
found that staff turnover to be a significant contributor to data
quality issues. New social workers must be trained on how to
effectively use the systems and what elements are important from a data
requirement perspective, while new technical staff must re-learn the
intricacies of the national reporting requirements. ACF remains firmly
committed to continue to support state technical staff and to provide
technical assistance and other guidance as our resources will permit.
We are continuously exploring new opportunities to extend our limited
and shrinking resources to provide the types of technical assistance
suggested in the GAO report to states and their staffs.
Other Comments:
Page 1, first paragraph, third sentence.
"Through these federal matching funds, states can develop and implement
Statewide Automated Child Welfare Information Systems (SACWIS) to
manage their child welfare cases as well as to report child abuse and
neglect, foster care, and adoption information to the federal
government.":
This sentence could be enhanced by noting that states have the option
of building SACWIS or a non-SACWIS system to support their business
processes. The GAO should consider replacing the current sentence with
the following:
"Through these federal matching funds, states can develop and implement
information systems to manage their child welfare cases as well as to
report child abuse and neglect, foster care, and adoption information
to the federal government. If the state elects to pursue implementing a
comprehensive case management system to support the different child
welfare programs that are administered by the state, the state may
elect to develop a system that meets the requirement for a Statewide
Automated Child Welfare Information System (SACWIS). If a state elects
to implement a SACWIS-compliant system, the state is eligible to
receive additional funding to build and operate that system. Regardless
of the type of system that a state elects to implement, child welfare
caseworkers at the county or local level are the key personnel who
collect and document information on children and families served by
child welfare agencies, in addition to performing a wide range of
services to protect children, such as investigating child abuse and
neglect reports or providing support services to maintain the children
in their homes.":
* Page 3, second paragraph, last sentence and page 4, first sentence. We
are concerned about the example cited at the bottom of the page, which
indicates that one state waited a year for technical assistance.
"For example, state officials in Oklahoma said that it took HHS close
to one year to provide written guidance on how to report the amount of
monthly adoption assistance subsidies. As a result, the state may have
misreported the amount of money the state awarded in subsidy payments
to adoptive families.":
We question whether the summary of the concern really conveys what
happened. This is particularly true because issues or questions
concerning the adoption assistance payment data elements are not
generally complex or difficult to answer.
Page 12, first paragraph, second complete sentence.
"As a result, Iowa designed a SACWIS to reflect this work process by
linking two databases - one to record child abuse and neglect
information and one to record ongoing case records - that share
information with one another.":
We recommend that GAO modify this sentence to read as follows:
"As a result, Iowa designed its SACWIS to have an interface with the
state child abuse and neglect data system so that information can be
shared between the two systems. Most states have built functionality
that supports the investigation of child abuse and neglect allegations
into their SACWIS.":
ACF does not view the child abuse and neglect system in Iowa as part of
its SACWIS. However, the state has met the SACWIS requirements in this
area by building an interface between the two systems.
* Page 12, second paragraph, first sentence.
"Forty-three states reported in our survey that they have or are
planning to incorporate child welfare and administrative components
into their SACWIS (see Table 3).":
This sentence is somewhat confusing since many of the functional areas
noted in the reference on page 13, Table 3, must be included in a
SACWIS for it to be considered SACWIS-compliant. The issue is further
complicated because some of the required functional areas identified in
the table may be addressed through an interface to another system,
while some must be included within the SACWIS application. The way this
section is written, it may leave a reader with the impression that
some states are building SACWIS systems that do not
meet all of the SACWIS requirements. We recommend that the first
sentence in this paragraph be modified as follows:
"Forty three states reported ... they are planning to incorporate
optional child welfare and administrative components into their SACWIS
(see Table 3).":
This approach would probably necessitate the addition of a footnote
explaining that some SACWIS components are mandatory while others are
optional. Also, please see our comments below related to Table 3.
* Page 12, second paragraph, second sentence.
"Some of these components, such as child protection and determining IV-
E eligibility, are required by SACWIS regulations, to the extent
practicable.":
The phrase "to the extent practicable" is used in the regulation and
statute in relation to certain mandatory interfaces. The use of the
phrase here is incorrect and it should be deleted.
* Page 12, second paragraph, second sentence from bottom.
"Currently, 27 states reported in our survey that they are at some
stage of using their SACWIS to track independent living services, and
an additional 14 states plan to include this component in their system
in preparation for the legislative requirements.":
The word "legislative" should be deleted. ACF is responsible for
promulgating any National Youth in Transition Database (NYTD)
requirements.
* Page 13, Table 3. As previously noted, this table may leave the reader
with the misunderstanding that some states are building non-compliant
SACWIS systems. We provide the table below to identify our concerns.
At the very least, we recommend that the GAO add the following
disclaimer to this table: "HHS has indicated that a state's failure to
support most of these functional components would be a problem that the
state would need to correct.":
[See PDF for image]
[End of figure]
Page 14, last sentence. This sentence states that North Dakota did not
report in the GAO survey the reason for stopping SACWIS development. It
is our understanding that North Dakota stopped its SACWIS initiative
due to a combination of issues, including programmatic structure.
Page 17, Table 6, titled, "Number of Months States Delayed in SACWIS
Development," indicates states were asked to report the number of
months the delays exceeded the time line outlined in their Advance
Planning Document (ADP). Based on our experience with the states
identified in Table 6, we believe that the respondents may have replied
to the survey using different definitions of what "delayed" meant and
different methodologies for counting the number of months that they
were delayed. ACF recommends that GAO explore ways to enhance the
reliability of the data presented in its report.
Page 23, footnote 13. This footnote states, "The analysis of survey
responses about reporting data to HHS is based on responses from 49
states and the District of Columbia." The GAO should consider
explaining why Puerto Rico was excluded from the survey.
Page 24. The GAO should consider enhancing this section of the report
to clarify the concerns being described. For example, the first two
sentences state:
"Almost every state responding to our survey and all the states we
visited
reported that insufficient training for caseworkers and inaccurate and
incomplete data entry affect the quality of the data reported to AFCARS
and NCANDS (see fig. 1.). Although most states did not report that
insufficient training contributed to inaccurate and incomplete data
entry, HHS and the states we visited found that these issues are often
linked.":
The first and second sentences appear to be contradictory. In sentence
one the report indicates that the states reported that insufficient
training for caseworkers affects the quality of the data, while the
second sentence notes that most states did not report this to be the
case. We recommend that the GAO enhance this area of its report to
describe what was found. Furthermore, if this conclusion is primarily
supported by empirical analysis, the GAO should consider downplaying
the importance of this finding.
Page 25, Figure 1 and Page 33, Figure 2. To clarify the survey results,
we recommend that the bar chart include demarcation lines for states
indicating the various factors had "some" or a "moderate" affect on the
quality of data. The current presentation approach appears to present a
more negative view of the impact these factors have on the quality of
data than the survey questions may have solicited.
* Page 26, first paragraph, first sentence.
"Caseworkers were inaccurately recording a child's race as unknown even
though this option should be selected only if the child's parents or
relatives cannot provide the information, such as when a child is
abandoned.":
The finding from AFCARS reviews indicates that information is often
defaulted to the response of "unable to determine" in order for the
element not to fail the missing data standard, not that workers are
recording "unknown.":
Pages 27 and 28. Having identified the difficulty of collecting
information from multiple state and county programs into a single
national database, ACF requests that GAO recommend solutions for this
concern. Specifically, we request that the GAO describe methodologies
on how to merge divergent data from multiple state and county programs
with different program requirements and data definitions into a single
common database in a manner that will result in better data. We welcome
suggestions for an approach that would be more efficient, effective and
economical than the approach currently used.
* Pages 28 through 35. ACF has increased the accessibility of technical
assistance resources available to the states through its support of the
National Resource Centers (NRCs), specifically the National Resource
Center for Information Technology in Child Welfare (NRC-ITCW). The
Children's Bureau (CB) increased funding to the NRC for FY 2003. The
NRC-ITCW provides substantial technical assistance to states in data-
related matters such as programming, input, interpretation, analysis,
and reporting for AFCARS and NCANDS data. The increased funding will
improve ACF's ability to provide this assistance to our state partners.
Concerning the states' comment on the lack of state input in suggesting
changes to AFCARS, it should be noted that AFCARS is defined in
regulation. Therefore, it cannot be changed or modified without a
process of public comment and a regulation change.
All publicly available AFCARS documentation is accessible through ACF's
web page. In order to enable self-sufficiency and to sustain the
state's learning curve, state staff are asked to first read the
guidance available through ACF's web page. In most instances, the
available guidance addresses their concern and we do not hear back from
them.
* Pages 32 and 33. ACF issued policy clarifications regarding placement
information on July 5, 2002. Additionally, the AFCARS report is in the
regulation process to further standardize the information that states
are to report. We recognize that the definitions need to be updated and
revised.
* Page 32, first paragraph, fourth sentence. The statement, "However,
HHS officials reported that they are not mandated to conduct the AFCARS
reviews and that priority is placed on other reviews, such as the CFSR
and SACWIS review," is incorrect. SACWIS reviews do not take precedence
over AFCARS reviews.
* Page 34. ACF wishes to advise the GAO that information pertaining to
the AFCARS review process, specifically the improvement plan, is
incorrect. States do not submit a proposed AFCARS improvement plan
(AIP). The AIP is included in the final report to the state following
an AFCARS assessment review (AAR). The AIP is based on the findings of
the AAR and indicates how states are to proceed to correct the errors.
States are given an opportunity to review the preliminary findings on-
site and make corrections, negotiate changes, and ask for clarification
at that time. The findings in the final report reflect the preliminary
findings. In some instances, additional findings are made regarding the
quality of the data after an analysis of the case file review is
completed. States are to provide ACF with estimated due dates for the
tasks listed in the AIP.
Page 34, first paragraph, fifth sentence. This sentence, indicating
that AFCARS reviews are relatively new, is incorrect. Reviews have been
conducted either as pilots or actual reviews since 1996. The statement
that the agency is still developing a process to respond to the states
is also not accurate. There is a defined process for assessing state
progress during the AFCARS improvement phase. This process was not part
of the pilot program; therefore, it is being fine tuned and is a fluid
process. Since AFCARS reviews are not mandatory, work related to the
reviews on the federal level has to be adjusted to accommodate work on
mandated priority items.
Page 34, first paragraph, sixth sentence. This section discusses HHS
taking steps to gather comments from states and should be a separate
paragraph. This is a separate topic than what is being discussed in the
paragraph where it is located.
* Page 35, footnote 22 has a few errors.
1) The first sentence should be changed, as the juvenile justice agency
does not claim federal title IV-E funds. The title IV-E agency claims
costs incurred by the juvenile justice agency.
2) The second sentence indicates that delinquents who meet IV-E
eligibility criteria and "present with child protection and/or
dependency issues, in addition to their delinquent status, may be
eligible for title IV-E foster care." This sentence should be revised
to read that delinquents who meet title N-E eligibility criteria are
eligible for the program.
3) The following clarification should be added at the end of the last
sentence: "for children who are receiving title IV-E funding or for
children in foster care under the placement and care responsibility of
the child welfare agency." As the sentence is written, it indicates
that all adjudicated delinquents must receive IV-E/IV-B protections,
but that is only true if they are receiving IV-E funds or meet our
definition of foster care.
There are a couple of states that have an umbrella agency where the IV-
B/IV-E agency has responsibility for all juvenile justice children.
Therefore, although those children may not receive a foster care
maintenance payment, the state may still have to meet the state plan
requirements and protections for those youth if they are in foster
care.
[End of section]
Appendix IV: GAO Contacts and Acknowledgments:
GAO Contacts:
Diana Pietrowiak, (202) 512-6239 Sara L. Schibanoff, (202) 512-4176:
Staff Acknowledgments:
In addition to those named above, Leah DeWolf and Rachel Seid made key
contributions to this report. Avrum Ashery, Patrick DiBattista, Barbara
Johnson, Valerie Melvin, and Rebecca Shea also provided key technical
assistance.
[End of section]
Bibliography:
The American Public Welfare Association. Statewide Automated Child
Welfare Information Systems: Survey of State Progress. Washington,
D.C., July 1997.
The American Public Welfare Association. Child Welfare Information
Systems: Some Concepts and Their Implications. Washington, D.C., July
1994.
The American Public Welfare Association. Survey of State Child Welfare
Information Systems: Status of AFCARS and SACWIS. Washington, D.C.,
April 1995.
Caliber Associates, Analysis of State Child Welfare Data: VCIS Survey
Data from 1990 through 1994, May 1998, Department of Health and Human
Services.
Center for Technology in Government, University of Albany, SUNY. Tying
a Sensible Knot: A Practical Guide to State-Local Information Systems.
Albany, N.Y., June 1997.
Child Welfare League of America. National Working Group Highlights,
"Child Maltreatment in Foster Care: Understanding the Data."
Washington, D.C., October 2002.
Child Welfare League of America. National Working Group Highlights,
"Placement Stability Measure and Diverse Out-of-Home Care Populations."
Washington, D.C., April 2002.
U.S. Department of Health and Human Services, Administration for
Children and Families, Administration on Children, Youth and Families,
Children's Bureau. Child Maltreatment 2001. Washington, D.C., 2003.
U.S. Department of Health and Human Services, Administration for
Children and Families, Administration on Children, Youth and Families,
Children's Bureau. Child Welfare Outcomes 1999: Annual Report.
Washington, D.C., n.d.
U.S. Department of Health and Human Services, Office of Inspector
General. Adoption and Foster Care Analysis and Reporting System
(AFCARS): Challenges and Limitations. Washington, D.C., March 2003.
[End of section]
Related GAO Products:
Child Welfare and Juvenile Justice: Federal Agencies Could Play a
Stronger Role in Helping States Reduce the Number of Children Placed
Solely to Obtain Mental Health Services. GAO-03-397. Washington, D.C.:
April 21, 2003.
Child Welfare: HHS Could Play a Greater Role in Helping Child Welfare
Agencies Recruit and Retain Staff. GAO-03-357. Washington, D.C.: March
31, 2003.
Human Services: Federal Approval and Funding Processes for States'
Information Systems. GAO-02-347T. Washington, D.C.: July 9, 2002.
Foster Care: Recent Legislation Helps States Focus on Finding Permanent
Homes for Children, but Long-Standing Barriers Remain. GAO-02-585.
Washington, D.C.: June 28, 2002.
Human Services Integration: Results of a GAO Cosponsored Conference on
Modernizing Information Systems. GAO-02-121. Washington, D.C.: January
31, 2002.
District of Columbia Child Welfare: Long-Term Challenges to Ensuring
Children's Well-Being. GAO-01-191. Washington, D.C.: December 29, 2000.
Child Welfare: New Financing and Service Strategies Hold Promise, but
Effects Unknown. GAO/T-HEHS-00-158. Washington, D.C.: July 20, 2000.
Welfare Reform: Improving State Automated Systems Requires Coordinated
Federal Effort. GAO/HEHS-00-48. Washington, D.C.: April 27, 2000.
Foster Care: States' Early Experiences Implementing the Adoption and
Safe Families Act. GAO/HEHS-00-1. Washington, D.C.: December 22, 1999.
Foster Care: HHS Could Better Facilitate the Interjurisdictional
Adoption Process. GAO/HEHS-00-12. Washington, D.C.: November 19, 1999.
Foster Care: Effectiveness of Independent Living Services Unknown. GAO/
HEHS-00-13. Washington, D.C.: November 5, 1999.
Foster Care: Kinship Care Quality and Permanency Issues. GAO/HEHS-99-
32. Washington, D.C.: May 6, 1999.
Juvenile Courts: Reforms Aim to Better Serve Maltreated Children. GAO/
HEHS-99-13. Washington, D.C.: January 11, 1999.
Child Welfare: Early Experiences Implementing a Managed Care Approach.
GAO/HEHS-99-8. Washington, D.C.: October 21, 1998.
Foster Care: Agencies Face Challenges Securing Stable Homes for
Children of Substance Abusers. GAO/ HEHS-98-182. Washington, D.C.:
September 30, 1998.
Managing Technology: Best Practices Can Improve Performance and Produce
Results. GAO/T-AIMD-97-38, January 31, 1997.
Child Welfare: HHS Begins to Assume Leadership to Implement National
and State Systems. GAO/AIMD-94-37. Washington, D.C.: June 8, 1994.
Executive Guide: Improving Mission Performance Through Strategic
Information Management and Technology. GAO/AIMD-94-115. May 1, 1994.
FOOTNOTES
[1] Throughout this report, references to state survey responses
include the District of Columbia. Forty-six of these states reported
that they are developing or operating a SACWIS. Nevada, which HHS
reported has an operational SACWIS, did not respond to our survey.
[2] When states choose to develop information systems that include
other human services, such as food stamps, child support enforcement,
or Medicaid, states must submit APDs to each cognizant federal agency.
In a hearing held last year before the Subcommittee on Technology and
Procurement Policy, House Committee on Government Reform, we testified
that the federal agencies do not have systems to monitor states
requests for federal approval and funding through the life cycle of a
state request. (See U.S. General Accounting Office, Human Services:
Federal Approval and Funding Processes for States' Information System,
GAO-02-347T (Washington, D.C: July 9, 2002.))
[3] Forty-four states provided information on the total amount of
federal funds they received to develop and operate SACWIS. Alaska,
Hawaii, Missouri, North Carolina, Texas, and Vermont did not report
federal funding information. Nevada did not respond to our survey.
State-reported figures may include some funding allocated in fiscal
year 2003 since the survey was issued in October 2002 and completed as
late as December 2002.
[4] Forty-four states provided information on the total amount of state
funds used to develop and operate SACWIS. Arkansas, Hawaii, Missouri,
North Carolina, Texas, and Vermont did not report state funding
information. Nevada did not respond to our survey. State-reported
figures may include some funding allocated in fiscal year 2003 since
the survey was issued in October 2002 and completed as late as December
2002.
[5] This figure includes developmental funds allocated by HHS to 49
states and the District of Columbia. Hawaii did not take any federal
money for SACWIS development.
[6] This figure includes operational funds allocated to 35 states.
States begin claiming operational costs when some or all components of
their SACWIS are operating in local offices. Operational activities
include routine maintenance, minor enhancements, and other changes that
do not significantly increase or modify the functionality of the
system.
[7] According to HHS officials, prior to fiscal year 2000, states
reported SACWIS operational expenses as part of their Title IV-E
administrative expenses because the claims sheet states used for
reporting did not have a separate column for SACWIS operational
expenditures. In fiscal year 2000, states were required to use a claims
sheet that was reformatted to provide space for SACWIS operational
expenditures. In addition, an HHS official explained that the
difference between the state-reported figures and the federal figures
may be due to states claiming some SACWIS expenses under different
programs, such as Title IV-E administrative funds, rather than
separately as SACWIS expenses.
[8] Although the Iowa state officials described their SACWIS as
including the child abuse and neglect system, HHS commented on a draft
of this report that it does not view the child abuse and neglect system
as part of the state's SACWIS. However, HHS said that the state has met
the SACWIS requirement in this area by building an interface between
the two systems.
[9] The Foster Care Independence Act of 1999 increased federal support
to states for independent living programs--programs designed to assist
youth who are identified as likely to remain in foster care until age
18. Independent living services can include education or training
necessary for the youth to obtain employment.
[10] Twelve of the 46 states reporting that they are developing or
operating a SACWIS reported that they have not experienced delays in
developing their systems. In response to the length of the delays
reported by 26 states in our survey, ACF commented on a draft of this
report that these states may be using different definitions in defining
their delays. However, ACF did not provide further information on how
the delays represented in this report differ from its perception of
states' experiences. In our survey, we asked states to report on the
delays that exceeded the time line outlined in their initial APD.
[11] With regard to the budget difficulties that states reported
facing, since 1994 the federal government has made a commitment to help
states develop and maintain their SACWIS by matching 75 percent of
states' development funds through 1997 and providing an ongoing match
of 50 percent of state funding for the development and maintenance of
their systems. However, since the states' legislatures must make the
initial commitment to fund SACWIS, the federal government cannot assist
state child welfare agencies with this challenge.
[12] In commenting on a draft of this report, HHS indicated that a Web
resource is available to states interested in learning about other
states' efforts to develop human services--child welfare, food stamps,
Temporary Assistance to Needy Families, child care, and child support
enforcement--information systems at http://www.acf.hhs.gov/nhsitrc.
[13] Data are reliable when they are complete and accurate. A
subcategory of accuracy is consistency. Consistency refers to the need
to obtain and use data that are clear and well-defined enough to yield
similar results in similar analysis. See U.S. General Accounting
Office, Assessing the Reliability of Computer-Processed Data,
GAO-02-15G (Washington, D.C.: Sept. 2002).
[14] States were asked the extent to which certain problems may
decrease the quality of the data submitted to AFCARS and NCANDS using
the following scale: very great, great, moderate, some, and no affect.
[15] The analysis of survey responses about reporting data to HHS is
based on responses from 49 states and the District of Columbia. All
states, regardless of SACWIS development, were asked to complete these
questions.
[16] We reviewed AFCARS reports from 6 of the 8 states assessed by HHS-
-Arkansas, Connecticut, New Mexico, Texas, Vermont, and Wyoming. HHS
conducted reviews in Delaware and West Virginia after we completed our
analysis.
[17] In commenting on a draft of this report, ACF said that the finding
from the AFCARS reviews indicates that information is often defaulted
to the response "unable to determine" in order for the element not to
fail the missing data standard, not that workers are recording
"unknown"; however, the report findings we used in this analysis
instruct states to fix the defaults and address caseworker practice by
enhancing training on the correct use of "unable to determine" when
noting a child's race.
[18] See U.S. General Accounting Office, Child Welfare: HHS Could Play
a Greater Role in Helping Child Welfare Agencies Recruit and Retain
Staff, GAO-03-357 (Washington, D.C.: Mar. 31, 2003).
[19] Department of Health and Human Services, Office of Inspector
General, Adoption and Foster Care Analysis and Reporting System
(AFCARS): Challenges and Limitations, OEI-07-01-00660 (Washington,
D.C.: Mar. 2003).
[20] Guardianship arrangements occur when permanent legal custody of a
child is awarded to an individual, such as a relative, but the child is
not legally adopted.
[21] Child Welfare League of America. National Working Group
Highlights, "Placement Stability Measure and Diverse Out-of-Home Care
Populations" (Washington, D.C., Apr. 2002).
[22] Respite care provides temporary childcare for children away from
their caretakers.
[23] Although the findings from the Inspector General's report and our
study are more recent, ACF commented on a draft of this report that it
issued policy clarifications regarding placement information on July 5,
2002.
[24] A rating of 2 indicates that the state's information system does
not fully meet the requirements for AFCARS reporting, whereas if a
state receives a 3, the information system requirements for AFCARS
reporting are in place, but there are data entry problems affecting the
quality of data. According to an HHS official, data elements that have
a combination of technical and data entry problems are rated as 2 until
the technical issues are resolved. HHS will then rate the element as a
3 until the data entry practices are changed. A state receives a rating
of 4 if in compliance with the AFCARS requirements or a 1 if the
requirement is not addressed in the state's information system.
[25] Federal guidance states that children who are (1) removed from
their home and initially placed in a juvenile justice facility are not
to be included in AFCARS reporting; (2) in a foster care setting who
are moved to a juvenile justice facility and who are expected to be
returned to a foster care setting should continue to be included in the
AFCARS reporting population; and (3) in a foster care setting and are
moved to a juvenile justice facility and who become the responsibility
of another agency should not be included in the AFCARS reporting
population. However, these studies show that the arrangements between
child welfare and juvenile justice agencies vary, which affect the
population of children reported to AFCARS. For example, some state
child welfare agencies have responsibility for all children in the
juvenile justice system and include these children in their AFCARS
reports, while other states only report children who are in the custody
of the juvenile justice system, but receiving Title IV-E funding.
[26] According to HHS, state child welfare agencies can claim Title IV-
E foster care maintenance funds for eligible children for costs
incurred by juvenile justice agencies that have entered into an
agreement with the child welfare agencies. Delinquent children served
by these agencies who meet the Title IV-E eligibility criteria are
eligible for Title IV-E foster care maintenance funds. States must meet
all Titles IV-B and IV-E program and/or eligibility requirements with
respect to the children who are adjudicated delinquent and are
receiving Title IV-E funding or for children in foster care under the
placement and care responsibility of the child welfare agency.
[27] Children in the juvenile justice system may enter care because of
behavior problems and return home when the behavior is controlled.
However, the problem may arise again and the child could re-enter state
custody. Also, the children may experience a greater number of
placements while in foster care because of their behavior problems or a
"step-down" approach that some states use to gradually decrease the
level of security required to care for the children.
[28] See U.S. General Accounting Office, Executive Guide: Improving
Mission Performance Through Strategic Information Management and
Technology, GAO/AIMD-94-115 (Washington, D.C.: May 1, 1994); Center for
Technology in Government, University of Albany, SUNY. Tying a Sensible
Knot: A Practical Guide to State-Local Information Systems. Albany,
N.Y., June 1997.
[29] The Child Support Enforcement Program is a joint federal, state,
and local partnership that was established in 1975 under Title IV-D of
the Social Security Act. Each state runs a child support program, which
provides four major services: locating non-custodial parents,
establishing paternity, establishing child support obligations, and
collecting child support for families.
[30] In 1996, the Congress created the block grant Temporary
Assistance for Needy Families program replacing the Aid to Families
with Dependent Children (AFDC) and related welfare programs. States
were given increased flexibility in designing the eligibility criteria
and benefit rules, which require work in exchange for time-limited
benefits.
[31] See U.S. General Accounting Office, Welfare Reform: Improving
State Automated Systems Requires Coordinated Federal Effort, GAO/
HEHS-00-48 (Washington, D.C.: Apr. 27, 2000).
[32] See U.S. General Accounting Office, Managing Technology: Best
Practices Can Improve Performance and Produce Results, GAO/T-AIMD-97-38
(Washington, D.C.: Jan. 31, 1997).
[33] See U.S. General Accounting Office, Child Welfare: HHS Could Play
a Greater Role in Helping Child Welfare Agencies Recruit and Retain
Staff. GAO-03-357. (Washington, D.C.: Mar. 31, 2003).
[34] The Child Welfare League of America is an association of almost
1,200 public and private nonprofit agencies that assist abused and
neglected children and their families with a wide range of services.
The American Public Human Services Association is an organization of
individuals and agencies concerned with human services, including state
and local human service agencies and individuals who work in or
otherwise have an interest in human service programs.
GAO's Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office
441 G Street NW,
Room LM Washington,
D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.
General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.
20548:
