Highway Safety
Further Opportunities Exist to Improve Data on Crashes Involving Commercial Motor Vehicles Gao ID: GAO-06-102 November 18, 2005Large trucks make up 3 percent of the nation's registered vehicles, but they were involved in 11 percent of all fatal crashes in 2003. To reduce the fatality rate, the Federal Motor Carrier Safety Administration (FMCSA) sets national safety goals and works in partnership with states to reach them. Crash data collected by states and submitted to FMCSA is key to these efforts, and to be fully useful, this data must be complete, timely, accurate, and collected in a consistent manner. GAO addressed (1) what is known about the quality of commercial motor vehicle crash data, and what states are doing to improve it, and (2) the results of FMCSA's efforts to help states make improvements.
Overall, commercial motor vehicle crash data does not yet meet general data quality standards of completeness, timeliness, accuracy, and consistency. For example, FMCSA estimates that nearly one-third of commercial motor vehicle crashes that states are required to report to the federal government were not reported, and those that were reported were not always accurate, timely, or consistent. States are undertaking four types of activities to improve data quality, including analyzing existing data to identify problems and develop plans for addressing them, reducing backlogs of data that have not been entered into state-level databases, developing and implementing electronic data systems, and providing training. As a result of these efforts, states have recently improved both the timeliness and the number of reportable crashes submitted to FMCSA. FMCSA has two main efforts to support states in improving their reporting of commercial motor vehicle crash information--a commercial vehicle crash data improvement program and a data quality rating system--and both appear to be beneficial. Through the data improvement program, FMCSA has provided nearly $21 million in discretionary grants to 34 states from 2002 through 2005. These grants have ranged from $2,000 to $2 million and have helped states conduct a variety of data improvement activities. GAO did not find problems with FMCSA's oversight of the program, but we did note that FMCSA does not have formal guidelines for awarding grants to states. As state participation in the program increases, formal guidelines and systems would likely assist FMCSA in prioritizing states' requests and ensuring consistency in grant awards. FMCSA's second major effort, a tool for rating states' data quality, has proven to be an important tool for states to use in improving their crash data as well. These results are presented in a map that rates each state's data quality as "good," "fair," or "poor." According to both FMCSA and state officials, the map and the underlying rating system serve as an incentive for states to improve their crash data. While the map is useful, GAO identified problems in the methodology used for developing ratings. These problems may potentially lead to drawing erroneous conclusions about the extent of improvements that have been made, and discourage states from continuing to devote attention and resources to areas needing improvement.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-06-102, Highway Safety: Further Opportunities Exist to Improve Data on Crashes Involving Commercial Motor Vehicles
This is the accessible text file for GAO report number GAO-06-102
entitled 'Highway Safety: Further Opportunities Exist to Improve Data
on Crashes Involving Commercial Motor Vehicles' which was released on
November 18, 2005.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Committees:
November 2005:
Highway Safety:
Further Opportunities Exist to Improve Data on Crashes Involving
Commercial Motor Vehicles:
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-102]
GAO Highlights:
Highlights of GAO-06-102, a report to congressional committees:
Why GAO Did This Study:
Large trucks make up 3 percent of the nation‘s registered vehicles, but
they were involved in 11 percent of all fatal crashes in 2003. To
reduce the fatality rate, the Federal Motor Carrier Safety
Administration (FMCSA) sets national safety goals and works in
partnership with states to reach them. Crash data collected by states
and submitted to FMCSA is key to these efforts, and to be fully useful,
this data must be complete, timely, accurate, and collected in a
consistent manner. GAO addressed (1) what is known about the quality of
commercial motor vehicle crash data, and what states are doing to
improve it, and (2) the results of FMCSA‘s efforts to help states make
improvements.
What GAO Found:
Overall, commercial motor vehicle crash data does not yet meet general
data quality standards of completeness, timeliness, accuracy, and
consistency. For example, FMCSA estimates that nearly one-third of
commercial motor vehicle crashes that states are required to report to
the federal government were not reported, and those that were reported
were not always accurate, timely, or consistent. States are undertaking
four types of activities to improve data quality, including analyzing
existing data to identify problems and develop plans for addressing
them, reducing backlogs of data that have not been entered into state-
level databases, developing and implementing electronic data systems,
and providing training. As a result of these efforts, states have
recently improved both the timeliness and the number of reportable
crashes submitted to FMCSA.
FMCSA has two main efforts to support states in improving their
reporting of commercial motor vehicle crash information”a commercial
vehicle crash data improvement program and a data quality rating
system”and both appear to be beneficial. Through the data improvement
program, FMCSA has provided nearly $21 million in discretionary grants
to 34 states from 2002 through 2005. These grants have ranged from
$2,000 to $2 million and have helped states conduct a variety of data
improvement activities. GAO did not find problems with FMCSA‘s
oversight of the program, but we did note that FMCSA does not have
formal guidelines for awarding grants to states. As state participation
in the program increases, formal guidelines and systems would likely
assist FMCSA in prioritizing states‘ requests and ensuring consistency
in grant awards.
FMCSA‘s second major effort, a tool for rating states‘ data quality,
has proven to be an important tool for states to use in improving their
crash data as well. These results are presented in a map that rates
each state‘s data quality as ’good,“ ’fair,“ or ’poor.“ According to
both FMCSA and state officials, the map and the underlying rating
system serve as an incentive for states to improve their crash data.
While the map is useful, GAO identified problems in the methodology
used for developing ratings. These problems may potentially lead to
drawing erroneous conclusions about the extent of improvements that
have been made, and discourage states from continuing to devote
attention and resources to areas needing improvement.
FMCSA‘s June 2005 data quality map showing each state‘s overall data
quality rating for crash and inspection data:
[See PDF for image]
[End of figure]
What GAO Recommends:
To ensure uniformity in awarding data improvement funds to states,
FMCSA should establish specific guidelines for assessing and awarding
state funding requests.
Also, in order to address limitations in its data quality map, FMCSA
should develop a plan for assessing and improving the map‘s
methodology, and it should provide a crash specific data rating and
limitations of the map on its Web site.
The Department of Transportation agreed with our findings and
recommendations in this report.
www.gao.gov/cgi-bin/getrpt?GAO-06-102.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Katherine Siggerud, (202)
512-6570, Siggerudk@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
States Continue to Have Problems with CMV Crash Data Quality, but Are
Pursuing a Variety of Improvement Efforts:
Data Quality Problems Often Reflect Difficulties in Collection and
Processing of Crash Reports:
FMCSA's Efforts Have Contributed to CMV Data Quality Improvements:
FMCSA's SaDIP Has Supported State Efforts to Improve Data Quality:
Data Quality Map Has Spurred Improvements, but Limitations Curb Map's
Continued Usefulness:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendixes:
Appendix I: Objectives, Scope and Methodology:
Appendix II: State Safety Data Quality Map Limitations:
Completeness: Overall Completeness is Based on Fatal Crashes Only:
Timeliness: Timeliness is Not Based on All Reported Crashes:
Accuracy: Accuracy is Based on Only One Variable:
Appendix III: FMCSA Reportable Crash, CMV, and Carrier Identification
Visor Cards:
Appendix IV: SaDIP Grant and Cooperative Agreement Distribution by
State:
Appendix V: SaDIP Case Study States:
Appendix VI: GAO Contact and Staff Acknowlegdments:
Tables:
Table 1: Examples of Problems with Commercial Vehicle Crash Data
Quality:
Table 2: Annual Distribution of SaDIP funds:
Table 3: Comparison of Data Quality Standards and State Safety Data
Quality Map Measures:
Table 4: Overall Data Quality Rating:
Table 5: State Safety Data Quality Map Measures for CMV Crashes:
Table 6: Distribution of SaDIP Grants by State:
Table 7: Distribution of SaDIP Cooperative Agreements by State:
Table 8: Georgia SaDIP Funding History:
Table 9: Georgia Crash Data Quality Statistics (Percentages):
Table 10: Minnesota SaDIP Funding History:
Table 11: Minnesota Crash Data Quality Statistics (Percentages):
Table 12: North Carolina SaDIP Funding History:
Table 13: North Carolina Crash Data Quality Statistics (Percentages):
Table 14: Ohio SaDIP Funding History:
Table 15: Ohio Crash Data Quality Statistics (Percentages):
Table 16: Oklahoma SaDIP Funding History:
Table 17: Oklahoma Crash Data Quality Statistics (Percentages):
Table 18: Washington SaDIP Funding History:
Table 19: Washington Crash Data Quality Statistics (Percentages):
Figures:
Figure 1: Criteria for Selecting Truck and Bus Crashes to Report to
FMCSA:
Figure 2: State Participation in CMV Crash Data Improvement Efforts
from Fiscal Year 2002 through Fiscal Year 2005:
Figure 3: FMCSA Reportable Crashes:
Figure 4: Reportable Commercial Motor Vehicle Configurations and Cargo
Body Type:
Figure 5: Responsible Carrier and Correct DOT Number Identification:
Abbreviations:
CMV: commercial motor vehicle:
CVARS: Commercial Vehicle Analysis Reporting System:
DOT: Department of Transportation:
FARS: Fatality Analysis Reporting System:
FMCSA: Federal Motor Carrier Safety Administration:
GSA: General Services Administration:
MCMIS: Motor Carrier Management Information System:
MCSAP: Motor Carrier Safety Assistance Program:
NHTSA: National Highway Traffic Safety Administration:
SaDIP: Safety Data Improvement Program:
SafeStat: Motor Carrier Safety Status Measurement System:
SAFETEA-LU: Safe, Accountable, Flexible, and Efficient Transportation
Equity Act of 2005: A Legacy for Users:
UMTRI: University of Michigan Transportation Research Institute:
[See PDF for image]
[End of figure]
Letter November 18, 2005:
The Honorable Christopher S. Bond:
Chairman:
The Honorable Patty Murray
Ranking Minority Member:
Subcommittee on Transportation, Treasury, the Judiciary, Housing and
Urban Development, and Related Agencies:
Committee on Appropriations:
United States Senate:
The Honorable Joe Knollenberg:
Chairman:
The Honorable John W. Olver:
Ranking Minority Member:
Subcommittee on Transportation, Treasury, and Housing and Urban
Development, the Judiciary, District of Columbia:
Committee on Appropriations:
House of Representatives:
Large trucks make up only 3 percent of the nation's registered
vehicles, but they were involved in 11 percent of all fatal crashes in
2003, the last year for which complete data is available. That year,
large trucks were involved in more than 430,000 crashes, killing
approximately 5,000 people.[Footnote 1] In 1999, Congress established
the Federal Motor Carrier Safety Administration (FMCSA) within the
Department of Transportation[Footnote 2] and mandated it with reducing
crashes, injuries, and fatalities involving large trucks and buses.
Currently, FMCSA has the goal of reducing commercial motor vehicle
crash fatalities to 1.65 fatalities per 100 million miles of travel by
2008. As of fiscal year 2003, the commercial motor vehicle fatality
rate was 2.3 fatalities per 100 million miles traveled, the lowest
recorded since the Department of Transportation initiated tracking in
1975--but still 40 percent above the 2008 goal.
FMCSA works in partnership with states to reach commercial motor
vehicle safety goals. States are the gatekeepers for the collection and
reporting of commercial motor vehicle crash information. They receive
crash reports completed by law enforcement personnel in local
jurisdictions, compile them, and then submit crash reports to FMCSA. At
the federal level, FMCSA manages a database which provides data that is
used in rating motor carriers according to various safety indicators.
Based on this rating, motor carriers are selected for safety
inspections and reviews as part of FMCSA's enforcement efforts. While
the data collected is primarily for federal use, states use the
information to assist overall crash safety efforts and in setting
commercial motor vehicle safety goals for themselves. Because the data
is used in both federal and state decision-making on a variety of
safety-related issues, it is important that it adequately meets data
quality standards.
To be useful to both federal and state decision-makers, crash data must
be complete, timely, accurate, and collected in a consistent manner.
However, there have been concerns about the quality of the information
FMCSA and the states use to direct their efforts. Beginning with the
Motor Carrier Safety Improvement Act of 1999, Congress directed the
Department of Transportation to improve the collection and analysis of
data on commercial motor vehicle crashes. This resulted in the creation
of a commercial motor vehicle data improvement program.[Footnote 3] The
program was reauthorized in 2005.[Footnote 4] Since 2002, about $21
million has been awarded to states to improve their crash data quality,
but data quality problems have persisted.[Footnote 5] In February 2004,
the Department of Transportation Inspector General released a report
discussing limitations of the commercial motor vehicle crash
data.[Footnote 6]
In a Senate report accompanying the fiscal year 2005 appropriation for
the Department of Transportation,[Footnote 7] Congress asked that we
review FMCSA's program for helping states improve their commercial
motor vehicle crash data. The report directed us to describe the
benefits obtained through the program, identify what can be done to
improve the effectiveness of the program, and address concerns
regarding crash data raised in the February 2004 Department of
Transportation Inspector General's report. Accordingly, this report
examines (1) what is known about the quality of commercial motor
vehicle crash data and what states are doing to improve it, and (2) the
results of FMCSA's efforts to facilitate the improvement of the quality
of commercial motor vehicle crash data sent to the federal government.
To describe the quality of commercial motor vehicle crash data, we
reviewed a number of sources, including data reported by FMCSA and
existing studies on the quality of commercial motor vehicle crash data.
We interviewed officials from FMCSA, contractors that develop FMCSA
crash data tools, and commercial vehicle industry researchers and
public interest organizations to gain their perspective on commercial
motor vehicle crash data quality. To provide information on states'
efforts to improve commercial motor vehicle crash data, we reviewed
grant documentation for 34 states that participated in FMCSA's Safety
Data Improvement Program (SaDIP) as of September 2005.[Footnote 8] We
also conducted case studies in six states that participated in the
program.[Footnote 9] States were chosen based on a wide variety of
factors including crash data quality and level of crash reporting.
Additionally, we conducted phone interviews with states that did not
participate, or are no longer participating, in the program.[Footnote
10] To provide results of FMCSA efforts to facilitate the improvement
of commercial motor vehicle crash data quality, we conducted interviews
with officials from FMCSA on the administration and management of the
SaDIP program. We also analyzed the guidance and support FMCSA provided
to states and assessed FMCSA's role in coordinating commercial motor
vehicle data quality initiatives. We reviewed studies conducted by the
University of Michigan Transportation Research Institute and determined
that the methodologies used in assessing the quality of the data states
submit to FMCSA were sound and that the studies provided sufficiently
reliable results for our purposes. Through site visits, a review of
grant applications, interviews with relevant stakeholders and experts,
and these studies, we were able to determine shortcomings in the
reliability of FMCSA's commercial motor vehicle crash data. However, we
determined that the data was sufficiently reliable for the purpose of
case study selection. Our work was conducted from February 2005 through
November 2005 in accordance with generally accepted government auditing
standards. See appendix I for more details regarding our objectives,
scope, and methodology.
Results in Brief:
Overall, commercial motor vehicle crash data does not yet meet general
data quality standards of completeness, timeliness, accuracy, and
consistency. For example, according to FMCSA, as of fiscal year 2004
nearly one-third of commercial motor vehicle crashes that states are
required to report to the federal government were not reported, and
those that were reported were not always accurate, timely, or
consistent. Data quality problems most often stem from errors or
omissions either by law enforcement officers at the scene of a crash or
in the processing of crash reports to a state level database. To
address data quality problems, a number of states are undertaking four
major types of data improvement activities:
* Analyzing existing data to identify where problems are and to develop
plans for addressing them;
* Reducing backlogs of data that have not been entered into state-level
databases in order to create more complete state crash files, through
steps such as hiring contract employees;
* Developing and implementing electronic data systems for collecting
and processing crash information in a more timely, accurate, and
consistent manner; and:
* Providing training, such as educating law enforcement officers on the
definitions and criteria for commercial motor vehicle crashes, to
create more accurate and consistent data.
These state efforts are resulting in some progress. Based on analysis
of FMCSA data, a great number of crashes are being reported to FMCSA.
Overall, the total number of commercial motor vehicle crashes being
reported to FMCSA has increased by 59 percent between fiscal year 2000
and fiscal year 2004, while the length of time it takes states to
report these crashes to FMCSA has decreased as well.
FMCSA has two main efforts to support states in improving their
reporting of commercial motor vehicle crash information--a commercial
vehicle crash data improvement program and a data quality rating
system--and both appear to be beneficial. Through the data improvement
program, FMCSA has provided nearly $21 million in discretionary grants
to 34 states between 2002 and 2005. These grants have ranged from
$2,000 to $2 million and have helped states conduct all four data
improvement activities previously described. The six states in our case
studies generally improved their data quality, mainly through projects
funded in whole or in part through the grant program. While we did not
find problems with FMCSA's oversight of the program, we did note that
FMCSA does not have formal guidelines for awarding funds to states.
Because these grants are discretionary, and because more states are
expected to participate in the program in the future, having formal
guidelines and systems in place would likely assist in prioritizing
states' requests and ensuring consistency in awarding funds. FMCSA's
second major effort, the State Safety Data Quality map, has proven to
be an important tool for states to use in improving their crash data as
well. This map, created by FMCSA and the Volpe National Transportation
Systems Center, is a color-coded display depicting the overall data
quality for each state in one of three rating categories--"good"
(green), "fair" (yellow), or "poor" (red). According to both FMCSA and
state officials, the map and the underlying rating system serve as an
incentive for states to make improvements to their crash data. Despite
the map's utility thus far, we identified potential problems both in
the methodology used for developing ratings and the risk of drawing
erroneous conclusions from the map. One example of a problem with the
current methodology is that the overall ratings combine information
about crashes with information stemming from FMCSA's inspections of
motor carriers. Combining ratings for both crash and inspection data
quality tends to make it difficult to determine how states are doing
specifically with their crash data. In addition, some states with a
"good" rating in completeness are not reporting all crashes to FMCSA.
Rating states as "good" when in fact they have problems may discourage
states from continuing to devote attention and resources to areas
needing improvement and possibly misdirect program efforts. FMCSA is
aware of many of the limitations of the map, but has not yet developed
and implemented a formal plan to improve it. Addressing these
limitations will strengthen the data quality map as a tool for
improving commercial motor vehicle crash data.
To ensure that FMCSA is able to target limited funds as effectively as
possible, we are recommending that FMCSA create specific guidelines and
criteria for awarding commercial motor vehicle crash data improvement
funding. We are also recommending that FMCSA develop a plan to improve
the data quality map, including assessing the methodology for
developing ratings, providing results in greater detail, and
documenting any limitations associated with the map. These enhancements
will provide users with a more useful tool to view the condition and
progress made in states' commercial motor vehicle crash data.
Background:
FMCSA was established as a separate administration within the U.S.
Department of Transportation (DOT) on January 1, 2000, pursuant to the
Motor Carrier Safety Improvement Act of 1999. FMCSA issues and enforces
the federal motor carrier safety regulations that govern many aspects
of specified commercial trucking and bus operations, including the
interstate operation and maintenance of commercial motor vehicles
(CMV). Regulations promulgated by FMCSA specify requirements that must
be met by drivers of these vehicles. FMCSA conducts compliance
reviews[Footnote 11] of truck and bus companies, and performs safety
audits of new entrants[Footnote 12] into the industry. In addition,
FMCSA trains inspectors to conduct safety audits, while states and
local authorities are responsible for conducting the inspections and
submitting the results to FMCSA.[Footnote 13] This partnership between
FMCSA and the states annually results in about 3 million truck and bus
inspections, 7,000 to 13,000 compliance reviews, and more than 19,000
new-entrant safety audits.
CMV crash data is key to FMCSA's efforts. CMV crash data is collected
by local law enforcement, sent to the state, and then processed and
uploaded by the state into FMCSA's data system. FMCSA maintains a
database management system in each state so they can submit crash
reports into FMCSA's central data system, the Motor Carrier Management
Information System (MCMIS). FMCSA uses the information in its Motor
Carrier Safety Status Measurement System, also known as SafeStat, to
target carriers for compliance reviews to ensure that they are
following safety regulations.[Footnote 14] SafeStat uses a variety of
data to rank carrier safety, but it places the heaviest weight on crash
data.
Federal and state data quality guidelines call for CMV crash data to
meet four basic quality standards:[Footnote 15]
* Completeness: To support adequate decision-making for identifying
problems and developing appropriate countermeasures, data should be
collected for all reportable CMV crashes in the state, and data on all
appropriate crash variables such as the carrier's identification
number, should be submitted to FMCSA.
* Timeliness: To make decisions about current safety problems, identify
trends, or target carriers that pose immediate threats to highway
safety, CMV crash data should be available for state and federal
analytical purposes within a useful timeframe.
* Accuracy: To adequately assess CMV crash problems and target the
appropriate carriers for enforcement, all data within reportable CMV
crash records should be accurate and reliable.
* Consistency: To target carriers nationwide and to compare state
results, CMV crash data should be collected uniformly using the same
standards and applying the same criteria across jurisdictions.
FMCSA has provided more specific guidelines and criteria for meeting
each of these standards. On timeliness, for example, FMCSA calls for
states to submit all CMV crash data to MCMIS within 90 days of when the
crash occurs. In order to facilitate complete reporting of CMV crashes,
FMCSA recommends data elements, such as the identity of the carrier,
vehicle configuration, cargo body type, etc., as the minimum
information to be:
collected in order to have complete information on CMV
crashes.[Footnote 16] FMCSA has also created criteria to assist in
consistent reporting of crash information. These published criteria are
used for identifying reportable CMV crashes to be submitted to MCMIS
(see fig. 1).
Figure 1: Criteria for Selecting Truck and Bus Crashes to Report to
FMCSA:
[See PDF for image]
Note: Exceptions include crashes that involve: 1) a personally-owned
truck or passenger vehicle meant for personal use only as the sole
vehicle meeting the criteria above, or 2) a driver with a disease
condition (e.g., stroke, heart attack, diabetic coma or epileptic
seizure) and no other injury or damage occurs, or 3) deliberate intent
(suicide, self-inflicted injury, homicide, etc.), with no unintentional
injury or damage. Of the 430,000 CMV crashes that occur each year,
about 150,000 meet reporting criteria.
[End of figure]
With the Motor Carrier Safety Improvement Act of 1999, Congress
established a program to improve the collection and analysis of CMV
crash data.[Footnote 17] This resulted in the creation of the
Commercial Motor Vehicle Analysis Reporting System (CVARS). CVARS,
which is now known as SaDIP was originally intended to be a standalone
data collection system. After determining that a separate system would
duplicate existing efforts, however, FMCSA decided to use SaDIP as a
federal funding tool to support state efforts to collect and report CMV
crash data.[Footnote 18]
Besides changes in scope, SaDIP has also changed greatly in how the
program is administered. Since fiscal year 2001, FMCSA has received
funding from the Congress to implement this program. As a new agency,
FMCSA did not have the appropriate contracting infrastructure in place
to administer the funds. Therefore it transferred funds and the
administrative duties of the program to the National Highway Traffic
Safety Administration (NHTSA), which awarded several grants to states
in fiscal year 2002. NHTSA also provided some of the funding to the
General Services Administration (GSA) to enter into cooperative
agreements with several states to fund multi-year data improvement
projects. In fiscal year 2003, however, FMCSA assumed responsibility
for all oversight of SaDIP funding.
FMCSA provides SaDIP funds to states through two different methods.
States can receive 1-year grants to fund specific projects, or they can
enter into multi-year cooperative agreements in order to fund multiple
efforts that are necessary to identify and reduce data quality
problems. States can receive both of these types of funds from FMCSA
and can apply for grant funding multiple times. SaDIP funding is
discretionary in nature, allowing states to request the amount of
funding they need to conduct data improvement projects. Since 2002,
SaDIP funding has been provided to 34 states, and as of September 2005,
awards ranged from $2,000 to $2 million totaling approximately $21
million. (See app. IV for funding distribution by state.)
In February 2004, the Department of Transportation Inspector General
issued a report on SafeStat, the system FMCSA uses to target its
compliance reviews.[Footnote 19] The report identified a number of
problems with SafeStat, much of which stemmed from the quality of the
crash data being used in calculating carrier ratings. The State Safety
Data Quality map, discussed later in the report, was developed in
response to the recommendations in the 2004 report.
States Continue to Have Problems with CMV Crash Data Quality, but Are
Pursuing a Variety of Improvement Efforts:
When measured against generally accepted standards for completeness,
timeliness, accuracy, and consistency, states continue to have
challenges with the quality of their CMV crash data. Many of these
challenges are based in the collection and processing of data at the
local level, and often reflect broader crash data quality problems for
all types of vehicle crashes. To address remaining limitations, and
improve the completeness, timeliness, accuracy, and consistency of the
data, states are undertaking four main types of efforts: analyzing data
to identify problem areas, eliminating backlogs of data not yet
entered, creating electronic systems to expedite data entry, and
training law enforcement officers in ways to improve the data they
submit.
Challenges Remain in Meeting Data Quality Standards:
The completeness, timeliness, accuracy, and consistency of CMV crash
data is currently not meeting generally accepted data quality
standards. Table 1 provides examples of some of the overall problems we
(and others) have identified. Appendix V, which contains summaries of
the six states we visited, provides more specific examples.
Table 1: Examples of Problems with Commercial Vehicle Crash Data
Quality:
Completeness;
As of 2004, FMCSA estimates that about one-third of the reportable CMV
crashes are not being submitted to their data system-- the Motor
Carrier Management Information System (MCMIS).[A] In addition, FMCSA
estimates 20 percent of nonfatal crashes are not being reported.
Further, studies indicate that even those crashes reported to MCMIS
often have missing data.b.
Timeliness;
The average length of time from when a crash occurs to when the crash
data is uploaded to MCMIS is 99 days--9 days over the required time
limit. While this is not far from the goal, there is substantial
variation in timeliness among states. For example, our analysis of
state CMV crash data shows timeliness ranges from 13 days to 339
days.[C].
Accuracy;
FMCSA assesses accuracy by determining the number and percentage of
interstate crashes uploaded to MCMIS without enough information to
determine a carrier's DOT number (known as a non-match). As of fiscal
year 2004, 15 percent of CMV crash records in MCMIS can not be matched
to a carrier's DOT number.[D].
Consistency;
According to an analysis by FMCSA, 33 of 50 states have crash reports
that do not adequately follow the criteria for reporting commercial
motor vehicle crashes to FMCSA.[E].
[A] MCMIS contains information on the safety fitness of commercial
motor carriers and hazardous material shippers subject to the Federal
Motor Carrier Safety Regulations and the Hazardous Materials
Regulations. States upload their crash information to MCMIS via FMCSA's
SafetyNet.
[B] Source: FMCSA:
[C] Source: Analysis of MCMIS data from fiscal year 2000 through fiscal
year 2004.
[D] Source: Analysis of fiscal year 2004 MCMIS data.
[E] Source: Comparison of State Crash Reports with SafetyNet Selection
Criteria and Key Data Secondary Elements Chart, FMCSA, August 12, 2005.
[End of table]
Data Quality Problems Often Reflect Difficulties in Collection and
Processing of Crash Reports:
CMV crash data quality problems often stem from issues that occur when
data is initially collected at the scene of a crash and later processed
through the state. We reviewed reports on crash data quality including
individual state reviews conducted by the University of Michigan
Transportation Research Institute (UMTRI).[Footnote 20] We also
discussed these matters with state officials. We identified two key
causes of poor data quality: (1) problems in interpreting how to fill
out crash reports at the scene and (2) crash report processing issues
ranging from competing priorities at the local level to complex routing
of crash reports.
Misinterpretation of Criteria and Definitions by Officers Filling Out
Crash Reports:
According to studies and our discussions with state officials, data
collected at the scene of a crash can be flawed because of law
enforcement misinterpretation of reporting criteria and
definitions.[Footnote 21] Misinterpretation can occur for several
reasons, such as infrequent opportunities for officers to receive
training on filling out crash reports or unfamiliarity resulting from
infrequent occurrences of CMV crashes in an officer's jurisdiction.
Below are common problems with properly reporting CMV crashes:[Footnote
22]
Identifying reportable crashes. While crashes that result in a fatality
are easily identifiable as a reportable crashes, tow-away or injury
crashes are more difficult to identify. For example, UMTRI's review of
eight states showed that five of those states experienced problems with
reporting crashes that did not involve a fatality. According to UMTRI,
this is likely due to a lack of understanding of criteria for reporting
CMV crashes to FMCSA (see fig. 1).
Identifying reportable commercial motor vehicles. FMCSA is responsible
for enforcing safety regulations for interstate carriers, but collects
crash information on a variety of CMVs that meet certain criteria.
According to UMTRI reports, states had problems identifying smaller
trucks and they were less likely to be reported because law enforcement
officials are less likely to recognize them as qualifying CMVs.
Further, law enforcement officials may be confused about reporting
interstate and intrastate carriers to FMCSA. UMTRI's review of 8 states
showed that five of the states encountered problems in identifying
appropriate vehicle type for reporting to FMCSA (see fig. 1).
Crash Report Processing Issues:
Several other issues relate to the processing of crash reports. These
tend to fall into the following three main categories:
Competing priorities at the officer level. Data collection is
complicated because at the scene of a crash, an officer's first
priority is to ensure the safety of those involved; data collection is
often a lesser concern. Only after the situation has been stabilized
can the officer fill out the crash report. Competing priorities can
make it difficult for law enforcement to adequately complete a crash
report. According to a report by NHTSA, crash reporting is often a low
priority when traffic safety data is not perceived as relevant to the
work of law enforcement. This was confirmed in states we visited. For
example, one state official with whom we spoke said that submitting
crash data is often dependent on how much priority the local police
chief places on data quality.
Manual crash report forms. Typically, law enforcement officers complete
handwritten crash reports, which are then manually submitted for data
processing at a local or state agency. According to an FMCSA official,
nearly all states use manual crash reporting forms to some extent.
Completing crash reports by hand and manually processing the
information has the potential to lead to errors in the data. Data
quality may be further compromised by the use of a supplemental CMV
crash report. States may have a general crash report for use in all
crashes and a supplemental CMV crash report to be completed if a CMV is
involved. For example, four of the six case study states we reviewed
used a supplemental form to report a CMV accident. When data has to be
captured in a separate form, law enforcement officers may not always
complete the form, and sometimes it may be separated from the original
crash report or even lost. UMTRI found in one state that use of the
supplemental crash report may be leading to CMV crashes not being
submitted to FMCSA.
Complex processing. Even with a correctly completed crash report that
is properly identified as a CMV crash, states may have a complicated
process in place to route CMV crash data to FMCSA and this may
contribute to lengthy delays. UMTRI state assessments found that
processing issues in some states led to delays and resulted in
incomplete submission of crash data. One official told us that all law
enforcement agencies in his state send crash reports to the state
patrol office where the report is scanned into electronic format. The
electronic version of the report is then sent to the state department
of transportation where additional location data is entered into the
report. From here, the report enters the state database and then is
periodically uploaded to FMCSA. This process can take significant time,
especially when the original crash report is missing information.
States Are Pursuing a Variety of Efforts to Address Data Quality
Issues:
States are undertaking a variety of projects in order to improve the
completeness, timeliness, accuracy, and consistency of CMV crash data.
These projects encompass four general types of efforts: research and
analysis, reduction of reporting backlogs, electronic processing of
crash records, and training. Figure 2 shows state participation in each
of these four efforts. These projects directly affect the completeness,
timeliness, accuracy, and consistency of the states' CMV crash data.
Figure 2: State Participation in CMV Crash Data Improvement Efforts
from Fiscal Year 2002 through Fiscal Year 2005:
[See PDF for image]
Note: Data available for the 34 states that have participated in
FMCSA's data improvement program. Several states are participating in
more than one effort.
[End of figure]
Data Analysis to Identify Problems:
Eleven of the 34 states have research projects to evaluate crash data
collection and reporting practices.[Footnote 23] Assessing data
collection and reporting processes allows states to determine how well
their data meet each of the data quality standards of completeness,
timeliness, accuracy, and consistency, providing them with a useful
tool for determining where they are going to concentrate their
improvement efforts.
Out of 34 states for which we have data, 11 have conducted research to
identify data problems.During our case studies, we found examples of
states conducting data analysis to identify their CMV crash data
collection and reporting problems. (See app. V for more information on
our case study states.) For example, one state conducted an analysis in
order to determine why approximately half of the eligible crashes in
the state crash file were not being reported to FMCSA. The analysis
found that the state's crash file was not in the correct format to be
correctly transferred to FMCSA. In addition, frequent errors in the
state or zip code fields, and incomplete or inaccurate data, were
leading to problems. A federally funded study of the state's crash data
found approximately 50 percent of the state's crash reports for one
year (2003) were not reported to the federal crash file due to problems
at the state level coding and preparing crash data for transfer to
FMCSA. Another state conducted a traffic records assessment that found
the major deficiency in the state's crash file was the lack of timely
data. It also identified the lack of effective quality controls during
data entry and non-reporting of eligible crashes as deficiencies.
Following the state's assessment, researchers made recommendations to
help the state prioritize projects, including planning for the eventual
implementation of electronic crash data collection.
Reduction of Crash Database Backlogs:
Fourteen of the 34 states have projects to reduce the backlog of crash
reports that need to be entered into state crash files. In most states,
until crashes are included in the state file, they are not reported to
FMCSA. Hence, backlog reduction is essential for creating a complete
state crash file, and with providing FMCSA with complete data about CMV
crashes. A complete data file is critical for performing crash and
trend analysis, for making informed policy decisions, and for gaining a
better understanding of CMV safety issues.
Of the 34 states for which we have data, 14 are participating in
backlog reduction projects, including four of our six case study
states. In these four states, temporary employees were hired to
eliminate data backlogs. For example, in one state, contract employees
were working to reduce a backlog of 600,000 crash reports. Data entry
work began in July 2004 and state officials expect it to be completed
by September 2005. Another state used temporary employees to eliminate
a backlog of approximately 2 million crash records. This effort began
in 2002, and as of June 2005, the data backlog has been completely
eliminated and the state is now processing crash reports within 3 days
of receipt.
Using Electronic Systems to Expedite CMV Crash Data:
Twenty-seven of the 34 states have projects to collect or input CMV
crash data through electronic systems. Electronic reporting allows for
more accurate and timely transfer of crash data to state and federal
crash files. Electronic transfer of crash data can reduce duplication
and data entry error, because paper-based data collection systems are
subject to human error and time delays. While these projects enhance
the quality of CMV crash data, often they are large in scope and
require law enforcement agencies to purchase hardware. Uniform
electronic crash reporting systems are heavily dependent on acceptance
from all stakeholders and often some jurisdictions may already have
their own systems.
Of the 34 states for which we have data, 27 are participating in
electronic crash record projects, including all six of the case study
states. For example, one state has developed software that facilitates
a nightly transfer of CMV crash records from the Division of Driver and
Vehicle Services to the state patrol agency. The state patrol agency
then submits the crash reports to FMCSA. The state is also working to
provide computer hardware and Internet access so that state patrol
officers can complete crash reports online. Another state is developing
electronic crash reporting capabilities using handheld computers, which
will eliminate handwritten crash reports, and the need to manually
enter crash report data into an electronic system.
Improving Law Enforcement Training:
States are providing training to law enforcement officers in CMV crash
data collection. Training for law enforcement officers on how to
correctly identify a CMV and how to accurately complete a police
accident report can help to improve the completeness, accuracy, and
consistency of CMV crash data since many of the mistakes in reporting
are made at the crash site.[Footnote 24] Training is also an
opportunity to highlight the link between data collection and its end
use in planning and prevention.
Seventeen of the 34 states have projects aimed at providing law
enforcement training initiatives, and all six of our case study states
had some form of CMV training. In addition, all case study states we
visited recognized problems with the quality of CMV crash data
resulting from issues at the collection point. For example, one state
provides on-going CMV training to teach officers to properly identify
and report commercial vehicle crashes. This state also tries to show
law enforcement officers how the data is used in planning and
prevention in order to show the significance of good data. However,
state officials told us that there has not been a lot of participation
in the training from local law enforcement agencies. Another state is
planning to develop a training video and visor cards[Footnote 25] for
statewide dissemination. The visor cards will assist officers in CMV
identification and provide information that will help police properly
complete all the necessary crash reporting documentation.
These state efforts are resulting in some progress. An analysis of
FMCSA data shows that a greater number of crashes are being reported to
FMCSA since 2000. Overall the total number of commercial motor vehicle
crashes being reported to FMCSA has increased by 59 percent between
fiscal year 2000 and fiscal year 2004, while the length of time it
takes states to report these crashes to FMCSA has decreased as well.
FMCSA's Efforts Have Contributed to CMV Data Quality Improvements:
FMCSA's efforts appear to improve CMV crash data quality and have had a
positive impact on state crash data. Specifically two efforts, the
Safety Data Improvement Program (SaDIP) and the State Safety Data
Quality map, have contributed to the changes. SaDIP funding has allowed
states to improve data completeness, timeliness, accuracy, and
consistency by supporting the implementation of new activities or
increasing the scope and timeliness of existing projects. While FMCSA
has provided adequate administration and oversight of SaDIP, the
management of the program with regard to awarding grants raises some
concerns. There are no formal guidelines in place for awarding funding
to the states, and while this has not yet presented a problem, it may
in the future. Funding continues to be made available, and more states
continue to request funds for new projects. The State Safety Data
Quality map is an evaluation tool that provides ratings for states'
crash and inspection data quality, and displays the ratings so that a
state's performance relative to other states is apparent. The map,
which is publicized on FMCSA's Analysis and Information (A&I) Online
Web site, allows state officials and the public to view the status of
states' CMV crash and inspection data quality.[Footnote 26] The map is
being used as an indicator of states' progress in improving the
completeness, timeliness, and accuracy[Footnote 27] of crash data and
by virtue of its public nature it is an incentive for states to improve
their crash information. However, the methodology used for the ratings
has limitations that may hinder the map's effectiveness in monitoring
the status of CMV crash data quality.
FMCSA's SaDIP Has Supported State Efforts to Improve Data Quality:
As of July 2005, FMCSA had provided about $21 million to 34 states in
order to assist in improving the quality of CMV crash data.[Footnote
28] Several states have received funding more than once, and in fiscal
year 2005, FMCSA made available to states almost twice the amount of
money than it had at the beginning of the program ($3.9 million in
fiscal year 2000 versus $7.3 million in fiscal year 2005).
Additionally, recently passed transportation legislation authorizes a
total of $11 million to be used for SaDIP funding for fiscal years 2006-
2009.[Footnote 29] The states that have participated in SaDIP account
for about two-thirds of all CMV crashes occurring between April 1,
2004, and March 31, 2005, reported to FMCSA's MCMIS database, and 70
percent of the 2003 fatalities reported to MCMIS. FMCSA's goal is to
provide funding for projects that will have the largest impact on
improving state data describing CMV crashes. FMCSA also encourages
states that are participating in data improvement projects that are
already funded to apply for SaDIP funds in order to provide additional
assistance to these larger efforts. (See table 2 for the annual
distribution of SaDIP funds and app. IV for fund distribution by
state.)
Table 2: Annual Distribution of SaDIP funds:
Fiscal year: 2000;
Funds budgeted for SaDIP: $3,865,000;
Funds awarded to states: $0.
Fiscal year: 2001;
Funds budgeted for SaDIP: $2,748,000;
Funds awarded to states: $0.
Fiscal year: 2002;
Funds budgeted for SaDIP: $5,462,614;
Funds awarded to states: $5,207,014.
Fiscal year: 2003;
Funds budgeted for SaDIP: $4,967,500;
Funds awarded to states: $854,732.
Fiscal year: 2004;
Funds budgeted for SaDIP: $4,913,990;
Funds awarded to states: $5,449,467.
Fiscal year: 2005;
Funds budgeted for SaDIP: $7,340,800;
Funds awarded to states: $9,180,181.
Total;
Funds budgeted for SaDIP: $29,297,904;
Funds awarded to states: $20,691,394.
Source: FMCSA.
Note: These data are current as of September 2005. In addition to funds
provided directly to states, FMCSA provided $1,200,000 to contractors
to assist states in improving data quality, bringing the total amount
spent on state data improvement efforts to $21,891,394.
[End of table]
SaDIP has Yielded Several Positive Results:
States have used the SaDIP funds to conduct a variety of projects,
including those discussed earlier in this report. These funds have
benefited states in several different ways, including increased focus
on CMV data quality and advancement and expansion of ongoing broader
data quality projects.
SaDIP increased national attention to the problems associated with the
quality of CMV crash data reported to the federal government. Several
state officials we interviewed stated that they have noticed an
increase in the amount of focus given to CMV crash data issues. For
example, presentations made by FMCSA at several national conferences
and workshops have highlighted the importance of data quality, and
informed states of the various types of assistance, such as training
and funding, that are available to them. During our case studies, we
were told the following:
* One state official said that by providing funds to be used for
specific purposes, SaDIP had the effect of focusing attention on data
quality improvement. SaDIP has sustained a high level of interest in
data quality and has been a catalyst to improving traffic records
coordination across state agencies.
* Another state official said SaDIP helped to improve communication
between the state patrol and the state's department of vehicle
services. Because grants were provided to both agencies, analysts at
the agencies are working together and have a better understanding of
each other's data needs and share access to their respective databases.
In half of our case study states, SaDIP has also allowed states to
expedite data improvement projects that were already planned for
implementation.
* One state official told us that the state had been considering a plan
to improve traffic safety data, including CMV crash data, and SaDIP
funds provided it with the means to implement this plan. The long-term
funding provided by the SaDIP cooperative agreement to support a full
time employee was also a crucial element in gaining support from state
decision-makers.
* One state official said the SaDIP grant made it significantly easier
to prioritize and expedite a data project that the state was already
considering implementing. The additional funding allowed the state to
specifically address CMV crash data and use other funding to continue
to address broader crash data issues.
Finally, SaDIP has allowed states to increase the scope of ongoing
projects or develop new initiatives. Two of our six case study states
have used SaDIP funds in addition to other resources to develop
comprehensive and long-term data quality initiatives to address the
completeness, timeliness, accuracy, and consistency of the CMV crash
data they report to FMCSA.
* Officials in one state told us that SaDIP funding supports elements
of the state's electronic traffic information processing initiative.
SaDIP funding has allowed the state to resolve a considerable backlog
in crash records, creating a more complete crash file, and work has
begun to develop a new electronic crash report that will be used--first
by the state police--and then by local law enforcement in the rest of
the state. Without resolution of this backlog, the state was unable to
fully implement the CMV data component of its electronic traffic
information processing initiative. Doing this allowed the state to have
more timely, accurate, and consistent crash data.
* SaDIP funding has helped another state develop an electronic crash
report and traffic records system. As previously stated, this will
allow the state to have more timely, accurate, and consistent crash
data. The state's goal is to develop the new system by 2008 and the
state is evaluating and using multiple sources of funding to achieve
this goal. Early in the process, the state made a strategic decision to
target all crash data, not just commercial vehicle data. While the
state uses SaDIP to develop electronic crash reporting software, SaDIP
funding has also been leveraged to help the state accomplish its data
quality improvement goals more quickly. For example, the new crash
reporting capabilities developed through the SaDIP grant allow for much
better crash analysis and targeting of safety efforts along major
commercial vehicle corridors in the state.
FMCSA Has Made Improvements in the Administration of SaDIP; However,
Management of Grant Awards Raises Concerns:
SaDIP has evolved over time and FMCSA has made several efforts to
improve the administration of the program. For example, beginning in
fiscal year 2006, FMCSA will be implementing a new application package
for states to use in applying for SaDIP. While SaDIP had some
preexisting application requirements for states, this application
package provides a uniform application for states to use, and it is the
first time states are going to be required to submit quantifiable
project objectives and program measures. This will allow FMCSA to begin
to measure the effectiveness of state improvement efforts.
Additionally, SaDIP will be posted on Grants.gov ([Hyperlink,
http://www.grants.gov]) in 2006.[Footnote 30] This federal website
provides a single source for grant applicants to electronically find
and apply for federal funds. FMCSA will also be updating its State
Program Manager's manual to include guidelines for the roles and
responsibilities of state program managers in administering SaDIP.
Finally, FMCSA will be issuing funds at designated times during the
year. Currently, funds are awarded to states on a rolling basis, but
beginning with the implementation of the new application package the
funds will be awarded on specific dates to all applicants. This is
expected to improve the program's organization and FMCSA's ability to
keep track of grant progress as the number of program participants
increases. Throughout these administrative changes, FMCSA has
maintained sufficient oversight of the states participating in SaDIP.
FMCSA has contracted with a company that is responsible for monitoring
SaDIP participant states, and ensuring that they are submitting
quarterly progress reports containing sufficient detail to FMCSA. This
contractor also has regular conversations with the FMCSA Division
Administrators in the SaDIP states, and it maintains copies of SaDIP-
related paperwork.
While these efforts are positive steps, we have concerns with FMCSA's
lack of guidelines for awarding funds to states. FMCSA has not yet
established formal guidelines for determining how much money a state
should receive, or if the state should receive the funds in the form of
a grant or a cooperative agreement. Since the beginning of the program,
funds awarded to states have ranged from $2,000 to $2 million for
projects ranging from specific activities to broader efforts that span
three or more years. (See app. IV for funds awarded by state.) These
awards have taken the form of both grants and cooperative agreements
between the state and the federal government. Currently these decisions
are made on a state-by-state basis, informally by a small review panel.
The state's application, data quality history, discussions with the
state's FMCSA division administrator, and any other relevant and
available data on the state are consulted when making funding awards.
While the absence of guidelines has not proven to be problematic to
date, having formal guidelines will better ensure consistency and
discipline in managing the grant program among states, particularly as
states' needs become more targeted or more states decide to participate
in the program. It will also add integrity to the grant management
system and assist in providing better administration and oversight of
SaDIP projects. For example, such guidelines would likely allow FMCSA
to better assess and prioritize states' funding requests, including
more formally considering whether proposed activities adequately
address problems states identify in the proposal, whether the amount of
funding requested is appropriate for the proposed activities, and
whether multiple-year versus one-year funding is appropriate. Further,
formal guidelines would provide a more structured framework to evaluate
the effectiveness of different SaDIP project activities and assist in
guiding future state improvement efforts.
Data Quality Map Has Spurred Improvements, but Limitations Curb Map's
Continued Usefulness:
The State Safety Data Quality map has encouraged states to improve
their CMV data quality, but limitations exist that may hinder the map's
usefulness as a tool for monitoring and measuring CMV crash data
quality. The State Safety Data Quality map was created by FMCSA and the
Volpe National Transportation Systems Center principally to provide
context for both crash and inspection data used in the SafeStat system;
however, it has evolved into a tool to evaluate state-reported crash
and inspection data.[Footnote 31] It is based on a system that rates
crash and inspection data quality--completeness, timeliness, and
accuracy--as "good," "fair," or "poor."[Footnote 32] The map has proven
to be a major incentive for states to initiate CMV crash data
improvements, for gaining support in implementing these improvements,
and as a tool to monitor CMV crash data quality. However, we have
identified some important limitations that can affect the data quality
map's future usefulness.
The State Safety Data Quality Map is an Incentive for States to Make
CMV Crash Data Improvements:
Since the State Safety Data Quality map is accessible by the public and
presents data quality ratings in a simplified form, the map is a
motivator for states to improve their CMV crash data. According to
officials at FMCSA, they said the map has been very influential in
encouraging states to improve their CMV crash data. Corroboratory
comments came from the state level, where many of the officials in case
study states provided anecdotes of how the data quality map served as
an impetus for initiating improvements and gaining support.[Footnote
33] Here are examples of how the map assisted states in initiating or
expediting improvements in CMV crash information.
* One state includes the data quality map ratings in weekly status
reports to agency heads and in reports to the governor. An official in
this state said that once the data quality map was shown to the
governor, it raised the importance of improving CMV crash data. This
state now posts the data quality map on state agency Web sites and uses
it as a tool to compare its data quality efforts to the rest of the
nation.
* One state official told us that when agency leaders understand that
their state has a "poor" rating, they are likely to make data
improvement a priority and focus attention and resources on the issue.
* Another official said that the map helped officials to "see the
light." In this state, the data quality map helped initiate a process
that led to improved communication and coordination for data quality at
the highest levels for all safety and security projects within the
state.
The Data Quality Map is Important in Addressing Data Quality:
FMCSA and states we reviewed use the State Safety Data Quality map to
measure states' CMV data quality and progress being made in their data
improvement efforts. Our review indicated that although the map was
created as a tool for providing context to FMCSA's SafeStat system,
both FMCSA and the states were using it specifically for monitoring the
status of CMV crash data quality. State officials were also using it to
identify data quality problems and assist states in targeting state
improvement efforts. Here are examples from the states we contacted:
* In one state an official reported that it was very helpful for law
enforcement officials and department heads to instantly recognize their
state's data quality status on the map.
* In another state, officials said they knew they had a data quality
problem, but were unable to identify the specific issue. The data
quality map indicated that the state's accuracy was poor, and the
problem was specifically with matching carriers in the state database
with crashes in MCMIS.
* In another state, officials said the data quality map focused
attention on the state's data quality spending and results, and it
helped state officials re-prioritize spending.
Both FMCSA and state officials in many of the states we spoke with
recognized the map as an important tool in measuring progress in their
crash data improvement efforts. Further, FMCSA officials said the map
was used as an assessment of a state's success in the SaDIP program.
While officials also said they reviewed data in the MCMIS system to
monitor state progress, an important measure of success in the program
was a state's status in the State Safety Data Quality map and the
underlying CMV crash measures.
The State Safety Data Quality Map Has Limitations that May Affect its
Effectiveness as a Tool for Monitoring and Measuring CMV Crash Data
Quality:
Given the importance that FMCSA and the states attach to the data
quality map, it appears to be a good first step in monitoring states'
data quality. However, we found limitations with many features of the
map that diminish its usefulness as an effective tool for specifically
monitoring and measuring state progress in their improvement efforts.
Specifically, we identified several limitations with the methodology
used to develop the data quality map's ratings, as well as with the
measures themselves that should be addressed. Below are some of the key
limitations. A more detailed discussion of limitations is located in
appendix II.
* The data quality map measures used to calculate the completeness,
timeliness, and accuracy of CMV crash data falls short of providing a
complete measure of CMV crash data quality. While each data quality
measure has some limitations, one key measure--completeness--has a
number of difficulties. Most importantly, the completeness measure is
limited in the data it is assessing. The completeness measure only
evaluates fatal CMV crashes, which represents about 3 percent of all
reportable CMV crashes.[Footnote 34] In addition, the completeness
measure does not assess the completeness of the information contained
within the crash report.
The completeness measure's methodology relies solely on comparing CMV
fatal crash data in MCMIS to data states submit to the Fatality
Analysis Reporting System (FARS).[Footnote 35] This particular approach
provides additional limitations to the completeness measure. First,
there are some definitional differences between FARS and MCMIS data
that can account for about 4 percent of the crash records. Secondly,
FARS data is not timely. For example, the June 2005 map relies on 2003
FARS data; thus the completeness measure does not reflect the current
status of CMV crash data. In addition, since FARS data is only released
once a year, quarterly issuances of the map do not necessarily reflect
changes in the number of fatal crashes. Table 3 below describes in more
detail the limitations for each crash data quality measure.
Table 3: Comparison of Data Quality Standards and State Safety Data
Quality Map Measures:
Data quality standard: Completeness;
Data quality standard: All reportable CMV crashes in the state and data
on all appropriate crash variables are submitted to FMCSA. FMCSA
recommends 20 CMV crash data variables that should be reported on.[A];
State Safety Data Quality Map measure: Percentage of Fatal Crash
Records Reported. Compares the number of large trucks in crashes
involving a fatality in MCMIS versus those in the FARS.[B];
Limitations: Includes only reported, fatal crashes; reflects only about
3 percent of all CMV crashes required to be reported to FMCSA. Bases
completeness on FARS data, though some differences exist in state
definitions of a CMV fatal crash and may result in an inflated or
deflated rating. Does not represent current status of completeness.
Does not measure the completeness of CMV information within a crash
report (missing variables).
Data quality standard: Timeliness;
Data quality standard: All reportable CMV crash records are available
for state and federal analytical purposes in a useful timeframe. FMCSA
recommends that CMV crash data be reported within 90 days of the crash
occurrence;
State Safety Data Quality Map measure: Percentage of Crash Records
Reported within 90 Days. The percentage of State-reported fatal and non-
fatal crash records reported within 90 days in the MCMIS database for
carriers over a 12-month time period.[C];
Limitations: Only reflects records that are uploaded into MCMIS and
have not been changed or edited.[D]. Backlogs of crash reports, once
entered, can negatively affect timeliness rating.
Data quality standard: Accuracy;
Data quality standard: All data within reportable CMV crash records are
accurate and reliable;
State Safety Data Quality Map measure: Percentage of Matched Crash
Records. The percentage of State-reported fatal and non-fatal crash
records in the MCMIS database for interstate carriers and intrastate
hazardous material carriers over a 12-month time period that were
matched to a motor carrier in MCMIS;
Limitations: Only measures accuracy of one variable (identity of the
motor carrier--the U.S. DOT number); neglects to measure the accuracy
for other recommended variables.
Data quality standard: Consistency;
Data quality standard: Crash data should be collected uniformly.
Officials should use the same standards and apply the same criteria
uniformly across jurisdictions. FMCSA provides guidelines and criteria
for reporting CMV crashes;
State Safety Data Quality Map measure: None;
Limitations: N/A.
Source: FMCSA and Volpe National Transportation Systems Center. Data
Quality Standards are based on GAO's review of data quality guidelines
from a variety of sources including FMCSA, NHTSA, and Data Nexus Inc.,
and includes references to the Model Minimum Uniform Crash Criteria,
National Governors' Association Elements and Criteria, and the American
National Standard Manual for Classification of Motor Vehicle Traffic
Records.
[A] In 1992, FMCSA adopted the National Governors' Association (NGA)
recommended data elements, requiring these data to be collected and
reported on motor carrier crashes.
[B] FMCSA created an "Crash Consistency Overriding Indicator" to
indicate consistency in reporting of non-fatal CMV crashes. The
indicator flags states that may be experiencing major problems in
reporting crash data. The Crash Consistency Overriding Indicator is the
percentage of state-reported non-fatal crashes as compared to a three-
year average of reported non-fatal crashes. States that have reported
fewer than 50 percent of non-fatal crash records for the current year
based on the previous three-year average of non-fatal crash records are
flagged and receive a rating of "poor" regardless of their ratings in
any of the other data quality indicators. This indicator is also
limited because it only identifies extreme cases of under-reporting and
does not assess if there is a substantial increase in reported non-
fatal crash records.
[C] Includes both interstate carriers (those carriers that operate
between states) and intrastate carriers (carriers that operate only
within one state) though FMCSA is only responsible for regulating
interstate carriers.
[D] FMCSA's MCMIS currently does not have the ability to track the
original upload date when a crash record is edited or changed.
Therefore, records that are changed after the original upload are not
used in the calculation for timeliness. For additional information see
appendix II.
[End of table]
* The data quality map is limited in its ability to meaningfully
monitor and track CMV crash data quality over time. Since the State
Safety Data Quality map was first issued, a large majority of states
have been rated as "good" or "fair" for the completeness, timeliness,
and accuracy measures. Since the first issuance of the map, over 90
percent of the states have been rated "good" or "fair" for
completeness, and about three quarters of the states have been rated
"good" or "fair" for crash timeliness and crash accuracy. However, our
review shows that states need to continue to make improvements. In
addition, since the majority of states are rated as "good" or "fair" in
measures, it is also difficult to measure any progress made because
many states have already reached the highest rating. For example, as of
the June 2005 data quality map, 41 states were rated "good" in crash
completeness. Because these states have already reached the highest
rating for this measure, it may prevent measuring any subsequent
progress in this data quality standard.
There are other problems with using the data quality map to track
trends. Based on our review, a state's rating can temporarily decline
for a variety of reasons--even for implementing improvements. Hence,
for that period of time, the data quality map is not accurately
reflecting a state's true data quality status. Officials in one state
told us that the state was implementing a new electronic system which
in the long term would improve its data quality greatly. However,
current CMV crash reporting was slowed during implementation, and the
state's rating went down from "fair" to "poor."
* The data quality map's ratings for overall data quality combine data
from crashes with data from FMCSA's inspections, making it difficult
for map users to obtain an overall picture based solely on crash data.
For the overall data quality rating, individual crash measures for
completeness, timeliness, and accuracy are combined with inspection
measures for timeliness and accuracy. For each of the five individual
crash and inspection measures a state will receive a rating of "good,"
"fair," or "poor."[Footnote 36] A state's overall rating depends on how
well the state did across all five measures (see table 4).
Table 4: Overall Data Quality Rating:
Rating: Good;
Thresholds for overall data quality rating: No "poor" and a minimum of
one "good" in a crash or inspection measure.
Rating: Fair;
Thresholds for overall data quality rating: Maximum of one "poor" in a
crash or inspection measure.
Rating: Poor;
Thresholds for overall data quality rating: Two or more "poor" ratings
in crash or inspection measures or a red flag in the Crash Consistency
Overriding Indicator.
Source: FMCSA.
[End of table]
Currently, users can only view CMV crash data quality by individual
measures (completeness, timeliness, and accuracy). Separating the
inspection data and presenting a specific overall CMV crash data
quality rating (a combined rating composed of completeness, timeliness,
and accuracy for crash data only) would enhance a state's ability to
understand its crash data status and to monitor progress in improving
the information.
* While the State Safety Data Quality map provides a description of the
methodology used, it does not identify limitations to the methodology.
While many of these limitations to the map are acknowledged by FMCSA,
they are not publicly displayed on the State Safety Data Quality map
Web site. The absence of this information limits users' understanding
of the map's data and increases the potential for incorrect deductions
and improper map-based decisions.
FMCSA officials are aware of many of the limitations that we have
identified and we recognize their efforts to improve the State Safety
Data Quality map to date. However, they do not have a formal plan in
place to implement improvements. Further, it is important to
acknowledge these issues so that users understand the limitations of
the data quality map as a tool. During our review we learned that not
only was the data quality map consulted in awarding SaDIP funds, it was
also consulted when awarding Motor Carrier Safety Assistance Program
(MCSAP) High Priority funds.[Footnote 37] Further, according to
officials at FMCSA and Volpe National Transportation Systems Center,
there has been some discussion of expanding the usage of the map in the
future for CMV crash data efforts. As the data quality map gains wider
use, it will become even more important that these limitations are
addressed.
Conclusions:
The grant program and FMCSA's collaborative efforts with states have
had a positive impact on improving the quality of states' crash data,
therefore ultimately enhancing the ability of both federal and state
governments to make highway planning and safety enforcement decisions.
While states have made progress in improving the quality of this data-
-in terms of timeliness, completeness, accuracy, and consistency--much
remains to be done. With additional funding through the reauthorization
of the Safe, Accountable, Flexible, Efficient Transportation Equity
Act: A Legacy for Users and as states refine and target areas needing
further improvement with respect to their crash data, it is expected
that additional states will participate in the program. FMCSA will need
a more formal framework to better ensure that the decision-making
process for awarding funds to SaDIP program applicants is conducted
uniformly.
FMCSA's efforts to improve the quality of commercial motor vehicle
crash data have brought considerable attention to the issues associated
with poor commercial motor vehicle crash data. Providing states with
funding to improve their CMV data quality and publicizing a rating
through a data quality map are incentives that work in tandem to
maximize states' efforts. It is clear that states pay attention to
their ratings of "good," "fair," or "poor" on the data quality map.
However, the limitations we identified highlight some important
concerns with the data quality map's ability to measure progress and
accurately portray states' commercial motor vehicle crash data quality.
As FMCSA continues to make improvements, it will be important for these
ratings to become more precise, so that FMCSA and the states can obtain
the clearest picture possible of the progress being made.
Recommendations for Executive Action:
We recommend that the Secretary of Transportation direct the
Administrator of FMCSA do the following:
* Establish specific guidelines for assessing state proposals for SaDIP
grants in order to better assess and prioritize states' funding
requests and provide uniformity in awarding funds.
* Increase the State Safety Data Quality map's usefulness as a tool for
monitoring and measuring commercial motor vehicle crash data by
ensuring that it adequately reflects the condition of the states'
commercial motor vehicle crash data and continues to motivate states in
their improvement efforts. Specifically, FMCSA should develop a plan
for assessing and improving the data quality map's methodology. In
addition, FMCSA should display an overall crash data rating separately
from the inspection rating, and provide information on the limitations
of the State Safety Data Quality map and the underlying ratings on
FMCSA's Analysis and Information (A&I) Online Web site.
Agency Comments and Our Evaluation:
We provided a draft of this report to the Department of Transportation
for its review and comment. The department agreed with our findings and
recommendations in this report. Department officials provided some
technical comments and some minor additions to provide more detail on
FMCSA's training efforts.
We will send copies of this report to the interested congressional
committees, the Secretary of Transportation, and other interested
parties. We will make copies available to others upon request. In
addition, the report will be available at no charge on GAO's Web site
at [Hyperlink, http://www.gao.gov].
If you or your staff has any questions about this report, please call
me at (202) 512-6570. Contact points for our Offices of Congressional
Relations and Public Affairs may be found on the last page of this
report. Key contributors to this report are listed in appendix VI.
Signed by:
Katherine Siggerud:
Director, Physical Infrastructure:
[End of section]
Appendixes:
Appendix I: Objectives, Scope and Methodology:
Congress asked us to review the Federal Motor Carrier Safety
Administration's (FMCSA) program for helping states improve their
commercial motor vehicle crash data. As part of this review, we were
asked to describe the benefits obtained through the program, identify
what can be done to improve the effectiveness of the program, and
address concerns regarding crash data raised in a February 2004
Department of Transportation (DOT) Inspector General's report. The
specific objectives of this report were to explain (1) what is known
about the quality of commercial motor vehicle crash data and what
states are doing to improve it, and (2) the results of FMCSA's efforts
to facilitate the improvement of the quality of commercial motor
vehicle crash data sent to the federal government.
To provide information on the quality of states' commercial motor
vehicle (CMV) crash data and efforts to improve it, we reviewed grant
applications and project information from 39 states. Of these 39
states, 34 participated in the Safety Data Improvement Program (SaDIP)
and 5 states either chose not to participate in the program or their
proposals were not accepted. We also conducted site visits to six of
these states--Georgia, Minnesota, North Carolina, Ohio, Oklahoma, and
Washington. We chose our case study states based on a variety of
criteria, including participation in the SaDIP grant program, the type
of agencies with which the state works under SaDIP, the number of CMV
crashes in the state, the number of reported CMV crash fatalities, and
data quality map ratings. To help ensure that our states reflected a
variety of experiences, we chose states that had different combinations
of these criteria. To understand the insights and experiences of states
that no longer participated--or had never participated--in SaDIP, we
also interviewed officials in Michigan, Missouri, New Hampshire, and
New Jersey.[Footnote 38]While the results from the case studies and
interviews cannot be projected to the universe of states, they are
nonetheless useful in illustrating the uniqueness and variation of CMV
crash data systems and the challenges states face in improving them.
During our case study visits we met and discussed the status of state
crash data systems with a variety of traffic safety data
officials.[Footnote 39] The discussions included gathering information
on FMCSA's data quality criteria[Footnote 40] but also included, for
those participating in the program, state objectives and progress made
with SaDIP funds. For additional perspective, we also interviewed key
experts from organizations responsible for the development of crash
data systems and models used by FMCSA, carrier industry officials, and
public interest organizations. Finally, we conducted a literature
review of studies published by the University of Michigan
Transportation Research Institute (UMTRI). The Institute plans to
conduct studies in all states to determine where problems are occurring
in the collection of CMV crash data and in the reporting of this data
to FMCSA. As of September 2005, studies had been conducted in eight
states. We reviewed the studies and determined the methodologies used
in determining whether Motor Carrier Management Information System
(MCMIS) and state CMV crash data were sound. We reviewed the studies
conducted by UMTRI and determined that the methodologies used in
assessing the quality of the data states submit to FMCSA were sound and
that the studies provided sufficiently reliable results for our
purposes. Through site visits, a review of grant applications,
interviews with relevant stakeholders/experts, and these studies, we
were able to determine shortcomings in the reliability of FMCSA's CMV
crash data. However, we determined that the data was sufficiently
reliable for our purpose of case study selection.
To provide results of FMCSA's efforts to facilitate the improvement of
the state CMV crash data quality, we conducted interviews with
officials from participating states and from FMCSA concerning the
administration and management of SaDIP. We also analyzed the guidance
and support provided by FMCSA to states for CMV data improvement
efforts and assessed FMCSA's role in coordinating CMV data quality
initiatives. In addition, we reviewed FMCSA's State Safety Data Quality
map and assessed the methodology used by FMCSA in evaluating states'
crash data quality. We interviewed officials and key experts at FMCSA
and the Volpe National Transportation Systems Center responsible for
developing and managing the data quality map. We also interviewed state
officials from states participating--and not participating--in the
SaDIP program to find out their views on the data quality map and its
use as a monitoring tool for CMV crash data improvements.
We conducted our review from February 2005 through November 2005 in
accordance with generally accepted government auditing standards.
[End of section]
Appendix II: State Safety Data Quality Map Limitations:
This appendix explains, in greater detail than the body of our report,
the concerns we (and others) have raised about the limitations in the
methodology FMCSA uses to develop ratings for the State Safety Data
Quality map. The measures FMCSA employs to measure the completeness,
timeliness, and accuracy of CMV crash data quality are limited, and do
not provide comprehensive measurements of these attributes as
established by the general data standards discussed in the body of this
report. As a result, the ability to draw conclusions about the actual
quality of a state's data is limited.
The definitions FMCSA uses for each of the crash measures are shown in
table 5, together with the criteria that constitute a rating of "good,"
fair," or "poor." In the sections that follow we explain the
limitations associated with each measure, followed by other limitations
that stem from the current methodology.
Table 5: State Safety Data Quality Map Measures for CMV Crashes:
Data quality standard: Completeness;
Measure: The measure compares CMV fatality data in the MCMIS database
against those in the Fatality Analysis and Reporting System (FARS). The
FARS data is available through 2003;
Criteria: Good = match to FARS is greater than or equal to 90 percent;
Fair = match to FARS is between 80 and 89 percent; Poor
= match to FARS is below 80 percent.
Data quality standard: Timeliness;
Measure: This measure reflects the percentage of state-reported crash
records uploaded to the MCMIS database within the 90-day standard;
Criteria: Good = the percentage of reported records within 90 days is
greater than or equal to 85 percent; Fair = the percentage of reported
records within 90 days is between 60-84 percent; Poor = the percentage
of reported records is less than 60 percent.
Data quality standard: Accuracy;
Measure: This measure reflects the percentage of state-reported crash
records (fatal and non-fatal) that were matched to a motor carrier in
MCMIS over a 12-month period;
Criteria: Good = the percentage of matched records is greater than or
equal to 95 percent; Fair = the percentage of matched records is
between 85 and 94 percent; Poor = the percentage of matched records is
below 85 percent.
Source: FMCSA.
[End of table]
Completeness: Overall Completeness is Based on Fatal Crashes Only:
A key limitation in FMCSA's measure of crash data completeness is the
inability to evaluate completeness against nonfatal CMV crashes.
Currently, the total number of nonfatal crashes occurring on a state-
by-state basis is unknown; no baseline exists against which to measure
these records.[Footnote 41] Consequently, FMCSA is limited to measuring
crash data completeness with fatal CMV crashes, which is approximately
3 percent of all CMV crashes.[Footnote 42]
Even within this narrowed dataset, the use of FARS[Footnote 43] as the
basis of comparison poses other limitations:
* There are notable differences in definitions used in the two
databases. Most importantly, MCMIS can be subject to individual state
definitions that may differ from FARS. According to officials at the
Volpe National Transportation Systems Center, the range of these state-
driven definitional differences is unknown. However, based on anecdotal
evidence received from states and reported by Volpe, these differences
can vary by state and can account for about 4 percent of crash
records.[Footnote 44] Some examples are below:
* A crash fatality resulting from the private use of a large truck may
not meet the criteria as a reportable crash for MCMIS, but it is
considered a large truck crash fatality by FARS.
* MCMIS defines a large truck as any truck greater than a 10,000 gross
vehicle weight rating. Many states do not collect the gross vehicle
weight rating of vehicles and instead define a large truck based on an
older FMCSA reporting criterion of greater than six tires or another
definition of their choosing. As such, counts of large trucks derived
from state crash databases (and reported to FARS) may be inconsistent
with counts of vehicles that were reported to MCMIS.[Footnote 45]
The effect of these definitional differences is evident when comparing
MCMIS fatal CMV crashes to FARS data--the state's index on completeness
for crash data rises above 100 percent. The extent to which this occurs
is substantial: in our analysis of the June 2005 issuance of the map,
24 of 51 states (47 percent) had a completeness measure of greater than
100 percent.[Footnote 46]
* FARS data is not current. FARS is released once per year, detailing
the prior calendar year's crash statistics. The lag between the most
recently available FARS data and the most recent map issuance can be
considerable. For example, the 2004 FARS annual report is not scheduled
to be released until fall 2005 and as a result, the baseline for
completeness data in the June 2005 map is based upon 2003 crash data.
Timeliness: Timeliness is Not Based on All Reported Crashes:
The timeliness measure currently relies on a subset of records states
submit to MCMIS. Any record that has been edited or changed since it
was originally entered into MCMIS is not included in the
calculations.[Footnote 47] If a record is edited, the initial upload
date is replaced with the date it was updated. Because FMCSA's
timeliness rating is based on the percentage of crash reports uploaded
within 90 days of the crash, and edited records no longer reflect their
initial upload date, they cannot be used in the timeliness calculation
without distorting a state's rating. The consequence, however, is that
timeliness is not measured against the entire universe of crashes in
MCMIS--the more records a state edits, the fewer records its timeliness
rating represents. FMCSA has acknowledged this problem with the edited
records and is taking steps to resolve it.
Another limitation with this measure is that efforts to reduce backlogs
of crash records--a positive effort--can have a negative effect on the
timeliness rating. If a state submits a backlog of reports from CMV
crashes that occurred more than 90 days previously, and the crash took
place during the period of time which the FMCSA rating covers, then the
state's timeliness rating will be negatively affected. Conversely, if a
state has a large backlog, its current timeliness rating may not be a
meaningful representation of timeliness because the methodology has no
way to capture those records that are accruing in the backlog.
Accuracy: Accuracy is Based on Only One Variable:
The measure for accuracy on the data quality map is based only on a
match of a CMV crash report in MCMIS, against a registered carrier in
MCMIS. The U.S. Department of Transportation (DOT) assigns each
interstate motor carrier a unique identifier number--the DOT number.
For this measure, accuracy is evaluated primarily based on the degree
to which a carrier's DOT number is the same as identified for a crash
in MCMIS.[Footnote 48] Therefore, a crash report with a missing or
invalid DOT number may be considered inaccurate, even if the rest of
the information on the report provides accurate information on the
crash. Currently, the data quality map does not assess the accuracy of
any other data elements; however, FMCSA has been working with states to
improve their collection of CMV crash information. For example,
recently FMCSA distributed visor cards to state officials that explain
how to determine who the carrier is, and where the correct carrier's
DOT number can be found. See appendix III for copies of these cards.
[End of section]
Appendix III: FMCSA Reportable Crash, CMV, and Carrier Identification
Visor Cards:
The following are copies of visor identification cards that FMCSA
created as educational tools for law enforcement officers. The cards
can be placed in officer's sun visor and referenced to determine
whether a CMV crash should be reported, for identifying a vehicle as a
CMV, and for identifying the correct carrier involved in the crash.
FMCSA provided these to states to distribute to enforcement officers
with the intention they will increase the officers' ability to properly
identify a CMV and a reportable CMV crash.
Figure 3: FMCSA Reportable Crashes:
[See PDF for image]
[End of figure]
Figure 4: Reportable Commercial Motor Vehicle Configurations and Cargo
Body Type:
[See PDF for image]
[End of figure]
Figure 5: Responsible Carrier and Correct DOT Number Identification:
[See PDF for image]
[End of figure]
[End of section]
Appendix IV: SaDIP Grant and Cooperative Agreement Distribution by
State:
Table 6: Distribution of SaDIP Grants by State:
State: Alaska;
FY2002: $198,000;
FY2003: $0;
FY2004: $0;
FY2005: $218,626;
Total: $416,626.
State: California;
FY2002: $0;
FY2003: $0;
FY2004: $100,000;
FY2005: $0;
Total: $100,000.
State: Connecticut;
FY2002: $234,056;
FY2003: $0;
FY2004: $0;
FY2005: $0;
Total: $234,056.
State: Georgia;
FY2002: $269,820;
FY2003: $0;
FY2004: $409,000;
FY2005: $553,733;
Total: $1,232,553.
State: Indiana;
FY2002: $179,321;
FY2003: $242,423;
FY2004: $0;
FY2005: $0;
Total: $421,744.
State: Kansas;
FY2002: $0;
FY2003: $0;
FY2004: $0;
FY2005: $373,436;
Total: $373,436.
State: Kentucky;
FY2002: $199,000;
FY2003: $0;
FY2004: $0;
FY2005: $0;
Total: $199,000.
State: Maryland;
FY2002: $145,600;
FY2003: $0;
FY2004: $0;
FY2005: $0;
Total: $145,600.
State: Massachusetts;
FY2002: $100,000;
FY2003: $249,972;
FY2004: $0;
FY2005: $0;
Total: $349,972.
State: Michigan;
FY2002: $0;
FY2003: $115,845;
FY2004: $0;
FY2005: $0;
Total: $115,845.
State: Minnesota;
FY2002: $363,000;
FY2003: $0;
FY2004: $620,000;
FY2005: $0;
Total: $983,000.
State: Montana;
FY2002: $0;
FY2003: $0;
FY2004: $0;
FY2005: $328,564;
Total: $328,564.
State: Nebraska;
FY2002: $0;
FY2003: $0;
FY2004: $2,000;
FY2005: $3,342;
Total: $5,342.
State: Nevada;
FY2002: $427,443;
FY2003: $0;
FY2004: $0;
FY2005: $350,000;
Total: $777,443.
State: New Hampshire;
FY2002: $0;
FY2003: $98,068;
FY2004: $0;
FY2005: $0;
Total: $98,068.
State: New Mexico;
FY2002: $0;
FY2003: $0;
FY2004: $0;
FY2005: $250,000;
Total: $250,000.
State: New York;
FY2002: $0;
FY2003: $0;
FY2004: $300,000;
FY2005: $0;
Total: $300,000.
State: North Carolina;
FY2002: $0;
FY2003: $0;
FY2004: $193,350;
FY2005: $0;
Total: $193,350.
State: Oklahoma;
FY2002: $0;
FY2003: $0;
FY2004: $131,111;
FY2005: $0;
Total: $131,111.
State: Pennsylvania;
FY2002: $0;
FY2003: $0;
FY2004: $0;
FY2005: $2,000,000;
Total: $2,000,000.
State: Rhode Island;
FY2002: $25,000;
FY2003: $0;
FY2004: $26,000;
FY2005: $0;
Total: $51,000.
State: South Carolina;
FY2002: $350,000;
FY2003: $0;
FY2004: $0;
FY2005: $0;
Total: $350,000.
State: South Dakota;
FY2002: $270,000;
FY2003: $0;
FY2004: $0;
FY2005: $0;
Total: $270,000.
State: Tennessee;
FY2002: $0;
FY2003: $0;
FY2004: $436,027;
FY2005: $149,334;
Total: $585,361.
State: Texas;
FY2002: $0;
FY2003: $148,424;
FY2004: $533,611;
FY2005: $0;
Total: $682,035.
State: Utah;
FY2002: $0;
FY2003: $0;
FY2004: $433,500;
FY2005: $128,853;
Total: $562,353.
State: Vermont;
FY2002: $0;
FY2003: $0;
FY2004: $0;
FY2005: $134,000;
Total: $134,000.
State: Washington;
FY2002: $0;
FY2003: $0;
FY2004: $803,935;
FY2005: $0;
Total: $803,935.
State: West Virginia;
FY2002: $157,500;
FY2003: $0;
FY2004: $0;
FY2005: $603,943;
Total: $761,443.
Total;
FY2002: $2,918,740;
FY2003: $854,732;
FY2004: $3,988,534;
FY2005: $5,093,831;
Total: $12,855,837.
Source: FMCSA.
Note: This data is current as of September 2005.
[End of table]
Table 7: Distribution of SaDIP Cooperative Agreements by State:
State: Colorado;
FY2002: $0;
FY2003: $0;
FY2004: $0;
FY2005: $1,460,000;
Total: $1,460,000.
State: Georgia;
FY2002: $0;
FY2003: $0;
FY2004: $0;
FY2005: $612,051;
Total: $612,051.
State: Iowa;
FY2002: $728,065;
FY2003: $0;
FY2004: $0;
FY2005: $226,400;
Total: $954,465.
State: Louisiana;
FY2002: $829,625;
FY2003: $0;
FY2004: $0;
FY2005: $569,000;
Total: $1,398,625.
State: Ohio;
FY2002: $406,330;
FY2003: $0;
FY2004: $91,000;
FY2005: $224,400;
Total: $721,730.
State: Oklahoma;
FY2002: $0;
FY2003: $0;
FY2004: $1,150,390;
FY2005: $0;
Total: $1,150,390.
State: Tennessee;
FY2002: $281,954;
FY2003: $0;
FY2004: $219,543;
FY2005: $426,382;
Total: $927,879.
State: Vermont;
FY2002: $0;
FY2003: $0;
FY2004: $0;
FY2005: $136,500;
Total: $136,500.
State: Virginia;
FY2002: $42,300;
FY2003: $0;
FY2004: $0;
FY2005: $431,617;
Total: $473,917.
State: Total;
FY2002: $2,288,274;
FY2003: $0;
FY2004: $1,460,933;
FY2005: $4,086,350;
Total: $7,835,557.
Source: FMCSA.
Note: This data is as of September 2005.
[End of table]
[End of section]
Appendix V: SaDIP Case Study States:
As part of our work, we conducted six case studies to examine how
states are working to improve commercial motor vehicle crash data. Our
visits to the six states yielded additional information about crash
data quality improvement activities, the nature of their efforts, and
the extent of progress made. States were chosen on a wide variety of
factors, including crash data quality and participation in the SaDIP
program.
Georgia:
Georgia received a total of $1,844,604 between 2002 and 2005 (see table
8) to conduct crash data improvement projects. During this period,
Georgia made significant improvement in its crash data quality, despite
undergoing a major state government reorganization process.
Table 8: Georgia SaDIP Funding History:
Award year: 2005;
Funds awarded: $612,051;
Agency: FMCSA;
Award type: Cooperative agreement.
Award year: 2005;
Funds awarded: $553,733;
Agency: FMCSA;
Award type: Grant.
Award year: 2004;
Funds awarded: $409,000;
Agency: FMCSA;
Award type: Grant.
Award year: 2002;
Funds awarded: $269,820;
Agency: NHTSA;
Award type: Grant.
Source: FMCSA.
[End of table]
SaDIP Projects:
Georgia received its first SaDIP grant from the National Highway
Traffic Safety Administration (NHTSA), and this grant was used to
accomplish several projects, including: (1) hiring temporary employees
to extract CMV crash reports from backlogged paper copies and microfilm
them; (2) precoding the paper crash reports in preparation for their
entry into the state repository system; (3) renting a mobile trailer to
house temp employees; (4) developing a system to electronically
transfer crash data from the state repository to the national database;
(5) adding edit checks to the state crash database; (6) purchasing a
new microfilm scanner/reader; and, (7) hiring a contractor to update
the state's crash report manual. Progress on these projects was
reported quarterly to NHTSA.
Georgia received two additional grants after FMCSA became the lead
agency for the SaDIP program. The first grant was used to maintain the
temporary employees. Additionally, these funds were used to hire a
statistician as a SaDIP advocate. This grant allowed Georgia to
eliminate its reporting backlog from 2003 to the present. The second
grant that Georgia received was used to fund its electronic crash
records system. This system will electronically transfer crash records
from local jurisdictions to the state repository system. This project
has been put on hold following a state government reorganization.
Crash Data Collection/Reporting Issues:
Georgia historically has had the worst CMV crash data quality rating in
the nation (see crash data quality statistics, table 9). This is due to
the fact that until recently, Georgia was not submitting crash data to
FMCSA. Georgia's current overall rating is "good," and it is rated
"good" for completeness and "fair" for timeliness and accuracy. For all
other rating periods prior to March 2004, Georgia had an overall rating
of "poor." This improvement in its data quality is due in large part to
the SaDIP program. Georgia's non-reporting was due to a technical
problem, but the state was able to correct it in a timely manner
because of the SaDIP funding it received.
Table 9: Georgia Crash Data Quality Statistics (Percentages):
Completeness;
March 2004: 117;
June 2004: 115;
September 2004: 95;
December 2004: 95;
March 2005: 95;
June 2005: 95.
Timeliness;
March 2004: 6;
June 2004: 8;
September 2004: 27;
December 2004: 48;
March 2005: 65;
June 2005: 80.
Accuracy;
March 2004: 84;
June 2004: 82;
September 2004: 82;
December 2004: 83;
March 2005: 94;
June 2005: 93.
Source: FMCSA.
[End of table]
SaDIP funds are the only federal funds currently being used to address
data quality improvement in Georgia. Because the state's data quality
was so poor, both FMCSA and the Federal Highway Administration
indicated they would withhold federal funding unless Georgia improved
its reporting. It was at this point that Georgia applied for SaDIP, and
began focusing on its data quality. As of July 1, 2005, Georgia began a
major state government reorganization process, and officials were
unsure of how this would affect the quality of its crash data because
all collection and reporting functions are moving to new departments.
Officials expect to see a decrease in the state's rating during the
transition period, but do not expect long-term problems.
State/Federal Coordination Issues:
State and federal coordination on issues related to traffic safety data
has not always been effective in Georgia. Between 1998 and 2002,
Georgia had only submitted 250 fatal CMV crash records to FMCSA, yet
between 2002 through 2003 Georgia submitted 410 CMV fatalities to FARS.
Georgia's Department of Public Safety, the agency responsible for crash
reporting, did not have a functional computer system to upload the
state's crash file to FMCSA. The problem was discovered in 2002 and
FMCSA offered assistance to fix the problem, but the Department of
Public Safety did not accept the offer. FMCSA then threatened to
withhold Georgia's highway safety funds until the data issues were
resolved; at this point the Department of Public Safety applied for--
and received--SaDIP funding, and began work to resolve the crash data
problems.
Data Quality Map:
Georgia officials stated that the data quality map provided an
incentive for the state to make data quality improvements in order to
raise its rating from "poor" to "good." State and federal officials
acknowledge that the map is one indication of the progress the state
has made in improving its CMV data quality.
Georgia officials never disagreed with their state's rating on the data
quality map. However, officials felt that the map did not recognize
improvements that were taking place within the state to improve crash
data. The state plans to continue to make improvements to the crash
records system even after the state reaches the "good" rating.
Minnesota:
Minnesota received a total of $983,000 in 2002 and 2004 (see table 10)
to conduct crash data improvement projects. During this period,
Minnesota received safety data improvement grants from NHTSA and FMCSA.
Table 10: Minnesota SaDIP Funding History:
Award year: 2004;
Funds awarded: $370,000;
Agency: FMCSA;
Award type: Grant.
Award year: 2004;
Funds awarded: $250,000;
Agency: FMCSA; Award type: Grant.
Award year: 2002;
Funds awarded: $363,000;
Agency: NHTSA;
Award type: Grant.
Source: FMCSA.
[End of table]
SaDIP Projects:
Minnesota received its first SaDIP grant while NHTSA was administering
the program. This grant went to the Department of Public Safety, Office
of Traffic Safety--the agency that receives NHTSA highway safety funds.
The grant was used to fund two projects, including an electronic post-
crash inspection form (Minnesota conducts in-depth post-crash
inspections for crashes involving CMVs), and to purchase software that
will allow for electronic transfer of CMV crash records between Driver
and Vehicle Services and the state patrol (both agencies within the
Department of Public Safety). In Minnesota, Driver and Vehicle Services
collects crash reports from law enforcement agencies, and the state
patrol is the office in charge of reporting crashes to FMCSA.
The Minnesota state patrol received subsequent funding after FMCSA
became the lead agency on the SaDIP program. These funds have been used
to provide computer hardware so that state patrol officers can access
the Internet and submit their crash reports electronically through an
online form. The second component of the grant is to provide ongoing
CMV crash training to law enforcement officers throughout the state.
This training will focus on correctly identifying CMVs and teaching
officers why CMV crash data is important, and what it is used for at
the state and national levels.
Crash Data Collection/Reporting Issues:
Before SaDIP funds were received, the Minnesota state patrol had
limited access to Driver and Vehicle Services records. As a result, the
state patrol was only able to report crashes to FMCSA that were
investigated by their officers or were sent directly to the state
patrol by local jurisdictions. Now all CMV records that are available
at the Driver and Vehicle Services are reported to FMCSA (see crash
data quality statistics, table 11).
Table 11: Minnesota Crash Data Quality Statistics (Percentages):
Completeness;
March 2004: 117;
June 2004: 117;
September 2004: 116;
December 2004: 123;
March 2005: 121;
June 2005: 121.
Timeliness;
March 2004: 72;
June 2004: 59;
September 2004: 58;
December 2004: 50;
March 2005: 67;
June 2005: 81.
Accuracy;
March 2004: 78;
June 2004: 74;
September 2004: 82;
December 2004: 86;
March 2005: 91;
June 2005: 93.
Source: FMCSA.
[End of table]
The State of Minnesota encountered difficulty receiving crash reports
from local law enforcement agencies. In addition, law enforcement
officers incorrectly identified CMVs on crash reports. If law
enforcement officers do not correctly identify the vehicle involved in
the crash as a CMV, then the crash may not be extracted from the state
crash file for submission to FMCSA.
State/Federal Coordination Issues:
Minnesota officials said that the state now has a crash data users
group as a subsection of its Traffic Records Coordinating Committee.
This group allows all users of crash data to discuss their data needs
and limitations, and provides a forum for making collaborative
recommendations regarding crash data improvements.
Minnesota officials and the FMCSA Division Administrator work very
closely with each other to monitor and improve CMV crash data quality.
Neither of these groups feels that the FMCSA data quality rating map is
an accurate portrayal of a state's data quality. For example,
Minnesota's rating decreased due to the fact that it was implementing
its electronic data transfer software, and during that period crash
records were not reported to FMCSA in a timely fashion. The map showed
the state's rating as "poor" during this period, which did not reflect
the fact that the state was making strides in improving its data
quality.
Minnesota uses several sources of funding to improve its overall crash
data and CMV crash data. In addition to FMCSA funding, NHTSA and state
funds have been used to improve the state's data quality.
Data Quality Map:
Minnesota crash data has gone from an overall "fair" rating to "poor"
to its current rating as "good." The drop in the state's rating was
actually a result of the state not being able to report crashes during
the implementation of SaDIP-funded improvement projects. After the
electronic systems were fully implemented, the state's rating improved
to "good" for completeness, and "fair" for timeliness and accuracy.
Officials in Minnesota recognize that the map is an effective tool for
getting the attention of decision-makers. While the map may not be an
accurate assessment of the state's current data quality, both state and
federal officials recognize that it does help to spotlight Minnesota's
data quality status. Federal and state officials use the data quality
map as a measure of the state's progress in improving data quality.
State officials also use the map as an indicator of the success of the
state's SaDIP project. Finally, officials in Minnesota said that the
data quality map has been used to identify data quality problems in the
state, including timeliness and accuracy, and to create projects to
improve those measures.
North Carolina:
North Carolina received a total of $193,350 in 2004 (see table 12) to
conduct crash data improvement projects. While North Carolina has
excellent crash data reporting at the state level, North Carolina
continues to have complications reporting that data to FMCSA. North
Carolina is a priority state for FMCSA given the fact that it is one of
the top five states for commercial motor vehicle accidents in the
nation.
Table 12: North Carolina SaDIP Funding History:
Award year: 2004;
Funds awarded: $193,350;
Agency: FMCSA;
Award type: Grant.
Source: FMCSA.
[End of table]
SaDIP Projects:
North Carolina is using its SaDIP grant to conduct two projects. First,
the state conducted an analysis to determine what differences existed
between the state data file and the federal crash file, and to try to
determine why these differences were occurring. The completed analysis
did not appear to fully address North Carolina's biggest problem, which
is the data transfer between the two files.
North Carolina is also using SaDIP to update a backlog it has in
entering crash reports into the state data file by providing overtime
pay to employees.
Crash Data Collection/Reporting Issues:
Historically, North Carolina has had poor CMV crash data (see table
13). North Carolina is also one of FMCSA's priority states because of
the large amount of crashes that take place in the state. It was rated
as having "poor" data for each period since FMCSA began rating states.
Table 13: North Carolina Crash Data Quality Statistics (Percentages):
Completeness;
March 2004: 82;
June 2004: 82;
September 2004: 77;
December 2004: 77;
March 2005: 77;
June 2005: 77.
Timeliness;
March 2004: 0;
June 2004: 100;
September 2004: 0;
December 2004: 0;
March 2005: 1;
June 2005: 2.
Accuracy;
March 2004: 81;
June 2004: 100;
September 2004: 81;
December 2004: 80;
March 2005: 80;
June 2005: 81.
Source: FMCSA.
[End of table]
North Carolina is unique in that it does not have a problem receiving
crash reports from localities across the state. In other states, this
is the biggest problem contributing to the states' data quality
issues.[Footnote 49] In North Carolina the problem is getting records
from the state crash file into the FMCSA data systems. FMCSA officials
worked extensively with North Carolina to fix the file compatibility
issues that are creating the problem exporting the data to FMCSA, but
North Carolina has failed to correct its information technology
problems. According to state officials, the biggest issue in North
Carolina is backlog. Even if the state corrects its backlog, the
state's data rating will not improve because the records are not
transferring correctly into the federal database. Additionally, North
Carolina does not require certain elements to be captured on its crash
report or in its state database that are required in the federal
database. Specifically, North Carolina does not require that motor
carrier DOT numbers are collected, which may contribute to the state's
"poor" accuracy rating.
While North Carolina has very good data at the state level, the state's
largest problem is reporting that data to the federal database. It
appears that North Carolina's SaDIP projects are not focusing on this
issue.
State/Federal Coordination Issues:
North Carolina coordination between state agencies involved in CMV
crash data has not always been effective. A major reorganization of the
state's highway safety agency contributed to this lack of coordination,
but the state is working to improve cooperation among these agencies
with regards to traffic records. The largest factor contributing to
North Carolina's poor crash data quality appears to be a lack of
understanding by some state officials regarding how to convert a crash
data file to the correct format in order to submit it to FMCSA. The
SaDIP grant has provided the state the opportunity to review its crash
data problems, a project that the state would not have conducted
otherwise. The state is also using its own funds to implement an
electronic crash reporting system in order to get crash data into the
state data file more quickly.
Data Quality Map:
State and federal officials in North Carolina use the data quality map
to measure the progress the state is making with its SaDIP grant and
with data improvements overall.
North Carolina state officials use the map as an incentive for
implementing data improvements and they appear motivated to change
North Carolina's rating for the better. The map has also been used in
North Carolina to identify data quality problems and target improvement
efforts, although more still needs to be done.
Both federal and state officials expect to continue to make
improvements to the state's traffic records infrastructure after the
state achieves the highest data rating.
Ohio:
Ohio received a total of $721,730 between 2002 and 2005 (see table 14)
to conduct crash data improvement projects. Ohio has been very
proactive in addressing traffic safety data concerns and the state has
paid particular attention to using crash data for planning.
Table 14: Ohio SaDIP Funding History:
Award year: 2005;
Funds awarded: $224,400;
Agency: FMCSA;
Award type: Cooperative agreement.
Award year: 2004;
Funds awarded: $91,000;
Agency: FMCSA;
Award type: Cooperative agreement.
Award year: 2002;
Funds awarded: $406,330;
Agency: NHTSA/FMCSA;
Award type: Cooperative agreement.
Source: FMCSA.
[End of table]
SaDIP Projects:
Ohio was one of the first 5 pilot states to participate in the SaDIP
program. It received funds through a cooperative agreement with the
General Services Administration (GSA) and NHTSA. Through this
agreement, the state created electronic crash reporting capabilities,
purchased handheld devices, modified crash reporting software, and is
providing training to law enforcement officers to help them properly
identify CMV crashes.
Until recently, when FMCSA gained authority for the SaDIP cooperative
agreements, Ohio had not submitted any progress reports to NHTSA on the
status of these projects.
Crash Data Collection/Reporting Issues:
Ohio has a "fair" overall data rating, with "good" ratings for
completeness and accuracy, but a "poor" rating for timeliness (see
crash data quality statistics, table 15). Ohio has been very aware of
the importance of CMV crash data and state officials have been working
to improve it for a long time, even before the SaDIP program.
Table 15: Ohio Crash Data Quality Statistics (Percentages):
Completeness;
March 2004: 95;
June 2004: 95;
September 2004: 95;
December 2004: 95;
March 2005: 95;
June 2005: 95.
Timeliness;
March 2004: 15;
June 2004: 17;
September 2004: 11;
December 2004: 23;
March 2005: 40;
June 2005: 46.
Accuracy;
March 2004: 84;
June 2004: 97;
September 2004: 97;
December 2004: 98;
March 2005: 98;
June 2005: 98.
Source: FMCSA.
[End of table]
The state of Ohio has over 1,000 local law enforcement jurisdictions
that are responsible for reporting CMV accidents. Both large and small
jurisdictions are likely responsible for Ohio's "poor" crash data
timeliness rating. The state has reduced the time lag in receiving
crash reports from local jurisdictions from 62 days in 2004 to 30 days
in 2005.
State/Federal Coordination Issues:
Ohio state agencies involved in the collection and reporting of CMV
crash data appear coordinated with each other, and have an active
Traffic Records Coordinating Committee, which includes participation by
FMCSA division officials. Ohio uses multiple sources of funding to
address data quality issues, including state funds, and the state is
proactive in data improvement projects.
Data Quality Map:
State and federal officials said that the map provided one gauge of
Ohio's data quality, but they felt the map would be more useful if it
were updated more often. The map has brought more attention to data
quality issues in Ohio and is included in reports to state leaders.
State and federal officials in Ohio said that data quality improvements
would still be a top priority regardless of the data quality map. Ohio
officials also stated that the inaccuracies in the map had a big effect
on staff morale. Local officials said they are working hard to improve
their state's data quality, but the map does not accurately capture
that improvement. State officials recognize the map provides a major
incentive for implementing data quality improvements and maintaining
the state's standing as a leader in traffic data quality.
Oklahoma:
Oklahoma received a total of $1,281,501 in 2004 (see table 16) to
conduct crash data improvement projects. Oklahoma's traffic records
coordinating committee has taken the lead in the coordination of crash
data projects in the state.
Table 16: Oklahoma SaDIP Funding History:
Award year: 2004;
Funds awarded: $1,150,390;
Agency: FMCSA;
Award type: Cooperative agreement.
Award year: 2004;
Funds awarded: $131,111;
Agency: FMCSA;
Award type: Grant.
Source: FMCSA.
[End of table]
SaDIP Projects:
Oklahoma began participation in the SaDIP program in July 2004 when it
received a grant to purchase computer equipment, and then entered into
a four year cooperative agreement with FMCSA in December 2004 to
support its long term data quality improvement plans. This includes
conducting a traffic records assessment, hiring a SaDIP coordinator,
funding data entry for the record backlog, revising the state's crash
report, and initiating a mobile data collection pilot project.
Oklahoma officials said that the projects that are taking place using
SaDIP funds would not have been funded otherwise. The SaDIP grant and
long-term cooperative agreement have allowed the state to focus on CMV
data and to begin to make data-driven decisions in its highway safety
planning.
Crash Collection/Reporting Issues:
Until recently Oklahoma had a nine month crash report backlog to be
entered into the state system (see crash data quality statistics, table
17). The backlog was primarily due to insufficient state resources.
Overtime hours funded by SaDIP has helped to alleviate this backlog.
Table 17: Oklahoma Crash Data Quality Statistics (Percentages):
Completeness;
March 2004: 104;
June 2004: 104;
September 2004: 102;
December 2004: 102;
March 2005: 101;
June 2005: 101.
Timeliness;
March 2004: 93;
June 2004: 92;
September 2004: 93;
December 2004: 92;
March 2005: 90;
June 2005: 83.
Accuracy;
March 2004: 84;
June 2004: 86;
September 2004: 87;
December 2004: 86;
March 2005: 86;
June 2005: 84.
Source: FMCSA.
[End of table]
State/Federal Coordination Issues:
Oklahoma has been working with the GSA to receive payments on its
cooperative agreement, and officials said that this has caused the
state some confusion.
Oklahoma's crash data divisions are housed in the state's Department of
Public Safety and Department of Transportation. While crash records
staff in both agencies work closely together to make sure that the
state's crash file is complete, Oklahoma officials were unaware of the
value of CMV crash data at the national level, or of the criteria that
FMCSA uses to rate CMV crash data.
Oklahoma is using funds from several sources to improve its entire
crash data system, including funds provided by NHTSA and the Federal
Highway Administration. SaDIP funds are the only funds that the state
is using specifically for CMV data improvements.
Data Quality Map:
While high-ranking officials in Oklahoma's Department of Public Safety
and the Highway Patrol are familiar with the data quality map, state
officials involved with the SaDIP program were less familiar. We found
that state officials involved with the SaDIP program had a detailed
understanding of their state's data quality, but had not used the data
quality map as an indicator of progress for the SaDIP cooperative
agreement.
Oklahoma officials stated that they would undertake their data
improvement program regardless of whether or not the state was ranked
"good." State officials, for example, recognized that more improvements
could be made in the state's timeliness measure even though the state
had a "good" rating in this category.
Washington:
Washington received a total of $803,935 in 2004 (see table 18) to
conduct crash data improvement projects. The state's traffic records
coordinating committee is leading an electronic information processing
initiative designed to reduce crash reporting inefficiencies and help
the state meet national traffic safety goals.
Table 18: Washington SaDIP Funding History:
Award year: 2004;
Funds awarded: $188,460;
Agency: FMCSA;
Award type: Grant.
Award year: 2004;
Funds awarded: $615,475;
Agency: FMCSA;
Award type: Grant.
Source: FMCSA.
[End of table]
SaDIP Projects:
SaDIP is primarily being used to implement an electronic data feed
between the Washington Department of Transportation and the state
patrol. This will allow records to be submitted instantly to the state
patrol when they are entered into the state department of
transportation database. It will also make records searchable so that
eligible CMV crashes that are misidentified and not sent to the state
patrol can be identified as CMV crashes, improving the completeness of
the state's CMV crash data.
SaDIP is also being used to eliminate a six-month backlog of crash
reports that need to be entered into the state's database housed at the
Washington Department of Transportation. Until reports are entered into
the state Department of Transportation system, they cannot be
transferred to the state patrol electronically, nor can those reports
incorrectly identified as non-CMV crashes be easily identified. Until
this takes place, Department of Transportation employees identify CMV
reports among all crash reports and provide the state patrol with paper
copies of the reports.
Crash Collection/Reporting Issues:
The largest problem that Washington State has with its crash data is
receiving the entire crash report, including supplemental form, from
the law enforcement offices that generate them (see crash data quality
statistics, table 19).
Table 19: Washington Crash Data Quality Statistics (Percentages):
Completeness;
March 2004: 96;
June 2004: 96;
September 2004: 103;
December 2004: 103;
March 2005: 108;
June 2005: 108.
Timeliness;
March 2004: 99;
June 2004: 98;
September 2004: 98;
December 2004: 99;
March 2005: 99;
June 2005: 99.
Accuracy;
March 2004: 95;
June 2004: 96;
September 2004: 96;
December 2004: 97;
March 2005: 97;
June 2005: 97.
Source: FMCSA.
[End of table]
Washington State also has a problem with incorrect identification of
CMVs on police accident reports. If police do not correctly identify
the vehicle involved in the crash as a CMV, then it may not get
extracted from the state crash file for submission to FMCSA.
State/Federal Coordination Issues:
Washington State has very good cooperation among state agencies
involved in crash data collection and reporting. The state also has a
good working relationship with its FMCSA division office. It also uses
FMCSA and state funds to address data quality issues, making decisions
on how to use these funds effectively through an active Traffic Records
Coordinating Committee.
While SaDIP initiatives in Washington State are a topic of discussion
at the Traffic Records Coordinating Committee meetings, the committee
coordinator was not specifically aware of FMCSA's grant-making process
and how those grants are accessed and then allocated by the state.
Whereas NHTSA funds are accessed via the Traffic Records Coordinating
Committee forum, FMCSA funds are processed within the Washington State
Patrol's Commercial Vehicle Division. To date there has been no problem
targeting safety data funding to immediate priorities as identified by
the Traffic Records Coordinating Committee, but this is an area that
may need to be addressed in the future.
Data Quality Map:
Washington State officials recognize that the data quality map is an
important indicator of the state's progress in improving its crash
records system. According to state officials, the data quality map has
been used to measure the progress of the state's data quality
improvements in general.
State officials also indicated that they would continue to make
improvements to their traffic records systems regardless of the state's
data quality rating. Specifically, the Traffic Records Coordinating
Committee is helping to coordinate the state's electronic reporting
system.
Even though the state has consistently ranked "good" on the map since
its inception, state officials report that the map has been helpful for
identifying areas where the state can make data quality improvements.
[End of section]
Appendix VI: GAO Contact and Staff Acknowledgments:
GAO Contact:
Katherine Siggerud (202) 512-6570 or [Hyperlink, siggerudk@gao.gov]
Staff Acknowledgments:
In addition to the contact named above, Randall Williamson, Assistant
Director; Tida Barakat; Jonathan Carver; Bert Japikse; Sara Ann
Moessbauer; Fred Swenson; Sharon Silas; and Stan Stenersen made key
contributions to this report.
(542054):
FOOTNOTES
[1] In 2003, there were also 289 fatal crashes involving buses.
[2] FMCSA was formerly a part of the Federal Highway Administration.
Its creation as a separate entity was pursuant to the Motor Carrier
Safety Improvement Act of 1999. Pub. L. No. 106-159, § 101.
[3] In 1999, Pub. L. No. 106-159, § 225, directed the Secretary of
Transportation to carry out a program, which became known as the
Commercial Vehicle Analysis Reporting System (CVARS). It is currently
known as the Safety Data Improvement Program (SaDIP).
[4] Safe, Accountable, Flexible, and Efficient Transportation Equity
Act of 2005: A Legacy for Users (SAFETEA-LU), Pub. L. No. 109-59.
[5] FMCSA began SaDIP in FY2000, but did not begin awarding funds to
states until FY2002. In August 2005, Congress, through the
reauthorization of surface transportation programs, authorized FMCSA's
SaDIP an additional $11 million over the next 4 years. This will be
used in addition to FMCSA funds to provide assistance to states.
[6] DOT Inspector General, Improvements Needed in the Motor Carrier
Safety Status Measurement System, MH-2004-034, (Washington, D.C.:
February 13, 2004).
[7] Senate Report 108-342.
[8] The 34 states are Alaska, California, Colorado, Connecticut,
Georgia, Indiana, Iowa, Kansas, Kentucky, Louisiana, Maryland,
Massachusetts, Michigan, Minnesota, Montana, Nebraska, Nevada, New
Hampshire, New Mexico, New York, North Carolina, Ohio, Oklahoma,
Pennsylvania, Rhode Island, South Carolina, South Dakota, Tennessee,
Texas, Utah, Vermont, Virginia, Washington, and West Virginia.
Participation was voluntary.
[9] The six case studies were conducted in Georgia, Minnesota, North
Carolina, Ohio, Oklahoma, and Washington.
[10] Telephone interviews were conducted with FMCSA officials from New
Hampshire, New Jersey, Michigan, and Missouri. We found during our
interviews that New Hampshire is still participating in the program.
[11] Compliance reviews are detailed audits of carriers, the results of
which sometimes lead to enforcement action.
[12] Safety audits review safety management systems and initial
operating performance of new motor carriers.
[13] Roadside inspections are conducted primarily by states. The
information from these inspections is collected and reported to FMCSA.
[14] The SafeStat data are also used to target carriers for roadside
inspections.
[15] Along with definitions developed by FMCSA, the National Governors'
Association also provides guidance for submitting complete and accurate
data on commercial vehicle crashes. In addition, the National Highway
Traffic Safety Administration has provided guidance for reporting crash
data based on the Model Minimum Uniform Crash Criteria and the American
National Standard Manual for Classification of Motor Vehicle Traffic
Records.
[16] There are 20 data elements that were originally recommended by the
National Governors' Association in 1989 and later adopted by FMCSA in
1992.
[17] Pub. L. No. 106-159, § 225.
[18] The program was designed to be a system similar to the National
Highway Transportation Safety Administration's Fatality Analysis
Reporting System (FARS).
[19] The SafeStat data are also used by to target carriers for roadside
inspections. Department of Transportation Inspector General,
Improvements Needed in the Motor Carrier Safety Status Measurement
System, MH-2004-034 (Washington, D.C.: February 13, 2004).
[20] FMCSA contracts with the UMTRI to conduct individual state data
quality assessments. The assessments focused on the completeness of the
data but also reviewed some accuracy and timeliness issues. To date,
they have completed assessments of the following eight states:
California (2005), Florida (2004), Illinois (2005), Michigan (2004),
Missouri (2004), New Jersey (2005), North Carolina (2005), and Ohio
(2003). UMTRI assessed 2003 data for all states except for Ohio, which
used 2000 data and Missouri which used 2001 data.
[21] Appendix III provides detailed information on what is a
"reportable crash" and what is a "reportable commercial motor vehicle."
[22] While FMCSA regulates interstate motor carriers (i.e., carriers
operating across states), information on crashes involving both
interstate or intrastate (i.e., carriers operating only within a state)
should be submitted to FMCSA. CMVs that are being used only for private
use and are involved in a crash do not need to be submitted to FMCSA.
[23] Additional research and analysis is also being conducted by other
public and private research entities.
[24] FMCSA provides training classes for collecting and coding state
crash data to representatives of the enforcement community in a growing
number of states. In addition FMCSA has developed and maintained
customized large truck crash data collection training materials for
each state, including train-the-trainer visuals and student workbook
materials, to enable the states to carry the training forward to
officers throughout their state and to incorporate the training into
their academies.
[25] These are information cards that can be attached to a law
enforcement officer's sun visor.
[26] The A&I Online website, developed and maintained by FMCSA Analysis
Division provides useful motor carrier safety information and analysis
over the Internet. The Web site increases FMCSA's effectiveness in
carrying out its programs and provides the motor carrier industry and
the public with information to make safety-minded decisions. The
website is a valuable information resource to FMCSA in promoting motor
carrier safety and is widely used by FMCSA and state field staff in
their preparation for and in conducting on-site motor carrier safety
audits.
[27] The data quality map includes an overriding indicator of crash
consistency of a state's non-fatal crash data submitted to FMCSA,
however, it is not measuring the consistency of data; rather it is
another indicator of completeness.
[28] These data are current as of September 2005. In addition to funds
provided directly to states, FMCSA provided $1,200,000 to contractors
to assist states in improving data quality, bringing the total amount
spent on state data improvement efforts to $21,891,394.
[29] FMCSA's Office of Information Management plans on requesting funds
in line with SAFETEA-LU to include $2 million in grant funds for
FY2006, and $3 million for FY2007-2009, as provided for in SAFETEA-LU
for a total of $11 million.
[30] The creation of this Web site (http://www.grants.gov) is part of
the federal government's grants streamlining initiative. The initiative
is the government-wide set of organizations and activities responsible
for implementing the Federal Financial Assistance Management
Improvement Act of 1999. Pub. L. No. 106-107, §113.
[31] Crash data includes information about CMV crashes as reported by
the states. Inspection data is submitted by the states to FMCSA and
includes information collected through roadside inspections. Crash data
is assessed for completeness, timeliness, and accuracy. Inspection data
is only assessed for timeliness and accuracy.
[32] The map's methodology includes a "Crash Consistency Overriding
Indicator." A state is flagged if it submits less than 50 percent of
the average number of reported non-fatal crash records for the current
year based on the previous three-year average of non-fatal crash
records.
[33] This is based on our six case study visits to states that are
currently participating in SaDIP and telephone interviews with state
officials that had previously participated or had not participated in
SaDIP.
[34] Of the 436,000 police-reported crashes involving large trucks in
2003, about 150,000 were required to be reported to FMCSA and of these,
4,289 resulted in at least one fatality.
[35] NHTSA's National Center for Statistics and Analysis created and
developed the Fatality Analysis Reporting System (FARS). Fatality
information derived from FARS includes motor vehicle traffic crashes
that result in the death of an occupant of a vehicle or a nonmotorist
within 30 days of the crash. FARS contains data on all fatal traffic
crashes within the 50 states, the District of Columbia, and Puerto
Rico. Each state employs a federal analyst who conducts a number of
quality control procedures to ensure correct information about the
fatality crash. A final FARS file is completed once a year.
[36] They are also assessed on whether a state's reporting meets a
minimum threshold based on past reporting (i.e., the "Crash Consistency
Overriding Indicator").
[37] MCSAP high priority funds are funds provided to states and local
governments to carry out activities and projects that directly support
the MCSAP, including supporting, enriching, or evaluating state
commercial motor vehicle safety programs. These funds are allocated at
the discretion of FMCSA. States apply for funding and are awarded funds
in an 80/20 split with the state (80 percent of funding is provided via
federal sources and states are required to provide the other 20 percent
of the funds) except if the funds are used for education and outreach
activities. Currently if a state is rated as "poor" in its crash data
quality and it applies for High Priority funding, the state is required
to use its High Priority funding for crash data improvements.
[38] New Hampshire was intended to represent a state that no longer
participated in SaDIP; however, we learned during our interview that
the original grant expiration date had recently been extended to 2006.
[39] These officials, in general, included representatives from state
traffic records coordinating committees, the governor's highway safety
offices, departments of public safety, departments of transportation,
and departments of motor vehicles.
[40] FMCSA's data quality criteria include completeness, timeliness,
accuracy, and consistency.
[41] FMCSA is currently engaged in two efforts to assess baselines for
non-fatal crashes in individual states. One effort is being conducted
by Data Nexus and another is being conducted through the UMTRI
assessments mentioned earlier in the report.
[42] Of the 436,000 police-reported crashes involving large trucks in
2003, about 150,000 were required to be reported to FMCSA and of these,
4,289 resulted in at least one fatality.
[43] FARS is a database of all fatal vehicle crashes maintained by the
National Highway Traffic Safety Administration.
[44] Also, according to Volpe, one state official estimated that
definitional differences account for about 10 percent of fatal CMV
crashes in their state.
[45] Recently, FMCSA distributed visor cards to state officials that
provide illustrations of FMCSA reportable CMVs. See appendix III for
copies of these cards.
[46] This includes the District of Columbia.
[47] Editing might be needed, for example, if activity at the state
level discloses errors or incomplete information in the record as
originally submitted to MCMIS.
[48] In general, if a crash report is missing a DOT number, FMCSA
attempts to use other identifiers in the report to match it to a
registered carrier, but officials told us that this has been met with
limited success.
[49] Additionally, an FMCSA official told us that this problem is
unique to North Carolina.
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
