Motor Carrier Safety
A Statistical Approach Will Better Identify Commercial Carriers That Pose High Crash Risks Than Does the Current Federal Approach Gao ID: GAO-07-585 June 11, 2007The Federal Motor Carrier Safety Administration (FMCSA) has the primary federal responsibility for reducing crashes involving large trucks and buses that operate in interstate commerce. FMCSA decides which motor carriers to review for compliance with its safety regulations primarily by using an automated, data-driven analysis model called SafeStat. SafeStat uses data on crashes and other data to assign carriers priorities for compliance reviews. GAO assessed (1) the extent to which changes to the SafeStat model could improve its ability to identify carriers that pose high crash risks and (2) how the quality of the data used affects SafeStat's performance. To carry out its work, GAO analyzed how SafeStat identified high-risk carriers in 2004 and compared these results with crash data through 2005.
While SafeStat does a better job of identifying motor carriers that pose high crash risks than does a random selection, regression models GAO applied do an even better job. SafeStat works about twice as well as (about 83 percent better than) selecting carriers randomly. SafeStat is built on a number of expert judgments rather than using statistical approaches, such as a regression model. For example, its designers decided to weight more recent motor carrier crashes twice as much as less recent ones on the premise that more recent crashes were stronger indicators of future crashes. GAO estimates that if FMCSA used a negative binomial regression model, FMCSA could increase its ability to identify high-risk carriers by about 9 percent over SafeStat. Carriers identified by the negative binomial regression model as posing a high crash risk experienced 9,500 more crashes than those identified by the SafeStat model over an 18 month follow-up period. The primary use of SafeStat is to identify and prioritize carriers for FMCSA and state compliance reviews. FMCSA measures the ability of SafeStat to perform this role by comparing the crash rate of carriers identified as posing a high crash risk with the crash rate of other carriers. Using a negative binomial regression model would further FMCSA's mission of reducing crashes through the more effective targeting of compliance reviews to the set of carriers that pose the greatest crash risk. Late-reported, incomplete, and inaccurate data reported to FMCSA by states have been a long-standing problem. However, GAO found that late reported data had a small effect on SafeStat's ability to identify carriers that pose high crash risks in 2004. If states had reported all crash data within 90 days after occurrence, as required by FMCSA, a net increase of 299 carriers (or 6 percent) would have been identified as posing high crash risks of the 4,989 that FMCSA identified. Reporting timeliness has improved, from 32 percent of crashes reported on time in fiscal year 2000, to 89 percent in fiscal year 2006. Regarding completeness, GAO found that data for about 21 percent of the crashes (about 39,000 of 184,000) exhibited problems that hampered linking crashes to motor carriers. Having complete information on crashes is important because SafeStat treats crashes as the most important factor for assessing motor carrier crash risk, and crash information is also the crucial factor in the statistical approaches that we employed. Regarding accuracy, a series of studies by the University of Michigan Transportation Research Institute covering 14 states found incorrect reporting of crash data is widespread. GAO was not able to quantify the effect of the incomplete or inaccurate data on SafeStat's ability to identify carriers that pose high crash risks because it would have required gathering crash records at the state level--an effort that was impractical for GAO. FMCSA has acted to improve crash data quality by completing a comprehensive plan for data quality improvement, implementing an approach to correct inaccurate data, and providing grants to states for improving data quality, among other things.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-07-585, Motor Carrier Safety: A Statistical Approach Will Better Identify Commercial Carriers That Pose High Crash Risks Than Does the Current Federal Approach
This is the accessible text file for GAO report number GAO-07-585
entitled 'Motor Carrier Safety: A Statistical Approach Will Better
Identify Commercial Carriers That pose high Crash Risks Than Does the
Current Federal Approach' which was released on June 11, 2007.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
United States Government Accountability Office:
GAO:
June 2007:
Motor Carrier Safety:
A Statistical Approach Will Better Identify Commercial Carriers That
Pose High Crash Risks Than Does the Current Federal Approach:
GAO-07-585:
GAO Highlights:
Highlights of GAO-07-585, a report to congressional requesters
Why GAO Did This Study:
The Federal Motor Carrier Safety Administration (FMCSA) has the primary
federal responsibility for reducing crashes involving large trucks and
buses that operate in interstate commerce. FMCSA decides which motor
carriers to review for compliance with its safety regulations primarily
by using an automated, data-driven analysis model called SafeStat.
SafeStat uses data on crashes and other data to assign carriers
priorities for compliance reviews.
GAO assessed (1) the extent to which changes to the SafeStat model
could improve its ability to identify carriers that pose high crash
risks and (2) how the quality of the data used affects SafeStat…s
performance. To carry out its work, GAO analyzed how SafeStat
identified high-risk carriers in 2004 and compared these results with
crash data through 2005.
What GAO Found:
While SafeStat does a better job of identifying motor carriers that
pose high crash risks than does a random selection, regression models
GAO applied do an even better job. SafeStat works about twice as well
as (about 83 percent better than) selecting carriers randomly. SafeStat
is built on a number of expert judgments rather than using statistical
approaches, such as a regression model. For example, its designers
decided to weight more recent motor carrier crashes twice as much as
less recent ones on the premise that more recent crashes were stronger
indicators of future crashes. GAO estimates that if FMCSA used a
negative binomial regression model, FMCSA could increase its ability to
identify high-risk carriers by about 9 percent over SafeStat. Carriers
identified by the negative binomial regression model as posing a high
crash risk experienced 9,500 more crashes than those identified by the
SafeStat model over an 18 month follow-up period. The primary use of
SafeStat is to identify and prioritize carriers for FMCSA and state
compliance reviews. FMCSA measures the ability of SafeStat to perform
this role by comparing the crash rate of carriers identified as posing
a high crash risk with the crash rate of other carriers. Using a
negative binomial regression model would further FMCSA‘s mission of
reducing crashes through the more effective targeting of compliance
reviews to the set of carriers that pose the greatest crash risk.
Late-reported, incomplete, and inaccurate data reported to FMCSA by
states have been a long-standing problem. However, GAO found that late
reported data had a small effect on SafeStat‘s ability to identify
carriers that pose high crash risks in 2004. If states had reported all
crash data within 90 days after occurrence, as required by FMCSA, a net
increase of 299 carriers (or 6 percent) would have been identified as
posing high crash risks of the 4,989 that FMCSA identified. Reporting
timeliness has improved, from 32 percent of crashes reported on time in
fiscal year 2000, to 89 percent in fiscal year 2006. Regarding
completeness, GAO found that data for about 21 percent of the crashes
(about 39,000 of 184,000) exhibited problems that hampered linking
crashes to motor carriers. Having complete information on crashes is
important because SafeStat treats crashes as the most important factor
for assessing motor carrier crash risk, and crash information is also
the crucial factor in the statistical approaches that we employed.
Regarding accuracy, a series of studies by the University of Michigan
Transportation Research Institute covering 14 states found incorrect
reporting of crash data is widespread. GAO was not able to quantify the
effect of the incomplete or inaccurate data on SafeStat‘s ability to
identify carriers that pose high crash risks because it would have
required gathering crash records at the state level”an effort that was
impractical for GAO. FMCSA has acted to improve crash data quality by
completing a comprehensive plan for data quality improvement,
implementing an approach to correct inaccurate data, and providing
grants to states for improving data quality, among other things.
What GAO Recommends:
GAO is recommending that FMCSA use a negative binomial regression model
to identify carriers that pose high crash risks.
In commenting on a draft of this report, the Department of
Transportation agreed that the use of a negative binomial regression
model looked promising for selecting carriers for compliance reviews,
but expressed some reservation about the greater sensitivity of this
approach to problems with reported crash data.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-585].
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Sidney H. Schwartz at
(202) 512-7387 or schwartzsh@gao.gov, or Susan A. Fleming at (202) 512-
2834 or flemings@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
A Statistical Approach Would Better Identify Carriers That Pose High
Crash Risks Than Does FMCSA's Current Approach:
Despite Quality Problems, FMCSA's Crash Data Can Be Used to Compare
Methods for Identifying Carriers That Pose High Crash Risks:
Conclusion:
Recommendation for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Results of Other Assessments of the SafeStat Model's
Ability to Identify Motor Carriers That Pose High Crash Risks:
Assessments of SafeStat's Predictive Capability:
Impact of Data Quality on SafeStat's Predictive Capability:
Appendix II: Scope and Methodology:
Appendix III: Additional Results from Our Statistical Analyses of the
SafeStat Model:
Overview of Regression Analyses:
Technical Explanation of the Negative Binomial Regression Model:
Tables:
Table 1: SafeStat Categories and Their Priority for Compliance Reviews:
Table 2: Size Distribution of Carriers Receiving a SafeStat Rating of A
through G:
Table 3: Results for SafeStat Model and Regression Models:
Figures:
Figure 1: Commercial Motor Vehicle Fatality Rate, 1975 to 2005:
Figure 2: Percentage of Crashes Submitted to MCMIS within 90 Days of
Occurrence:
Abbreviations:
FMCSA: Federal Motor Carrier Safety Administration:
MCMIS: Motor Carrier Management Information System:
SafeStat: Motor Carrier Safety Status Measurement System:
United States Government Accountability Office:
Washington, DC 20548:
June 11, 2007:
The Honorable James L. Oberstar:
Chairman:
The Honorable John L. Mica:
Ranking Republican Member:
Committee on Transportation and Infrastructure:
House of Representatives:
The Honorable Peter A. DeFazio:
Chairman:
The Honorable John J. Duncan:
Ranking Republican Member:
Subcommittee on Highways and Transit:
Committee on Transportation and Infrastructure:
House of Representatives:
The Honorable Thomas E. Petri:
House of Representatives:
The Federal Motor Carrier Safety Administration (FMCSA) within the U.S.
Department of Transportation has the primary federal responsibility for
reducing crashes, deaths, and injuries involving large trucks and buses
operating in interstate commerce. While it carries out a number of
activities toward this end, an important tool at its disposal is the
compliance review--a detailed inspection of a motor carrier's
operations at its place of business. FMCSA decides which carriers to
inspect primarily by using an automated, data-driven analysis system
called the Motor Carrier Safety Status Measurement System (SafeStat).
SafeStat uses data on crashes, vehicle and driver violations, and other
information to develop numerical scores for carriers, and then SafeStat
assigns each carrier a priority to receive a compliance review.
Following an incident in which a bus company, with many driver
violations and a low priority for compliance review from the SafeStat
model, suffered a fire on one of its buses that resulted in 23 deaths,
you were interested in whether SafeStat could better identify
commercial motor carriers at risk for crashes. To address your
interest, we assessed (1) the extent to which changes to the SafeStat
model could improve its ability to identify these carriers and (2) how
the quality of the data used affects SafeStat's performance. These two
topics are the main focus of this report. We also examined the findings
of other studies on how SafeStat's ability to identify carriers at risk
for crashes can be improved. (See app. I.)
To determine whether statistical approaches could be used to improve
FMCSA's ability to identify carriers that pose high crash risks, we
tested a number of regression models and compared their performance
with SafeStat's results from June 2004. We chose 2004 because it
allowed us to examine actual crash data for the 18-month period
following June 2004 to determine the degree to which SafeStat
successfully identified carriers that proved to be of high risk for
crashes. It also allowed us to include crashes that occurred within the
18 months after June 2004 but had not yet been reported to FMCSA by
December 2005. Using regression models, we compared the predictive
performance of these statistical approaches to SafeStat's performance
to determine which method best identified carriers that pose high crash
risks. We also calculated crash rates from a series of random samples
of all carriers to determine if the SafeStat model did a better job
than random selection in identifying motor carriers that pose high
crash risks. To assess whether changes could be made to the SafeStat
model to improve its identification of carriers that pose high crash
risks, we tested changes to selected portions of the SafeStat model and
investigated the effect of changing decision rules used to construct
the four safety evaluation areas.[Footnote 1]
To assess the extent to which data quality affects SafeStat's ability
to identify carriers that pose high crash risks, we carried out a
series of analyses and surveyed the literature to identify findings
from other studies. To address timeliness, we measured the number of
days it took states to report crashes. We also added late-reported
crashes to FMCSA's June 2004 data and recalculated SafeStat scores to
determine the effect of late-reported crashes on carriers' rankings.
For completeness, we attempted to match crash records in FMCSA's Motor
Carrier Management Information System (MCMIS) crash master file to
motor carriers listed in the MCMIS census file and reviewed studies on
state reporting. To address accuracy, we reviewed a report that tested
the accuracy of electronic data on a sample of paper records and
studies that identified the impact of incorrectly reported crashes in
individual states on MCMIS data quality. While there are known problems
with the quality of the crash data reported to FMCSA for use in
SafeStat, we determined that the data were of sufficient quality for
our use, which was to compare the ability of regression models to
identify carriers that pose high crash risks to the current approach,
which is largely derived through professional judgment. We conducted
our work in accordance with generally accepted government auditing
standards from May 2006 through May 2007. Appendix II provides further
information on our scope and methodology.
Shortly, we expect to issue a related report that examines how FMCSA
identifies and takes action against carriers that are egregious safety
violators. In addition, that report examines how thoroughly and
consistently FMCSA conducts compliance reviews.
Results in Brief:
While SafeStat does a better job of identifying motor carriers that
pose high crash risks than does a random selection, regression models
we applied do an even better job. SafeStat works about twice as well as
(about 83 percent better than) selecting carriers randomly and,
therefore, has value for improving safety. SafeStat is built on a
number of expert judgments. For example, SafeStat's designers used
their judgment and experience to weight more recent crashes involving a
motor carrier twice as much as less recent crashes on the premise that
more recent crashes were stronger indicators that a carrier may have
crashes in the future. Using similar reasoning, fatal crashes were
weighted more heavily than less serious crashes. We found that if a
negative binomial regression model was used instead, FMCSA could
increase its ability to identify carriers that pose high crash risks by
about 9 percent over SafeStat.[Footnote 2] Moreover, according to our
analysis, this 9 percent improvement would enable FMCSA to identify
carriers with twice as many crashes in the following 18 months as those
carriers identified under its current approach.[Footnote 3] Carriers
identified by the negative binomial regression model as posing a high
crash risk experienced 9,500 more crashes than those identified by the
SafeStat model over an 18 month follow-up period. The primary use of
SafeStat is to identify and prioritize carriers for FMCSA and state
safety compliance reviews. FMCSA measures the ability of SafeStat to
perform this role by comparing the crash rate of carriers identified as
posing a high crash risk with the crash rate of other carriers. In our
view, using a negative binomial regression model would further FMCSA's
mission of reducing crashes through the more effective targeting of
safety improvement and enforcement programs to the set of carriers that
pose the greatest crash risk. Applying a regression model would be easy
to adapt to the existing SafeStat model and, in our opinion, would be
beneficial even if FMCSA makes major revisions to its compliance and
enforcement approach in the coming years under its Comprehensive Safety
Analysis 2010 initiative.[Footnote 4]
Crash data reported by the states from December 2001 through June 2004
have problems in terms of timeliness, accuracy, and completeness that
potentially hinder FMCSA's ability to identify high risk carriers.
Regarding timeliness, we found that including late-reported data had a
small impact on SafeStat--including late-reported data added a net of
299 (or 6 percent) more carriers to the original 4,989 carriers that
the SafeStat model ranked as highest risk in June 2004.[Footnote 5] The
timeliness of crash reporting has shown steady and marked improvement:
the percentage of crashes reported by states within 90 days of
occurrence jumped from 32 percent in fiscal year 2000 to 89 percent in
fiscal year 2006. Regarding completeness, data for about 21 percent of
the crashes (about 39,000 of 184,000) exhibited problems that hampered
linking crashes to motor carriers. Thirteen percent of the crashes
(about 24,000) involving interstate carriers reported to FMCSA from
December 2001 through June 2004 are missing the unique identifier that
FMCSA assigns to each carrier when the agency authorizes the carrier to
engage in interstate commerce. Crashes without a unique identifier to
link to a company are excluded from use in SafeStat. An additional 8
percent of the crashes (about 15,000) that were reported had an
identification number that could not be matched to a motor carrier in
the FMCSA database that contains census information on motor carriers.
Linking crashes to carriers is important because the current SafeStat
model treats crashes as the most important factor in assessing motor
carrier crash risk. Crash information is also the crucial factor in the
regression models that we employed. Regarding accuracy, a series of
University of Michigan Transportation Research Institute's reports on
crash reporting shows that, among the 14 states studied, incorrect
reporting of crash data is widespread. For example, in recent reports,
the researchers found that, in 2005, Ohio incorrectly reported 1,094
(22 percent) of the 5,037 cases it reported, and Louisiana incorrectly
reported 137 (5 percent) of the 2,699 cases it reported. In Ohio, most
of the 1,094 crashes did not qualify because they did not meet the
crash severity threshold.[Footnote 6] We were not able to quantify the
actual effect of the incomplete or inaccurate data on SafeStat's
ability to identify carriers that pose high crash risks, because it
would have required us to gather crash records at the state level--an
effort that was impractical. FMCSA has acted to improve the quality of
SafeStat's data by completing a comprehensive plan for data quality
improvement, implementing an approach to correct inaccurate data, and
providing grants to states for improving data quality, among other
things. We could not quantify the effects of FMCSA's efforts to improve
the completeness or accuracy of the data for the same reason as
mentioned above.
This report contains a recommendation to the Secretary of
Transportation aimed at applying a negative binomial regression model
to the four SafeStat safety evaluation areas that would result in
better identification of commercial motor carriers that pose high crash
risks. Because FMCSA has initiated efforts to improve the quality of
SafeStat's data, we are not making a recommendation in this area.
In commenting on a draft of this report, the department agreed that it
would be reasonable to consider the use of the negative binomial
regression model in order to better target compliance reviews to
carriers posing high crash risks, but expressed some concerns about
placing more emphasis on crash information and less on other factors,
such as driver, vehicle, or safety management issues. In addition,
FMCSA noted that, while it has devoted considerable efforts to
improving the quality of crash data submitted by states, the negative
binomial regression model is more sensitive than SafeStat to problems
with the crash data.
Background:
The interstate commercial motor carrier industry, primarily the
trucking industry, is an important part of the nation's economy. Trucks
transport over 11 billion tons of goods annually, or about 60 percent
of the total domestic tonnage shipped.[Footnote 7] Buses also play an
important role, transporting an estimated 631 million passengers
annually. There are approximately 711,000 commercial motor carriers
registered in MCMIS,[Footnote 8] about 9 million trucks and buses, and
more than 10 million drivers. Most motor carriers are small; about 51
percent operate one vehicle, and another 31 percent operate two to four
vehicles. Carrier operations vary widely in size, however, and some of
the largest motor carriers operate upwards of 50,000 vehicles. Carriers
continually enter and exit the industry. Since 1998, the industry has
increased in size by an average of about 29,000 interstate carriers per
year.
In the United States, commercial motor carriers account for less than 5
percent of all highway crashes, but these crashes result in about 13
percent of all highway deaths, or about 5,500 of the approximately
43,000 highway fatalities that occur nationwide annually. In addition,
about 106,000 of the approximately 2.7 million highway injuries per
year involve motor carriers. The fatality rate for trucks has generally
been decreasing over the past 30 years, but this decrease has leveled
off, and the rate has been fairly stable since the mid-1990s. The
fatality rate for buses has improved slightly from 1975 to 2005 but has
more annual variability than the fatality rate for trucks due to a much
smaller total vehicle miles traveled. (See fig. 1.)
Figure 1: Commercial Motor Vehicle Fatality Rate, 1975 to 2005:
[See PDF for image]
Source: GAO analysis of Department of Transportation data.
[End of figure]
Congress created FMCSA through the Motor Carrier Safety Improvement Act
of 1999 to reduce crashes, injuries, and fatalities involving
commercial motor vehicles. To accomplish this mission, FMCSA carries
out a number of enforcement, education, and outreach activities. FMCSA
uses enforcement as its primary approach for reducing the number of
crashes, fatalities, and injuries involving trucks and buses. Some of
FMCSA's enforcement programs include compliance reviews, which are on-
site reviews of carriers' records and operations to determine
compliance with regulations; safety audits of new interstate carriers;
and roadside inspections of drivers and vehicles.
FMCSA's education and outreach programs are intended to promote motor
carrier safety and consumer awareness. One of the programs is the New
Entrant program, which is designed to inform newly registered motor
carriers about motor carrier safety standards and regulations to help
them comply with FMCSA's requirements. Other programs are designed to
identify unregistered carriers and get them to register, promote
increased safety belt use among commercial drivers, and inform
organizations and individuals that hire buses how to make safe choices.
FMCSA plans to make major revisions to its compliance and enforcement
approach under an initiative called Comprehensive Safety Analysis 2010.
Compliance reviews are an important enforcement tool because they allow
FMCSA to take an in-depth look at carriers that have been identified as
posing high crash risks because of high crash rates or poor safety
performance records. Motor carriers may be identified as high risk from
SafeStat or through calls to FMCSA's complaint hotline. Carriers are
given a satisfactory, conditional, or unsatisfactory safety rating. A
conditional rating means the carrier is allowed to continue operating,
but FMCSA may schedule a follow-up compliance review to ensure that
problems noted in the first compliance review are addressed. An
unsatisfactory rating must be addressed or the carrier is placed out of
service, meaning it is no longer allowed to do business, and the
carrier may face legal enforcement actions undertaken by FMCSA.
Compliance reviews can take several days to complete, depending on the
size of the carrier, and may result in enforcement actions being taken
against a carrier.
FMCSA uses both its own inspectors and state inspectors to carry out
its enforcement activities. In total, about 750 staff are available to
perform compliance reviews, and more than 10,000 staff do vehicle and
driver inspections at weigh stations and other points. Together, FMCSA
and its state partners perform about 16,000 compliance reviews a year,
which cover about 2 percent of the nation's 711,000 carriers.[Footnote
9]
Because the number of inspectors is small compared with the size of the
motor carrier industry, FMCSA prioritizes carriers for compliance
reviews. To do so, it uses SafeStat to identify carriers that pose high
crash risks. SafeStat is a model that uses information gathered from
crashes, roadside inspections, traffic violations, compliance reviews,
and enforcement cases to determine a motor carrier's safety performance
relative to that of other motor carriers that have similar exposure in
these areas. A carrier's score is calculated on the basis of its
performance in four safety evaluation areas:
* Accident safety evaluation area: The accident safety evaluation area
reflects a carrier's crash history relative to other motor carriers'
histories. The safety evaluation area is based on state-reported crash
data, vehicle data from MCMIS, and data on reportable crashes and
annual vehicle miles traveled from the most recent compliance review. A
carrier must have two or more reportable crashes within the last 30
months to have the potential to receive a deficient value and thus be
made a priority for a compliance review.
* Driver safety evaluation area: The driver safety evaluation area
reflects a carrier's driver-related safety performance and compliance
relative to other motor carriers. The driver safety evaluation area is
based on violations cited in roadside inspections that have been
performed within the last 30 months and compliance reviews that have
occurred within the last 18 months, together with the number of drivers
listed in MCMIS. A carrier must have three or more driver inspections,
three or more moving violations, or at least one acute or critical
violation of driver regulations[Footnote 10] from a compliance review
to have the potential to receive a deficient value and thus be made a
priority for a compliance review.
* Vehicle safety evaluation area: The vehicle safety evaluation area
reflects a carrier's vehicle-related safety performance and compliance
relative to other motor carriers. The vehicle safety evaluation area is
based on violations identified during vehicle roadside inspections that
have occurred within the last 30 months or vehicle-related acute and
critical violations of regulations discovered during compliance reviews
that have occurred within the last 18 months. A carrier must have
either three or more vehicle inspections or at least one acute or
critical violation of vehicle regulations from a compliance review to
have the potential to receive a deficient value and thus be made a
priority for a compliance review.
* Safety management safety evaluation area: The safety management
safety evaluation area reflects a carrier's safety management relative
to other motor carriers. It is based on the results of violations cited
in closed enforcement cases in the past 6 years or violations of
regulations related to hazardous materials and safety management
discovered during a compliance review performed within the last 18
months. A carrier must have had at least one enforcement case initiated
and closed or at least two enforcement cases closed within the past 6
years, or at least one acute, critical, or severe violation of
hazardous material or safety management regulations[Footnote 11]
identified during a compliance review within the last 18 months to have
the potential to receive a deficient value and thus be made a priority
for a compliance review.
A motor carrier's score is based on its relative ranking, indicated as
a value, in each of the four safety evaluation areas. For example, if a
carrier receives a value of 75 in the accident safety evaluation area,
then 75 percent of all carriers with sufficient data for evaluation
performed better in that safety evaluation area, while 25 percent
performed worse. The calculation used to determine a motor carrier's
SafeStat score is as follows:
SafeStat Score = (2.0x accident value) + (1.5x driver value) + vehicle
value + safety management value:
As shown in the formula, the accident and driver safety evaluation
areas have 2.0 and 1.5 times the weight, respectively, of the vehicle
and safety management safety evaluation areas. Safety evaluation area
values less than 75 are ignored in the formula used to determine the
SafeStat score. For example, a carrier with values of 74 for all four
safety evaluation areas has a SafeStat score of 0. FMCSA assigned more
weight to these safety evaluation areas because, according to FMCSA,
crashes and driver violations correlate relatively better with future
crash risk. In addition, more weight is assigned to fatal crashes and
to crashes that occurred within the last 18 months. In consultation
with state transportation officials, insurance industry
representatives, safety advocates, and the motor carrier industry,
FMCSA used its expert judgment and professional knowledge to assign
these weights, rather than determining them through a statistical
approach, such as regression modeling.
FMCSA assigns carriers categories ranging from A to H according to
their performance in each of the safety evaluation areas. A carrier is
considered to be deficient in a safety evaluation area if it receives a
value of 75 or higher in that particular safety evaluation area.
Although a carrier may receive a value in any of the four safety
evaluation areas, the carrier receives a SafeStat score only if it is
deficient in one or more safety evaluation areas. Carriers that are
deficient in two or more safety evaluation areas and have a SafeStat
score of 225 or more are considered to pose high crash risks and are
placed in category A or B. (See table 1.) Carriers that are deficient
in two safety evaluation areas but have a SafeStat score of less than
225 are placed in category C and receive a medium priority for
compliance reviews. Carriers that are deficient in only one of the
safety evaluation areas are placed in category D, E, F, or G. Carriers
that are not deficient in any of the safety evaluation areas do not
receive a SafeStat score and are placed in category H.
Table 1: SafeStat Categories and Their Priority for Compliance Reviews:
Category: A;
Condition: Deficient in all four safety evaluation areas; or; Deficient
in three safety evaluation areas that result in a weighted SafeStat
score of 350 or more;
Priority for compliance review: High.
Category: B;
Condition: Deficient in three safety evaluation areas that result in a
weighted SafeStat score of less than 350; or; Deficient in two safety
evaluation areas that result in a weighted SafeStat score of 225 or
more;
Priority for compliance review: High.
Category: C;
Condition: Deficient in two safety evaluation areas that result in a
weighted SafeStat score of less than 225;
Priority for compliance review: Medium.
Category: D;
Condition: Deficient in the accident safety evaluation area (accident
safety evaluation area value between 75-100);
Priority for compliance review: Low.
Category: E;
Condition: Deficient in the driver safety evaluation area (driver
safety evaluation area value between 75-100);
Priority for compliance review: Low.
Category: F;
Condition: Deficient in the vehicle safety evaluation area (vehicle
safety evaluation area value between 75-100);
Priority for compliance review: Low.
Category: G;
Condition: Deficient in the safety management safety evaluation area
(safety management safety evaluation area value between 75-100);
Priority for compliance review: Low.
Category: H;
Condition: Not deficient in any of the safety evaluation areas (value
below 75 in each of the safety evaluation areas);
Priority for compliance review: Low.
Source: GAO summary of FMCSA data.
[End of table]
Of the 622,000 motor carriers listed in MCMIS as having one or more
vehicles in June 2004, about 140,000, or 23 percent, received a
SafeStat category A through H. There are several reasons why a small
proportion of carriers receive a score. First, approximately 305,900,
or about 42 percent, of the carriers have crash, vehicle inspection,
driver inspection, or enforcement data of any kind. SafeStat relies on
these data to calculate a motor carrier's score, so carriers without
such data are not rated by SafeStat. It is likely that some of the
carriers listed in MCMIS are no longer in business, but it is also
possible that these carriers had no crashes, inspections, or compliance
reviews in the 30-month period prior to June 2004. Second, a carrier
must meet the minimum requirements to be assigned a value in a given
safety evaluation area.[Footnote 12] If, for example, a carrier had
only one reportable crash within the last 30 months, then the carrier
would not be assigned an accident safety evaluation area value. Of the
305,900 carriers that have any safety data in SafeStat, 140,000 met the
SafeStat minimum requirements in one or more safety evaluation areas.
Of these 140,000 carriers, 45,000 were rated in categories A through G.
The other carriers were placed in category H because they were not
considered deficient, meaning they did not receive a value of 75 or
more in any of the safety evaluation areas.
The design of SafeStat and its data sufficiency requirements increase
the likelihood that larger motor carriers will be deficient in one of
the safety evaluation areas, in other words, rated in categories A
through G, than are small carriers. About 51 percent of all carriers
listed in MCMIS operate one vehicle, and about 3 percent of them
received a SafeStat rating in categories A through G. (See table 2.) In
contrast, fewer than 1 percent of the carriers listed in MCMIS have
more than 100 vehicles, and nearly 25 percent of them received a
SafeStat rating in categories A through G.
Table 2: Size Distribution of Carriers Receiving a SafeStat Rating of A
through G:
Carrier size (number of vehicles): 1;
Number of carriers (percentage[A] ): 317,037 (51%);
Number of carriers within size category receiving A through G SafeStat
rating (percentage of carriers in size category): 8,697 (3%).
Carrier size (number of vehicles): >1 to 4;
Number of carriers (percentage[A] ): 191,739 (31%);
Number of carriers within size category receiving A through G SafeStat
rating (percentage of carriers in size category): 14,430 (8%).
Carrier size (number of vehicles): >4 to 10;
Number of carriers (percentage[A] ): 66,422 (11%);
Number of carriers within size category receiving A through G SafeStat
rating (percentage of carriers in size category): 10,595 (16%).
Carrier size (number of vehicles): >10 to 25;
Number of carriers (percentage[A] ): 28,780 (5%);
Number of carriers within size category receiving A through G SafeStat
rating (percentage of carriers in size category): 6,504 (23%).
Carrier size (number of vehicles): >25 to 100;
Number of carriers (percentage[A] ): 14,148 (2%);
Number of carriers within size category receiving A through G SafeStat
rating (percentage of carriers in size category): 3,550 (25%).
Carrier size (number of vehicles): >100;
Number of carriers (percentage[A] ): 3,903 (1%);
Number of carriers within size category receiving A through G SafeStat
rating (percentage of carriers in size category): 909 (23%).
Source: GAO analysis of FMCSA data.
Note: The table only includes those carriers listed as having one or
more vehicles.
[A] Percentages do not equal 100 because of rounding.
[End of table]
A Statistical Approach Would Better Identify Carriers That Pose High
Crash Risks Than Does FMCSA's Current Approach:
We found that FMCSA could improve SafeStat's ability to identify
carriers that pose high crash risks if it applied a statistical
approach, called a negative binomial regression model, to the four
SafeStat safety evaluation areas instead of its current approach.
Through this change, FMCSA could more efficiently target compliance
reviews to the set of carriers that pose the greatest crash risk.
Applying a negative binomial regression model would improve the
identification of high risk carriers over SafeStat's performance by
about 9 percent,[Footnote 13] compared with the current approach, which
incorporates safety data weighted in accordance with the professional
judgment and experience of SafeStat's designers. Moreover, according to
our analysis, this 9 percent improvement would enable FMCSA to identify
carriers with almost twice as many crashes in the following 18 months
as those carriers identified under its current approach. Targeting
these high-risk carriers would result in FMCSA giving compliance
reviews to carriers that experienced both a higher crash rate and, in
conjunction with the higher crash rate, 9,500 more crashes over an 18-
month period than those identified by the SafeStat model. Applying a
negative binomial regression model approach to the SafeStat safety
evaluation areas would be easy to implement and, in our opinion, would
be consistent with other FMCSA uses for SafeStat beyond identifying
carriers that pose high risks for crashes. In addition, adopting a
negative binomial regression model approach would be beneficial even if
FMCSA makes major revisions to its compliance and enforcement approach
in the coming years under its Comprehensive Safety Analysis 2010
initiative. Overall, other changes to the SafeStat model that we
explored, such as modifying decision rules used in the construction of
the safety evaluation areas, did not improve the model's overall
performance.
Regression Models Identify Carriers That Pose High Crash Risks Better
Than Expert Judgment:
Although SafeStat is nearly twice as effective as (83 percent better
than) random selection in identifying carriers that pose high crash
risks[Footnote 14] and, therefore, has value for improving safety, we
found that FMCSA could improve SafeStat's ability to identify such
carriers by about 9 percent if it applied a negative binomial
regression model approach to its analysis of motor carrier safety data.
The use of a regression model does not entail assigning the letter
categories currently assigned by the SafeStat model. Rather, the model
predicts carriers' crash risks, sorts the carriers according to their
risk level, and assigns a high priority for a compliance review to the
highest risk carriers. The improvement in identification of high-risk
carriers, which we observed with the negative binomial regression
model, is consistent with results obtained in an earlier analysis of
MCMIS data performed by a team of researchers at Oak Ridge National
Laboratory.[Footnote 15]
To compare the effectiveness of regression models and SafeStat in
identifying carriers that pose high crash risks, we applied several
regression models to the four safety evaluation areas (accident,
driver, vehicle, and safety management) used by the SafeStat model. We
recalculated SafeStat's June 2004 accident safety evaluation area
values because the data FMCSA provided on the number of crashes for
each carrier differed in 2006 from the data used in the model in
2004.[Footnote 16] Using our accident safety evaluation area value and
the original driver, vehicle, and safety management safety evaluation
area values from June 2004, we selected the 4,989 carriers that our
regression models identified as the highest crash risks,[Footnote 17]
calculated the crash rate per 1,000 vehicles for these carriers over
the next 18 months, and compared this rate with the crash rate per
1,000 vehicles for the 4,989 carriers identified by the SafeStat model
as posing high crash risks (categories A and B).
All of the regression models that we estimated were at least as
effective as SafeStat in identifying motor carriers that posed high
crash risks. (See app. III for these results.) Of these, the negative
binomial regression approach gave the best results and proved 9 percent
more effective than SafeStat, as measured by future crashes per 1,000
vehicles. The set of carriers in SafeStat categories A and B had a
crash rate of 102 per 1,000 vehicles for the 18 months after June 2004
while the set of high-risk carriers identified by the negative binomial
regression model had 111 crashes per 1,000 vehicles. Even though this 9
percent improvement rate seems modest, it translates into nearly twice
as many "future crashes" identified. Specifically, the negative
binomial regression model identified carriers that had nearly twice as
many crashes (from July 2004 to December 2005) as the carriers
identified by SafeStat--19,580 crashes compared with 10,076.[Footnote
18]
SafeStat (categories A and B) and our negative binomial regression
model identified many of the same carriers--1,924 of the 4,989 (39
percent)--as posing high crash risks. However, our model also
identified a number of high-risk carriers that SafeStat did not
identify, and vice versa. For example, our model identified 2,244
carriers as posing high crash risks, while SafeStat placed these
carriers in category D (the accident area), assigning them a lower
priority for compliance reviews. One reason for this difference is the
decision rules that SafeStat employs. Under SafeStat, carriers must
perform worse than 75 percent of all carriers to be considered
deficient in any safety evaluation area. The regression approach
identifies the carriers with the highest crash risks regardless of how
they compare with their peers in individual areas. For example, we
identified as posing high crash risks 482 carriers that SafeStat did
not consider at all for compliance reviews because the carriers had not
performed worse than 75 percent of their peers in any of the four
safety evaluation areas.
FMCSA Can Apply a Regression Model Approach in the Short Term, Even
Though It Is Planning to Overhaul SafeStat:
In the short term, FMCSA could easily implement a regression model
approach for SafeStat.[Footnote 19] All the information required as
input for the negative binomial regression model is already entered
into SafeStat. In addition, a standard statistical package can be used
to apply the negative binomial approach to the four SafeStat safety
evaluation areas. Like SafeStat, the negative binomial regression model
would be run every month to produce a list of motor carriers that pose
high crash risks, and these carriers would then be assigned priorities
for a compliance review. As with SafeStat, the results of the negative
binomial model would change slightly each month with the addition of
new safety data to MCMIS.
In discussing the concept of adopting a negative binomial regression
model approach with FMCSA officials, they were interested in
understanding how the use of the negative binomial regression model
results could be used to identify and improve the safety of those
carriers that pose the greatest crash risks (much as the SafeStat
categories of A and B do now) and how it could employ the proposed
approach for current uses beyond identifying carriers that pose high
crash risks. These uses include providing an understandable public
display to shippers, insurers, and others who are interested in the
safety of carriers; selecting carriers for roadside inspections; and
trying to gain carriers' compliance with driver and vehicle safety
rules, when these carriers may not have crashes, consistent with agency
efforts.
* Identifying and improving the safety of carriers that pose high crash
risks. The negative binomial regression model approach would produce a
rank order listing of carriers by crash risk and by the predicted
number of crashes. For compliance reviews, FMCSA could choose those
carriers with the greatest number of predicted crashes. FMCSA would
choose the number of carriers to review based on the resources
available to it, much as it currently does.
Regarding improving the safety of carriers that pose high crash risks,
FMCSA currently enrolls carriers that receive a SafeStat category of A,
B, or C in the Motor Carrier Safety Improvement Program. This program
aims to improve the safety of high-risk carriers through (1) a
repetitive cycle of identification, data gathering, and assessment and
(2) progressively harsher treatments applied to carriers that do not
improve their safety. The use of a negative binomial regression model
would not affect the structure or workings of this program, other than
to better identify carriers that pose high crash risks. As discussed
above, FMCSA would use the regression model's results to identify the
highest risk carriers and then intervene using its existing approaches
(such as issuing warning letters, conducting follow-up compliance
reviews, or levying civil penalties) as treatment.
* Providing an understandable display to the public. FMCSA could choose
to provide a rank order listing of carriers together with the
associated number of predicted crashes or it could look for natural
breaks in the predicted number of crashes and associate a category--
such as "category A" to these carriers.
* Selecting carriers for roadside inspections. Safety rankings from the
SafeStat model are also used in FMCSA's Inspection Selection System to
prioritize carriers for roadside driver and vehicle inspections. The
negative binomial regression model optimizes the identification of
carriers by crash risk using safety evaluation area information. The
negative binomial regression model approach that we describe in this
report retains SafeStat's basic design with four safety management
areas (driver, vehicle, accident, and safety management). Therefore,
FMCSA could use the negative binomial regression model results to
identify carriers that pose a high crash risk, the results from the
driver and vehicle safety evaluation areas, or both, to target carriers
or vehicles for roadside driver and vehicle inspections.
* Furthering agency efforts to gain compliance with driver and vehicle
safety rules for carriers that do not experience crashes (or a
sufficient number of crashes to pose a high risk for crashes). FMCSA
was interested in understanding how, if at all, the negative binomial
regression model approach would affect gaining compliance against
carriers that may routinely violate safety rules (such as drivers'
hours of service requirements), but where these violations do not lead
to crashes. As discussed above, the negative binomial regression model
approach retains SafeStat's four safety evaluation areas. Where it
differs, is that it assigns different weights to those areas based on a
statistical procedure, rather than having the weights assigned by
expert judgment. As a result, FMCSA would still be able to identify
carriers with many driver, vehicle, and safety management violations.
Other opportunities also exist for FMCSA to improve the ability of
regression models to identify carriers that pose high crash risks. In
2005, a FMCSA compliance review work group reported a positive
correlation between driver hours of service violations and crash
rates.[Footnote 20] Because FMCSA can link violations of specific
regulatory provisions, including those limiting driver hours of
service, to the crash experience of the carriers involved, it has the
opportunity to improve the violation severity weighting used in
constructing the driver and vehicle safety evaluation areas. FMCSA has
detailed violation data from roadside inspections and can statistically
analyze these data to find other strong relationships with carriers'
crash risks. Changes made to the safety evaluation area methodology to
strengthen the association with crash risk will improve the ability of
the negative binomial regression model to identify carriers that pose
high crash risks.
FMCSA has expressed doubts in the past when analysts have proposed
switching to a regression model approach. For example, Oak Ridge
National Laboratory advocated using a regression model approach in
place of SafeStat in 2004, but FMCSA was reluctant to move away from
its expert judgment model because it believed that the regression model
approach would place undue weight on the accident safety evaluation
area in determining priorities for compliance reviews,[Footnote 21]
thereby diminishing the incentive for motor carriers to comply with the
many safety regulations that feed into the driver, vehicle, and safety
management safety evaluation areas. In FMCSA's view, carriers would be
less likely to comply with these regulations because violations in the
driver, vehicle, and safety management areas would be less likely to
lead to compliance reviews under a regression model approach that
placed a heavy emphasis on crashes. Our view is that adopting a
negative binomial regression model approach would better identify
carriers that pose high crash risks and would thus further FMCSA's
primary mission of ensuring safe operating practices among commercial
interstate motor carriers.
Over the longer term, FMCSA is considering a complete overhaul of its
safety fitness determinations with its Comprehensive Safety Analysis
2010 initiative. This planned comprehensive review and analysis of the
agency's compliance and enforcement programs may result in a new
operational model for identifying drivers and carriers that pose safety
problems and for intervening to address those problems.[Footnote 22]
FMCSA expects to deploy the results of this initiative in 2010. In our
opinion, given the relative ease of adopting the regression modeling
approach discussed in this report,[Footnote 23] and the immediate
benefits that can be achieved, there is no reason to wait for FMCSA to
complete its initiative, even if the initiative results in major
revisions to the SafeStat model.
Modifications of SafeStat Did Not Improve Crash Identification:
Besides investigating whether the use of regression models could
improve SafeStat's ability to identify carriers that pose high crash
risks, we explored whether the existing model could be improved by
changing several of its decision rules. Overall, these changes did not
enhance the model's ability to identify carriers that pose high crash
risks. As long as FMCSA continues to estimate the safety evaluation
area values with its present methodology, the rules we investigated
help make the identification of high-risk motor carriers more efficient
for both SafeStat and the negative binomial regression model.
Because the SafeStat model is composed of many components, we selected
three decision rules for analysis. We chose these three rules because
they are important pillars of the SafeStat model's methodology for
constructing the safety evaluation areas and because we could complete
our analysis of them during the time we had to perform our work. A
fuller exploration of areas with high potential to improve the
identification of carriers that pose high crash risks would be a long-
term effort, and FMCSA plans to address this work as part of the
Comprehensive Safety Analysis 2010 initiative.
* Removing comparison groups. As part of its methodology for
calculating the accident, driver, and vehicle safety evaluation area
values, SafeStat divides carriers into comparison groups. For example,
in the driver safety evaluation area, SafeStat groups carriers by the
number of moving violations they have, placing them in one of four
groups (3 to 9, 10 to 28, 29 to 94, and 95 or more).[Footnote 24]
SafeStat uses the comparison groups to control for the size of the
carrier. We removed all the comparison groups in each of the three
safety evaluation areas, recalculated their values, and compared the
number of crashes in which the carriers were involved and their crash
rates, for each of the SafeStat categories A through H, with the
SafeStat results in which comparison groups were retained.
* Removing minimum event requirements. SafeStat imposes minimum event
requirements. For example, as noted, SafeStat does not consider a
carrier's moving violations if, in the aggregate, its drivers had fewer
than three moving violations over a 30-month period. FMCSA does not
calculate a safety evaluation area value for carriers with fewer than
three events in an attempt to control for carriers that have
infrequent, rather than possibly systemic, safety problems.[Footnote
25] We removed the requirement to have a minimum number of events (such
as moving violations, crashes, and inspections), recalculated the three
safety evaluation values, and compared the number of crashes in which
the carriers were involved and their crash rates, for each of the
SafeStat categories A through H, with the SafeStat results in which
minimum event requirements were retained.
* Removing time and severity weights. The SafeStat formula weights more
recent events and more severe events more heavily than less recent or
less severe events in the accident, driver, and vehicle safety
evaluation areas. For example, the results of vehicle roadside
inspections performed within the latest 6 months receive three times
the weight of inspections performed 2 years ago. Similarly, crashes
involving deaths or injuries receive twice as much weight as those that
resulted in property damage only. We removed the time and severity
weights for the three safety evaluation areas, recalculated these
values, and compared the number of crashes in which the carriers were
involved and their crash rates, for each of the SafeStat categories A
through H, with the SafeStat results in which time and severity weights
were retained.
* Simultaneous changes to comparison group, event, and time severity
requirements. Finally, we simultaneously removed comparison groups,
minimum event requirements, and time and severity weights and compared
the number of crashes in which the carriers were involved and their
crash rates, for each of the SafeStat categories A through H, with the
SafeStat results in which comparison groups, minimum event
requirements, and time and severity weights were retained.
The results of each of our individual analyses and of making all
changes simultaneously produced one of two outcomes, neither of which
was considered more desirable. Relaxing the minimum data requirements
greatly increased the number of carriers identified as high risk
without increasing the overall number of predicted crashes over the
subsequent 18 months, thus reducing the effectiveness of the SafeStat
model. Removing comparison groups and removing time and severity
weights had the effect of reducing the future crashes per 1,000
vehicles among those carriers identified as high risk, also reducing
the effectiveness of the SafeStat model. As a result, we are not
reporting on these results in detail. Trying to modify the decision
rules used in SafeStat did highlight the balance that FMCSA has to
strike between maximizing the identification of companies with the
largest number of crashes (usually larger carriers) and those carriers
with the greatest safety risk (which can be of any size).
Despite Quality Problems, FMCSA's Crash Data Can Be Used to Compare
Methods for Identifying Carriers That Pose High Crash Risks:
The quality of crash data is a long-standing problem that potentially
hindered FMCSA's ability to accurately identify carriers that pose high
crash risks.[Footnote 26] Despite the problems of late-reported crashes
and incomplete and inaccurate data on crashes during the period we
studied, we determined that the data were of sufficient quality for our
use, which was to assess how the application of regression models might
improve the ability to identify high-risk carriers over the current
approach--not to determine absolute measures of crash risk. Our
reasoning is based on the fact that we used the same data set to
compare the results of the SafeStat model and the regression models.
Limitations in the data would apply equally to both results. FMCSA has
recently undertaken a number of efforts to improve crash data quality.
Late Reporting Had a Small Effect on SafeStat's Ability to Identify
High-Risk Carriers:
FMCSA's guidance provides that states report all crashes to MCMIS
within 90 days of their occurrence. Late reporting can cause SafeStat
to miss some of the carriers that should have received a SafeStat
score. Alternatively, since SafeStat's scoring involves a relative
ranking of carriers, a carrier may receive a SafeStat score and have to
undergo a compliance review because crash data for a higher risk
carrier were reported late and not included in the calculation.
Late reporting affected SafeStat's ability to identify all high-risk
carriers to a small degree--about 6 percent---for the period that we
studied. Late reporting of crashes by states affected the safety
rankings of more than 600 carriers, both positively and negatively.
When SafeStat analyzed the 2004 data, which did not include the late-
reported crashes, it identified 4,989 motor carriers as highest risk,
meaning they received a category A or B ranking. With the addition of
late-reported crashes, 481 carriers moved into the highest risk
category, and 182 carriers dropped out of the highest risk category,
resulting in a net increase of 299 carriers (6 percent) in the highest
risk category. After the late-reported crashes were added, 481 carriers
that originally received a category C, D, E, F, or G SafeStat rating
received an A or B rating. These carriers would not originally have
been given a high priority for a compliance review because the SafeStat
calculation did not take into account all of their crashes. On the
other hand, a small number of carriers would have received a lower
priority if the late-reported crashes had been included in their score.
Specifically, 182 carriers - or fewer than 4 percent of those ranked,
fell from the A or B category into the C, D, E, F, or G category once
the late-reported crashes were included.[Footnote 27] These carriers
would not have been considered high priority for a compliance review if
all crashes had been reported on time. This does not have a big effect
on the overall motor carrier population, however, as only 4 percent of
carriers originally identified as highest risk were negatively affected
by late reporting.
The timeliness of crash reporting has shown steady and marked
improvement. The median number of days it took states to report crashes
to MCMIS dropped from 225 days in calendar year 2001 to 57 days in 2005
(the latest data available at the time of our analysis).[Footnote 28]
In addition, the percentage of crashes reported by states within 90
days of occurrence has jumped from 32 percent in fiscal year 2000 to 89
percent in fiscal year 2006. (See fig. 2.)
Figure 2: Percentage of Crashes Submitted to MCMIS within 90 Days of
Occurrence:
[See PDF for image]
Source: GAO analysis of FMCSA data.
[End of figure]
Incomplete Data from States Potentially Limit SafeStat's Identification
of All Carriers That Pose High Crash Risks:
FMCSA uses a motor carrier identification number, which is unique to
each carrier, as the primary means of linking inspections, crashes, and
compliance reviews to motor carriers. Approximately 184,000 (76
percent) of the 244,000 crashes reported to MCMIS between December 2001
and June 2004 involved interstate carriers. Of these 184,000 crashes,
nearly 24,000 (13 percent) were missing this identification number. As
a result, FMCSA could not match these crashes to motor carriers or use
them in SafeStat. In addition, the carrier identification number could
not be matched to a number listed in MCMIS for 15,000 (8 percent) other
crashes that involved interstate carriers. Missing data or being unable
to match data for nearly one quarter of the crashes during the period
of our review potentially has a large impact on a motor carrier's
SafeStat score because SafeStat treats crashes as the most important
source of information for assessing motor carrier crash risk.
Theoretically, information exists to match crash records to motor
carriers by other means, but such matching would require too much
manual work to be practicable.
We were not able to quantify the actual effect of either the missing
data or the data that could not be matched for MCMIS overall. To do so
would have required us to gather crash records at the state level--an
effort that was impractical. For the same reason, we cannot quantify
the effects of FMCSA's efforts to improve the completeness of the data
(discussed later). However, a series of reports by the University of
Michigan Transportation Research Institute sheds some light on the
completeness of the data submitted to MCMIS by the states.[Footnote 29]
One of the goals of the research was to determine the states' crash
reporting rates. Reporting rates varied greatly among the 14 states
studied, ranging from 9 percent in New Mexico in 2003 to 87 percent in
Nebraska in 2005. It is not possible to draw wide-scale conclusions
about whether state reporting rates are improving over time because
only two of the states--Missouri and Ohio---were studied in multiple
years. However, in these two states, the reporting rate did improve.
Missouri experienced a large improvement in its reporting rate, with 61
percent of eligible crashes reported in 2001, and 83 percent reported
in 2005. Ohio's improvement was more modest, increasing from 39 percent
in 2000 to 43 percent in 2005.
The University of Michigan Transportation Research Institute's reports
also identified a number of factors that may affect states' reporting
rates. One of the main factors affecting reporting rates is the
reporting officer's understanding of crash reporting requirements. The
studies note that reporting rates are generally lower for less serious
crashes and for crashes involving smaller vehicles, which may indicate
that there is some confusion about which crashes are reportable. Some
states, such as Missouri, aid the officer by explicitly listing
reporting criteria on the police accident reporting form, while other
states, such as Washington, leave it up to the officer to complete
certain sections of the form if the crash is reportable, but the form
includes no guidance on reportable crashes. Yet other states, such as
North Carolina and Illinois, have taken this task out of officers'
hands and include all reporting elements on the police accident
reporting form. Reportable crashes are then selected centrally by the
state, and the required data are transmitted to MCMIS.
Inaccurate Data Potentially Limit SafeStat's Ability to Identify
Carriers That Pose High Crash Risks:
Inaccurate data, such as reporting a nonqualifying crash to FMCSA,
potentially has a large impact on a motor carrier's SafeStat score
because SafeStat treats crashes as the most important source of
information for assessing motor carrier crash risk. For the same
reasons as discussed in the preceding section, we were neither able to
quantify these effects nor determine how data accuracy has improved for
MCMIS overall.
The University of Michigan Transportation Research Institute's reports
on crash reporting show that, among the 14 states studied, incorrect
reporting of crash data is widespread. In recent reports, the
researchers found that, in 2005, Ohio incorrectly reported 1,094 (22
percent) of the 5,037 cases, and Louisiana incorrectly reported 137 (5
percent) of the 2,699 cases. In Ohio, most of the incorrectly reported
crashes did not qualify because they did not meet the crash severity
threshold. In contrast, most of the incorrectly reported crashes in
Louisiana did not qualify because they did not involve vehicles
eligible for reporting. Other states studied by the institute had
similar problems with reporting crashes that did not meet the criteria
for reporting to MCMIS. These additional crashes could cause some
carriers to exceed the minimum number of crashes required to receive a
SafeStat rating and result in SafeStat's mistakenly identifying
carriers as posing high crash risks. Because each report focuses on
reporting in one state in a particular year, it is not possible to
identify the number of cases that have been incorrectly reported
nationwide and, therefore, it is not possible to determine the impact
of inaccurate reporting on SafeStat's calculations.
As noted in the University of Michigan Transportation Research
Institute's reports, states may be unintentionally submitting incorrect
data to MCMIS because of difficulties in determining whether a crash
meets the reporting criteria. For example, in Missouri, pickups are
systematically excluded from MCMIS crash reporting, which may cause the
state to miss reportable crashes. However, some pickups may have
vehicle weights above the reporting threshold, making crashes involving
them eligible for reporting. There is no way for the state to determine
which crashes involving pickups qualify for reporting without examining
the characteristics of each vehicle. In this case, the number of
omissions is likely to be relatively small, but this example
demonstrates the difficulty states may face when identifying reportable
crashes.
In addition, in some states, the information contained in the police
accident report may not be sufficient for the state to determine if a
crash meets the accident severity threshold. It is generally
straightforward to determine whether a fatality occurred as a result of
a crash, but it may be difficult to determine whether an injured person
was transported for medical attention or a vehicle was towed because of
disabling damage. In some states, such as Illinois and New Jersey, an
officer can indicate on the form if a vehicle was towed by checking a
box, but there is no way to identify whether the reason for towing was
disabling damage. It is likely that such uncertainty results in
overreporting because some vehicles may be towed for other reasons.
FMCSA Has Undertaken Efforts to Improve Crash Data Quality:
FMCSA has taken steps to try and improve the quality of crash data
reporting. As we noted in November 2005, FMCSA has undertaken two major
efforts to help states improve the quality of crash data.[Footnote 30]
One program, the Safety Data Improvement Program, has provided funding
to states to implement or expand activities designed to improve the
completeness, timeliness, accuracy, and consistency of their data.
FMCSA has also used a data quality rating system to rate and display
ratings for states' crash and inspection data quality. Due to its
public nature, this map serves as an incentive for states to make
improvements in their data quality.
To further improve these programs, FMCSA has made additional grants
available to states and implemented our recommendations to (1)
establish specific guidelines for assessing states' requests for
funding to support data improvement in order to better assess and
prioritize the requests and (2) increase the usefulness of its state
data quality map as a tool for monitoring and measuring commercial
motor vehicle crash data by ensuring that the map adequately reflects
the condition of the states' commercial motor vehicle crash data.
In February 2004, FMCSA implemented Data Q's, an online system that
allows for challenging and correcting erroneous crash or inspection
data. Users of this system include motor carriers, the general public,
state officials, and FMCSA. In addition, in response to a recent
recommendation by the Department of Transportation Inspector General,
FMCSA is planning to conduct a number of evaluations of the
effectiveness of a training course on crash data collection that it
will be providing to states by September 2008.
While the quality of crash reporting is sufficient for use in
identifying motor carriers that pose high crash risks and has started
to improve, commercial motor vehicle crash data continue to have some
problems with timeliness, completeness, and accuracy. These problems
have been well-documented in several studies, and FMCSA is taking steps
to address the problems through studies of each state's crash reporting
system and grants to states to fund improvements. As a result, we are
not making any recommendations in this area.
Conclusion:
Interstate commerce involving large trucks and buses has been growing
substantially, and this growth is expected to continue. While the
number of fatalities per million vehicle miles traveled has generally
decreased over the last 30 years, the fatality rate has leveled off and
remained fairly steady since the mid-1990s. FMCSA could more
effectively address fatalities due to crashes involving a commercial
motor vehicle if it better targeted compliance reviews to those
carriers that pose the greatest crash risks. Using a negative binomial
regression model would further FMCSA's mission of reducing crashes
through the more effective targeting of compliance reviews to the set
of carriers that pose the greatest crash risks. In light of possible
changes to FMCSA's safety fitness determinations resulting from its
Comprehensive Safety Analysis 2010 initiative, we are not suggesting
that FMCSA undertake a complete and thorough investigation of SafeStat.
Rather, we are advocating that FMCSA apply a statistical approach that
employs the negative binomial regression model rather than relying on
the current SafeStat formula that was determined through expert
judgment. In our view, the substitution of a statistically based
approach would likely yield a markedly better ability to identify
carriers that pose high crash risks with relatively little time or
effort on FMCSA's part.
Recommendation for Executive Action:
We recommend that the Secretary of Transportation direct the
Administrator of FMCSA to apply a negative binomial regression model,
such as the one discussed in this report, to enhance the current
SafeStat methodology.
Agency Comments and Our Evaluation:
We provided a draft of this report to the Department of Transportation
for its review and comment. In response, departmental officials,
including FMCSA's Director of the Office of Enforcement and Compliance
and Director of the Office of Research and Analysis, noted that our
report provided useful insights and offered a potential avenue for
further improving the effectiveness of FMCSA's efforts to reduce
crashes involving motor carriers. The agency indicated that it is
already working to improve upon SafeStat as part of its Comprehensive
Safety Analysis 2010 initiative. FMCSA agreed that it would be useful
for it to consider whether there are both short and longer term
measures that would incorporate the type of analysis identified in our
report, as an adjunct to the SafeStat model, in order to better target
compliance reviews so as to make the best use of FMCSA's resources to
reduce crashes.
The agency expressed some concerns with the negative binomial
regression analysis, noting that its intent is to effectively target
its compliance activities based on a broader range of factors than is
considered in the negative binomial regression analysis approach
described in our draft report, which increases reliance on past crashes
as a predictor of future crashes while apparently de-emphasizing known
driver, vehicle, or safety management compliance issues. FMCSA told us
that it incorporates a broad range of information including driver
behavior, vehicle condition, and safety management in an attempt to
capture and enable the agency to act on accident precursors in order to
reduce crashes.
FMCSA is correct in concluding that the use of the negative binomial
regression approach could tilt enforcement heavily toward carriers that
have experienced crashes and away from other aspects of its problem
areas, such as violation of vehicle safety standards, that are intended
to prevent crashes. That is because the present SafeStat model does not
statistically assign weights to the accident, driver, vehicle, and
safety management areas. In addition, the negative binomial regression
approach fully considers information on the results of driver and
vehicle inspection data and safety management data. We used the same
data that FMCSA used, with some adjustments as new information became
available. While we found that the driver, vehicle, and safety
management evaluation area scores are correlated with the future crash
risk of a carrier, the accident evaluation area correlates the most
with future crash risk. We recognize that FMCSA selects carriers for
compliance reviews for multiple reasons, such as to respond to
complaints, and we would expect that it would retain this flexibility
if it adopted the negative binomial regression approach.
FMCSA also indicated that greater reliance on crash data increases
emphasis on the least reliable available data set, and one that is out
of the organization's direct control--crash reporting. While our draft
report found that crash reporting has improved, and that late reporting
has not significantly impaired FMCSA's use of the SafeStat model, FMCSA
noted that the reliance on previous crashes in the negative binomial
regression analysis described in our draft report could result in
greater sensitivity to the crash data quality issues.
As FMCSA noted in its comments, our results showed that the effect of
late-reported data was minimal. Also, as mentioned in our draft report
and in this final report, it was not practical to determine the effect,
if any, on SafeStat rankings of correcting inaccurate data or adding
incomplete data. Since June 2004, FMCSA has devoted considerable
efforts to improving the quality of the crash data it receives from the
states. States are now tracked quarterly for the completeness,
timeliness, and accuracy of their crash reporting. As FMCSA continues
its efforts to have states improve these data, any sensitivity of
results to crash data quality issues for the negative binomial
regression approach should diminish.
We are sending copies of this report to congressional committees and
subcommittees with responsibility for surface transportation safety
issues; the Secretary of Transportation; the Administrator, FMCSA; and
the Director, Office of Management and Budget. We also will make copies
available to others upon request. In addition, this report will be
available at no charge on the GAO Web site at http://www.gao.gov.
If you have any questions about this report, please either contact
Sidney H. Schwartz at (202) 512-7387 or Susan A. Fleming at (202) 512-
2834. Alternatively, they may be reached at schwartzsh@gao.gov or
flemings@gao.gov. Contact points for our Offices of Congressional
Relations and Public Affairs may be found on the last page of this
report. Staff who made key contributions to this report are Carl
Barden, Elizabeth Eisenstadt, Laurie Hamilton, Lisa Mirel, Stephanie
Purcell, and James Ratzenberger.
Signed by:
Sidney H. Schwartz:
Director Applied Research and Methods:
Signed by:
Susan A. Fleming:
Director Physical Infrastructure Issues:
[End of section]
Appendix I: Results of Other Assessments of the SafeStat Model's
Ability to Identify Motor Carriers That Pose High Crash Risks:
Several studies by the Volpe National Transportation Systems Center
(Volpe), the Department of Transportation's Office of Inspector
General, the Oak Ridge National Laboratory (Oak Ridge), and others have
assessed the predictive capability of the Motor Carrier Safety Status
Measurement System (SafeStat) model and the data used by that model. In
general, those studies that assessed the predictive power of SafeStat
offered suggestions to increase that power, and those studies that
assessed data quality found weaknesses in the data that the Federal
Motor Carrier Safety Administration (FMCSA) relies upon.
Assessments of SafeStat's Predictive Capability:
The studies we reviewed covered topics such as comparing SafeStat with
random selection to determine which does a better job of selecting
carriers that pose high crash risks, assessing whether statistical
approaches could improve that selection, and analyzing whether carrier
financial positions or driver convictions are associated with crash
risk.
Predictive Capability of SafeStat Compared with Random Selection:
In studies of the SafeStat model published in 2004 and 1998,[Footnote
31] Volpe analyzed retrospective data to determine how many crashes the
carriers in SafeStat categories A and B experienced over the following
18 months. The 2004 study used the carrier rankings generated by the
SafeStat model on March 24, 2001. Volpe then compared the SafeStat
carrier safety ratings with state-reported data on crashes that
occurred between March 25, 2001, and September 24, 2002, to assess the
model's performance. For each carrier, Volpe calculated a total number
of crashes, weighted for time and severity, and then estimated a crash
rate per 1,000 vehicles for comparing carriers in SafeStat categories A
and B with the carriers in other SafeStat categories. The 1998 Volpe
study used a similar methodology. Each study used a constrained subset
of carriers rather than the full list contained in the Motor Carrier
Management Information System (MCMIS).[Footnote 32] Both studies found
that the crash rate for the carriers in SafeStat categories A and B was
substantially higher than the other carriers during the 18 months after
the respective SafeStat run. On the basis of this finding, Volpe
concluded that the SafeStat model worked.
In response to a recommendation by the Department of Transportation's
Office of Inspector General,[Footnote 33] FMCSA contracted with Oak
Ridge to independently review the SafeStat model. Oak Ridge assessed
the SafeStat model's performance and used the same data set (for March
24, 2001), provided by Volpe, that Volpe had used in its 2004
evaluation. Perhaps not surprisingly, Oak Ridge obtained a similar
result for the weighted crash rate of carriers in SafeStat categories A
and B over the 18-month follow-up period. As with the Volpe study, the
Oak Ridge study was constrained because it was based on a limited data
set rather than the entire MCMIS data set.
Application of Regression Models to Safety Data:
While SafeStat does better than simple random selection in identifying
carriers that pose high crash risks, other methods can also be used to
achieve this outcome. Oak Ridge extended Volpe's analysis by applying
regression models to identify carriers that pose high crash risks.
Specifically, Oak Ridge applied a Poisson regression model and a
negative binomial model using the safety evaluation area values as
independent variables to a weighted count of crashes that occurred in
the 30 months before March 24, 2001. (For more information on
statistical analyses, see app. III.)
In addition, Oak Ridge applied the empirical Bayes method to the
negative binomial regression model and assessed the variability of
carrier crash counts by estimating confidence intervals. Oak Ridge
found that the negative binomial model worked well in identifying
carriers that pose high crash risks. However, the data set Oak Ridge
used did not include any carriers with one reported crash in the 30
months before March 24, 2001. Because data included only carriers with
zero or two or more reported crashes, the distribution of crashes was
truncated.
Since the Oak Ridge regression model analysis did not cover carriers
with safety evaluation area data and one reported crash, the findings
from the study are limited in their generalizability. However, other
analyses of crashes at intersections and on road segments have also
found that the negative binomial regression model works well.[Footnote
34] In addition, our analysis using a more recent and more
comprehensive data set supports the finding that the negative binomial
regression model performs better than the SafeStat model.
The studies carried out by other authors advocate the use of the
empirical Bayes method in conjunction with a negative binomial
regression model to estimate crash risk. Oak Ridge also applied this
model to identify motor carriers that pose high crash risks. We applied
this method to the 2004 SafeStat data and found that the empirical
Bayes method best identified the carriers with the largest number of
crashes in the 18 months after June 25, 2004. However, the crash rate
per 1,000 vehicles was much lower than that for carriers in SafeStat
categories A and B. We analyzed this result further and found that
although the empirical Bayes method best identifies future crashes, it
is not as effective as the SafeStat model or the negative binomial
regression model in identifying carriers with the highest future crash
rates. The carriers identified with the empirical Bayes method were
invariably the largest carriers. This result is not especially useful
from a regulatory perspective. Companies operating a large number of
vehicles often have more crashes over a period of time than smaller
companies. However, this does not mean that the larger company is
necessarily violating more safety regulations or is less safe than the
smaller company. For this reason, we do not advocate the use of the
empirical Bayes method in conjunction with the negative binomial
regression model as long as the method used to calculate the safety
evaluation area values remains unchanged. If changes are made in how
carriers are rated for safety, this method may in the future offer more
promise than the negative binomial regression model alone.
Relationship of Carrier Financial Data and Safety Risk:
Conducted on behalf of FMCSA, a study by Corsi, Barnard, and Gibney in
2002 examined how a carrier's financial performance data correlate with
the carrier's score on a compliance review.[Footnote 35] The authors
selected those motor carriers from MCMIS in December 2000 that had
complete data for the accident, driver, vehicle, and safety management
safety evaluation areas. Using these data, the authors then matched a
total of 700 carriers to company financial statements in the annual
report database of the American Trucking Associations.[Footnote 36] The
authors created a binary response variable for whether the carrier
received a satisfactory or an unsatisfactory outcome on the compliance
review. The authors then assessed how this result correlated with
financial measures derived from the company financial statements. In
general, the study found that indicators of poor financial condition
correlated with an increased safety risk.
Two practical considerations limit the applicability of the findings
from this study to SafeStat. First, the 700 carriers in the study
sample are not necessarily representative of the motor carriers that
FMCSA oversees. Only about 2 percent of the carriers evaluated by the
SafeStat model in June 2004 had a value for the safety management
safety evaluation area. Of these carriers, not all had complete data
for the other three safety evaluation areas. Second, FMCSA does not
receive annual financial statements from all motor carriers.[Footnote
37] For these reasons, we did not consider using carrier financial data
in our analysis of the SafeStat data.
Relationship of Commercial Driver License Convictions and Crash Risk:
A series of studies by Lantz and others examined the effect of
incorporating conviction data from the state-run commercial driver
license data system into the calculation of a driver conviction
measure.[Footnote 38] The studies found that the driver conviction
measure is weakly correlated with the crash per vehicle rate.[Footnote
39] However, the studies did not incorporate the proposed driver
conviction measure into one of the existing safety evaluation areas and
use the updated measure to estimate new SafeStat scores for carriers.
While the use of commercial driver license conviction data may have
potential for future incorporation into a model for identifying
carriers that pose high crash risks, there is no assessment of its
impact at this time.
Impact of Data Quality on SafeStat's Predictive Capability:
The 2004 Office of Inspector General report, the 2004 Oak Ridge study,
and reports by the University of Michigan Transportation Research
Institute on state crash reporting all examined the impact of data
quality on SafeStat's ability to identify carriers that pose high crash
risks. These studies looked at issues such as late reporting and
incomplete or inaccurate reporting of crash data and found weaknesses.
Late Reporting of Crash Data:
To determine whether states promptly report SafeStat data, the Office
of Inspector General conducted a two-stage statistical sample in which
it selected 10 states for review and then selected crash and inspection
reports from those states for examination. It sampled 392 crash records
and 400 inspection records from July through December 2002. In 2 of the
10 states selected, Pennsylvania and New Mexico, no crash records were
available for the sample period, so it selected samples from earlier
periods. The Office of Inspector General also discussed reporting
issues with state and FMCSA officials and obtained crash records from
selected motor carriers. In addition, the Office of Inspector General
used the coefficient of variation to analyze data consistency and
trends in reporting timeliness across geographic regions.[Footnote 40]
Our review of the study indicates that it was based on sound audit
methodology.
The study found that, as of November 2002, states submitted crash
reports in fiscal year 2002 an average of 103 days after the crash
occurred and that states varied widely in the timeliness of their crash
data reporting. (FMCSA requires that states report crashes no more than
90 days after they occur.) In addition, the study found that 20 percent
of the crashes that occurred in fiscal year 2002 were entered into
MCMIS 6 months or more after the crash occurred. On the basis of this
information, the Office of Inspector General concluded that the
calculation of the accident safety evaluation area value was affected
by the location of the carrier's operations but did not estimate the
degree of this effect.
We also assessed the extent of late reporting. We measured how many
days, on average, it took each state to report crashes to MCMIS in each
calendar year and found that the amount of time taken to report crashes
declined from 2000 to 2005. Our findings were similar in nature to the
Office of Inspector General's findings. However, our results are
broader because they are based on all crash data rather than a sample.
In addition, since our work is more recent, it reflects more current
conditions. We both came to the conclusion, although to varying
degrees, that late reporting of crash data by states negatively affects
SafeStat's identification of carriers that pose high crash risks.
Oak Ridge also examined the impact of late reporting. Using data
provided by Volpe, Oak Ridge looked at the difference between the date
a crash occurred and the date it was entered into MCMIS. The
researchers found that after 497 days, 90 percent of the reported
crashes were entered into MCMIS.
The Oak Ridge study also reran the SafeStat model for March 2001 with
the addition of crash data from March 2003 to see how more complete
data changed SafeStat scores. The study found that the addition of late-
reported data increased the number of carriers in the high-risk group
by 18 percent. This late reporting affected the rankings of 8 percent
of all the carriers ranked by SafeStat in March 2001. Of these affected
carriers, 3 percent moved to a lower SafeStat category, and 5 percent
moved to a higher category. Including the late-reported crash data
available in March 2003 for the period from September 1998 through
March 2001 resulted in a 35 percent increase in the available crash
data.
We performed the same analysis as the Oak Ridge study and obtained
similar results. We used SafeStat data from June 2004, which include
carrier safety data from December 2001 through June 2004. Using FMCSA's
master crash file from June 2006, we found that, with the addition of
late-reported crashes, 481 carriers moved into the highest risk
category, and 182 carriers dropped out of the highest risk category
resulting in a net increase of 299 carriers (6 percent) being added to
the highest risk category.
The University of Michigan Transportation Research Institute issued a
series of reports examining crash reporting rates in 14 states. These
reports looked at late reporting as a potential source of low crash
reporting rates but did not specifically examine the extent of late
reporting or the impact of late reporting on SafeStat scores. The
institute looked at reporting rates in each of the states by month to
determine if reporting rates were lower in the latter part of the year
because of late reporting. It found that reporting rates were lower in
the latter part of the year in 6 of the 14 states studied. This issue
was not a focus of our efforts, so we did not conduct a similar
analysis.
Incomplete and Inaccurate Reporting of Crash Data:
The Office of Inspector General's study found several instances of
incomplete or inaccurate data on crashes and carriers. The study
reviewed MCMIS reporting for all states and found that 6 of them did
not report any crashes to FMCSA in the 6-month period from July through
December 2002. In addition, the study found that MCMIS listed about 11
percent of carriers as having no vehicles, and 15 percent as having no
drivers. Finally, from a sample of crash records, the study estimated
that 13 percent of the crash reports and 7 percent of the inspection
reports in MCMIS contained errors that would affect SafeStat results.
In particular, the study concluded that the database identified the
wrong motor carrier as having been involved in a crash or as having
received a violation in 11 percent of the erroneous records.
The University of Michigan Transportation Research Institute also
examined the accuracy of states' crash data reporting. To determine if
crashes were reported accurately, the institute compared information
contained in the individual states' police accident reporting files
with crash data reported to MCMIS. Some states, such as Ohio, had
enough information captured in the police accident file to determine if
individual crashes were eligible for reporting, and, therefore, the
institute was able to use these data in its analyses. In other states,
not enough information was available to make a determination, and the
institute had to project results on the basis of other states'
experience. The institute also carried out a number of analyses, such
as comparing reporting rates for different reporting jurisdictions, in
an attempt to identify reporting trends in the individual states.
The institute identified several problems with the accuracy of states'
crash reporting. All 14 states that it studied reported ineligible
crashes to MCMIS. These crashes were ineligible because they either
involved vehicles not eligible for reporting or they did not meet the
crash severity threshold. In total, the 14 states reported nearly 5,800
ineligible crashes to MCMIS out of almost 68,000 crashes reported (9
percent). The states also failed to report a number of eligible
crashes: the 14 states studied reported from 9 percent to 87 percent of
eligible crashes.
Our review of the institute's methodology indicates that its findings
are based on sound methodology and that its analyses were very
thorough. However, its studies are limited to the 14 states studied and
to the particular year studied. (Not all studies covered the same
year.) These states' experience may or may not be representative of the
experiences of the entire country, and there is no way to determine if
the reporting for this year is representative of the state's reporting
activities over a number of years or if the results were unique to that
particular year. The exceptions to this are the studies for Missouri,
which covered calendar years 2001 and 2005, and Ohio, which covered
calendar years 2000 and 2005.
We did not attempt to assess the extent of inaccurate reporting in
individual states, but we did find examples of inaccurate data
reporting. To analyze the completeness of reporting, we attempted to
match all crash records in the MCMIS master crash file for crashes
occurring between December 26, 2001, and June 25, 2004, to the list of
motor carriers in the MCMIS census file. We found that Department of
Transportation numbers were missing for 30 percent of the crashes that
were reported, and the number did not match a Department of
Transportation number listed in MCMIS for 8 percent of reported
crashes. We also compared the number of crashes in MCMIS with the
number in the General Estimates System produced by the National Highway
Traffic Safety Administration and found evidence of underreporting of
crashes to MCMIS.[Footnote 41]
[End of section]
Appendix II: Scope and Methodology:
To determine whether statistical approaches could be used to improve
FMCSA's ability to identify carriers that pose high crash risks, we
tested a variety of regression models and compared their results with
results from the existing SafeStat model. The models we tested, using
MCMIS data used by SafeStat in June 2004 to identify carriers that pose
high crash risks, include the Poisson, negative binomial, zero-inflated
negative binomial, zero-inflated Poisson, and empirical Bayes. We chose
these regression models because crash totals for a company represent
count outcomes, and these statistical models are appropriate for use
with count data. In addition, we explored logistic regression to assess
the odds of having a crash. Based on the results of the statistical
models, we ranked the predicted means (or predicted probabilities in
the logistic regression) to see which carriers would be at risk during
the 18-month period after June 2004. We selected June 2004 because this
date enabled us to examine MCMIS data on actual crashes that occurred
in the 18-month period from July 2004 through December 2005.[Footnote
42] We used these data to determine the degree to which SafeStat
identified carriers that proved to pose high crash risks. We then
compared the predictive performance of the regression models with the
performance of SafeStat to determine which method best identified
carriers that pose high crash risks. Using a series of simple random
samples,[Footnote 43] we also calculated the crash rates of all
carriers listed in the main SafeStat summary results table in MCMIS for
comparison with the crash rates of carriers identified by SafeStat as
high risk. We did this analysis to determine whether the SafeStat model
did a better job than random selection of identifying motor carriers
that pose high crash risks.
In addition, we tested changes to selected portions of the SafeStat
model to see whether improvements could be made in the identification
of high-risk motor carriers. In one analysis, we modified the
calculation of the safety evaluation area values and compared the
number of high-risk motor carriers identified with the number
identified by the unmodified safety evaluation areas. For example, we
included carriers with only one crash in the calculation of the
accident safety evaluation area whereas the unmodified SafeStat model
includes only carriers with two or more crashes. We also investigated
the effect of removing the time and severity weights from the indexes
used to construct the accident, driver, and vehicle safety evaluation
areas. We then compared the result of using the modified and unmodified
safety evaluation area values to determine if this modification
improved the model's ability to identify future crash risks.
To assess the extent to which the timeliness, completeness, and
accuracy of MCMIS and state-reported crash data affect SafeStat's
performance, we carried out a series of analyses with the MCMIS crash
master file and MCMIS census file, as well as surveying the literature
to assess findings on MCMIS data quality from other studies. To assess
the effect of timeliness, we first measured how many days on average it
was taking each state to report crashes to FMCSA by year for calendar
years 2000 through 2005. We also recalculated SafeStat scores from the
model's June 25, 2004, run to include crashes that had occurred more
than 90 days before that date but had not been reported to FMCSA by
that date. We compared the number and rankings of carriers from the
original SafeStat results with those obtained by adding in data for the
late-reported crashes. In addition, we reviewed the University of
Michigan Transportation Research Institute's studies of state crash
reporting to MCMIS to identify the impact of late reporting in
individual states on MCMIS data quality.
To assess the effect of completeness, we attempted to match all crash
records in the MCMIS crash file for crashes occurring from December
2001 through June 2004 to the list of motor carriers in the MCMIS
census file. In addition, we reviewed the University of Michigan
Transportation Research Institute's studies of state crash reporting to
MCMIS to identify the impact of incomplete crash reporting in
individual states on MCMIS data quality.
To assess the effect of accuracy, we reviewed a report by the Office of
Inspector General that tested the accuracy of electronic data by
comparing records selected in the sample with source paper documents.
In addition, we reviewed the University of Michigan Transportation
Research Institute's studies of state crash reporting to MCMIS to
identify the impact of incorrectly reported crashes in individual
states on MCMIS data quality.
While the limitations in the data adversely affect the ability of any
method to identify carriers that pose high crash risks, we determined
that the data were of sufficient quality for our use, which was to
assess how the application of regression models might improve the
ability to identify high-risk carriers over the current approach--not
to determine absolute measures of crash risk. Our reasoning is based on
the fact that we used the same data set to compare the results of the
SafeStat model and the regression models. Limitations in the data would
apply equally to both results. Methods to identify carriers that pose
high crash risk will perform more efficiently once the known problems
with the quality of state-reported crash data are addressed.
To understand what other researchers have found about how well SafeStat
identifies motor carriers that pose high crash risks, we identified
studies through a general literature review and by asking stakeholders
and study authors to identify high-quality studies. Studies included in
our review were (1) the 2004 study of SafeStat done by Oak Ridge
National Laboratory, (2) the SafeStat effectiveness studies done by the
Department of Transportation Office of Inspector General and Volpe
Institute, (3) the University of Michigan Transportation Research
Institute's studies of state crash reporting to FMCSA, and (4) the 2006
Department of Transportation Office of Inspector General's audit of
data for new entrant carriers.[Footnote 44] We assessed the methodology
used in each study and identified which findings are supported by
rigorous analysis. We accomplished this by relying on information
presented in the studies and, where possible, by discussing the studies
with the authors. When the studies' methodologies and analyses appeared
reasonable, we used those findings in our analysis of SafeStat. We
discussed with FMCSA and industry and safety stakeholders the SafeStat
methodology issues and data quality issues raised by these studies. We
also discussed the aptness of the respective methodological approaches
with FMCSA. Finally, we reviewed FMCSA documentation on how SafeStat is
constructed and assessments of SafeStat conducted by FMCSA.
[End of section]
Appendix III: Additional Results from Our Statistical Analyses of the
SafeStat Model:
This appendix contains technical descriptions and other information
related to our statistical analyses.
Overview of Regression Analyses:
To study how well statistical methods identify carriers that pose high
crash risks, we carried out a series of regression analyses. The safety
evaluation area values for the accident, driver, vehicle, and safety
management areas served as the independent variables to predict crash
risks.[Footnote 45] We used the state-reported crash data in MCMIS for
crashes that occurred during the 30 months preceding June 25, 2004, as
the dependent variable in each model. We used the results of the
SafeStat model run from June 25, 2004, to benchmark the performance of
the regression models with the crash records for the identified high-
risk carriers over the succeeding 18 months.
We matched the state-reported crashes that occurred from December 26,
2001, through June 25, 2004, to the carriers listed in
SafeStat.[Footnote 46] We checked our match of crashes for carriers
with those carriers used by FMCSA in June 2004 and found that the
reported numbers had changed for about 10,700 carriers in the
intervening 2 years. We found this difference even though we used only
crashes that occurred from December 26, 2001, through June 25, 2004,
and were reported to FMCSA before June 25, 2004. Because of this
difference in matched crashes, we recalculated the accident safety
evaluation area using our match of the crashes. This is discussed later
in more detail.
Using our recalculation of the accident safety evaluation area values
and the original driver, vehicle, and safety management safety
evaluation area values for the carriers, we fit a Poisson regression
model and a negative binomial regression model to the crash counts.
Both of these models are statistically appropriate for use when
modeling counts that are positive and integer valued. The two models
differ in their assumptions about the mean and variance. Whereas the
Poisson model assumes that the mean and the variance are equal, the
negative binomial model assumes the mean is not equal to the variance.
The crash data in MCMIS fit the assumptions of the negative binomial
distribution better than those of the Poisson.[Footnote 47]
We also tried to estimate zero-inflated Poisson and zero-inflated
negative binomial models with the SafeStat data. These models are
appropriate when the count values include many zeros, as is the case
with the values in this data set (because many carriers do not have
crash records). However, we could not estimate the parameters for these
models with the MCMIS data. We also considered using logistic
regression to model the carrier's odds of experiencing a crash.
However, the parameter estimates of the four safety evaluation area
values could not be estimated, so we did not use the results of this
model.[Footnote 48]
Finally, we used the results from the negative binomial model to assess
the expected carrier crash counts using the empirical Bayes estimate.
In safety applications, the empirical Bayes method[Footnote 49] is used
to increase the precision of estimates and correct for the regression-
to-mean bias.[Footnote 50] In this application, the empirical Bayes
method calculates a weighted average of the rate of crashes for a
carrier from the prior 30 months with the predicted mean number of
crashes from the negative binomial regression. This method optimizes
the identification of carriers with the highest number of future
crashes. This optimization of total crashes, however, resulted in the
identification of primarily the largest companies. The crash rate
(crashes per 1,000 vehicles per 18 months) was not as high for this
group as for the carriers placed by the SafeStat model in its A and B
categories.
Technical Explanation of the Negative Binomial Regression Model:
This section provides the technical details for the negative binomial
regression model fit to the SafeStat data. This section also explains
how we handled incomplete safety evaluation area data for carriers in
the regression model analyses.
The basic negative binomial probability distribution function for count
data is expressed as:
[See PDF for equation]
for The term represents the dispersion parameter. It is not assumed to
equal one, as in the Poisson distribution. The represents the crash
count for the motor carrier, and the represents the observed safety
evaluation areas. To formulate the negative binomial regression model
and control for differences in exposure to events among the carriers,
we can express the functional relationship between the safety
evaluation areas and the mean number of crashes as:
[See PDF for equation]
With complete data for a motor carrier, where none of the safety
evaluation area values are equal to missing, the regression model of
interest is as follows:
[See PDF for equation]
This equation models the log of the expected mean number of crashes for
each motor carrier using the four safety evaluation area values, but
most commercial motor companies listed in MCMIS do not have values for
all four safety evaluation areas.[Footnote 51] To account for this, it
is necessary to define four indicator variables. Let:
[See PDF for equations]
The indicator variables will be used as main effects in the negative
binomial regression model to indicate cases for which information is
available. The effect of the safety evaluation area will be measured by
the interaction of the indicator function with the safety evaluation
area value. This gives us the following model specification:
[See PDF for equation]
With this parameterization, the estimate for the mean rate of crashes
for a carrier with no safety evaluation area information is . For a
carrier with information for just the accident safety evaluation area,
the estimate for the mean number of crashes is . Note that the effect
for each safety evaluation area will include a coefficient times the
safety evaluation area value for the carrier plus an offset to the
intercept for the indicator term (the coefficient for the indicator
function).
We used a similar parameterization to formulate the Poisson regression
model.
Evaluation of Regression Models' Performance:
We estimated regression models using the same data FMCSA used in its
application of the SafeStat model on June 25, 2004, with one exception
for the accident safety evaluation area. For that area, we used our own
match of crashes to carriers for December 26, 2001, through June 25,
2004. The MCMIS data we received in June 2006 produced different totals
in the match of crashes to carriers for about 10,700 carriers. MCMIS
data change over time because crash data are added, deleted, or changed
as more information about these crashes is obtained. The discrepancies
in matching arose even though we used the identical time interval and
counted crashes only when the record indicated they had been reported
to FMCSA before June 25, 2004. Because of these discrepancies, it was
necessary to calculate the accident safety evaluation area values using
our match of crashes and then recalculate the SafeStat carrier scores
for June 25, 2004, using our accident safety evaluation area values and
the original driver, vehicle, and safety management safety evaluation
area values.[Footnote 52] We used our accident safety evaluation area
values and the original driver, vehicle, and safety management safety
evaluation area values in the regression model analysis.
Using the revised accident safety evaluation area values and FMCSA's
original driver, vehicle, and safety management safety evaluation area
values, the SafeStat model identified 4,989 carriers that pose high
crash risks. For each regression model, we input the safety evaluation
area data for the carriers in our analysis data set and used the
regression model to calculate the predicted mean number of crashes. We
then sorted the predicted scores and selected the 4,989 carriers with
the worst predicted values as the set of high-risk carriers identified
by the regression model. Next, we used MCMIS to determine the crash
history of these 4,989 carriers between June 26, 2004, and December 25,
2005, and compared the aggregate crash history with the aggregate crash
history of the carriers identified by the SafeStat model during the
same period of time.
The regression models do not categorize carriers by letter; the
regression models produce a predicted crash risk for each carrier. The
regression models make use of the safety evaluation area values, but
they differ from the SafeStat model in this respect.
The results show that a negative binomial regression model estimated
with the safety evaluation area values outperforms the current SafeStat
model in terms of predicting future crashes and the future crash rate
among identified carriers that pose high crash risks. (See table 3.)
That is, our negative binomial and Poisson models show 111 and 109
crashes per 1,000 vehicles per 18 months, respectively, compared with
the 102 crashes per 1,000 vehicles per 18 months estimated by the
current SafeStat model. The Poisson model is not as appropriate since
the crash counts for carriers have variability that is significantly
different from the mean number of crashes.[Footnote 53] The empirical
Bayes method optimizes the selection of future crashes; however, it
does so by selecting the largest carriers. The largest carriers have a
lower crash rate per 1,000 vehicles per 18 months than the carriers
that pose high crash risks identified by the SafeStat model or by the
negative binomial regression model. Since the primary use of SafeStat
is to identify and prioritize carriers for FMCSA and state compliance
reviews, the empirical Bayes method did not identify carriers with the
highest safety risk.
Table 3: Results for SafeStat Model and Regression Models:
Method: SafeStat category A & B;
Crash rate: 102;
Number of crashes in 18 months: 10,076;
Number of vehicles: 98,619.
Method: Negative binomial;
Crash rate: 111;
Number of crashes in 18 months: 19,580;
Number of vehicles: 175,820.
Method: Poisson;
Crash rate: 109;
Number of crashes in 18 months: 21,532;
Number of vehicles: 198,396.
Method: Empirical Bayes;
Crash rate: 59;
Number of crashes in 18 months: 56,705;
Number of vehicles: 965,070.
Source: GAO analysis of FMCSA data.
Note: As discussed in the text, the zero inflated Poisson, the zero
inflated negative binomial, and the logistic regression approaches did
not provide useful results.
[End of table]
FOOTNOTES
[1] There are four safety evaluation areas--accident, driver, vehicle,
and safety management. They are used by the SafeStat model to assess a
carrier's safety. See the background section for a description of these
four areas. SafeStat is built on a number of expert judgments rather
than using statistical approaches, such as a regression model.
[2] Negative binomial regression is often used to model count data
(e.g., crashes). The results from this regression model can be
interpreted as the estimated mean number of crashes per carrier.
[3] The 9 percent improvement is in crash rate per 1,000 vehicles over
an 18-month period.
[4] The goal of this initiative is to develop an optimal operational
model that will allow FMCSA to focus its resources on improving the
safety performance of high-risk operators.
[5] We applied the SafeStat model to retrospective data. Because of
changes to the MCMIS crash file over the past 2 years, our number does
not correspond exactly to the number of carriers identified by FMCSA as
high risk on June 25, 2004. Had all crash data been reported within 90
days of when the crashes occurred, 182 of the carriers identified by
SafeStat as highest risk would have been excluded (because other
carriers had higher crash risks), and 481 carriers that were not
originally designated as posing high crash risks would have scored high
enough to be considered high risk, resulting in a net addition of 299
carriers.
[6] A reportable crash is one that meets both a vehicle and a crash
severity threshold. Generally, for a crash to be reported, it must
involve a truck with a gross vehicle weight rating of over 10,000
pounds; a bus with seating for at least nine people, including the
driver; or a vehicle displaying a hazardous materials placard.
Reportable accidents involve a fatality, an injury requiring transport
to a medical facility for immediate medical attention, or towing
required because the vehicle sustained disabling damage.
[7] This figure is for 2002, the most recent date for which data is
available.
[8] This includes an unidentified number of carriers that are
registered but are no longer in business.
[9] FMCSA completed 15,626 compliance reviews in 2006. The number of
companies reviewed was less because some carriers received more than 1
compliance review.
[10] Acute violations are violations so severe that FMCSA requires
immediate corrective actions by a motor carrier regardless of the
carrier's overall safety status. An example of an acute violation is a
carrier's failing to implement an alcohol or drug testing program for
drivers. Critical violations are serious, but less severe than acute
violations, and most often point to gaps in carriers' management or
operational controls. For example, a carrier may not maintain records
of driver medical certificates.
[11] Severe violations are violations of hazardous materials
regulations. Level I violations require immediate corrective actions.
An example of a level I violation is offering or accepting a hazardous
material for transportation in an unauthorized vehicle. Level II
violations indicate a breakdown in the management or operational
controls of the facility. An example of a level II violation is failing
to train hazardous materials employees as required.
[12] Minimum requirements in this context mean that the carrier has
enough safety data to receive a rating. Usually, the safety data are
associated with adverse safety events. However it is possible for a
carrier to have enough roadside inspections, even if none of the
inspections resulted in violations, to qualify for a driver and vehicle
safety evaluation area score.
[13] The 9 percent improvement is in the crash rate per 1,000 vehicles
over an 18-month period.
[14] Applying the SafeStat model to June 2004 data identifies 4,989
carriers as high risk (categories A or B). Using 10,000 randomly
selected samples of 4,989 carriers and considering the crashes that
these carriers had between June 2004 and December 2005, we found that
the crash rate per 1,000 vehicles in the ensuing 18 months was 83
percent higher among the carriers identified by the SafeStat model than
among the randomly selected carriers.
[15] Ken Campbell, Rich Schmoyer, and Ho-Ling Hwang, Review of the
Motor Carrier Safety Status Measurement System (SAFESTAT), Oak Ridge
National Laboratory, Final Report, October 2004. See appendix I for a
more detailed discussion of the findings from this report.
[16] This occurs because data were added, deleted, or modified as more
information became known over time. See appendix III for a more
detailed discussion.
[17] The threshold could be increased or decreased to align with the
resources that FMCSA and its state partners have available to perform
compliance reviews. As discussed earlier, FMCSA and its state partners
select carriers for these reviews because they pose high crash risks
and for other reasons.
[18] The carriers identified as high risk by SafeStat had a total of
98,619 vehicles while those identified by the negative binomial
regression model had 175,820 vehicles. The identification of larger
sized companies on average by the negative binomial regression model is
how a 9 percent increase in the crash rate translated into 9,500
additional crashes.
[19] FMCSA can use the current safety evaluation area values in
SafeStat and the number of state-reported crashes for each carrier in
the 30 preceding months in the negative binomial regression model.
[20] Federal Motor Carrier Safety Administration Compliance Review
Workgroup, Phase II Final Report: Proposed Operational Model for FMCSA
Compliance and Safety Programs Report, February 2005.
[21] Oak Ridge National Laboratory statistically measured the weights
for the safety evaluation areas and estimated the accident safety
evaluation area should have a weight of 57 in the SafeStat model
formula. This compares with the present weight of 2 that SafeStat gives
the accident safety evaluation area. Ken Campbell, Rich Schmoyer, and
Ho-Ling Hwang, Review of the Motor Carrier Safety Status Measurement
System (SAFESTAT), Oak Ridge National Laboratory, Final Report, October
2004.
[22] We expect to issue a report shortly that provides additional
discussion of FMCSA's initiative to identify and take action against
carriers that are egregious safety violators.
[23] Revisions to SafeStat are exempt from notice and comment under the
Administrative Procedure Act if they relate to FMCSA's internal
practices and procedures.
[24] SafeStat does not consider carriers with fewer than three moving
violations.
[25] Carriers with one or zero state-reported crashes do not receive an
accident safety evaluation area score unless the recordable accident
indicator is available from a recent compliance review. Carriers with
two or fewer driver inspections and two or fewer moving violations do
not receive a driver safety evaluation area score unless the driver
review indicator is available from a recent compliance review. Carriers
with two or fewer vehicle inspections do not receive a vehicle safety
evaluation area score unless the vehicle review indicator is available
from a recent compliance review. In the data we reviewed, almost 2
percent of the carriers had undergone a compliance review within the 18
months prior to the SafeStat run on June 25, 2004.
[26] For another assessment of data quality, see Office of Inspector
General, Improvements Needed in the Motor Carrier Safety Status
Measurement System, U.S. Department of Transportation, Report MH-2004-
034, 2004.
[27] These 182 carriers were no longer in the worst 25 percent for the
accident safety evaluation area after the addition of the late-reported
crashes.
[28] Part of the improvement in timeliness of reporting for the most
recent year is that an unknown number of crashes that occurred in 2005
had still not been reported as of June 2006, the date we obtained these
data.
[29] The University of Michigan Transportation Research Institute's
reports on state crash reporting can be found at
http://www.umtri.umich.edu. State reports issued by the University of
Michigan Transportation Research Institute cover California, Florida,
Illinois, Iowa, Louisiana, Maryland, Michigan, Missouri, Nebraska, New
Jersey, New Mexico, North Carolina, Ohio, and Washington. We included
all of these reports in our review.
[30] GAO, Highway Safety: Further Opportunities Exist to Improve Data
on Crashes Involving Commercial Motor Vehicles, GAO-06-102 (Washington,
D.C.: Nov. 18, 2005).
[31] David Madsen and Donald Wright, Volpe National Transportation
Systems Center, An Effectiveness Analysis of SafeStat (Motor Carrier
Safety Status Measurement System), Paper No. 990448, November 1998 and
John A. Volpe National Transportation Systems Center, Motor Carrier
Safety Assessment Division, SafeStat Effectiveness Study Update, March
2004.
[32] Volpe included only carriers with two or more crashes and/or three
or more inspections during the preceding 30 months, and/or an
enforcement action within the past 6 years, and/or a compliance review
within the previous 18 months. This is consistent with the SafeStat
minimum event requirements.
[33] Office of Inspector General, Improvements Needed, 2004.
[34] Ezra Hauer, Douglas Harwood, and Michael Griffith, The Empirical
Bayes Method for Estimating Safety: A Tutorial. Transportation Research
Record 1784, National Academies Press, 2002, 126-131.
[35] Thomas Corsi, Richard Barnard, and James Gibney, Motor Carrier
Industry Profile: Linkages Between Financial and Safety Performance
Among Carriers in Major Industry Segments, Robert H. Smith School of
Business at the University of Maryland, October 2002.
[36] The American Trucking Associations is a membership organization
with a mission to serve and represent the interests of the trucking
industry.
[37] The Annual Report Form M is required only for class 1 or class 2
carriers that have revenue exceeding $3 million for 3 consecutive
years.
[38] Brenda Lantz and David Goettee, An Analysis of Commercial Vehicle
Driver Traffic Conviction Data to Identify Higher Safety Risk Motor
Carriers, Upper Great Plains Transportation Institute and FMCSA, 2004.
Brenda Lantz, Development and Implementation of a Driver Safety History
Indicator into the Roadside Inspection Selection System, FMCSA, April
2006.
[39] Correlation = 0.085. (See FMCSA, Development and Implementation of
a Driver Safety History Indicator into the Roadside Inspection
Selection System, April 2006, 14).
[40] The Office of Inspector General used MCMIS data to estimate a
standard deviation for days to report a crash and then divided the
standard deviation by the average number of days. This number was
multiplied by 100 to derive the coefficient of variation. The obtained
value of about 77 indicates substantial variability relative to the
average number of days to report a crash.
[41] The General Estimates System collects all types of information
from all types of crashes. It is based on a nationally representative
probability sample from the estimated 6.4 million police-reported
crashes that occur annually. While the crash eligibility definitions
are not strictly comparable, the number of crashes reported to MCMIS is
below the lower bound for the 95 percent confidence interval around the
estimated total number of crashes for large trucks in 2004.
[42] We obtained crash data for this period that were reported to FMCSA
through June 2006. This allowed us to obtain data on late-reported
crashes for the July 2004 through December 2005 period.
[43] We drew 10,000 simple random samples of 4,989 carriers (the number
of carriers that SafeStat identified as being at highest risk for
crashes when we recalculated it) from the list of all carriers in the
MCMIS master file used by SafeStat on June 25, 2004, and for each
sample we calculated how many crashes the selected carriers reported to
MCMIS between June 26, 2004, and December 25, 2005.
[44] Campbell, Schmoyer, and Hwang, Review of The Motor Carrier Safety
Status Measurement System (SAFESTAT), 2004; U.S. DOT Office of
Inspector General, Improvements Needed In the Motor Carrier Safety
Status Measurement System, 2004; Madsen and Wright, U.S. DOT-Volpe
National Transportation Systems Center, An Effectiveness Analysis of
SafeStat, 1998; John A. Volpe National Transportation Systems Center,
SafeStat Effectiveness Study Update, 2004. University of Michigan
Transportation Research Institute MCMIS State Reports; U.S. DOT Office
of Inspector General, Significant Improvements in Motor Carrier Safety
Program Since 1999 Act But Loopholes For Repeat Violators Need Closing,
2006.
[45] In addition to the safety evaluation area scores, we included
indicator variables to flag any missing safety evaluation area scores.
[46] We used the carrier's Department of Transportation number recorded
in the crash record to match to the carrier's Department of
Transportation number listed in the SafeStat summary table.
[47] We checked this by estimating the mean and variance of the crashes
for the population of all carriers and determined that they were
significantly different.
[48] The coefficients in the model could not be reliably estimated (the
maximum likelihood of the model did not converge).
[49] Hauer, Harwood, Council, and Griffith, Estimating Safety by the
Empirical Bayes Method: A Tutorial, 2001.
[50] In the context of crashes, we wish to "treat" the most dangerous
companies with a compliance review to make them safer. But, crashes are
distributed with a fair degree of randomness. A company selected for a
compliance review may have suffered an atypical random grouping of
accidents in the preceding months. With or without a compliance review,
it is likely that the random grouping will not exist next year, and the
crash figures will improve. Statistical methods seek to control for
this regression-to-mean bias in order to better identify the effect of
a compliance review on a company's safety.
[51] A carrier has to have two or more reported crashes in the past 30
months to receive an accident safety evaluation area value. A carrier
has to have three or more roadside inspections to receive a driver or
vehicle safety evaluation area value. A driver has to have had a
compliance review in the past 18 months to receive a safety management
safety evaluation area value. There are other ways a carrier can
receive a value for one of these four safety evaluation areas, refer to
the description of each one provided in the Background.
[52] Our calculation of the accident safety evaluation area differed
slightly from that used by FMCSA. We did not add 1 to the severity
weights for crashes with an associated hazardous materials release due
to the rarity of this event.
[53] The equality of the variability in the number of crashes to the
average number of crashes is an assumption of the Poisson regression
model. This assumption does not hold for the MCMIS data we analyzed.
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site (www.gao.gov). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site.
To have GAO e-mail you a list of newly posted products every afternoon,
go to www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125
Washington, D.C. 20548:
Public Affairs:
Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548:
