FMCSA Drops IRT in Proposed Changes in Identifying Unsafe Motor Carriers
The Federal Motor Carrier Safety Administration announced long-awaited proposed changes to its Safety Measurement System and rejected the recommendation of the National Academy of Sciences to implement a new statistical model using Item Response Theory, or IRT.
FMCSA first implemented SMS in 2010, as part of the broader Compliance, Safety, Accountability (CSA) program (which at the time was known as Comprehensive Safety Analysis.) SMS replaced the previous SafeStat system.
The SMS uses data from roadside inspections, crash reports, and investigations to help FMCSA and its state partners identify and prioritize for intervention the motor carriers that pose the greatest risk to safety. Through the SMS, carriers are scored in seven categories known as BASICs. Carriers receive percentile scores in each of the BASICs, indicating how they are performing relative to their peers.
From the beginning, there were complaints that the system was not an accurate representation of motor carrier safety because it gave no credit for “clean” inspections or for crashes that were clearly not the motor carrier’s fault.
In light of these and other criticisms, in 2017, Congress demanded FMCSA commission an independent study of SMS. The agency engaged the National Academy of Sciences (NAS) to conduct that study, which it completed that year.
NAS recommended that FMCSA consider adopting a better statistical model as the backbone of the system: Item Response Theory, or IRT.
The agency created an IRT model using inspection data from FMCSA’s MCMIS database. In the Federal Register Notice, the agency said this IRT modeling work “revealed many limitations and practical challenges with using an IRT model. As a result, FMCSA has concluded that IRT modeling does not perform well for the agency’s use in identifying motor carriers for safety interventions, and therefore, does not improve overall safety.”
However, the agency did note that during the development and testing of the IRT model, it gained valuable insight and concluded that reorganizing the BASICs, now called “safety categories,” could make it easier for FMCSA and motor carriers to pinpoint and address safety issues.
FMCSA said its proposed changes aim to better identify the companies needing the most intervention, and also will help companies better understand how to use this data to drive safer behaviors.
Some of the proposed changes include:
- Reorganizing the SMS’s safety categories (currently known as “BASICs”).
- Organizing roadside violations into violation groups for prioritization purposes.
- Simplifying violation severity weights.
- Adjusting some of the Intervention Thresholds that identify companies for possible intervention.
- Changes aimed at comparing similar motor carriers to each other.
Bye-Bye, BASICs
FMCSA proposes reorganizing the Controlled Substances/Alcohol, Unsafe Driving, and Vehicle Maintenance safety categories as described below. FMCSA also proposes to segment the Driver Fitness and HM Compliance safety categories to account for differences in carrier operations.
The new safety categories (No longer called BASICs) would be:
- Unsafe Driving
- Crash Indicator
- Hours of Service (HOS) Compliance
- Vehicle Maintenance
- Vehicle Maintenance: Driver Observed
- HM (Hazardous Materials) Compliance
- Driver Fitness
The Controlled Substances/Alcohol has the fewest violations of any BASIC. The agency found that controlled substances and alcohol violations were strongly associated with the Unsafe Driving BASIC, so they rolled the drug and alcohol violations into the new Unsafe Driving safety
category.
In addition, FMCSA’s analysis found that violations for operating while under an out-of-service order belong in the new Unsafe Driving safety category. Currently, SMS places these types of violations across multiple BASICs based on the underlying OOS violation.
Vehicle Maintenance is the largest BASIC in terms of both the number of violation identifiers and the number of violations cited during inspections. The agency said its analysis showed that breaking this category into two separate categories would provide greater specificity to help carriers improve and enforcement officials to conduct targeted investigations.
Therefore, Vehicle Maintenance violations would be divided into two separate categories: Vehicle Maintenance: Driver Observed, which includes violations that may be identified by a driver during a pre- or post-trip inspection and/or while operating the vehicle; and Vehicle Maintenance, which includes all other vehicle maintenance violations.
Better Comparisons of Fleets
The agency also proposes segmenting some of the safety categories to make peer comparisons more meaningful.
In the Driver Fitness BASIC, carriers that operate straight trucks and similar vehicles have much higher violation rates than motor carriers that operate combination vehicles. Segmenting the Driver Fitness BASIC into Straight and Combination segments more effectively identifies carriers with higher crash rates in both segments.
The current HM Compliance BASIC compares cargo tank carriers to non-tank carriers, but these carriers have fundamentally different operations. FMCSA’s analysis found that segmenting carriers as Cargo Tank carriers and Non-Cargo Tank carriers in the HM Compliance safety category, in conjunction with adjusting the HM Compliance threshold from the 80th to 90th percentile, identifies a group of carriers that has an HM inspection violation rate that is 2% higher and an HM A/C violation rate that is 46% higher than carriers identified for intervention under the current HM Compliance BASIC.
The proposal also would consolidate the universe of violations that filter into the system. According to the notice, SMS currently accounts for 973 separate violations. The revised system will group similar violations together. For example, the 73 separate hours-of-service violations will be combined into a total of nine violations. (A complete list of the new violations is included in the agency’s docket.)
Another significant revision would be a simplification of violation severity weights. Under the current system, violations are assigned a severity weight of 1-10. Many have criticized these over the years as overly subjective. Instead, the agency is proposing to assign each violation group a weight of either 1 or 2. Out-of-service violations and violations in the Unsafe Driving category that are disqualifying will receive a weight of 2, and all other violations in the system will have a weight of 1.
How Would Your Fleet Fare Under New System?
“While the FMCSA’s proposed revisions to SMS are not nearly as substantial as they would have been had the agency adopted an IRT model, they still have the potential of significantly impacting the way motor carriers are prioritized for enforcement,” said Brandon Wiseman of Trucksafe Consulting in his analysis of the proposal.
A new website, the Compliance Safety Accountability Prioritization Preview, which is now live, is the first phase of planned updates to the agency’s SMS. Motor carriers can visit the website to preview how their data would appear under the proposed changes. Companies are encouraged to preview these results and submit feedback on the proposed changes to FMCSA at the Federal Register website. Other users will be able to view sample pages.
Learn More and Comment:
The proposed changes are explained in a Federal Register notice (2023-02947). Feedback on the proposed changes must be submitted to the Federal Docket Management System (https://www.regulations.gov/), Docket ID Number: FMCSA-2022-0066. Comments are due by May 16, 2023.
FMCSA will hold four public online question and answer webinars, during which participants will be able to ask questions about the preview and proposed changes and receive real-time answers, time permitting. Registration is required. Visit the CSA Prioritization Preview website for more information.
Why Did FMCSA Reject the IRT Model?
The agency outlined five problems in its modeling of item response theory:
- IRT is heavily biased towards identifying smaller carriers that have few inspections with violations and limited on-road exposure to crash risk. When the safety event groups and data sufficiency standards used in SMS were applied to the IRT model, IRT produced similar results to SMS.
- IRT does not use vehicle miles traveled or the number of power units in a fleet to adjust for differences in on-road exposure in the Unsafe Driving BASIC. As a result, IRT identified carriers with much lower crash rates in that BASIC compared to SMS.
- IRT modeling is not readily understandable by most stakeholders or the public. While SMS results can be reproduced and explained using simple math, IRT requires an advanced understanding of statistical modeling and analysis.
- A motor carrier cannot independently compute its IRT results. A carrier would not be able to identify how specific violations or areas of regulatory noncompliance impacted its prioritization status or how it could improve its status.
- IRT’s is too slow. The IRT model takes four weeks to run, compared to two days for SMS. The long runtime would make it difficult to make even minor changes to the system, according to the FMCSA.
Original article via: truckinginfo.com
Related Articles: