{"id":"3dd7b493-5d0f-4786-b731-19afa75550ba","arxiv_id":"2505.12545","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TrafficSafe fine-tunes LLMs on textualized multi-modal crash reports and claims a 41.7% F1 gain over baselines, then uses sentence-level Shapley attribution to interpret predictions and guide safety interventions.","lead":"This paper fine-tunes Llama 3.1 models on 58,903 textualized traffic crash reports to predict crash severity, injury count, and crash type, reporting a 41.7% average F1 improvement over tabular machine learning baselines. It also introduces a sentence-level Shapley attribution method that links model confidence to contributing factors such as alcohol-impaired driving and work-zone presence, to support conditional risk analysis and data collection decisions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 41.7% F1 gain is confounded: baselines are not documented to use the same feature set or the same class-imbalance handling as TrafficSafe, and evaluation on rebalanced test sets can depress baseline F1 independently of text reasoning.","rationale":"Good-faith reading: the paper's contribution is a new textualized crash dataset, a LoRA fine-tuning recipe, and a Shapley-based attribution method, and the headline empirical claim is the 41.7% F1 improvement together with the confidence-as-risk claim. The strongest internal support is that TrafficSafe is evaluated on held-out splits with two model sizes and two states, and the confusion matrices show more balanced predictions than baselines. However, the comparison is not controlled: the baseline feature set is absent from the methods, and the training and evaluation class distributions differ between TrafficSafe and baselines. Because F1 is highly sensitive to the predicted class distribution, the observed Table 1 pattern, high baseline accuracy with very low F1, is consistent with baselines predicting the majority class on a balanced test set. That would make the reported improvement an artifact of protocol asymmetry rather than text reasoning. This concern is concrete and testable by re-running baselines with matched features and class balancing. I do not see an internal contradiction that would demand rejection; the paper is reproducible in principle, and the zero-shot generalization results provide some independent support. But the central comparison is currently unverified, so the CONDITIONAL verdict should stand.","tokens_in":26307,"tokens_out":6832,"duration_ms":77813,"concrete_test":"Retrain every baseline (at least XGBoost and RandomForest) on the exact feature set encoded in the TrafficSafe prompts, including the satellite-derived annotations, and with the same uniform or class-balanced sampling used in Section 5.2.3; then re-evaluate on the un-rebalanced original test sets from Section 5.2.4. If the average F1 gap over Table 1 falls below a pre-registered threshold, for example 10 points, or loses statistical significance across random seeds, then the 41.7% improvement claim is not attributable to LLM text reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, that fine-tuning Llama 3.1 on textualized prompts yields a 41.7% average F1 improvement over tabular baselines (Table 1, Section 3.3), is not established because the comparison is not controlled on two variables that directly affect F1. First, Section 5.1.2 shows TrafficSafe prompts contain GPT-4o-derived satellite-image annotations and narrative descriptions of driver behavior, vehicle movement, and occupant details; Section 5.2.6 lists baseline algorithms and BayesSearchCV tuning but never states what feature set the baselines received. If baselines were trained on a subset of the raw HSIS fields, part of the gap is additional information, not LLM reasoning. Second, and more specifically, Section 5.2.3 says TrafficSafe LLM is trained with uniform class sampling, while Section 5.2.4 rebalances the validation and test sets (Washington: 1,428 S1 records removed; Illinois: 'all subsets' balanced). No equivalent class-imbalance handling is reported for baselines. On a balanced test set, a model trained on natural class priors will systematically predict the majority class and receive a low macro-F1 despite decent accuracy; the table's pattern, with baselines achieving Type accuracy around 0.74 but F1 around 0.27, is exactly this artifact. The reported 97.61% fatal-prediction precision also comes from these rebalanced test sets, so it is not a population risk estimate. The claims may survive, but currently the 41.7% gain cannot be attributed to text-based reasoning.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TrafficSafe, a framework that textualizes multi-modal traffic crash data (HSIS records, satellite images, driver and vehicle information) into natural-language prompts and fine-tunes Llama 3.1 (8B and 70B) with LoRA and task-specific special tokens for three prediction targets: number of injuries, injury severity, and crash type. The authors report that TrafficSafe achieves a 41.7% average F1-score improvement over seven tabular baselines (Section 3.3, Table 1), that higher confidence scores correlate with higher accuracy and that fatal-crash precision reaches 97.61% when confidence exceeds 0.6 (Figures 3e-3i), and that a sentence-level Shapley attribution method (TrafficSafe Attribution) supports conditional risk analysis, with findings such as alcohol-impaired driving and work-zone combinations being highest-risk. The paper also reports zero-shot generalization to North Carolina and Maine and provides code and processed data examples via GitHub.","tokens_in":26632,"tokens_out":4745,"duration_ms":48984,"significance":"The paper addresses a timely and societally relevant problem, and the idea of reframing crash prediction as a text-based reasoning task with interpretable attributions is a potentially valuable contribution to traffic safety analytics. The dataset construction (58,903 records, 12.74 million words), the use of real-world HSIS data, and the public availability of code and data examples are concrete assets for reproducibility. If the comparative claims and the confidence-as-risk calibration are established after the evaluation confounds are addressed, the framework could offer a meaningful advance over conventional tabular crash prediction. The attribution analysis, if carefully framed, could also be useful for guiding data collection. However, the central quantitative claims currently rest on an evaluation protocol whose controls are not fully documented.","major_comments":[{"comment":"The reported F1 improvements are measured on test sets that were rebalanced by removing 1,428 S1 records in the Washington test/validation split and by balancing all subsets in Illinois, while Section 5.2.6 does not report any equivalent class-imbalance handling for the tabular baselines. Since TrafficSafe was trained with uniform class sampling (Section 5.2.3) and the baselines are not described as using class weights or resampling, the comparison is not controlled for class-imbalance handling; the paper should report results on the original unmodified held-out test distribution as well, or specify and justify the same rebalancing procedure for all methods.","section":"Section 5.2.4 and Table 1"},{"comment":"The baseline feature sets are never specified. The TrafficSafe prompts contain GPT-4o-derived satellite-image annotations and narrative descriptions of driver behavior, vehicle movement, and occupant details, but the paper does not state which subset of raw HSIS fields the RandomForest, XGBoost, CatBoost, and other baselines received as input. Without this information, the 41.7% F1 gain cannot be attributed to text-based reasoning rather than to the inclusion of additional data modalities; the authors should document the exact features used for each baseline and, ideally, add a text-based baseline (e.g., TF-IDF with a linear classifier trained on the same textualized prompts) to isolate the contribution of the LLM.","section":"Section 5.2.6 and Section 5.1.2"},{"comment":"The claimed 97.61% precision for fatal predictions when confidence exceeds 0.6, and the related statement in the Discussion that this reflects 'real-world occurrence risk', are computed on the rebalanced test sets described in Section 5.2.4, where 1,428 majority-class S1 records were removed from the Washington split and the Illinois subsets were balanced. Because the test-set class priors were intentionally altered, this precision value is not a population-level risk estimate; the paper should either recompute the confidence-calibration analysis on an unmodified representative test set or explicitly state that the number is conditional on the rebalanced evaluation distribution.","section":"Section 4 and Figure 3i"},{"comment":"The conditional-risk findings, such as 'alcohol-impaired driving is the leading factor in severe crashes' and 'aggressive and impairment-related behaviors have nearly twice the contribution for severe crashes', are derived from the model's confidence score used as a proxy for risk and from Shapley attributions of that confidence (Equations 7-8). These are statements about the model's internal attribution and have not been validated as population causal risk factors; the abstract and Section 3.4 should be rephrased to make clear that these are model-attributed contributions to predicted confidence rather than established real-world risk factors, or the confidence-as-risk assumption should be validated on held-out outcome data.","section":"Section 3.4 and Abstract"}],"minor_comments":[{"comment":"The metrics in Equations (2)-(4) are defined for binary classification, but the tasks (Injury, Severity, Type) are multi-class; the paper should specify how precision and F1 are averaged across classes and whether the reported numbers are macro-, micro-, or class-weighted.","section":"Section 5.2.5"},{"comment":"No confidence intervals, error bars, or statistical significance tests are reported for the F1 differences in Table 1; reporting a bootstrap or repeated-seed analysis would strengthen the comparison, especially for the rare fatal and serious-injury classes.","section":"Section 3.3"},{"comment":"Equation (8) defines the inference-stage Shapley value using the LLM's predicted probability, but the text does not specify which class probability is used (e.g., the probability of the predicted special token) or how the stratified sampling estimator of Reference 49 is configured; a brief pseudocode or explicit estimator formula would improve reproducibility.","section":"Section 5.3.3"},{"comment":"The GPT-4o textualization process (satellite-image annotation and prompt generation) should explicitly state that this step is done before the train/test split and uses no label information, to reassure readers that the test prompts are not influenced by the training labels.","section":"Section 5.1.2"},{"comment":"The confusion matrices show that baselines collapse onto the majority classes; reporting classwise F1-scores for the rare classes (e.g., 'serious injury' and 'fatal') in the main text would help readers assess whether the TrafficSafe improvement is driven mainly by rare-class recall.","section":"Extended Data Figure 3 and Figure 3c/3d"}],"recommendation":"major_revision","confidential_remarks":"The paper has a plausible and interesting central idea, and the availability of code and processed data is a strength. The main barrier to publication is the evaluation protocol: the baseline feature sets are undocumented, the test-set rebalancing is not mirrored in the baselines, and the risk-level framing goes beyond what the rebalanced evaluation supports. I recommend that the editor request a revision that addresses these controls before further consideration, as the current 41.7% improvement claim is not yet established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is worth a look for its dataset, but the headline 41.7% F1 gain is not yet supported. The genuinely new thing is the TrafficSafe Event Dataset: 58,903 textualized crash-event prompts from Washington and Illinois, with code and processed data on GitHub. The sentence-level Shapley attribution for LLM prompts is a sensible extension, and the zero-shot evaluations on North Carolina and Maine add value.\n\nWhere it goes soft is the comparison itself. The LLM is trained with uniform class sampling and evaluated on rebalanced test sets (Washington removes 1,428 severity-S1 records; Illinois balances all subsets), while the tabular baselines get no documented equivalent. On a balanced test set, a model trained on natural class priors will show low F1 even with decent accuracy, so part of the gap is an artifact of the protocol. Second, the prompts include GPT-4o satellite-image annotations and narrative descriptions, but the baselines' feature set is never specified. The comparison may be giving the LLM more information, not just better reasoning. There is also no text-based baseline.\n\nThe 97.61% fatal-precision figure comes from the same rebalanced sets, so it is not a population risk estimate. And using model confidence as 'risk level' in the conditional analyses is circular, even if the identified factors (alcohol, work zones, aggressive driving) match domain knowledge.\n\nThe paper is transparent about its limitations, and the methods are detailed enough to reproduce in principle. I would send it to review—the dataset and framework deserve referee time—but the performance comparison needs serious rework: identical feature sets, identical class-imbalance handling, or results on the natural test distribution, plus a text-based baseline. As it stands, the central claim is likely inflated.","headline":"This paper is worth a look for its dataset, but the headline 41.7% F1 gain is not yet supported by the evaluation as described.","tokens_in":27162,"tokens_out":4075,"would_cite":false,"duration_ms":40182,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned LLMs beat tabular models at predicting crash severity, injury counts, and crash type, by reasoning over crash reports turned into text.","keywords":["traffic crash prediction","large language models","fine-tuning","crash severity","feature attribution","Shapley values","conditional risk analysis","text-based reasoning"],"falsifier":"A direct check: retrain the seven tabular baselines on the original feature set, then evaluate them on the exact same rebalanced test sets that TrafficSafe uses, after adding the satellite-image-derived and textual features that the prompts contain. If the baselines' F1-scores rise to near the LLM's levels, the 41.7% gain is attributable to added information or evaluation-set changes rather than to LLM text reasoning.","tokens_in":26084,"feed_emoji":"🚗","tokens_out":3099,"duration_ms":26191,"temperature":0.7,"pith_summary":"This paper tries to show that reframing traffic-crash outcome prediction as a text-reasoning task—rather than feeding tabular features into classical machine-learning classifiers—produces substantially more accurate and more interpretable predictions. The authors build a large textual dataset from real crash records, fine-tune Llama 3.1 models to predict injury count, severity, and crash type, and report an average 41.7% F1-score improvement over seven tabular baselines. They also claim the model's confidence score is a calibrated risk proxy, reaching 97.61% precision for fatal-crash predictions when confidence exceeds 0.6. If true, this would give traffic-safety agencies a single pipeline that both forecasts crash outcomes and attributes risk to specific contributing factors, supporting targeted interventions.","feed_headline":"Fine-tuned LLMs outpredict tabular models on crash severity","feed_subtitle":"Text reasoning over 58,903 crash reports lifts F1 by 41.7% and flags fatal cases with 97.61% precision.","key_machinery":"The load-bearing machinery is the TrafficSafe Event dataset plus the supervised fine-tuning of Llama 3.1 with special output tokens. Raw multi-modal crash data (crash reports, infrastructure, vehicle, person, and satellite-image-derived text) is converted through an AI-expert cooperative textualization process into natural-language prompts. The prediction targets are added to the model's vocabulary as special tokens, and the model is fine-tuned with LoRA to generate these tokens, turning classification into next-token generation. The second key mechanism is TrafficSafe Attribution, which computes Shapley values over sentence-level feature groups to assign contributions that sum to the model's confidence score, enabling conditional risk analysis.","core_discovery":"The central claim is that a language model fine-tuned on textualized multi-modal crash records (the TrafficSafe Event dataset, 58,903 prompts, about 12.74 million words) outperforms standard tabular machine-learning baselines across three crash-outcome prediction tasks: number of injuries, injury severity, and crash type. The paper reports an average F1-score improvement of 41.7% over the baselines, with the largest gains on crash-type prediction (e.g., F1 of 0.759 in Washington versus at most 0.329 for the best baseline). The paper further claims that the model's confidence score is a reliable indicator of risk: accuracy rises with confidence, and for fatal-crash predictions precision reaches 97.61% when confidence exceeds 0.6. The paper also introduces TrafficSafe Attribution, a Shapley-based sentence-level attribution method, and uses it to identify alcohol-impaired driving as the leading contributor to severe crashes, with work-zone plus alcohol combinations and multi-factor co-occurrence markedly raising predicted risk.","pith_inferences":["The 41.7% F1 improvement may partially reflect that the baselines were evaluated with class-imbalanced data, while the LLM's test sets were rebalanced by removing 1,428 S1 (no-apparent-injury) records for Washington; a fair apples-to-apples comparison would rerun baselines on the identical rebalanced test sets with the same information content.","A testable extension would be to probe whether the Shapley-based attributions track the causal contributions of factors (e.g., alcohol, work zones) or merely the statistical associations learned from the training corpus; controlled counterfactual generations with edited prompts could separate these.","The confidence-precision calibration claim suggests a promising practical safeguard: refusal-to-predict or deferral-to-human-review rules based on confidence thresholds could be embedded in real-time crash-response or safety-policy workflows, and could be validated prospectively on new crash years."],"forward_implications":["Traffic agencies could deploy a single fine-tuned LLM to predict injury counts, severity, and crash type from text-based incident descriptions, replacing or complementing tabular classifiers.","The confidence score could be used as a decision threshold: a fatal-crash prediction with confidence above 0.6 corresponds to 97.61% precision, enabling prioritization of high-risk cases for review or intervention.","Sentence-level attribution could be used to run what-if risk analyses for rare or unseen conditions (e.g., hypothetical alcohol-plus-work-zone scenarios), guiding policy changes even where crash data is sparse.","Training-stage attribution identifies which data components (unit information, event information) most improve model performance, providing a data-collection priority list for states building crash datasets.","The framework's zero-shot generalization results (F1 around 0.52-0.54 on unseen North Carolina and Maine data) suggest the model captures transferable crash patterns across states."],"supporting_citations":[{"why":"HSIS is the primary data source for crash, infrastructure, vehicle, and person records.","marker":"[30]"},{"why":"Google Maps satellite images are textualized to supplement infrastructure and environmental context.","marker":"[31]"},{"why":"LoRA is the fine-tuning method used to adapt Llama 3.1 efficiently, providing the resource-feasibility of the approach.","marker":"[32]"},{"why":"Llama 3.1 is the base model family that is fine-tuned to create TrafficSafe LLM.","marker":"[24]"},{"why":"Shapley theory is the mathematical foundation for sentence-level feature attribution in TrafficSafe Attribution.","marker":"[43]"},{"why":"XGBoost is one of the strongest tabular baselines the paper must beat, representing the standard gradient-boosting approach.","marker":"[41]"},{"why":"National baseline provides the existing statistical severity-prediction benchmark used for comparison.","marker":"[42]"}],"fun_headline_variants":["LLMs outpredict tabular models on crash severity by 42% F1","Fine-tuned LLMs lift crash prediction F1 by 42%","LLM attribution: alcohol-impaired driving tops severe crash risk","Crash AI flags fatal cases with 97.6% precision","Textualizing crash data: LLMs improve prediction and interpretability"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline improvement over tabular baselines assumes that those baselines were given the same information content and the same rebalanced evaluation sets as the LLM, so the gain comes from text reasoning rather than from extra text or satellite data or from the evaluation-set rebalancing.","fun_headline_variants_meta":{"raw":{"variants":["LLMs outpredict tabular models on crash severity by 42% F1","Fine-tuned LLMs lift crash prediction F1 by 42%","LLM attribution: alcohol-impaired driving tops severe crash risk","Crash AI flags fatal cases with 97.6% precision","Textualizing crash data: LLMs improve prediction and interpretability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000716,"raw_usage":{"total_tokens":3271,"prompt_tokens":1049,"completion_tokens":2222,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":665,"completion_tokens_details":{"reasoning_tokens":2128}},"tokens_in":665,"tokens_out":2222,"duration_ms":18592,"temperature":1.0,"reasoning_tokens":2128,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:31:45.865907+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct check: retrain the seven tabular baselines on the original feature set, then evaluate them on the exact same rebalanced test sets that TrafficSafe uses, after adding the satellite-image-derived and textual features that the prompts contain. If the baselines' F1-scores rise to near the LLM's levels, the 41.7% gain is attributable to added information or evaluation-set changes rather than to LLM text reasoning.","supporting_citations":[{"cited_title":"Department of Transportation, Federal Highway Administration","cited_arxiv_id":null,"evidence_quote":"HSIS is the primary data source for crash, infrastructure, vehicle, and person records."},{"cited_title":"Google Maps Static API Documentation https://developers.google.com/maps/ documentation/maps-static","cited_arxiv_id":null,"evidence_quote":"Google Maps satellite images are textualized to supplement infrastructure and environmental context."},{"cited_title":"& von Luxburg, U","cited_arxiv_id":null,"evidence_quote":"Shapley theory is the mathematical foundation for sentence-level feature attribution in TrafficSafe Attribution."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"XGBoost is one of the strongest tabular baselines the paper must beat, representing the standard gradient-boosting approach."},{"cited_title":"S., Khazraee, S","cited_arxiv_id":null,"evidence_quote":"National baseline provides the existing statistical severity-prediction benchmark used for comparison."}],"review_version":1}