{"id":"86d0257c-7274-4c2c-ab4d-74dfefc04aa0","arxiv_id":"2506.19342","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Applying a BERT classifier to Iowa crash narratives yields an estimated 24.03% alcohol-inference mismatch rate among model-detected alcohol crashes, plus associated crash factors and county clusters.","lead":"A BERT text classifier scans 371,062 Iowa crash narratives and flags 2,767 crashes where the narrative suggests alcohol but the database does not, an inferred alcohol inference mismatch rate of 24.03%. The paper then uses regression and spatial analysis to identify crash types and counties with the most mismatches.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Full-data recall implied by the paper's own counts is ~57%, not 97%; the 24.03% AIM estimate is therefore unsupported.","rationale":"The reader's weakest assumption is exactly the load-bearing point: the BERT test accuracy is assumed to transfer to the full dataset, and the paper's own counts contradict that assumption. Summing Table 2 gives 15,282 originally alcohol-labeled crashes; the filtered positive set of 11,517 includes 8,750 of them, so full-data recall is about 57%, far below the reported 97% test recall. This discrepancy is not a minor calibration issue. The AIM rate is computed as 2,767 / 11,517 = 24.03%, so the numerator and denominator both come from model predictions whose operating point is unvalidated under class imbalance. If the model misses a substantial number of narrative-alcohol crashes, the true AIM denominator changes, and if the threshold were adjusted the 2,767 count would change as well. The paper provides no external validation (hospital records, manual review of a full-data sample, or code/data release) to resolve this. The probit and LISA analyses inherit the same label noise, so the regression coefficients and spatial clusters cannot be interpreted as explaining verified underreporting. These issues are internal to the paper's reported numbers rather than merely a disagreement with the literature, so the reader's REJECT verdict stands. My independent concern is the same one, and no additional adjustment is needed beyond what the reader recommended.","tokens_in":12486,"tokens_out":4746,"duration_ms":49449,"concrete_test":"Recompute the full-data confusion matrix from the published counts: 15,282 original alcohol labels, 11,517 model-positive predictions, and 8,750 overlap imply recall 57.3% and precision 76.0%. Then re-estimate the AIM rate after calibrating the decision threshold to the full-data prevalence, and manually review a random sample of 500 model-negative narratives to estimate missed AIM cases. If the corrected AIM percentage moves by more than a few points relative to 24.03%, the headline estimate is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the 24.03% AIM rate. That estimate depends entirely on the BERT classifier's performance transferring from the manually corrected, balanced training sample to the full 371,062-crash population. Table 5 reports a test recall of 0.97 and precision of 0.99 for alcohol, but those numbers come from a test set with roughly equal class sizes (898 alcohol vs. 902 non-alcohol). Applying the model to the full data yields 11,517 predicted-alcohol crashes, of which 8,750 overlap with the 15,282 crashes originally labeled alcohol (summing Table 2). That implies full-data recall of 8750/15282 = 57.3% and precision of 8750/11517 = 76.0%, directly contradicting the reported test-set metrics. The AIM count of 2,767 is the number of model-positive, originally non-alcohol crashes; it is conditioned on a decision threshold that behaves very differently at the 4% base rate of alcohol-labeled crashes than under the balanced validation sample. The paper never reports full-data precision/recall, never calibrates the threshold, and uses only predicted-positive crashes as the denominator in the AIM rate. Because the regression and spatial outcomes are built from these same machine-generated labels, they describe the classifier's disagreement patterns rather than verified underreporting. Additionally, the training and test support counts in Table 5 sum to 8,501, not the stated 8,914, so even the internal arithmetic of the validation procedure is not fully consistent.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a BERT-based text-classification framework to identify \"alcohol inference mismatch\" (AIM) in Iowa crash data, defined as crashes whose narrative mentions alcohol but whose structured ALCOHOL_REL field is non-alcohol. The authors train a BERT Large Uncased model on a manually corrected stratified sample of 8,914 narratives, report 98% test accuracy, apply it to all 371,062 crashes, and obtain 11,517 predicted-alcohol crashes, of which 2,767 are originally non-alcohol and are labeled AIM, yielding an AIM rate of 24.03%. They then fit a probit-logit model with county-level random intercepts to the machine-generated AIM outcome and perform LISA spatial clustering to identify counties with high mismatch. The paper concludes that fatal and nighttime crashes have lower AIM, while older drivers, unknown vehicle types, and vulnerable road users have higher AIM.","tokens_in":12750,"tokens_out":3138,"duration_ms":32341,"significance":"If the central AIM estimate were valid, the paper would provide a practical, scalable tool for improving crash-data quality and targeting enforcement or training resources. The application to a full state database is a useful proof of concept, and the integration of NLP labeling with regression and spatial analysis is a reasonable design template for similar data-quality problems. However, the manuscript does not make code or data available, and the reported validation metrics do not transfer to the full-data application, which undermines the quantitative headline claims.","major_comments":[{"comment":"The full-data predictions directly contradict the reported test-set performance. Table 5 reports test recall of 0.97 and precision of 0.99 for the alcohol class, but the text states that applying the model to all 371,062 crashes yields 11,517 predicted-alcohol crashes, of which 8,750 are originally alcohol-labeled. Since Table 2 sums to 15,282 originally alcohol-labeled crashes, the implied full-data recall is 8,750/15,282 = 57.2% and precision is 8,750/11,517 = 76.0%, far below the validation metrics. Because the AIM rate is computed as 2,767/11,517, both the numerator and denominator depend on the model's decision threshold at the true 4% base rate of alcohol-labeled crashes, which is not a regime represented by the balanced validation set. The paper never reports full-data precision or recall, never calibrates the threshold, and therefore the 24.03% AIM estimate is unsupported.","section":"Analysis on the whole data; Table 5"},{"comment":"The support counts in Table 5 are internally inconsistent with the stated sample size. Training supports are 3,559 alcohol and 3,142 non-alcohol, summing to 6,701; test supports are 898 and 902, summing to 1,800; the total is 8,501, not the 8,914 stated in the text. This discrepancy suggests missing or duplicate records in the stratified sample, and it is not explained. The reported accuracy, recall, and precision are therefore computed on a sample whose composition is unclear.","section":"Preparing Training Dataset and Training the Model; Table 5"},{"comment":"The outcome variable \"Mismatch Category\" is defined solely by the BERT model's prediction combined with the original structured label (Table 4). The probit-logit model is then fitted to this machine-generated outcome, so the estimated coefficients and associated interpretations describe the classifier's disagreement patterns, not independently verified underreporting. Without a human-validated random sample of the full-data predictions (e.g., a sample of the 2,767 AIM crashes and the 8,750 non-AIM crashes), the regression and LISA spatial results cannot be interpreted as identifying factors of true alcohol reporting mismatches.","section":"Regression modeling of predicted 'Mismatch Category'; Table 4"},{"comment":"The definition of the AIM percentage is inconsistent across the paper. The Methodology first says the AIM percentage is 'the ratio of the number of AIM crashes to the total number of alcoholic crashes in a given region during a specific time period,' then later says it is computed as the number of AIM crashes divided by the total number of crashes, 'which includes both Non-AIM Crashes and those identified as AIM.' The reported 24.03% uses 2,767/11,517, i.e., the denominator is all model-positive crashes, not all alcoholic crashes (which would include 15,282 originally alcohol-labeled crashes plus an unknown number of true positives among originally non-alcohol crashes). The ambiguity materially changes the headline statistic and must be resolved.","section":"Methodology; 'AIM Percentage Estimation'"}],"minor_comments":[{"comment":"The paper states that TF-IDF weighted features were used to train BERT Large Uncased, but BERT is normally trained on raw token sequences, not TF-IDF-weighted vectors. Please clarify the actual input representation, as the described pipeline is not standard.","section":"Text Classification subsection"},{"comment":"The confusion matrices in Figure 2 are referenced but not described in the text. Please include the exact cell counts so that readers can verify the reported precision, recall, and F1 values.","section":"Table 5 and Figure 2"},{"comment":"Table 7 lists 'DriveDistracted' as a variable with 1,703 yes and 6,087 no, but Table 8 does not include this variable in the final model. Please explain whether it was excluded and why.","section":"Table 7 and Table 8"},{"comment":"Reference 18 is a Google Drive link for 'Probit and Logit Models Example'; this is not a stable or citable source. Please replace it with a proper textbook or peer-reviewed citation for probit/logit modeling.","section":"References"},{"comment":"There are several typographical errors and inconsistent terms, e.g., 'Unkown' in Table 2, 'Probit Logit' vs 'probit logistic regression', and 'alcohal' in the abstract. A careful proofread is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The central quantitative claim is not supported by the paper's own numbers, but the underlying framework is potentially sound and the issues are fixable with a re-analysis: the authors should report full-data precision/recall, calibrate the decision threshold appropriately, validate a random subset of predictions against human review, and re-estimate the AIM rate and regression with that corrected outcome. If the full-data validation cannot be performed, the paper should be substantially reframed as a methodological demonstration rather than a quantitative estimate of AIM in Iowa."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something worthwhile: it applies a BERT classifier to 371,000 crash narratives from Iowa and uses the disagreements between the model and the officer-reported alcohol flag to identify probable underreported alcohol crashes. The data scale and the downstream regression/spatial analysis are real contributions. The 24.03% AIM rate is in the same ballpark as prior mismatch estimates, so the ballpark is not crazy. But the paper's own presentation undermines the central claim.\n\nThe biggest problem is the denominator. The AIM rate is computed as 2,767 AIM crashes divided by 11,517 model-predicted alcohol crashes. In other words, it is the share of model positives that the database had flagged as non-alcohol — a false-discovery rate, not the rate of underreporting among all alcohol-involved crashes. The abstract and conclusion call it an underreporting rate, but it is a different quantity. That distinction matters if an agency wants to know how many alcohol crashes are being missed.\n\nSecond, the stress-test note's recall arithmetic (57% vs. the reported 97% test recall) is partly unfair, because it treats the original database labels as ground truth — but the whole point of the manual correction was to fix those labels. However, the paper never reports how the model performs on a full-distribution validation set. The test set is balanced 50/50 while the population has ~4% positive labels; precision and recall are expected to shift, and the paper neither calibrates the threshold nor reports full-data metrics. That is a legitimate gap.\n\nThird, there are internal inconsistencies. Table 5's training and test support counts sum to 8,501, but the text says the sample was 8,914. Table 4 defines Non-AIM crashes as predicted non-alcoholic/original alcohol, but the actual analysis treats Non-AIM as predicted alcoholic/original alcohol. These are fixable, but they shake confidence in the numbers.\n\nFinally, the probit/logit and spatial analysis use the machine-generated mismatch labels as the outcome. The factors identified therefore describe correlates of the classifier's disagreement patterns, not independently verified underreporting. The limitations section acknowledges the need for expert validation, but that is an afterthought rather than part of the design.\n\nFor whom is this paper? Transportation safety analysts who want a scalable way to flag likely underreported alcohol crashes. It deserves a serious referee — not a desk reject — because the problem is important and the framework is plausible. But the referee should demand a clean definition of the AIM rate, a validation sample of manually reviewed narratives from the full population, full-data precision/recall, and release of code/data. In its current form, the 24.03% estimate is not ready to be cited as a reliable underreporting rate.","headline":"Useful framework with a central estimate that is not actually an underreporting rate; deserves a serious referee but needs major methodological cleanup.","tokens_in":13299,"tokens_out":3891,"would_cite":false,"duration_ms":40380,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By aligning police crash narratives with structured alcohol flags, the paper identifies 2,767 alcohol-inference-mismatch crashes in Iowa and estimates an overall AIM rate of 24.03%.","keywords":["alcohol inference mismatch","crash narrations","crash data quality","impaired crashes","BERT model","NLP","mismatched crashes","traffic crash underreporting"],"falsifier":"Take a random sample of the 2,767 AIM crashes, or of the 11,517 model-flagged alcohol crashes, and have independent human reviewers read the original narratives plus any available hospital or toxicology records without seeing the model's label. If a large share of the 2,767 flags do not actually describe alcohol involvement, the AIM rate is largely a classifier artifact; if independent records confirm the flags, the framework's estimate holds.","tokens_in":12297,"feed_emoji":"🚗","tokens_out":8325,"duration_ms":78665,"temperature":0.7,"pith_summary":"The paper tries to show that a large share of alcohol-involved crashes are hidden in plain sight: the structured database says \"non-alcohol,\" but the officer-written narrative describes drinking, impairment, or odor. It builds a BERT-based text classifier that reads those narratives, trains it on manually corrected reports, and applies it to all 371,062 Iowa crashes from 2016 to 2022. The result is an estimated 2,767 alcohol inference mismatch (AIM) crashes, an overall AIM rate of 24.03%, with fatal and nighttime crashes less likely to be mismatched and crashes involving older drivers, trucks, pedestrians, and bicyclists more likely. If correct, this gives crash databases a practical correction layer and pinpoints counties and report types where officer training could improve data quality.","feed_headline":"BERT finds 2,767 alcohol crashes Iowa reports mislabeled","feed_subtitle":"Reading officers' own words catches underreported drunk driving and maps where reporting falls short.","key_machinery":"The load-bearing mechanism is database-narrative alignment: a fine-tuned BERT classifier reads the free-text crash narrative, predicts whether alcohol was involved, and compares that prediction with the structured alcohol flag; when the narrative says alcohol and the flag says non-alcohol, the crash is scored as an AIM crash. BERT is a bidirectional transformer language model pretrained on large text corpora, here fine-tuned for binary alcohol classification on 8,914 manually corrected, stratified narratives. Around that core sit text preprocessing (personally identifiable information removal, punctuation and number stripping, lemmatization, TF-IDF weighting), a two-stage pipeline that filters model-predicted alcohol crashes before computing AIM rates, and a county-level analysis combining Local Moran's I spatial clustering with a probit model using random county intercepts to identify where and why mismatch concentrates.","core_discovery":"The paper's central claim is that alcohol involvement can be recovered from the officer-written narrative even when the structured database flag says \"no,\" and that doing so at scale changes the picture of alcohol-related crashes in Iowa. Using a fine-tuned BERT text classifier, the study labels each narrative as alcohol-related or not, then compares that prediction with the ALCOHOL_REL field. Of 371,062 crashes from 2016 through 2022, the model flags about 11,517 as alcohol-related; 8,750 of those were already coded alcohol, and 2,767 were originally coded non-alcohol. Those 2,767 are the paper's AIM crashes, giving an AIM percentage of 24.03%, lower than the roughly 40% mismatch in earlier police-hospital comparisons. The mismatch is not uniform: fatal crashes have the lowest AIM rate at 16.79%, unknown-injury crashes the highest at 32.48%, and a county-random-intercept probit model finds higher mismatch odds for heavy trucks, other vehicles, drivers aged 15-24 and 65 or older, pedestrians, bicyclists, and daylight or dusk conditions.","pith_inferences":["Editorial inference: the paper's full-data application implies the classifier's real-world recall is far below its 97% test recall, since only 8,750 of the 15,282 crashes originally coded alcohol were predicted alcohol, about 57%, so the AIM count and rate would need recalibration on the imbalanced population before being used as a definitive statistic.","An extension the authors leave implicit: the same narrative-versus-structured-field alignment could be applied to other underreported factors such as driver distraction, drug impairment, or seatbelt use, because the mechanism does not depend on alcohol specifically.","A testable next step would be to link the flagged AIM crashes to hospital emergency records or toxicology results for a sample of cases; if external records agree with the narrative-based flags, the 24.03% estimate gains independent support."],"forward_implications":["Agencies can run the classifier on incoming crash reports and automatically flag narratives that mention alcohol but were coded non-alcohol, creating a shortlist for officer review instead of relying on manual audits.","The 24.03% AIM estimate implies Iowa's official alcohol-crash counts understate alcohol involvement by roughly a quarter among crashes whose narratives contain alcohol signals, which would shift the denominator for alcohol-crash rates and policy targets.","The severity pattern, with fatal crashes lowest at 16.79% and possible or unknown-injury crashes highest at 32.48%, suggests reporting reforms should target ambiguous, low-severity crashes rather than fatal ones.","Regression results point to concrete training priorities: older drivers, young drivers, heavy trucks, pedestrians, and bicyclists are the report types most likely to be mismatched, and spatially clustered High-High counties deserve the most attention."],"supporting_citations":[{"why":"Supplies the BERT model the paper fine-tunes to classify crash narratives as alcohol-related or not.","marker":"Devlin et al., 2019"},{"why":"Earlier large-language-model framework for uncovering underreported crashes; the study extends this line of work to a full statewide database with a fine-tuned classifier.","marker":"Arteaga and Park, 2025"},{"why":"Police-hospital capture-recapture estimates of alcohol underreporting; provides the roughly 40% mismatch benchmark the paper contrasts with its 24.03%.","marker":"Miller et al., 2012"},{"why":"Documents how often officers identify alcohol involvement by blood alcohol concentration level; underpins the assumption that underreporting is systematic.","marker":"Blincoe et al., 2023"},{"why":"Provides alcohol inference mismatch rates by crash severity and a method to estimate underreporting; the paper's severity findings are compared against this pattern.","marker":"Wood et al., 2016"},{"why":"Shows crash-report templates and officer narratives create mismatches; motivates using narrative text as the signal for mismatch.","marker":"Lopez et al., 2022"},{"why":"Supplies the TF-IDF weighting used to boost informative terms before BERT classification.","marker":"Geetha and Karthika Renuka, 2021"}],"fun_headline_variants":["BERT flags 2,767 missed alcohol crashes in Iowa","Text analysis uncovers 2,767 unreported drunk driving crashes","Iowa crash reports miss alcohol in 24% of cases","Machine learning spots 2,767 alcohol crashes police missed","BERT narrative analysis finds 2,767 hidden alcohol-involved crashes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a model trained and tested on a balanced, manually corrected sample classifies the entire imbalanced crash database just as accurately as it classifies the test set, even though the paper's own full-data counts suggest real-world recall is far lower.","fun_headline_variants_meta":{"raw":{"variants":["BERT flags 2,767 missed alcohol crashes in Iowa","Text analysis uncovers 2,767 unreported drunk driving crashes","Iowa crash reports miss alcohol in 24% of cases","Machine learning spots 2,767 alcohol crashes police missed","BERT narrative analysis finds 2,767 hidden alcohol-involved crashes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000814,"raw_usage":{"total_tokens":3580,"prompt_tokens":969,"completion_tokens":2611,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":2539}},"tokens_in":585,"tokens_out":2611,"duration_ms":20849,"temperature":1.0,"reasoning_tokens":2539,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:33:16.487986+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the 2,767 AIM crashes, or of the 11,517 model-flagged alcohol crashes, and have independent human reviewers read the original narratives plus any available hospital or toxicology records without seeing the model's label. If a large share of the 2,767 flags do not actually describe alcohol involvement, the AIM rate is largely a classifier artifact; if independent records confirm the flags, the framework's estimate holds.","supporting_citations":[{"cited_title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","cited_arxiv_id":null,"evidence_quote":"Supplies the BERT model the paper fine-tunes to classify crash narratives as alcohol-related or not."},{"cited_title":"A Large Language Model Framework to Uncover Underreporting in Traffic Crashes","cited_arxiv_id":null,"evidence_quote":"Earlier large-language-model framework for uncovering underreported crashes; the study extends this line of work to a full statewide database with a fine-tuned classifier."},{"cited_title":"Underreporting of Driver Alcohol Involvement in United States Police and Hospital Records: Capture-Recapture Estimates","cited_arxiv_id":null,"evidence_quote":"Police-hospital capture-recapture estimates of alcohol underreporting; provides the roughly 40% mismatch benchmark the paper contrasts with its 24.03%."},{"cited_title":"February 2023 6","cited_arxiv_id":null,"evidence_quote":"Documents how often officers identify alcohol involvement by blood alcohol concentration level; underpins the assumption that underreporting is systematic."},{"cited_title":"A method to account for and estimate underreporting in crash frequency research","cited_arxiv_id":null,"evidence_quote":"Provides alcohol inference mismatch rates by crash severity and a method to estimate underreporting; the paper's severity findings are compared against this pattern."},{"cited_title":"Police narrative reports: Do they provide end- users with the data they need to help prevent bicycle crashes? Accid Anal Prev 164","cited_arxiv_id":null,"evidence_quote":"Shows crash-report templates and officer narratives create mismatches; motivates using narrative text as the signal for mismatch."},{"cited_title":"Improving the performance of aspect based sentiment analysis using fine-tuned Bert Base Uncased model","cited_arxiv_id":null,"evidence_quote":"Supplies the TF-IDF weighting used to boost informative terms before BERT classification."}],"review_version":2}