{"id":"4ebf521b-91cc-4021-9c60-fba9ce83253f","arxiv_id":"2509.07845","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"The paper claims narrative text improves person-level crash injury prediction in Kentucky, with TF-IDF and XGBoost best, but the structured-only baseline comparison is not shown.","lead":"Police-written crash narratives, when added to structured crash data, are claimed to improve machine learning predictions of person-level injury severity across Kentucky road classes. The paper compares TF-IDF and Word2Vec text features with three ensemble models, but does not show the structured-only baseline needed to verify the main claim.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim is untestable: no structured-only baseline results are reported, so narrative-data superiority cannot be assessed.","rationale":"The reader's verdict is REJECT, and I agree with that conclusion, but the single most load-bearing concern is somewhat different from the reader's stated weakest_assumption. The reader emphasizes representativeness of the labeled subset (Section 3.2), which is a real external-validity threat. However, the more fundamental problem is that the paper's headline claim—narratives add value—requires a comparison against structured-only models, and that comparison is absent from the results. Section 1 explicitly says such structured-only models were developed, but Section 5 reports only narrative-enhanced results. This is not a matter of minor reporting omission; it means the central claim is not supported by any evidence presented in the manuscript. The ISS validation is a good-faith effort at external validation, and I credit it, but it validates a trained model's severity ordering against ISS, not the incremental contribution of narrative text. The data-accounting inconsistency between Section 6 ('only ~3200 cases had reported severities') and Table 1 (roughly 32,000 labeled records) further undermines confidence in the practical imputation claim, though it is secondary. A concrete remedy is straightforward: report the structured-only baselines under identical conditions. Until then, the central claim is untestable, and the paper should not be accepted as is.","tokens_in":17828,"tokens_out":2585,"duration_ms":23320,"concrete_test":"Re-run the same 102 model configurations (3 algorithms × 17 datasets × TF-IDF/Word2Vec) with the narrative-derived features removed, using identical train/validation/test splits and evaluation metrics, and report macro-F1 for all 17 structured-only baselines. Then compare each narrative-augmented model against its structured-only counterpart; if narrative models do not win a clear, consistent majority (or a paired significance test, e.g., Wilcoxon signed-rank on macro-F1 across the 17 datasets), the abstract's central claim fails.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central claim is that models incorporating narrative data consistently outperform models relying solely on structured data (Abstract; Section 1). Section 1 states that 'injury prediction models are first developed using only structured crash data' and that narrative information is then incorporated, but the Results (Section 5) never reports any structured-only baseline. Figures 4 and 5 and Tables 2 and 3 rank only narrative-enhanced models across the 17 datasets; no macro-F1, precision, recall, or accuracy for a structured-only XGBoost/RF/AdaBoost model appears anywhere. Without this baseline, the comparative claim is not merely weak—it is unmeasured. The ISS validation in Section 5.2 is a positive element, but it validates the narrative-enhanced XGBoost model against clinical data; it does not establish the added value of narratives. A second inconsistency compounds the problem: Section 6 states that in the 67,000-record dataset only ~3,200 cases had reported severities, yet Section 3.2 and Table 1 describe and tabulate roughly 32,000 persons with severity labels. If the conclusion's number is literal, the training/test design described in Section 4.2 cannot be reconciled with the reported support values. Either way, the practical imputation claim rests on unresolved data-accounting issues. The load-bearing defect remains the absence of the structured-only control group, which makes the central claim untestable from the manuscript as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops person-level injury severity prediction models for Kentucky crash records (2019–2023) by combining structured crash data with police-written crash narratives. Two text representations, TF-IDF and Word2Vec, are paired with three ensemble classifiers (XGBoost, Random Forest, AdaBoost) under three roadway classification schemes, producing 102 models. To handle class imbalance, SMOTE is applied to training data. The central claim is that narrative-augmented models consistently outperform structured-only models, with TF-IDF plus XGBoost best in most subgroups. The paper also validates the group-3 XGBoost model against trauma-registry ISS scores and proposes the framework for imputing missing severity labels.","tokens_in":18142,"tokens_out":6734,"duration_ms":58049,"significance":"If the central claim were established, the paper would offer a practical framework for crash severity imputation and a useful comparison of roadway stratification strategies, and its external validation against ISS data is a genuine strength. The multi-dataset design across 17 roadway subgroups and the use of a held-out test set for final evaluation are also positive elements. However, the reported results do not currently support the headline added-value claim, because no structured-only baseline is presented. The external ISS validation is informative about the narrative-enhanced model's clinical alignment but does not substitute for a controlled comparison. The practical imputation claim additionally rests on unresolved data-accounting and missingness issues. With the addition of the missing comparison and clarification of the pipeline, the contribution could be meaningful for transportation safety practice.","major_comments":[{"comment":"The paper's central claim (Abstract and §1) is that narrative-augmented models consistently outperform structured-only models, yet no structured-only baseline is reported anywhere in §5. Figures 4–5 and Tables 2–3 rank only narrative-enhanced models by group, and the text does not report macro-F1, precision, recall, or accuracy for a structured-only XGBoost, Random Forest, or AdaBoost model on any of the 17 datasets. Since §1 states that models were first developed using only structured crash data, these results should already exist; they need to be reported with the same pipeline, evaluation protocol, and metrics as the narrative-enhanced models. Without this baseline, the added-value claim is unmeasured.","section":"§5.1, Figures 4–5, Tables 2–3"},{"comment":"The Conclusion states that in the 67,000-record dataset only about 3,200 cases had reported severities, whereas §3.2 and Table 1 describe a person-level dataset with roughly 32,000 severity-labeled records (e.g., 16,583 male and 15,713 female records), and Table 5 lists test-set supports that sum to 26,800 for the group-3 model. These numbers cannot all be correct. The authors must reconcile the data accounting, because the practical imputation claim depends on knowing how many records are labeled and how the modeling subset relates to the full dataset.","section":"§6 vs. §3.2 and Table 1"},{"comment":"The Word2Vec pipeline is incomplete: the paper states that the embedding dimension is 100 and that a 200-word narrative yields a 200×100 matrix, but it never specifies how the per-word vectors are aggregated into a single document-level vector for the classifiers. Options such as mean pooling, max pooling, concatenation, or training a document-embedding model are not described. The training corpus, window size, minimum count, and number of epochs are also omitted. Without this information, the Word2Vec results in Figure 4 and Table 2 cannot be reproduced.","section":"§4.6.2"},{"comment":"The feature-selection procedure does not state whether the Random Forest importance scores used to select the top 100 features are computed on the training split only. If the importance ranking is computed on the full dataset before the 70/15/15 split into training, validation, and test sets, then information from the validation and test sets leaks into the feature selection and the reported metrics are optimistically biased. Please clarify that importance is computed inside the training portion, or modify the procedure accordingly.","section":"§4.4"},{"comment":"Because the 'No Injury' class is excluded and only records with a reported severity are used for modeling, the proposal to use the model to impute missing severities in the full dataset assumes that the labeled subset is representative of all persons involved in crashes. If missingness is correlated with severity or with crash characteristics, the imputed distribution will be biased. The manuscript should either provide evidence that labels are missing at random or restrict the imputation claim to the population of records with reported severities.","section":"§3.2 and §6"}],"minor_comments":[{"comment":"There is a typo in the phrase 'sem antic meaning'; please proofread the text for similar spacing errors.","section":"Abstract"},{"comment":"In the Speed Limit 30–50 row, the percentage '51.36' appears as '5136'; the missing decimal point should be restored so that the row sums to 100.","section":"Table 1"},{"comment":"The sentence beginning 'As shown in Tables 4 and 5, except for Group 2 within the Urban class...' refers to the ISS cross-tabulations, not to the TF-IDF/Word2Vec comparison; the intended references appear to be Tables 2 and 3 or Figures 4 and 5.","section":"§5.1"},{"comment":"The citation 'Jiang et al. (6)' does not match reference [6], which is Christoforou et al.; the transformer-based discussion appears to refer to reference [4] (Jiang et al.). In addition, the text 'Do et al.' corresponds to reference [10] by Du et al.; please correct the author names in the citations.","section":"§2"},{"comment":"Figures 2 and 3 are not described in sufficient detail in the text. In particular, Figure 2 does not illustrate how a document-level representation is obtained from per-word Word2Vec embeddings, which is a central methodological question.","section":"Figures 2 and 3"},{"comment":"The paper would benefit from a data and code availability statement; no repository or release is mentioned, which limits reproducibility given the complexity of the preprocessing pipeline.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the reader's reject recommendation is understandable, but the primary defect—the missing structured-only baseline—is an omitted experiment rather than an unfixable flaw. If the authors can rerun the same pipeline without narrative features and report comparable metrics, the central comparison can be completed within the scope of a major revision. The data-accounting mismatch in §6, however, must be resolved: if the 67,000-record dataset truly has only ~3,200 labeled severities, then the tables and training description describe a different dataset, and a rejection would be warranted. The Word2Vec aggregation and feature-selection leakage concerns are also fixable with clarifications or additional experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The systematic comparison across 17 road-class subsets is a legitimate extension, and the ISS validation in Section 5.2 is genuine evidence: average ISS declines monotonically from Fatal to Possible Injury. But the abstract's central claim—that narrative data consistently beat structured data—never appears in the results. There is no structured-only baseline anywhere in Section 5; Figures 4 and 5 and Tables 2 and 3 rank only narrative-enhanced models. So the paper's main assertion is simply unmeasured.\n\nWhat is new: the road classification stratification (eight detailed classes, four paired categories, unified) and the external validation against linked trauma registry ISS scores. The split into train/validation/test, SMOTE applied only to training, and feature selection via RF importance are all standard but correctly executed. The Word2Vec dimension is stated, and the claim that TF-IDF generally beats Word2Vec is at least internally consistent.\n\nSoft spots, in order of severity. First, the missing structured-only baseline makes the comparative claim untestable. This is fixable: the authors presumably have those numbers. Second, a data-accounting inconsistency: Table 1 and Section 3.2 imply roughly 32,000 labeled persons, but Section 6 says only ~3,200 reported severities in a 67,000-record dataset. That is a factor-of-ten discrepancy and needs explanation. Third, the Word2Vec document-level aggregation is unspecified: a 200-word narrative gives a 200x100 matrix, but the paper never says how that becomes a single vector for concatenation with structured features. Fourth, no code or data are provided, which is unfortunate for a reproducibility-focused venue. The exclusion of 'No Injury' is understandable given the data, but it limits the practical imputation claim, since the labeled subset may not represent the missing records.\n\nThe citation pattern is honest: the authors acknowledge prior work (refs 1, 3, 21, 22) that already established narrative value, and they position their contribution as a comparative application. That is fair, though it further reduces the novelty of the core claim.\n\nBottom line: this is a competent applied study with a real external validation, but the central comparison is missing. I would not cite it in its current form, but I would send it to peer review because the fix is straightforward and the ISS validation gives it enough substance to warrant referee time.","headline":"Systematic road-class comparison and a credible ISS validation are real positives, but the paper's central claim—narrative beats structured—is never tested because no structured-only baseline is reported.","tokens_in":18652,"tokens_out":3115,"would_cite":false,"duration_ms":26206,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding police-written crash narratives to structured crash data improves person-level injury severity prediction, with TF-IDF plus XGBoost performing best.","keywords":["individual injury prediction","crash severity","narrative data","natural language processing","TF-IDF","Word2Vec","road classification","SMOTE oversampling"],"falsifier":"Take a random sample of crash-involved persons, obtain their true injury outcomes from hospital or trauma records (including no-injury cases), and compare TF-IDF plus XGBoost predictions against those ground-truth labels; if the narrative-augmented model does not outperform structured-only models, or if the predicted severity gradient against ISS disappears, the paper's central claim would be refuted.","tokens_in":17673,"feed_emoji":"🚑","tokens_out":6986,"duration_ms":56141,"temperature":0.7,"pith_summary":"This paper asks whether the free-text narratives police officers write at crash scenes carry predictive information about how badly each person involved was hurt, beyond what structured fields like vehicle type, speed limit, and weather already provide. Using five years of Kentucky crash records (2019–2023), the authors compare machine-learning models trained on structured data alone with models that also include narrative text encoded by TF-IDF or Word2Vec, across three roadway-classification schemes and three ensemble algorithms. They report that narrative-augmented models consistently outperform structured-only models, and that TF-IDF combined with XGBoost gives the best macro F1 scores in most road-type subgroups. They further validate the best model against clinical Injury Severity Scores from linked trauma registries, finding a clear severity gradient, and argue the framework can impute the missing severities that affect over half of Kentucky's crash records.","feed_headline":"Crash narratives beat structured data for injury prediction","feed_subtitle":"Adding police-written crash text improves person-level severity modeling; TF-IDF plus XGBoost leads across road classes.","key_machinery":"The load-bearing mechanism is the concatenation of TF-IDF term-weight vectors derived from crash narratives with the top-100 structured features selected by Random Forest importance, fed into XGBoost. TF-IDF assigns higher weight to words that are frequent within a narrative but rare across the corpus, so phrases like 'ejected', 'unresponsive', or 'head-on' can act as direct severity signals; Word2Vec serves as the semantic-context alternative. SMOTE oversampling is applied to the training partition to address the rarity of fatal and serious injuries, and the evaluation metric that carries the comparison is macro-averaged F1, which treats all four severity classes equally. External validation is provided by cross-tabulating predicted severity with Injury Severity Score categories from linked trauma records.","core_discovery":"On its own terms, the paper's central discovery is that structured crash variables alone leave predictive signal on the table, and that police-written narratives recover much of it. Across 17 roadway-stratified datasets (eight detailed functional classes, four binary pairings, and one pooled set) and 102 model configurations, models incorporating narrative data consistently outperformed those relying solely on structured data, and TF-IDF plus XGBoost won in most subgroups. The authors also show that the predicted severity categories align monotonically with clinical Injury Severity Scores: average ISS falls from 23.0 for predicted Fatal to 10.6 for Possible Injury, supporting the model's external validity and its proposed use for imputing missing severities in the ~67,000-record dataset.","pith_inferences":["The paper's result suggests that in this domain, discriminative vocabulary (TF-IDF) beats semantic embeddings (Word2Vec) for short police narratives; an untested extension is whether a transformer-based encoder fine-tuned on crash reports would further improve macro F1, or whether the narrative signal is mostly lexical.","Because the study excludes the 'No Injury' class and trains only on records with reported severity, the real-world performance depends on the labeled subset being representative; a direct test is to apply the model to a sample with complete clinical labels and compare class distributions.","The roadway-stratification design implicitly tests domain adaptation; one could also add road class as a categorical feature in a single pooled model, which would let the model share statistical strength across road types instead of training separate models."],"forward_implications":["Transportation agencies can apply the trained TF-IDF plus XGBoost model to impute injury severities for the over 50% of Kentucky crash records with missing labels, months before official statistics are released.","Per-roadway-class model selection matters: the paper shows that grouping rare road types into binary pairs (e.g., Freeway vs. Non-Freeway) stabilizes predictions for rare outcomes like fatal crashes, so practitioners should not default to a single pooled model.","Because predicted severity tracks clinical ISS, the framework can serve as a proxy for injury outcomes in settings where hospital data are unavailable, supporting triage and emergency-response planning.","The sensitivity-analysis capability lets safety officials simulate policy changes, such as posted speed limit adjustments, and estimate their effect on the distribution of predicted injury severities."],"supporting_citations":[{"why":"Supplies TF-IDF as a frequency-based NLP technique for extracting informative terms from accident narratives.","marker":"(1)"},{"why":"Supplies Word2Vec as a dense embedding method that preserves semantic and contextual meaning in accident narratives.","marker":"(2)"},{"why":"Provides prior evidence that combining narrative text with structured data improves crash analysis performance.","marker":"(3)"},{"why":"Shows that latent topics from crash narratives combined with XGBoost relate to severity outcomes, a direct precedent for the paper's approach.","marker":"(20)"},{"why":"Demonstrates hybrid predictive text analytics plus statistical modeling for pedestrian and bicyclist injury severity, supporting narrative-enhanced injury prediction.","marker":"(22)"},{"why":"Defines SMOTE, the oversampling method used to address class imbalance in the training data.","marker":"(30)"},{"why":"Introduces XGBoost, the ensemble algorithm that delivered the best performance in the paper's comparisons.","marker":"(42)"},{"why":"Provides the Word2Vec model used to generate dense word embeddings from crash narratives.","marker":"(45)"}],"fun_headline_variants":["Crash text lifts injury severity prediction beyond structured data","Narratives + structured data beat structured alone for severity","TF-IDF + XGBoost on crash narratives: top severity predictor","Police crash narratives sharpen person-level injury forecasts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the records with reported injury severity are representative of all persons in crashes, including the majority whose severity is missing and the excluded 'No Injury' cases; if that subset is biased, the model's performance and imputations will not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Crash text lifts injury severity prediction beyond structured data","Narratives + structured data beat structured alone for severity","TF-IDF + XGBoost on crash narratives: top severity predictor","Police crash narratives sharpen person-level injury forecasts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00047,"raw_usage":{"total_tokens":2375,"prompt_tokens":1020,"completion_tokens":1355,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":1288}},"tokens_in":636,"tokens_out":1355,"duration_ms":10705,"temperature":1.0,"reasoning_tokens":1288,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:09:38.886667+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of crash-involved persons, obtain their true injury outcomes from hospital or trauma records (including no-injury cases), and compare TF-IDF plus XGBoost predictions against those ground-truth labels; if the narrative-augmented model does not outperform structured-only models, or if the predicted severity gradient against ISS disappears, the paper's central claim would be refuted.","supporting_citations":[],"review_version":2}