{"id":"76b271ee-43f1-4d72-ae90-8ba4c87a8e48","arxiv_id":"2506.02242","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"UrbanX uses LLM-generated visual questions, MLLM answers, and linear regression to predict crash rates, claiming better performance than ResNet/ViT while keeping features interpretable.","lead":"This paper proposes UrbanX, a pipeline that uses multimodal language models to generate questions about street view images, answer them, and feed the answers into regressions that predict crash rates in Manhattan. It reports better predictive performance than pretrained vision models with interpretable features, but the statistical evidence is weakened by outcome-driven hypothesis selection and missing error bars.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported p-values in §4.4 are not valid evidence of discovery: Algorithm 1 selects hypotheses using the same crash-rate outcome that later produces the significance tests, and the paper provides no split-sample or selection-aware correction.","rationale":"I read the paper as a method demonstration whose central promise is automated, statistically rigorous discovery of interpretable safety factors from SVIs, i.e., the significance evidence in §4.4 is the payoff. There are real strengths: the pipeline is fully described, the code is released, the MLLM audit (84% correct on the case image, >90% on high-salience hypotheses) gives some credibility to the embeddings, and the robustness experiments show sensitivity to model capacity and hypothesis count. But the inferential claim is the load-bearing part, and it is not supported as written. The reader's weakest assumption is well placed. Algorithm 1 selects hypotheses by p-value thresholds on the training crash outcome, feeds those p-values back into the LLM prompt, and then the paper reports p-values and SHAP-based significance for the survivors without any selection adjustment. This is the classic winner's curse: the very act of keeping variables with small p-values makes later p-values uninterpretable. The lack of a described sample split for significance testing, or of any post-selection correction, is a mechanical gap, not a matter of disagreement with the current consensus. It is also easy to fix, but as written the Section 4.4 claims of 'statistically significant' factors are not justified. I additionally note Eq. 4's independence assumption is implausible for adjacent segments, which would further shrink standard errors; this compounds rather than replaces the selection problem. I do not think the predictive comparison, the SHAP analysis, or the qualitative audit would by themselves force rejection. The predictive comparison lacks error bars and could be sensitive to spatial leakage in the image-level split, but the paper at least gives a segment-level cross-validation for the spatial maps. The fatal step is the statistical significance evidence. The framework could be salvaged with a holdout-based or selection-valid inference protocol, but the current manuscript does not state one. Therefore the reader's REJECT verdict is appropriate; no verdict change is needed.","tokens_in":22516,"tokens_out":8190,"duration_ms":81765,"concrete_test":"Run a permutation-null calibration of Algorithm 1: keep the SVIs, the LLM/MLLM prompts, and the regression pipeline fixed, randomly permute the 16,000 crash rates across street-view images, and rerun the full iterative procedure (generation, embedding, pruning, final p-value reporting) for at least 50 independent permutations. Under a valid procedure with 50 final hypotheses and alpha = 0.05, the expected number of retained hypotheses with p < 0.05 is about 2.5 per run; if the median count is much larger (e.g., >10), the Section 4.4 p-values are inflated by outcome-dependent selection. To localize the source, also run the same permutation check with a single fixed hypothesis set (no iterative pruning); if the inflation disappears, the pruning loop is the culprit.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 (Section 3.5) iteratively generates hypotheses with an LLM conditioned on previous p-values, prunes hypotheses with p > 0.05 computed from the regression on the training outcome, and then reports the final p-values from a regression on the same data in Section 4.4 and Figure 6. This is outcome-dependent selection: under the null, variables that happen to look significant are preferentially retained, so the distribution of p-values for the retained set is no longer uniform. Five percent of noise variables do not become '5% of retained variables'; the retained set is enriched for small p-values, so the displayed -log10(p) values are inflated. The manuscript does not describe sample splitting, data carving, selective inference, or any multiple-testing correction across the 50 retained hypotheses, despite multiple rounds of LLM-conditioned hypothesis generation. Even if the final p-values were computed on the validation split, the paper's stated retention rule uses the validation set to decide which hypotheses to keep, so those same p-values would still be selection-conditional unless the test split is used and kept untouched. In addition, Eq. 4 assumes independent Gaussian errors; Manhattan road segments are spatially adjacent, so OLS standard errors are likely downward-biased. Both mechanisms affect the exact outputs used to claim 'statistically significant' safety factors. Because the paper's central claim is scientific discovery of statistically meaningful relationships, the reported significance evidence is not load-bearing as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes UrbanX, a framework that uses multimodal large language models (MLLMs) to generate natural-language hypotheses about visual features in street view images, converts the hypotheses into categorical embeddings via visual question answering, and evaluates them with linear regression in an iterative loop. The framework is applied to segment-level crash rates in Manhattan. The authors claim that UrbanX outperforms pretrained ResNet and ViT baselines while providing interpretable variables, and that the retained hypotheses are statistically significant correlates of crash rates. The central methodological contribution is an automated, LLM-driven hypothesis generation and refinement pipeline that is intended to support transparent, data-driven urban science.","tokens_in":22823,"tokens_out":3894,"duration_ms":35957,"significance":"If the claims were validated, UrbanX would be a useful template for transforming unstructured street-level imagery into interpretable, testable variables for urban safety research, and the idea of using MLLMs as semantic engines in an iterative statistical loop is timely and interesting. The authors provide a public code repository and a manual audit of MLLM answers, which are helpful for reproducibility. However, the current empirical evidence for both main claims—predictive superiority and statistically significant discovery—is undermined by statistical issues in the evaluation protocol. The paper's contribution is therefore promising but not yet established; the framework deserves publication only after the inferential problems are addressed.","major_comments":[{"comment":"The regression model in Eq. (4) assumes independent, normally distributed errors, but crash rates on spatially adjacent Manhattan road segments are likely spatially autocorrelated. The reported standard errors and p-values are therefore probably downward-biased, and the significance of the discovered factors is overstated. The authors should use spatially clustered standard errors, a spatial error model, or another approach that accounts for spatial dependence, and report how the results change.","section":"§3.4, Eq. (4)"},{"comment":"The claim that UrbanX 'outperforms pretrained deep learning models' is not supported empirically because Figure 3 reports only point estimates without error bars, confidence intervals, or significance tests. It is not possible to determine whether the differences between UrbanX (LR/LGBM) and the ResNet/ViT baselines are meaningful or due to random variation across data splits or model initializations. The authors should provide multiple runs or cross-validation folds with uncertainty estimates and a statistical comparison (e.g., paired tests) before claiming superiority.","section":"§4.2, Figure 3"},{"comment":"The robustness analysis reports 'convergence over 50 training epochs' for what appear to be linear regression and LightGBM models; linear regression has no epochs and LightGBM has boosting iterations, not epochs. This inconsistency makes it difficult to interpret the convergence curves. The authors should clarify the training procedure and the x-axis units.","section":"§4.5, Figure 7"}],"minor_comments":[{"comment":"The phrase 'full interpretability' is stronger than what is demonstrated: the framework uses SHAP values, which are approximate, and the final model includes both traditional features and hypothesis embeddings. Please temper the claim to 'interpretable' or specify the sense in which interpretability is full.","section":"Abstract and §2"},{"comment":"The pairwise correlation analysis reports that over 85% of variable pairs have absolute correlation below 0.2, but Appendix C acknowledges semantic overlap among several hypotheses (e.g., the multi-lane and road-width questions, and the two barrier questions). Clarify the relationship between low linear correlation and semantic redundancy.","section":"§4.4"},{"comment":"The manual audit is performed on a single panoramic SVI and a separate random sample of 50 SVIs, but the secondary inspection results are not quantified in the table. Providing a confusion matrix or summary statistics for the 50-SVI audit would strengthen the reliability claim.","section":"§4.6"},{"comment":"The y-axis label 'R20.0' appears to be a rendering artifact; the correct label should be 'R²'. Please fix the figure.","section":"Figure 3"},{"comment":"The limitations section acknowledges MLLM reliability and computational overhead, but it does not mention the selection-bias issue or spatial autocorrelation that affect the statistical conclusions. The appendix already notes redundancy and ambiguity in the hypothesis set; these concerns should be integrated into the main limitations discussion.","section":"§4.7 and Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a compelling framework idea, and the availability of code and the manual audit are positives. However, the inferential claims in Sections 4.3 and 4.4 are not valid as reported because of outcome-dependent hypothesis selection and ignored spatial dependence. These issues are fixable with a clean hold-out test set, selection-aware inference, and spatially robust standard errors, so I believe a major revision is appropriate rather than outright rejection. The predictive comparison also needs uncertainty quantification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the UrbanX pipeline is a real methodological novelty, but the paper's central discovery claim rests on p-values that are invalid as reported. The iterative selection loop generates and prunes hypotheses using the same crash-rate outcome that later produces the significance tests, so the retained p-values are inflated. That said, the framework itself is worth engaging with, and the authors include some honest self-criticism (Appendix C) and a small manual audit of MLLM answers that most similar papers skip.\n\nThe new thing is the loop: LLM proposes natural-language hypotheses, MLLM answers them as VQA on street view images, linear regression scores each, and the LLM gets the p-values back to generate the next batch. I haven't seen that exact loop before. The case study is substantial: 20,000 Manhattan street segments, several baseline vision models, a robustness analysis of model capacity and hypothesis count, and a reproducibility-minded GitHub link. The 84% manual accuracy on MLLM answers is an honest attempt to validate the weakest link. For a method paper, that's a solid foundation.\n\nThe soft spots. First, the selection problem is real and it's not minor. Algorithm 1 uses p-values on the training data to prune hypotheses, then Section 4.4 reports those same p-values as evidence of discovery. Under the null, variables that look significant are preferentially kept, so five percent of noise variables do not stay at 5% false positives — the retained set is enriched. The authors need a split-sample design (e.g., generate and prune on train, test the final p-values on a held-out split untouched during selection), or selective inference / a multiple-testing correction across 50 hypotheses. Without that, the 'statistically significant safety factors' claim falls.\n\nSecond, Figure 3 shows no error bars or significance tests for the RMSE/MAE/R² comparison, so the 'outperforms ResNet/ViT' claim is not established. It might be true, but as shown it's just point estimates.\n\nThird, Eq. 4 assumes independent errors across spatial segments. Adjacent Manhattan road segments are strongly spatially correlated, so the OLS standard errors are likely optimistic. This is the same direction as the selection bias — it inflates significance — so it reinforces the concern.\n\nWho's this for? Urban computing researchers, transportation safety people, and anyone building LLM-in-the-loop discovery pipelines. The framework is a genuine contribution even if the statistical packaging needs work. I'd cite it for the loop design and the prompt engineering, not for the specific discovered factors.\n\nRecommendation: send it to peer review. The flaws are mechanical and correctable, the method is novel, and the authors have already shown signs of engaging with limitations. I'd ask for a revision with a proper train/test split for the p-values, spatial correction on the regression, and error bars on the predictive comparison.","headline":"Novel LLM-driven hypothesis loop with an honest case study, but the reported p-values are inflated by outcome-dependent selection, so the discovery claim needs a split-sample fix.","tokens_in":23337,"tokens_out":2057,"would_cite":true,"duration_ms":18575,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an MLLM-driven loop of hypothesis generation, visual-question-answering embeddings, and significance-based pruning can discover interpretable road-safety factors that predict crash rates better than pretrained vision…","keywords":["Multimodal Large Language Models","Street View Imagery","Interpretable Modeling","Hypothesis Inference","Road Safety","Crash Rate Prediction","Urban Science","Visual Question Answering"],"falsifier":"Refit the final 50 hypothesis variables on a held-out set of Manhattan segments and test whether the same coefficients stay significant; if they do not, the discovery claim is not supported. A second check is to recompute standard errors with spatial clustering by neighborhood and see whether the significant variables survive.","tokens_in":22347,"feed_emoji":"🚦","tokens_out":8427,"duration_ms":69390,"temperature":0.7,"pith_summary":"This paper tries to establish that an automated pipeline can replace human hypothesis-making in urban science: a multimodal large language model (MLLM) proposes natural-language questions about street-view images, answers those questions, and feeds the answers as interpretable variables into regression models that predict crash rates. The framework, called UrbanX, prunes hypotheses whose coefficients are not significant and asks the language model to generate replacements, iterating until a compact set of transparent variables remains. On Manhattan road segments, the authors report that these interpretable embeddings predict crash rates better than fine-tuned ResNet and Vision Transformer baselines while preserving full traceability of each variable. If this holds, urban researchers would gain a scalable way to go from raw street imagery to statistically grounded, human-readable hypotheses about what makes streets safe.","feed_headline":"Street-view AI finds crash-risk factors, beats vision baselines","feed_subtitle":"Machines propose street-view questions, test them statistically, and beat black-box vision models.","key_machinery":"The carrying mechanism is the hypothesis-aligned embedding: each natural-language question is a hypothesis, and the MLLM's categorical answer to it is a dimension of the embedding vector for a street-view image. This makes every regression coefficient interpretable as the association between a concrete visual feature and crash rate. Around that embedding, Algorithm 1 runs an iterative posterior approximation: generate hypotheses from an LLM, extract embeddings via MLLM visual question answering, test each coefficient with a two-sided t-test, prune those with $p > \\alpha$, and generate replacements conditioned on the surviving hypotheses and their p-values. The paper also uses SHAP values on a regression that combines the discovered variables with 58 conventional built-environment features to rank which factors matter most.","core_discovery":"UrbanX's central discovery is that visual question answering by an MLLM can turn unstructured street-view imagery into structured, interpretable variables that carry more predictive signal about segment-level crash rates than latent representations from pretrained deep learning models. The claim is operationalized as an iterative loop: a language model proposes hypotheses of the form 'Is there a median strip separating opposing traffic?', an MLLM answers them from each image, linear regression measures each variable's association with crash rates via t-tests, and insignificant hypotheses are pruned and replaced. The final 50 variables include median strips, lane markings, pedestrian crossings, visible pedestrians, and billboards; the paper reports that over 85 percent of variable pairs have absolute correlation below 0.2 and that the top contributors match established traffic-safety knowledge. The authors further claim the same loop can be transplanted to other urban outcomes such as walkability, equity, and environmental quality.","pith_inferences":["The paper computes p-values after using the same crash-rate outcome to prune hypotheses, so the reported significance levels are likely overstated; confirming the final set on independent data would settle how much of the discovery is real.","The regression assumes independent errors across adjacent Manhattan segments; if nearby streets share unmeasured risk factors, standard errors may be too small and some retained hypotheses might flip under spatially clustered inference.","A direct testable extension is to run the identical pipeline in a second city; hypotheses that replicate across cities would be far stronger evidence of generalizable safety factors.","The manual audit blames most MLLM errors on image resolution rather than reasoning, which predicts that higher-resolution or zoomed imagery should improve the long tail of hypotheses with little architectural change."],"forward_implications":["An agency could train a crash-rate model whose inputs are plain-language questions about street views, so each prediction can be traced back to visible evidence.","The same generate-answer-test loop can be pointed at other street-view-linked outcomes, such as walkability, equity, or environmental quality, without manual variable engineering.","SHAP rankings of the discovered variables give planners a direct list of which visual features to change first, such as median strips, lane markings, or crossings.","On the Manhattan test set, the interpretable embeddings beat fine-tuned ResNet and Vision Transformer baselines on RMSE, MAE, and R-squared, indicating that transparency need not cost accuracy."],"supporting_citations":[{"why":"Supplies the ResNet-50 baseline whose predictive performance UrbanX must surpass in the experiments.","marker":"He et al. (2016)"},{"why":"Supplies the Vision Transformer baseline used in the same predictive comparison.","marker":"Dosovitskiy et al. (2020)"},{"why":"Documents GPT-4o, the language model used to generate hypotheses in the framework.","marker":"Hurst et al. (2024)"},{"why":"Introduces InternVL2.5, the MLLM used to answer the hypothesis questions from street-view images.","marker":"Chen et al. (2024)"},{"why":"Provides SHAP, the attribution method used to rank discovered variables against conventional built-environment features.","marker":"Lundberg and Lee (2017)"},{"why":"Supplies the crash-rate normalization formula used to define the target outcome.","marker":"Hou et al. (2020)"},{"why":"Also cited for the crash-rate definition and for modeling crash rates in safety research.","marker":"Zeng et al. (2017)"}],"fun_headline_variants":["AI proposes, tests street-safety hypotheses from images","Large language models turn street views into safety insights","Interpretable AI discovers road-safety factors in street views","Street-view AI offers transparent crash-risk findings","MLLM-driven urban science: from images to safety stats"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the final p-values remain valid evidence even though the hypotheses were chosen using the same crash-rate data, and that regression errors are independent across adjacent road segments.","fun_headline_variants_meta":{"raw":{"variants":["AI proposes, tests street-safety hypotheses from images","Large language models turn street views into safety insights","Interpretable AI discovers road-safety factors in street views","Street-view AI offers transparent crash-risk findings","MLLM-driven urban science: from images to safety stats"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000797,"raw_usage":{"total_tokens":3535,"prompt_tokens":1000,"completion_tokens":2535,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":2459}},"tokens_in":616,"tokens_out":2535,"duration_ms":17034,"temperature":1.0,"reasoning_tokens":2459,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:27:11.400024+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Refit the final 50 hypothesis variables on a held-out set of Manhattan segments and test whether the same coefficients stay significant; if they do not, the discovery claim is not supported. A second check is to recompute standard errors with spatial clustering by neighborhood and see whether the significant variables survive.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the crash-rate normalization formula used to define the target outcome."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Also cited for the crash-rate definition and for modeling crash rates in safety research."}],"review_version":1}