{"id":"449fb6c6-2b4a-4d1a-8ebd-86021363274a","arxiv_id":"2506.03360","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Multimodal LLMs infer earthquake damage levels from social media with city-level correlations up to 0.78 against USGS crowd-reported intensity, with strong language and modality dependence.","lead":"The authors used multimodal AI models that can read text and look at images to score earthquake damage from social media posts in English and Japanese. They found the models often match official crowd-reported shaking maps, but accuracy drops for Japanese posts, image-only inputs, and some models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline r=0.78 compares model MMI to DYFI, a crowd-perception measure that shares a perceptual basis with the model's prompt; without an independent structural benchmark the central claim overstates damage-assessment skill.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: DYFI is not an independent structural ground truth. My reading sharpens it further: the prompt design explicitly encodes the MMI perceptual scale, so model outputs and DYFI share a construct, and the city-level aggregation without confidence intervals amplifies the risk that r=0.78 is an artifact of small sample size and shared perceptual bias. The paper's independent contributions (released code, prompt library, and qualitative reasoning analysis) are real, but the quantitative headline should be interpreted as 'agreement with perceived-intensity reports' rather than as validation of damage assessment. This supports the reader's CONDITIONAL verdict and does not move it to ACCEPT or REJECT; hence UNCHANGED.","tokens_in":21063,"tokens_out":2548,"duration_ms":30861,"concrete_test":"For the same city-level units used in Figure 3, obtain USGS ShakeMap instrumental ground-motion estimates (PGA or instrumental MMI) and recompute Pearson r between model-estimated MMI and ShakeMap. Also compute r between DYFI and ShakeMap for those cities as a baseline. Report the number of cities and bootstrap 95% confidence intervals for the original r=0.78 and for the ShakeMap-based correlations. If model-to-ShakeMap r is substantially below model-to-DYFI r (or comparable to DYFI-to-ShakeMap baseline), the central claim of damage-assessment skill is overstated and the paper should be revised to describe the pipeline as predicting perceived intensity rather than structural damage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 validates model outputs against USGS DYFI, but DYFI is a self-reported measure of perceived shaking, and the model's MMI prompt (Appendix 7.4) explicitly asks it to infer the same kind of perceptual and observable cues ('felt by', 'frightens', 'damage to buildings'). Both sources are therefore keyed to the same latent human-perception construct, and both are crowd-generated, so high correlation may reflect shared perceptual/social bias rather than skill at estimating structural damage. Section 4.1 aggregates model predictions at the city level but reports no city counts or confidence intervals; the r=0.78 for Qwen on Ridgecrest could come from a small set of aggregated units given Qwen's limited spatial coverage shown in Figure 2(a). Tables 7-8 attempt a more structural 'exterior damage' subset, but the ground truth there is still DYFI MMI, not independent damage observations. The paper's own Limitations acknowledge DYFI subjectivity; that admission is not sufficient to support the abstract's 'strong correlation with ground-truth seismic data'. The claim would only be fully supported if an independent, non-perceptual ground truth (e.g., instrumental shaking or building inspection records) also correlated with model outputs.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a structured 'Multimodal, Multilingual, and Multidimensional' (3M) pipeline that uses multimodal large language models to assess earthquake damage from social media posts. The pipeline filters Twitter data with damage-related keywords, uses MLLMs to identify locations, verify event relevance, and assign Modified Mercalli Intensity (MMI) levels, and then validates model outputs against USGS 'Did You Feel It?' (DYFI) crowd-reported intensities. The authors evaluate three MLLMs (Gemini, LLaVA, Qwen) on two earthquakes (2019 Ridgecrest, 2021 Fukushima) and report city-level Pearson correlations between model-estimated damage and DYFI MMI, with the headline result of r=0.78 for Qwen on Ridgecrest. They also present micro-level analyses of input modality, prompt sensitivity, and model reasoning patterns, and they release code and data.","tokens_in":21263,"tokens_out":3017,"duration_ms":34670,"significance":"The paper addresses a practically important problem and introduces a reusable, open pipeline with real-world case studies in two languages. The design choices are mostly transparent: the prompts are fully specified, the filter terms are listed, and the reasoning traces are qualitatively analyzed. The prompt-sensitivity analysis and the reasoning-reliability taxonomy are useful contributions. However, the central quantitative claim—that MLLMs 'demonstrate a strong correlation with ground-truth seismic data'—rests on a ground-truth measure (DYFI) that is itself crowd-reported human perception, and the statistical reporting (city-level correlations without confidence intervals or sample sizes) is insufficient to support the strength of the claim. If the ground-truth validity and statistical issues are addressed, the pipeline could be a valuable benchmark for future work in crisis informatics.","major_comments":[{"comment":"The central claim of strong correlation with 'ground-truth seismic data' is overstated because the ground truth is DYFI, a crowdsourced measure of perceived shaking, while the MMI prompt explicitly asks models to infer perceptual and observable cues such as 'felt by', 'frightens', and 'damage to buildings'. Both the model inputs (social media text and images) and the validation labels are crowd-generated and keyed to the same human-perception construct, so the reported correlations (e.g., r=0.78 for Qwen on Ridgecrest) may reflect shared perceptual or reporting bias rather than independent skill at assessing structural damage. The Limitations section acknowledges DYFI's subjectivity, but this admission does not support the abstract's wording. The authors should either validate against an independent, non-perceptual benchmark (e.g., instrumental shaking records, building inspection data, or remote-sensing damage maps) or explicitly reframe all claims as correlations with crowd-perceived intensity.","section":"§3, §4.1, Appendix 7.4"},{"comment":"The city-level Pearson correlations are reported without confidence intervals, sample sizes, or significance tests. For example, the r=0.78 for Qwen on Ridgecrest is computed on an unspecified number of city-level aggregates, and Qwen's spatial coverage is visibly much smaller than the other models (Figure 2a), so this high correlation could be based on a small number of units and be unstable. The authors should report the number of cities/tweets per city and provide confidence intervals or p-values for each correlation, and consider Spearman correlations or bootstrap estimates to assess robustness. Without this, the headline numbers are not properly quantified.","section":"§4.1, Figure 3"},{"comment":"The exterior-damage comparison uses model-selected subsets of very different sizes: for Fukushima, Gemini's correlation is based on 1,207 tweets while LLaVA's is based on 24; for Ridgecrest, Gemini has 1,185 vs. LLaVA's 53. Because each model selects its own subset according to its own exterior-damage judgments, the Pearson R values are not directly comparable across models—the larger subsets may simply reflect broader recall, and the correlation on a tiny subset (e.g., 24 tweets) is statistically unreliable. The authors should evaluate all models on a fixed, common subset of tweets (e.g., tweets that all models or a gold standard identify as exterior-damage-relevant) and report confidence intervals for the correlations.","section":"§7.8, Tables 7–8"},{"comment":"The model-selection step reports accuracy values on a randomly selected sample of 50 tweets with an inter-annotator alpha of 0.67, but no confidence intervals are given, and one reported value is negative (GPT-4o-nano: -0.841). Accuracy is a proportion and cannot be negative, so this indicates a likely computational or reporting error. More importantly, the model ranking may be unstable on such a small, subjective sample; the authors should report the full contingency and provide a stability analysis (e.g., bootstrap) before using this ranking to select the three models for the main evaluation.","section":"§7.3, Table 5"}],"minor_comments":[{"comment":"The event is labeled the '2021 Fukushima earthquake' in the text and Figure 2, but Tables 1–2 and the Figure 8 caption say '2022 Fukushima earthquake'. Please correct the year consistently.","section":"Tables 1–2; Figure 8 caption"},{"comment":"There are typos in the prompts: 'accessment' should be 'assessment', and 'extual' should be 'textual'. Also, in Table 6, 'V oted candidate' appears to be a typo for 'Voted candidate'.","section":"Appendix 7.4"},{"comment":"The limitation paragraph beginning 'Second, the data retrieval process itself imposes restrictions' is duplicated verbatim; one copy should be removed.","section":"Limitations"},{"comment":"The column header 'Accuracy' with values including -0.841 and 0.957 needs clarification: state the evaluation metric explicitly and explain how accuracy can be negative.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely within scope for a computational social science or NLP venue, but the central quantitative claim needs to be reframed around crowd-perceived intensity unless an independent structural benchmark is added. The event-year inconsistency across tables and figures, while minor, suggests the final version should be carefully proofread. If the authors can address the ground-truth and statistical concerns, the released pipeline could be a useful resource for the community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful applied paper — the first systematic comparison I know of that runs MLLMs over English and Japanese earthquake social media, with modality and prompt ablations, and it ships code and prompts. It deserves a serious referee.\n\nWhat's new: the combination of MMI-anchored prompts, two events in two languages, three open/closed models, and the prompt-sensitivity analysis. None of the cited papers do that exact package. The TF-IDF reasoning traces are a nice interpretability add. The limitations section is honest: it names social-media sampling bias, keyword recall limits, and DYFI subjectivity.\n\nSoft spots, in order of size. First, the abstract's 'strong correlation with ground-truth seismic data' overstates what DYFI is. DYFI is crowd-reported perceived shaking, and the MMI prompt asks models to rate the same kind of perceptual/observable cues. So part of the r=0.78 for Qwen is plausibly shared perceptual bias rather than independent damage skill. That's a real concern, not a fatal one — DYFI is a standard macroseismic measure — but the paper should say 'perceived intensity' and either add an independent structural benchmark or soften the claim.\n\nSecond, the city-level correlations in Figure 3 have no confidence intervals or counts. Qwen's sparse spatial coverage makes its top correlation especially hard to interpret. Please give n and CI per city.\n\nThird, the exterior-damage comparison (Tables 7–8) uses subsets the models themselves selected, with very different sizes — Gemini 1207 vs LLaVA 24 on Fukushima. Those r values aren't comparable and should be framed as exploratory.\n\nMinor: the event is 2021 in the text and data collection, but Tables 1–2 and Figure 8 label it 2022. Fix the inconsistency.\n\nOverall: the central claim — multimodal fusion helps, language matters — is plausible and partially supported. The specific correlations are preliminary until the statistics are tightened. I'd accept this for review with major revisions, and I'd bring it to a reading group as a case study in MLLM evaluation for crisis informatics.","headline":"Useful multilingual MLLM disaster benchmark with honest limitations, but headline correlations need uncertainty stats and an independent validation target.","tokens_in":21816,"tokens_out":4155,"would_cite":true,"duration_ms":42679,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multimodal large language models can grade earthquake damage from social media posts, correlating with official intensity up to r=0.78.","keywords":["earthquake damage assessment","multimodal large language models","social media crowdsourcing","Modified Mercalli Intensity","few-shot chain-of-thought prompting","disaster response","multilingual evaluation","prompt sensitivity"],"falsifier":"Compare model-estimated MMI against independent structural damage records, such as post-earthquake building inspection databases or high-resolution satellite damage maps for the same cities; if the model's DYFI correlation stays high while its correlation with structural damage is near zero, the claimed skill is perceptual rather than physical. Alternatively, running the pipeline on a held-out earthquake in a low-resource language and checking whether the DYFI correlation survives would test the multilingual generalization.","tokens_in":20861,"feed_emoji":"🌍","tokens_out":6558,"duration_ms":66815,"temperature":0.7,"pith_summary":"This paper sets out to show that multimodal large language models can turn raw social media posts into fine-grained earthquake damage assessments. It builds a three-stage pipeline that filters damage-related tweets, has an MLLM locate the post, confirm it concerns the earthquake, and assign a Modified Mercalli Intensity level from text plus images, and then validates the result against DYFI crowd-reported intensity. Across the 2019 Ridgecrest and 2021 Fukushima earthquakes, the best model reaches r = 0.78 at city level, and text-image fusion consistently beats either modality alone. The payoff would be near-real-time, cross-lingual damage maps in the hours before official reports arrive, though the paper documents uneven performance across languages, distance from the epicenter, and prompt phrasing.","feed_headline":"r=0.78: Multimodal AI rates quake damage from tweets and photos","feed_subtitle":"Foundation models fuse text and images to estimate shaking intensity, pointing toward faster disaster response.","key_machinery":"The load-bearing object is the 3M pipeline itself. Its data stage collects tweets by keyword and filters them with a damage-term library; its evaluation stage uses few-shot chain-of-thought prompting to make an MLLM decide event relevance, geolocate the post, classify damage as interior/exterior/both/none, mark human impact, and output a Modified Mercalli Intensity level from 1 to 10; its validation stage compares model MMI to the 'Did You Feel It?' (DYFI) crowd reports using Pearson correlation, measures prompt sensitivity with Cramér's V, and inspects reasoning traces with TF-IDF. The fusion of text and image within the same prompt is what makes the correlation materialize; image-only inputs degrade sharply, especially for non-English content.","core_discovery":"The paper's central claim is that MLLMs effectively integrate image and text signals from social media and produce damage estimates that correlate with ground-truth seismic data. Concretely, model-estimated city-level damage correlates with DYFI intensity up to r = 0.78 for the 2019 Ridgecrest earthquake (Qwen), and at r = 0.47 for both LLaVA and Qwen on the 2021 Fukushima earthquake; Gemini collapses to r = 0.04 on the Japanese event. The models also recover expected physical attenuation, with estimated MMI falling as epicentral distance grows, and their reasoning traces show they lean on human-perception words at low intensities and structural words at high intensities. Performance depends on language familiarity, input modality (fusion helps), and prompt phrasing, so the authors frame the result as demonstrated capability with deployment caveats.","pith_inferences":["Because both the model inputs and the DYFI labels are crowd-generated perceptions, the r = 0.78 likely overstates agreement with physical structural damage; independent building-inspection or satellite damage data would be a harder test.","The same prompting scaffold could be adapted to other hazard types (wildfires, floods) and other languages, but the prompt-sensitivity results suggest each adaptation needs its own calibration study rather than a one-shot transfer.","A testable next step is to fuse the pipeline's MMI estimates with seismic sensor or remote-sensing data, which the paper's appendix hints at with Sentinel-2 scene classification; the combined signal could separate perceived shaking from actual structural impact.","The near-zero Gemini correlation on the Japanese event suggests the model may be relying on English-language cultural cues; probing with controlled Japanese/English paraphrases of the same content could isolate whether the gap is linguistic or visual."],"forward_implications":["If the correlation holds, a pipeline like this can produce city-level damage maps within hours of an earthquake, from public social media alone.","Text-image fusion is the configuration that tracks ground truth; text-only and image-only modes underperform, so deployment should route both modalities into the model.","Multilingual viability depends on model choice: Qwen holds up in Japanese while Gemini degrades, so language-aligned model selection matters for non-English crises.","Prompt standardization or ensembling would be needed in practice, since paraphrase variants shift categorical classifications (Cramér's V up to 0.77) even when MMI levels stay stable.","The pipeline's cost is low enough for open models to run at scale, making rapid, repeated assessment feasible during an event."],"supporting_citations":[{"why":"Supplies the DYFI crowd-reported intensity data used as ground truth for validating model MMI estimates.","marker":"Wald et al., 2011"},{"why":"Provides the few-shot prompting method used to elicit MMI classifications from the models.","marker":"Brown et al., 2020"},{"why":"Introduces chain-of-thought prompting, which the pipeline uses to generate reasoning traces and classifications.","marker":"Wei et al., 2022"},{"why":"Supplies the damage-related filter-term library and the prior social-media damage assessment framework the pipeline builds on.","marker":"Li et al., 2023"},{"why":"Defines the Gemini model family, one of the three MLLMs evaluated in the study.","marker":"Team et al., 2023"},{"why":"Defines Qwen 2.5-VL-7B, the open model that achieves the highest correlation.","marker":"Qwen Team, Alibaba Cloud, 2024"},{"why":"Defines LLaVA 3-8B, the open model whose reasoning trace is analyzed.","marker":"Labs, 2024"},{"why":"Motivates social media crowdsourcing and NLP methods for disaster damage assessment.","marker":"Ma et al., 2024b"},{"why":"Supports the tiered geolocation approach that prioritizes geotag metadata over content-based inference.","marker":"Stock, 2018"}],"fun_headline_variants":["AI fuses tweets and photos to estimate quake damage up to r=0.78","Multimodal LLMs match seismic shaking from social media posts","AI pipeline turns tweets and images into quake damage scores","Social media + multimodal AI gauges earthquake damage","Quake damage from social posts: AI matches seismic records"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation treats the DYFI crowd-reported shaking intensity as ground truth for damage; since those reports are self-reported perception and the model inputs are also crowd-generated social media, the measured agreement may reflect shared perceptual bias rather than true structural damage.","fun_headline_variants_meta":{"raw":{"variants":["AI fuses tweets and photos to estimate quake damage up to r=0.78","Multimodal LLMs match seismic shaking from social media posts","AI pipeline turns tweets and images into quake damage scores","Social media + multimodal AI gauges earthquake damage","Quake damage from social posts: AI matches seismic records"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000979,"raw_usage":{"total_tokens":4137,"prompt_tokens":904,"completion_tokens":3233,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":3146}},"tokens_in":520,"tokens_out":3233,"duration_ms":22622,"temperature":1.0,"reasoning_tokens":3146,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:04:24.890005+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare model-estimated MMI against independent structural damage records, such as post-earthquake building inspection databases or high-resolution satellite damage maps for the same cities; if the model's DYFI correlation stays high while its correlation with structural damage is near zero, the claimed skill is perceptual rather than physical. Alternatively, running the pipeline on a held-out earthquake in a low-resource language and checking whether the DYFI correlation survives would test the multilingual generalization.","supporting_citations":[],"review_version":1}