Pith. sign in

REVIEW 4 major objections 4 minor 12 references

A Multimodal, Multilingual, and Multidimensional Pipeline for Fine-grained Crowdsourcing Earthquake Damage Evaluation

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read Multimodal large language models can grade earthquake damage from social media posts, correlating with official intensity up to r=0.78.

desk verdict Useful multilingual MLLM disaster benchmark with honest limitations, but headline correlations need uncertainty stats and an independent validation target. read the letter →

arxiv 2506.03360 v1 pith:QKFG4FPB submitted 2025-06-03 cs.CL cs.CYcs.SI

classification cs.CLcs.CYcs.SI
keywords earthquakedamageassessmentmultimodallargelanguagemodelssocialmediacrowdsourcingModifiedMercalliIntensityfew-shotchain-of-thoughtpromptingdisasterresponsemultilingualevaluationpromptsensitivity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that multimodal large language models can turn raw social media posts into fine-grained earthquake damage assessments. It builds a three-stage pipeline that filters damage-related tweets, has an MLLM locate the post, confirm it concerns the earthquake, and assign a Modified Mercalli Intensity level from text plus images, and then validates the result against DYFI crowd-reported intensity. Across the 2019 Ridgecrest and 2021 Fukushima earthquakes, the best model reaches r = 0.78 at city level, and text-image fusion consistently beats either modality alone. The payoff would be near-real-time, cross-lingual damage maps in the hours before official reports arrive, though the paper documents uneven performance across languages, distance from the epicenter, and prompt phrasing.

What carries the argument

The load-bearing object is the 3M pipeline itself. Its data stage collects tweets by keyword and filters them with a damage-term library; its evaluation stage uses few-shot chain-of-thought prompting to make an MLLM decide event relevance, geolocate the post, classify damage as interior/exterior/both/none, mark human impact, and output a Modified Mercalli Intensity level from 1 to 10; its validation stage compares model MMI to the 'Did You Feel It?' (DYFI) crowd reports using Pearson correlation, measures prompt sensitivity with Cramér's V, and inspects reasoning traces with TF-IDF. The fusion of text and image within the same prompt is what makes the correlation materialize; image-only inputs degrade sharply, especially for non-English content.

What would settle it

Compare model-estimated MMI against independent structural damage records, such as post-earthquake building inspection databases or high-resolution satellite damage maps for the same cities; if the model's DYFI correlation stays high while its correlation with structural damage is near zero, the claimed skill is perceptual rather than physical. Alternatively, running the pipeline on a held-out earthquake in a low-resource language and checking whether the DYFI correlation survives would test the multilingual generalization.

Watch

Extended reading notes

Core claim

The paper's central claim is that MLLMs effectively integrate image and text signals from social media and produce damage estimates that correlate with ground-truth seismic data. Concretely, model-estimated city-level damage correlates with DYFI intensity up to r = 0.78 for the 2019 Ridgecrest earthquake (Qwen), and at r = 0.47 for both LLaVA and Qwen on the 2021 Fukushima earthquake; Gemini collapses to r = 0.04 on the Japanese event. The models also recover expected physical attenuation, with estimated MMI falling as epicentral distance grows, and their reasoning traces show they lean on human-perception words at low intensities and structural words at high intensities. Performance depends on language familiarity, input modality (fusion helps), and prompt phrasing, so the authors frame the result as demonstrated capability with deployment caveats.

Load-bearing premise

The evaluation treats the DYFI crowd-reported shaking intensity as ground truth for damage; since those reports are self-reported perception and the model inputs are also crowd-generated social media, the measured agreement may reflect shared perceptual bias rather than true structural damage.

Editorial extensions

If this is right

  • If the correlation holds, a pipeline like this can produce city-level damage maps within hours of an earthquake, from public social media alone.
  • Text-image fusion is the configuration that tracks ground truth; text-only and image-only modes underperform, so deployment should route both modalities into the model.
  • Multilingual viability depends on model choice: Qwen holds up in Japanese while Gemini degrades, so language-aligned model selection matters for non-English crises.
  • Prompt standardization or ensembling would be needed in practice, since paraphrase variants shift categorical classifications (Cramér's V up to 0.77) even when MMI levels stay stable.
  • The pipeline's cost is low enough for open models to run at scale, making rapid, repeated assessment feasible during an event.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because both the model inputs and the DYFI labels are crowd-generated perceptions, the r = 0.78 likely overstates agreement with physical structural damage; independent building-inspection or satellite damage data would be a harder test.
  • The same prompting scaffold could be adapted to other hazard types (wildfires, floods) and other languages, but the prompt-sensitivity results suggest each adaptation needs its own calibration study rather than a one-shot transfer.
  • A testable next step is to fuse the pipeline's MMI estimates with seismic sensor or remote-sensing data, which the paper's appendix hints at with Sentinel-2 scene classification; the combined signal could separate perceived shaking from actual structural impact.
  • The near-zero Gemini correlation on the Japanese event suggests the model may be relying on English-language cultural cues; probing with controlled Japanese/English paraphrases of the same content could isolate whether the gap is linguistic or visual.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a structured 'Multimodal, Multilingual, and Multidimensional' (3M) pipeline that uses multimodal large language models to assess earthquake damage from social media posts. The pipeline filters Twitter data with damage-related keywords, uses MLLMs to identify locations, verify event relevance, and assign Modified Mercalli Intensity (MMI) levels, and then validates model outputs against USGS 'Did You Feel It?' (DYFI) crowd-reported intensities. The authors evaluate three MLLMs (Gemini, LLaVA, Qwen) on two earthquakes (2019 Ridgecrest, 2021 Fukushima) and report city-level Pearson correlations between model-estimated damage and DYFI MMI, with the headline result of r=0.78 for Qwen on Ridgecrest. They also present micro-level analyses of input modality, prompt sensitivity, and model reasoning patterns, and they release code and data.

Significance. The paper addresses a practically important problem and introduces a reusable, open pipeline with real-world case studies in two languages. The design choices are mostly transparent: the prompts are fully specified, the filter terms are listed, and the reasoning traces are qualitatively analyzed. The prompt-sensitivity analysis and the reasoning-reliability taxonomy are useful contributions. However, the central quantitative claim—that MLLMs 'demonstrate a strong correlation with ground-truth seismic data'—rests on a ground-truth measure (DYFI) that is itself crowd-reported human perception, and the statistical reporting (city-level correlations without confidence intervals or sample sizes) is insufficient to support the strength of the claim. If the ground-truth validity and statistical issues are addressed, the pipeline could be a valuable benchmark for future work in crisis informatics.

major comments (4)
  1. [§3, §4.1, Appendix 7.4] The central claim of strong correlation with 'ground-truth seismic data' is overstated because the ground truth is DYFI, a crowdsourced measure of perceived shaking, while the MMI prompt explicitly asks models to infer perceptual and observable cues such as 'felt by', 'frightens', and 'damage to buildings'. Both the model inputs (social media text and images) and the validation labels are crowd-generated and keyed to the same human-perception construct, so the reported correlations (e.g., r=0.78 for Qwen on Ridgecrest) may reflect shared perceptual or reporting bias rather than independent skill at assessing structural damage. The Limitations section acknowledges DYFI's subjectivity, but this admission does not support the abstract's wording. The authors should either validate against an independent, non-perceptual benchmark (e.g., instrumental shaking records, building inspection data, or remote-sensing damage maps) or explicitly reframe all claims as correlations with crowd-perceived intensity.
  2. [§4.1, Figure 3] The city-level Pearson correlations are reported without confidence intervals, sample sizes, or significance tests. For example, the r=0.78 for Qwen on Ridgecrest is computed on an unspecified number of city-level aggregates, and Qwen's spatial coverage is visibly much smaller than the other models (Figure 2a), so this high correlation could be based on a small number of units and be unstable. The authors should report the number of cities/tweets per city and provide confidence intervals or p-values for each correlation, and consider Spearman correlations or bootstrap estimates to assess robustness. Without this, the headline numbers are not properly quantified.
  3. [§7.8, Tables 7–8] The exterior-damage comparison uses model-selected subsets of very different sizes: for Fukushima, Gemini's correlation is based on 1,207 tweets while LLaVA's is based on 24; for Ridgecrest, Gemini has 1,185 vs. LLaVA's 53. Because each model selects its own subset according to its own exterior-damage judgments, the Pearson R values are not directly comparable across models—the larger subsets may simply reflect broader recall, and the correlation on a tiny subset (e.g., 24 tweets) is statistically unreliable. The authors should evaluate all models on a fixed, common subset of tweets (e.g., tweets that all models or a gold standard identify as exterior-damage-relevant) and report confidence intervals for the correlations.
  4. [§7.3, Table 5] The model-selection step reports accuracy values on a randomly selected sample of 50 tweets with an inter-annotator alpha of 0.67, but no confidence intervals are given, and one reported value is negative (GPT-4o-nano: -0.841). Accuracy is a proportion and cannot be negative, so this indicates a likely computational or reporting error. More importantly, the model ranking may be unstable on such a small, subjective sample; the authors should report the full contingency and provide a stability analysis (e.g., bootstrap) before using this ranking to select the three models for the main evaluation.
minor comments (4)
  1. [Tables 1–2; Figure 8 caption] The event is labeled the '2021 Fukushima earthquake' in the text and Figure 2, but Tables 1–2 and the Figure 8 caption say '2022 Fukushima earthquake'. Please correct the year consistently.
  2. [Appendix 7.4] There are typos in the prompts: 'accessment' should be 'assessment', and 'extual' should be 'textual'. Also, in Table 6, 'V oted candidate' appears to be a typo for 'Voted candidate'.
  3. [Limitations] The limitation paragraph beginning 'Second, the data retrieval process itself imposes restrictions' is duplicated verbatim; one copy should be removed.
  4. [Table 5] The column header 'Accuracy' with values including -0.841 and 0.957 needs clarification: state the evaluation metric explicitly and explain how accuracy can be negative.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: model outputs are compared against external DYFI ground truth without fitted parameters; the DYFI subjectivity concern is a validity threat, not a circularity.

full rationale

The paper's derivation chain is self-contained. The pipeline filters tweets with a term library, prompts MLLMs for MMI levels, and compares the resulting estimates to USGS DYFI data. No parameters are fitted to the evaluation set, and no prediction is constructed from the ground-truth values. The filter terms borrowed from Li et al. (2023) are data-preparation heuristics rather than outputs of the central claim, so that self-citation is not load-bearing. The MMI prompt uses standard MMI scale descriptors, and DYFI is also MMI-based; evaluating with a shared perceptual scale is a construct-validity limitation, which the paper explicitly acknowledges in its Limitations section, rather than a circular reduction. No fitted quantity is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no known result is repackaged as novel. The strongest concern raised by the reader, that DYFI and the model share crowd-perceptual bias, concerns the validity of the benchmark, not the logical dependence of the result on its inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No fitted numeric parameters appear in the pipeline; the reported quantities are correlations, and the MMI scale is an input rubric, not a fitted parameter. The axioms are domain assumptions about data representativeness, ground-truth validity, aggregation, and zero-shot model behavior. The paper introduces no new physical or conceptual entities.

assumptions (4)
  • domain assumption Twitter keyword 'earthquake' plus a hand-built damage-term filter yields a representative sample of damage-related posts.
    Section 3 Data Preparation defines the dataset through this filter, so recall and representativeness depend entirely on the term list and single keyword.
  • domain assumption USGS DYFI crowd-reported MMI is an appropriate ground truth for earthquake damage assessment.
    Section 3 Model Selection and Validation treats DYFI as authoritative, but it is self-reported perception rather than measured structural damage and may share bias with social media reports.
  • domain assumption City-level averages of model MMI predictions can be meaningfully compared to city-level DYFI values via Pearson correlation.
    Section 4.1 Figure 3 uses this aggregation; small city counts and averaging hide per-post error and make correlation coefficients unstable.
  • domain assumption Zero-shot MLLM outputs are valid proxies for human damage assessment without task-specific fine-tuning.
    Section 3 and the Limitations acknowledge no fine-tuning was performed; the entire evaluation relies on this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Multimodal, Multilingual, and Multidimensional Pipeline for Fine-grained Crowdsourcing Earthquake Damage Evaluation." pith.science (2026). https://pith.science/paper/QKFG4FPB

@misc{pith2026250603360,
  author       = {Pith},
  title        = {Pith review of: A Multimodal, Multilingual, and Multidimensional Pipeline for Fine-grained Crowdsourcing Earthquake Damage Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QKFG4FPB}},
  note         = {Machine review of arXiv:2506.03360}
}
read the original abstract

Rapid, fine-grained disaster damage assessment is essential for effective emergency response, yet remains challenging due to limited ground sensors and delays in official reporting. Social media provides a rich, real-time source of human-centric observations, but its multimodal and unstructured nature presents challenges for traditional analytical methods. In this study, we propose a structured Multimodal, Multilingual, and Multidimensional (3M) pipeline that leverages multimodal large language models (MLLMs) to assess disaster impacts. We evaluate three foundation models across two major earthquake events using both macro- and micro-level analyses. Results show that MLLMs effectively integrate image-text signals and demonstrate a strong correlation with ground-truth seismic data. However, performance varies with language, epicentral distance, and input modality. This work highlights the potential of MLLMs for disaster assessment and provides a foundation for future research in applying MLLMs to real-time crisis contexts. The code and data are released at: https://github.com/missa7481/EMNLP25_earthquake

Figures

Figures reproduced from arXiv: 2506.03360 by the authors.

Figure 1
Figure 1. Proposed 3M pipeline, which integrates data preparation, damage evaluation framework, and model [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Spatial distribution of (a) Ridgecrest and (b) Fukushima data points identified by LLaVA, Qwen, and [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Correlation between model-estimated average damage levels and DYFI MMI levels for (a) Ridgecrest and [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Epicentral distance vs. model estimated MMI [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 6
Figure 6. Figure 6: Reasoning reliability evaluation for (a) Ridgecrest and (b) Fukushima. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Integrated map for the 2019 Ridgecrest earthquake from (a) LLaVA 3-8B, (b) Qwen-2.5-VL-7B, and (c) [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Integrated map for the 2022 Fukushima earthquake from (a) LLaVA 3-8B, (b) Qwen-2.5-VL-7B, and (c) [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 8 canonical work pages

  1. [8]

    traditional computer vision

    Deep learning vs. traditional computer vision. InAdvances in computer vision: proceedings of the 2019 computer vision conference (CVC), volume 1 1, pages 128–144. Springer. Pew Research Center. 2022. 10 facts about americans and twitter. Accessed: 2025-05-18. Qwen Team, Alibaba Cloud. 2024. Qwen2.5: The lat- est vision-language model from alibaba. Accesse...

  2. [10]

    Matthew M Torok, Mani Golparvar-Fard, and Kevin B Kochersberger

    Large language models (llm) in computa- tional social science: prospects, current state, and challenges.Social Network Analysis and Mining, 15(1):1–30. Matthew M Torok, Mani Golparvar-Fard, and Kevin B Kochersberger. 2014. Image-based automated 3d crack detection for post-disaster building assess- ment.Journal of Computing in Civil Engineering, 28(5):A401...

  3. [12]

    Chen Yu and Zhiguo Wang

    Crisissense-llm: Instruction fine-tuned large language model for multi-label social media text classification in disaster informatics.arXiv preprint arXiv:2406.15477. Chen Yu and Zhiguo Wang. 2024. Multimodal social sensing for the spatio-temporal evolution and assess- ment of nature disasters.Sensors, 24(18):5889. Jingming Zhuo, Songyang Zhang, Xinyu Fan...

  4. [2010]

    V olodymyr V Mihunov, Navid H Jafari, Kejin Wang, Nina SN Lam, and Dylan Govender

    Cross-modal congruency benefits for com- bined tactile and visual signaling.The American journal of psychology, 123(4):413–424. V olodymyr V Mihunov, Navid H Jafari, Kejin Wang, Nina SN Lam, and Dylan Govender. 2022. Disaster impacts surveillance from social media with topic modeling and feature extraction: Case of hurricane harvey.International Journal o...

  5. [2011]

    did you feel it?

    Usgs “did you feel it?” internet-based macro- seismic intensity maps.Annals of geophysics, 54(6). Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Sharon Li, and Neel Joshi. 2024. Is a picture worth a thousand words? delving into spatial reasoning for vision language models.Advances in Neural Information Processing Systems, 37:75392– 75421. T...

  6. [2012]

    In15th world conference on earthquake engineering, volume 10

    Fema p58: Next-generation building seismic performance assessment methodology. In15th world conference on earthquake engineering, volume 10. Amanda Lee Hughes and Holden Clark. 2025. Seeing the storm: Leveraging multimodal llms for disaster social media video filtering. InProceedings of the International ISCRAM Conference. J Jelének and Kopa ˇcková-Strnad...

  7. [2014]

    Accurate Local Estimation of Geo-Coordinates for Social Media Posts

    Accurate local estimation of geo- coordinates for social media posts.arXiv preprint arXiv:1410.4616. Paula García-Tapia-Mateo, Andrés Bueno-Crespo, Irene Garrigos, Jose-Norberto Mazón, José Cecilia, and Juan Morales-García. 2025. Explainable deep learn- ing for early detection of natural disasters through social media text classification.Available at SSRN...

  8. [2019]

    Aligning AI with Public Values: Deliberation and Decision-Making for Governing Multimodal LLMs in Political Video Analysis

    Disaster management 2.0: A real-time disaster damage assessment model based on mobile social media data—a case study of weibo (chinese twitter). Safety science, 115:393–413. Tanusree Sharma, Yujin Potter, Zachary Kilhoffer, Yun Huang, Dawn Song, and Yang Wang. 2024. From ex- perts to the public: Governing multimodal language models in politically sensitiv...

Show all 12 references
  1. [2020]

    Derek Doran, Swapna Gokhale, and Aldo Dagnino

    Machine-learning methods for identifying social media-based requests for urgent help during hurricanes.International Journal of Disaster Risk Reduction, 51:101757. Derek Doran, Swapna Gokhale, and Aldo Dagnino

  2. [2023]

    International Journal of Disaster Risk Reduction, 98:104062

    Exploring the potential of social media crowd- sourcing for post-earthquake damage assessment. International Journal of Disaster Risk Reduction, 98:104062. Lingyao Li, Zihui Ma, and Tao Cao. 2021. Data-driven investigations of using social media to aid evacua- tions amid weste...

  3. [2024]

    New York University

    Unveiling the dynamics of crisis events: Sen- timent and emotion analysis via multi-task learning with attention mechanism and subject-based intent prediction.Information Processing & Management, 61(4):103695. New York University. 2024. Nyu high performance computing. Accessed...

  4. [2025]

    Niall O’Mahony, Sean Campbell, Anderson Carvalho, Suman Harapanahalli, Gustavo Velasco Hernandez, Lenka Krpalkova, Daniel Riordan, and Joseph Walsh

    Ai in social good: Llm powered interventions in crisis management and disaster response.J Artif Intell Mach Learn & Data Sci 2025, 3(1):2353–2360. Niall O’Mahony, Sean Campbell, Anderson Carvalho, Suman Harapanahalli, Gustavo Velasco Hernandez, Lenka Krpalkova, Daniel Riordan,...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.