{"id":"f788e49a-3d46-44d1-a0f1-35097554b48b","arxiv_id":"2608.09772","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new controlled benchmark shows that four open-source vision-language models detect multimodal sarcasm largely from lexical, stylistic, and OCR surface cues, not from pragmatic image-text understanding.","lead":"PragMatch is a new 3,000-pair benchmark that tests whether image-language AI models detect sarcasm by really understanding images and captions or by grabbing surface cues such as hashtags, emojis, and visible text. It shows that several popular models flip their verdicts when harmless markers are added, suggesting they rely on shortcuts rather than on the relation between image and text.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Injection interventions are not validated to preserve the gold label: OCR and lexical cues plausibly alter the image-text relation, so the Table 3 flips do not yet establish shortcut reliance.","rationale":"After reading the paper in good faith, I find that its central diagnostic contribution is the masking/injection analysis in Section 5.2, which purports to show that surface cues flip predictions while the underlying image-text relation is unchanged. The load-bearing assumption is that the perturbations do not change the relation or the gold label. The paper states this assumption but does not substantiate it with human ratings of the perturbed pairs. The OCR template in Table 5 explicitly asserts text that is not present in the image ('Visible text in the image says: \"Yeah, right.\"'), which changes what a rational model should infer about the image. Lexical and style markers are also pragmatically loaded. Without a validation study, the strongest empirical claim of the paper is ungrounded. The reader's weakest assumption identifies the same issue, though focuses on OCR; I would extend it to all cue families. This is why agreement_with_reader is 'partial.' I considered whether the paper's other findings survive even if this concern lands. The paired-accuracy results (Table 1), the AUROC analysis (Table 2), and the text-only baseline (99.6% accuracy) are computed on clean examples and do not depend on the injection premise. They support the weaker claim that standard accuracy overestimates relational understanding. However, the paper's headline contribution, the shortcut diagnosis with causal language ('causing substantial changes... despite unchanged underlying image-text relationships'), is exactly what the injection analysis is supposed to establish. If the premise fails, those results are uninterpretable as shortcut evidence. I also note a separate factual issue: the text in Section 5.1 states 'no model is above chance on both P and M,' but Table 1 shows LLaVA-1.6-7B zero-shot achieves 62.5% on P and 85.9% on M, both above chance. This is a direct contradiction in the reported results and should be corrected, but it does not affect the injection concern. The appropriate verdict remains CONDITIONAL: the paper should not be trusted as a shortcut-learning testbed until the perturbation labels are validated, but its clean-condition metrics and diagnostic framework are still potentially valuable. The concrete test above would settle whether the central causal claim holds.","tokens_in":13861,"tokens_out":9899,"duration_ms":81298,"concrete_test":"Draw a random sample of 100 injected non-sarcastic examples (50 M and 50 L) and 100 masked sarcastic examples, stratified by cue type. Have two annotators who passed the original annotation protocol independently label each perturbed pair as sarcastic or not, and also judge whether the perturbation preserves the original image-text relation (unchanged vs. changed). Compute the proportion of injected examples where annotators assign 'sarcastic' or where they say the relation changed. If this proportion is significantly above zero (e.g., >5%), the 'gold label unchanged' assumption is violated and the Table 3 flips cannot be attributed to shortcut reliance. As a secondary check, for OCR injections, verify against the source images whether the claimed visible text actually appears; if it does not, the injection introduces a false assertion about the image, which is itself a relation change.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The causal interpretation of the shortcut diagnostics in Section 5.2 rests on the premise stated in Section 3.3 that masking and injection 'modify only surface form without altering the image-caption relationship or the gold label.' This premise is not validated for the injected cues. Table 5 shows that the OCR template appends 'Visible text in the image says: \"Yeah, right.\"' to the caption; this is not a surface edit but a new factual assertion about the image content, which can create a second layer of incongruity and thereby change the relation to a sarcastic one. Lexical templates such as 'lol' or 'ironic' and style templates such as '#nice' are also conventional sarcasm markers that can shift the perceived intent of an otherwise literal caption. Since no human rating of perturbed pairs is reported, we cannot know whether the gold label remains non-sarcastic after injection. If the label changes, then the prediction flips in Table 3 (e.g., LLaVA's M accuracy dropping from 85.9 to 9.4 after OCR injection) are exactly what a relation-tracking model should do, rather than evidence of shortcut reliance. The paper's own quality control (Section 3.4) validates original labels but not perturbed ones. This concern is most acute for OCR, which the paper reports as producing the largest effects, so the central claim 'injected surface signals causing substantial changes in model predictions despite unchanged underlying image-text relationships' is currently unsupported for the cue family that matters most.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PragMatch, a 3,000-pair benchmark derived from MMSD2.0 in which each source image is paired with a sarcastic pragmatic-incongruity caption (P), a literal caption (L), and a non-pragmatic mismatch caption (M). It evaluates four open LVLMs under zero-shot and chain-of-thought prompting, reporting standard accuracy, per-condition accuracy, paired/grouped accuracy, and AUROC. It further proposes masking and injection perturbations over lexical, style, and OCR cues to diagnose shortcut reliance. The central claim is that models are sensitive to surface cues and that injected cues change predictions even though the underlying image-text relation and gold label are supposedly unchanged; on this basis the authors argue that current LVLMs do not genuinely perform multimodal pragmatic reasoning.","tokens_in":14123,"tokens_out":7739,"duration_ms":69538,"significance":"If the central claim is supported, PragMatch is a valuable diagnostic resource: it separates pragmatic incongruity from generic cross-modal mismatch while holding the image fixed, and its paired/grouped accuracy metrics expose a genuine gap between standard accuracy and relation-level consistency. The paper also deserves credit for reporting AUROC separately from discrete decisions, for human validation of the original labels, and for constructing hard negatives with CLIP and RelSim rather than relying on trivially unrelated captions. However, the paper's headline shortcut-reliance conclusion currently rests on two unverified or internally inconsistent pieces of evidence: the injection interventions are not validated to preserve the gold label, and the perturbation results in Table 3 compare baselines and paired accuracies on different subsets in ways that produce impossible values. The benchmark itself is useful even if the causal shortcut claim is subsequently softened to a claim about cue sensitivity.","major_comments":[{"comment":"The claim that all interventions 'modify only surface form without altering the image-caption relationship or the gold label' is asserted but not validated. The OCR template appends 'Visible text in the image says: \"Yeah, right.\"' to the caption; this is not a purely surface edit because it introduces a new factual assertion about the image, and it can plausibly change the relation from non-sarcastic to sarcastic. Lexical and style templates such as 'lol', 'ironic', and '#nice' are conventional sarcasm markers that can shift the perceived intent of an otherwise literal caption. Section 3.4 reports human validation only for the original labels, not for the perturbed pairs. Therefore the prediction flips in Table 3 (e.g., LLaVA-v1.6-7B's M accuracy dropping from 85.9 to 9.4 after OCR injection) do not yet establish shortcut reliance; they could reflect a reasonable update to a changed image-text relation. I ask the authors to either provide human ratings confirming that the gold label is invariant under the injected perturbations, or reframe the injection results as evidence of sensitivity to cue presence rather than as evidence of shortcut reliance.","section":"Section 3.3 and Table 5"},{"comment":"The baseline comparisons in Table 3 are not computed on matched subsets. Section 3.5 states that masking is applied only when the relevant cue is present (n=19 for lexical, n=638 for style, n=505 for OCR), but the reported baseline P values are the full-set AccP from Table 1. Consequently, a paired accuracy such as LLaVA-v1.6-7B's P-P- of 100.0 cannot be compared with its full-set baseline P of 62.5, and the negative 'Drop' values cannot be interpreted as evidence of response bias. The same problem appears on the injection side: Qwen2.5-VL-7B CoT reports M-M+ of 79.1 for the lexical cue against a baseline M of 61.5, which is impossible if M-M+ requires the original M prediction to be correct on the same examples. The manuscript needs to recompute baselines on the exact subset used for each paired metric, report those subset-specific baselines, and then re-evaluate the 'Drop' analysis and the claims about masking and injection in Section 5.2.","section":"Table 3 and Section 4.1"},{"comment":"The sentence 'no model is above chance on both P and M' is directly contradicted by the paper's own Table 1: LLaVA-v1.6-7B under zero-shot prompting achieves P=62.5 and M=85.9, both above 50. The paper appears to have meant that no model is above chance on both P and L, or that paired accuracy is below chance for three of four models, but as written the statement is false and it weakens the argument that no model shows per-condition relational competence. This needs correction, and the surrounding discussion should be adjusted so that the claim about below-chance behavior refers specifically to paired/grouped accuracy rather than to per-condition accuracy.","section":"Section 5.1 and Table 1"}],"minor_comments":[{"comment":"In the per-condition discussion, LLaVA-v1.6-7B is described as scoring 62.5% on P but only 43.1% on L; the table also shows M=85.9, so the sentence should clarify that the contrast is with L, not with M.","section":"Section 5.1"},{"comment":"The notation 'P –P −', 'P −–M +', and 'P–P −' is visually confusing because the minus signs are sometimes rendered as separate dashes; please use a consistent notation such as Pmasked, Minjected, and paired metrics defined once in Section 4.1.","section":"Notation"},{"comment":"There are typographical spacing issues in the author line: 'V era Demberg' and 'LLaV A-v1.6-7B' should be corrected to 'Vera Demberg' and 'LLaVA-v1.6-7B'.","section":"Author line"},{"comment":"The Limitations paragraph correctly acknowledges that the three conditions differ in caption provenance and style, but the abstract's global wording ('current LVLMs do not perform genuine multimodal pragmatic reasoning') is stronger than what the cross-condition comparisons can support; consider softening the abstract to reflect the acknowledged confound.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and relation-level evaluation are a useful contribution, but the shortcut diagnostic analysis currently has two load-bearing problems: unvalidated label preservation under injection, and subset mismatches in Table 3 that produce impossible paired-accuracy values. Both are fixable within the scope of the paper, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for the pointer. I read PragMatch carefully. The core construction is genuinely useful: for each image they pair the original sarcastic caption with a literal description and a hard-negative mismatch from a visually similar image, then score models on P/M/L jointly. The main empirical finding is strong and doesn't depend on the perturbation machinery: standard accuracy overestimates pragmatic reasoning by a lot. LLaVA-v1.6 drops from 74.2% item accuracy to 19.0% paired P-M-L; three of four models sit below the 25% chance paired accuracy. Text-only inputs match or beat image+text for most models. That is a clean, reproducible observation about current LVLMs, and the AUROC analysis adds a useful separation between representation quality and decision bias.\n\nThe soft spot is exactly where the stress-test note lands. The shortcut diagnosis in Section 5.2 assumes that masking and injection 'modify only surface form without altering the image-caption relationship or the gold label.' The paper asserts this but never validates it. The OCR template appends 'Visible text in the image says: Yeah, right.' That is a new factual claim about image content, and it can reasonably flip a non-sarcastic pair to sarcastic. Similarly 'lol' and '#nice' are conventional sarcasm markers; their presence can change perceived speaker intent. So LLaVA's M accuracy dropping from 85.9 to 9.4 after OCR injection is exactly what a relation-tracking model should do if the label changed. Without human rating of the perturbed pairs, the central claim in the abstract — unchanged underlying relationships — is unsupported for the cue family that produces the largest effects. The paper's own limitations section is honest about cross-condition style differences, but then claims the perturbation results are unaffected; that is the gap.\n\nMinor points: the lexical masking subset is 19 examples, the benchmark is not yet released, and the hard-negative selection uses CLIP/RelSim thresholds that are reasonable but not deeply justified.\n\nOverall: the benchmark and the relation-level evaluation are worth taking seriously. The shortcut mechanism claims need either validation of perturbed labels or a qualified rewrite. I would send this to peer review — it deserves referee time — and I'd cite the paired-accuracy result. The perturbation section needs revision before the benchmark becomes a trusted testbed.","headline":"Solid relation-level benchmark and a striking paired-accuracy result, but the shortcut claim rests on perturbations that may change the label rather than only surface form.","tokens_in":14702,"tokens_out":2842,"would_cite":true,"duration_ms":24436,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large vision-language models fail to distinguish sarcastic incongruity from ordinary image–text mismatch.","keywords":["PragMatch","multimodal sarcasm detection","large vision-language models","shortcut learning","pragmatic incongruity","paired accuracy","surface cues","benchmark"],"falsifier":"Give human annotators the original and perturbed caption pairs and ask whether the image–caption relationship (and the sarcasm label) is preserved; if a substantial share of OCR-injected pairs are judged to change the relationship or become sarcastic, the shortcut attribution collapses for that cue family. Alternatively, a model that achieves high paired accuracy on P–M and stays invariant under all three perturbation families would directly refute the claim.","tokens_in":13664,"feed_emoji":"😏","tokens_out":5008,"duration_ms":40109,"temperature":0.7,"pith_summary":"PragMatch is a controlled test of whether large vision-language models (LVLMs) detect sarcasm by understanding pragmatic incongruity or by latching onto surface cues. The benchmark collects 3,000 image–caption pairs in three conditions that share the same image: sarcastic captions (P), literal descriptions (L), and non-pragmatic mismatches (M) drawn from visually similar images. The paper argues that standard accuracy overestimates pragmatic understanding because no model scores above chance on both P and M, and paired accuracy measuring joint correctness falls below 25% chance for three of four models. Masking and injection experiments show that lexical, stylistic, and OCR-derived cues can flip predictions even though the image–caption relationship and gold label are unchanged. If correct, this means current LVLMs are not performing genuine multimodal pragmatic reasoning, and evaluation must move to relation-level consistency and shortcut robustness.","feed_headline":"Vision-language models fail to tell sarcasm from simple mismatch","feed_subtitle":"A 3,000-pair benchmark shows high accuracy collapses on paired tests; injected cues flip verdicts.","key_machinery":"The central object is PragMatch, a benchmark that fixes the image and varies only the communicative relation among three captions: Pragmatic Incongruity (P), Non-Pragmatic Mismatch (M), and Literal Congruity (L). The load-bearing evaluation is paired and grouped accuracy, which credits a model only when it classifies all matched conditions from the same image correctly, complemented by AUROC over the continuous sarcasm score derived from yes/no token log-probabilities. The diagnostic machinery is a set of masking and injection interventions over three cue families — lexical markers, stylistic formatting, and simulated OCR text — that are asserted to modify only surface form while preserving the image–caption relation and gold label. A robust model should be invariant to these edits; prediction changes are attributed to shortcut reliance.","core_discovery":"On its own terms, the paper establishes that LVLM predictions in multimodal sarcasm detection are sensitive to lexical, OCR-derived, and stylistic cues, with injected surface signals causing substantial changes in model predictions despite unchanged underlying image–text relationships. No model reaches above-chance accuracy on both pragmatic incongruity (P) and non-pragmatic mismatch (M); models either default to non-sarcastic for both (InternVL2.5-8B, Qwen2.5-VL-7B, Idefics2-8B) or treat cross-modal inconsistency as sarcasm regardless of intent (LLaVA-v1.6-7B). Paired accuracy that requires both P and M to be correct for the same image falls below the 25% chance level for three of four models under zero-shot prompting, even when single-condition accuracy exceeds 96%. Score-level AUROC shows that some models (Qwen2.5-VL-7B at 90.0) separate the two classes in their continuous scores while still making wrong decisions, pointing to decision bias rather than missing relational information. Chain-of-thought prompting improves sarcastic-pair recognition but increases false sarcasm on non-pragmatic pairs and can lower AUROC, showing that reasoning traces do not fix the underlying relation modeling.","pith_inferences":["If the perturbation premise is relaxed — particularly for OCR injection, where \"Visible text in the image says: Yeah, right.\" adds new semantic content — some verdict flips are arguably correct responses to a changed caption rather than proof of shortcut reliance; human rating of perturbed pairs would settle this.","The same paired-accuracy design transfers to other pragmatic phenomena such as irony, humour, and indirect requests, where incongruity is communicative rather than arbitrary.","The benchmark's own text-only classifier result (99.6% accuracy) suggests that caption provenance and register differ across conditions, so cross-condition comparisons may partly measure style, and matching literal captions for register is a direct next step the paper itself flags.","One testable extension is to train or fine-tune a model on PragMatch's P–M contrast and check whether paired accuracy, not just item accuracy, improves; current results imply it would not without explicit relational supervision."],"forward_implications":["Standard item-level accuracy and F1 on sarcasm benchmarks overestimate relational understanding; paired and grouped metrics should be reported alongside them.","A model can separate sarcasm from literal content in its continuous scores yet fail at the decision boundary, so accuracy gains from prompting do not imply better representation.","OCR-derived surface cues are the strongest shortcut family in this testbed; models that treat injected visible-text phrases as speaker evidence will be fooled by adversarial captions.","Text-only inputs match or exceed multimodal inputs for most tested models, implying that apparent visual grounding in sarcasm detection can be an illusion.","Chain-of-thought prompting is not a cure: it raises sarcasm recall but increases false positives on non-pragmatic mismatches."],"supporting_citations":[{"why":"Supplies the source sarcastic examples from MMSD2.0 that form the Pragmatic Incongruity condition.","marker":"(Qin et al., 2023)"},{"why":"Provides the hard-positive sensitivity/invariance principle used to justify why models should stay stable under meaning-preserving surface edits.","marker":"(Kamath et al., 2024)"},{"why":"NaturalBench supplies the paired and grouped evaluation protocol and the threshold-adjustment analysis that motivates AUROC reporting.","marker":"(Li et al., 2024)"},{"why":"CLIP retrieves visually similar neighbor captions used to construct hard-negative mismatched captions.","marker":"(Radford et al., 2021)"},{"why":"RelSim filters candidate negatives by relational similarity to keep mismatches visually plausible but semantically mismatched.","marker":"(Nguyen et al., 2026)"},{"why":"PunchBench documents caption-only answerability and sensitivity to synonym and antonym substitution, supporting the shortcut diagnosis.","marker":"(Ouyang et al., 2025)"},{"why":"Shows VLMs can answer diagram questions above chance without the image, providing context for the text-only result.","marker":"(Chi et al., 2025)"}],"fun_headline_variants":["LVLMs mistake surface cues for sarcasm, not real reasoning","Injected cues flip LVLM verdicts on sarcasm vs mismatch","PragMatch: LVLMs fail to separate sarcasm from simple mismatch","Sarcasm test: LVLMs rely on shortcuts, not image-text reasoning","Benchmark shows LVLMs can't tell pragmatic incongruity from mismatch"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The diagnosis assumes that masking and injection change only surface form and leave the image–caption relationship and the gold label untouched; if an injection such as appended OCR text genuinely changes what the caption means, then prediction flips are not necessarily shortcuts.","fun_headline_variants_meta":{"raw":{"variants":["LVLMs mistake surface cues for sarcasm, not real reasoning","Injected cues flip LVLM verdicts on sarcasm vs mismatch","PragMatch: LVLMs fail to separate sarcasm from simple mismatch","Sarcasm test: LVLMs rely on shortcuts, not image-text reasoning","Benchmark shows LVLMs can't tell pragmatic incongruity from mismatch"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000983,"raw_usage":{"total_tokens":4187,"prompt_tokens":979,"completion_tokens":3208,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":3111}},"tokens_in":595,"tokens_out":3208,"duration_ms":20502,"temperature":1.0,"reasoning_tokens":3111,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:03:18.119119+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give human annotators the original and perturbed caption pairs and ask whether the image–caption relationship (and the sarcasm label) is preserved; if a substantial share of OCR-injected pairs are judged to change the relationship or become sarcastic, the shortcut attribution collapses for that cue family. Alternatively, a model that achieves high paired accuracy on P–M and stays invariant under all three perturbation families would directly refute the claim.","supporting_citations":[],"review_version":1}