{"id":"26a2556b-7c69-4660-a736-63dd07a844ee","arxiv_id":"2506.23122","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The best overall F1 scores come from larger models such as Qwen2.5-VL and DeBERTa-v3, but performance is dataset-dependent and victim detection remains poor across all model families.","lead":"This paper benchmarks more than 25 language and vision-language models on detecting narrative roles (hero, villain, victim, other) in English and English-Hindi memes from three test sets. It finds that victim detection stays weak even for the strongest models, and that prompt engineering gives only dataset-dependent improvements.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central negative result depends on untreated annotation and OCR noise; neither is measured, so 'Victim remains hard' may reflect data artifacts rather than model limits.","rationale":"The paper's positive contribution is a broad benchmark and a plausible negative result: victim roles are hard and code-mixed generalization is weak. The concern is load-bearing because the central claim is a statement about what current models can and cannot do, and the evidence for it is entirely observational F1 scores over a fixed dataset. If the gold labels are noisy for the most ambiguous class (Victim) or the OCR text is systematically worse for code-mixed memes, then every reported number in Tables III, IV, and V is shifted in a way that could manufacture the headline finding. The paper neither reports inter-annotator agreement nor performs an OCR error analysis, despite using OCR text in all prompt templates and acknowledging an OCR bottleneck for CLIP. This is not a disagreement with current consensus; it is a correctness risk internal to the benchmark design. The suggested audit would settle the concern decisively: if labels and transcripts are clean, the negative result stands; if errors concentrate in the victim and EnHi subsets, the conclusion needs to be reframed as a data-characterization result rather than a model-capability result. The reader's verdict of CONDITIONAL remains appropriate, and no upgrade or downgrade is needed.","tokens_in":23150,"tokens_out":9044,"duration_ms":98228,"concrete_test":"Run a two-part audit on 100 randomly sampled memes per test set (stratified by role): (1) manually transcribe embedded text and compute OCR word error rate against the released {ocr}; (2) have two fresh annotators re-label the narrative role from the image plus manual transcription, with adjudication, and compute Cohen's kappa. Then re-run the best models on manually transcribed text and on the adjudicated labels. If victim F1 moves by more than about 0.05 or the code-mixed gap closes, the benchmark's central claim is partly an artifact of OCR/annotation noise rather than model capability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that models consistently fail on Victim and on code-mixed content—is inferred from F1 numbers computed against a single gold label sequence and against OCR text. Section III introduces the datasets but reports no inter-annotator agreement, and the prompt templates in Section V insert {ocr} directly into the model input. For HVVMemes-2 (EnHi), code-mixed meme text is romanized, fragmented, and often stylized; if OCR introduces errors, or if the Victim label is the least stable under re-annotation (the paper's own confusion-matrix analysis in Section VII.C shows Victim being absorbed into Other), then the observed failures are not necessarily properties of the models. The paper even concedes an 'OCR-based performance bottleneck' in Section VI.A for CLIP, but never quantifies it. Without an audit separating label noise and OCR error from model error, the headline negative result is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks narrative-role classification (Hero, Villain, Victim, Other) in memes across three test sets: the original HVVMemes (English) and the HVVMemes-2 test sets in English and code-mixed English-Hindi. It reports lexical and structural dataset analyses, evaluates over 25 text-only and multimodal models in zero-shot and fine-tuned settings, and studies four prompt variants for LLaVA-NeXT. The main empirical claims are that the Victim class is consistently hard to identify, that code-mixed and culturally specific content hurts generalisation, and that hybrid prompts offer marginal but consistent improvements.","tokens_in":23345,"tokens_out":4921,"duration_ms":55192,"significance":"If the central negative result holds, the paper is a useful benchmark contribution: it assembles a broad model suite, connects two existing shared-task datasets, and provides qualitative, embedding, and confusion-matrix analyses that are largely absent from the original dataset papers. The finding that even the best zero-shot multimodal models plateau at macro F1 around 0.30–0.35, with victim F1 often below 0.20, would be a valuable calibration point for the community. However, the headline conclusions depend on two unverified data-quality assumptions: the gold labels are treated as noise-free, and the OCR text is treated as a faithful representation of meme content. Because neither assumption is audited, the central claim is not yet established.","major_comments":[{"comment":"The central negative claim about Victim detection and code-mixed generalisation is not separable from annotation and OCR noise. Section III introduces HVVMemes and HVVMemes-2 without reporting inter-annotator agreement or label-quality statistics, and every prompt in Section V inserts {ocr} directly into the model input. The paper itself concedes an 'OCR-based performance bottleneck' in Section VI.A but never quantifies OCR accuracy or its downstream effect. Since the confusion-matrix analysis in Section VII.C shows Victim being absorbed into Other, a plausible alternative explanation is that the Victim label is the least stable under re-annotation and that OCR errors disproportionately affect code-mixed romanised text. The authors should provide annotation agreement, an OCR error analysis, or a re-evaluation on a manually verified subset before the global 'Victim remains hard' conclusion can be accepted.","section":"III, V, VI.A"},{"comment":"There is a direct numerical inconsistency in the core result tables. Table I reports 1,084 total entity references in HVVMemes (49 Hero, 273 Villain, 95 Victim, 667 Other), but Tables II and III both show column totals of 2,433 with 1,917 instances of Other, and the majority-baseline precision of 0.79 in Table III is computed against 1,917/2,433. These numbers cannot describe the same test set. Since all macro-F1 comparisons, majority baselines, and class-wise conclusions in Table III rest on these counts, the authors must correct the headers or explain which superset the evaluations actually used. The same issue affects Table II, whose column headers are identical across rows for three different test sets.","section":"Table I vs Tables II and III"},{"comment":"The abstract and the Contributions section claim that hybrid prompts (P4) provide 'marginal yet consistent improvements', but the reported results contradict this. In Table II, on HVVMemes (En), P4 has macro F1 0.240, clearly below P3's 0.333; the same reversal appears in Table III. On HVVMemes-2 (EnHi), P4 is better than P3 overall, but it is still worse than P1 for Hero (0.310 vs 0.372) and for Villain (0.393 vs 0.468). The contribution statement that P4 'best generalises across roles, especially villain and hero' is therefore not supported by the data shown. The claim should be restricted to the settings where it holds, or the prompt analysis should be revised.","section":"Abstract, Contributions, V, Table II"},{"comment":"The headline comparison between fine-tuned transformers and zero-shot LLMs confounds architecture with training condition. In Table III, deberta-v3-large (fine-tuned on the task) reaches macro F1 0.543, while Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, and the multimodal models are evaluated zero-shot. The paper later attributes DeBERTa's advantage to 'reasoning capabilities of current LLMs' being limited, but the more parsimonious reading is that fine-tuning on the training split explains much of the gap. The authors should either add a fine-tuned LLM comparison or clearly frame the result as 'best fine-tuned model vs best zero-shot model' and soften the architectural conclusions.","section":"IV and VI"}],"minor_comments":[{"comment":"The model name is written inconsistently as LLaVA-NeXT, LLaVA-Next, and LlaVA-Next; please standardise.","section":"Throughout"},{"comment":"In the Introduction, 'Another ridiculous Rahul Gandhi's speech' should read 'Another meme ridicules Rahul Gandhi's speech'.","section":"I"},{"comment":"In Section VI.C, 'performes' should be 'performs'.","section":"VI.C"},{"comment":"Table II's column headers list counts that are identical for all three test-set rows; the table should instead show per-dataset class counts, e.g., 49/273/95/667 for HVVMemes (En).","section":"Table II"},{"comment":"In Section VI.B, the text refers to 'Qwen-2.5B-Instruct', but the model evaluated is listed as Qwen2.5-7B-Instruct; please correct the name.","section":"VI.B"},{"comment":"The paper does not report decoding parameters (temperature, top-p, number of runs, seed) for the instruction-tuned and multimodal LLMs; without this, differences such as 0.295 vs 0.242 in Table II may reflect sampling noise rather than prompt quality.","section":"V"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is likely to be a useful benchmark paper if the evaluation tables are corrected and the data-quality audit is added. I would not reject on the grounds that the datasets were created or co-organized by the same group, as that is common for shared-task benchmark papers. However, the internal count inconsistency between Table I and Tables II/III is serious enough that the authors must repair it before any comparison with prior work or majority baselines can be trusted. The OCR/annotation concern is the kind of issue that a careful reviewer for this venue should require as a revision condition rather than as a post-publication caveat."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a competent, useful benchmark study. The headline result — that Victim is consistently the hardest role for every model family tested, and that code-mixed memes are harder than English ones — looks robust across their three test sets. The paper does not introduce a new dataset or method, but it does provide a broad, systematic comparison of 25+ models on an established shared task, which has real reference value for people building moderation pipelines. The lexical and error analyses are also honest and informative.\n\nWhere it wobbles: the abstract's claim that hybrid prompts give \"marginal yet consistent improvements\" is not supported by their own Table III. On HVVMemes (En), the P4 hybrid macro-F1 (0.240) is well below P3 (0.333). The P4 advantage only appears on the HVVMemes-2 sets. So \"consistent\" is wrong; \"mixed but sometimes helpful\" is what the data say.\n\nThe bigger soft spot is one the paper itself gestures at but never quantifies: everything runs through OCR text, and the gold labels come from the authors' own shared-task annotations with no inter-annotator agreement reported. If the Victim label is unstable under re-annotation, or if OCR systematically mangles the romanized Hinglish, part of the \"Victim is hard\" story could be data artifacts rather than model limits. The stress-test note makes this point, and I think it holds up. That said, the sheer consistency of the victim failures across many models, including text-only and multimodal, suggests there is a real phenomenon underneath. It doesn't sink the paper, but a referee should ask for an error audit or at least an explicit acknowledgment of the limitation.\n\nAlso, no code or error bars are shipped, and prompt selection appears to have been done on one of the reported test sets (EnHi) before applying to the others. That's not fatal, but it should be stated clearly.\n\nBottom line: this is a paper for people who need a model-selection reference for meme role classification, and for shared-task organizers. It deserves a serious peer review; the central negative result is plausible and useful, and the overclaim about prompts is fixable with a more careful abstract and discussion.","headline":"Useful benchmark with a real negative result on Victim, but the abstract overstates prompt-engineering gains and the evaluation never quantifies OCR/label noise.","tokens_in":23821,"tokens_out":3261,"would_cite":false,"duration_ms":31221,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Across 25+ models, a benchmark finds no current model reliably identifies whether a meme casts an entity as hero, villain, victim, or other.","keywords":["narrative role classification","meme analysis","multimodal reasoning","code-mixed Hinglish","victim detection","benchmark","prompt engineering","vision-language models"],"falsifier":"A replication with a second round of annotations on a random sample of all three test sets, plus manual transcription of meme text instead of OCR, could settle the claim: if victim F1 jumps substantially under cleaner labels and transcripts, the reported difficulty is partly an artifact of annotation or OCR noise; if it stays near 0.2, the model failure is genuine.","tokens_in":22975,"feed_emoji":"🎭","tokens_out":6315,"duration_ms":63491,"temperature":0.7,"pith_summary":"This paper tries to establish a benchmark for a hard, under-studied task: deciding whether a person or group shown in a meme is cast as hero, villain, victim, or other, in English and code-mixed English-Hindi memes. It assembles 2,850 annotated examples across three test sets and evaluates more than 25 models, from fine-tuned multilingual transformers to text-only and vision-language LLMs in zero-shot settings. The central finding is that no tested model robustly performs the task: the victim role is consistently the most difficult, with F1 scores often below 0.35 even for the best systems, and performance drops on culturally specific and code-mixed content. The paper also shows that adding structured role definitions to prompts helps multimodal models only marginally. If this diagnosis is right, current automated tools cannot be trusted to infer who is framed as harmed in a meme without substantial task-specific adaptation.","feed_headline":"Victim role in memes stumps 25+ AI models","feed_subtitle":"Benchmark of English and Hinglish memes shows even vision-language LLMs miss who is framed as harmed; fine-tuned DeBERTa leads.","key_machinery":"The load-bearing machinery is a four-class annotation scheme (Hero, Villain, Victim, Other) applied to three meme test sets, combined with a uniform evaluation protocol in which OCR-extracted text, a queried entity, and prompt instructions are fed to each model and scored with macro-averaged precision, recall, and F1. The paper's four prompt variants (basic, other-emphasized, definition-optimized, and hybrid) function as the controlled intervention for multimodal models, while confusion matrices and embedding visualizations expose which classes collapse into which. The central contrast the comparison relies on is between real meme text, which is culture-specific and indirect, and synthetic hateful meme text, which exhibits explicit and repetitive lexical markers.","core_discovery":"On the author's own terms, the discovery is that narrative-role classification in memes is not solved by scale or multimodality alone. On the original HVVMemes test set (US politics and COVID-19, skewed toward the other class), the fine-tuned DeBERTa-v3-large model reaches the best macro F1 of 0.54, while the strongest zero-shot instruction-tuned LLM, Qwen2.5-7B-Instruct, reaches 0.37 and the best multimodal model, LLaVA-NeXT with a definition-rich prompt, reaches 0.33. On the more balanced English HVVMemes-2 set, Qwen2.5-VL-7B-Instruct leads with macro F1 0.35; on the code-mixed English-Hindi set, Qwen2.5-VL and Qwen2-VL both reach 0.30. Across all settings the victim class is the recurring failure point, with F1 values exceeding 0.4 only rarely and falling below 0.1 in many configurations, while hero detection also trails the random baseline in several settings. The paper interprets this as evidence that subtle, culturally grounded framing requires better multimodal reasoning, cultural grounding, and role-adaptive training rather than simply larger models.","pith_inferences":["If the benchmark is accepted, a practical reading for deployment is that zero-shot role labelling is not production-ready: systems should be fine-tuned with role-adaptive objectives, and any victim-class output should be treated as low-confidence.","Because synthetic hateful memes carry an explicit and repetitive lexicon, models evaluated only on such data may look better than they are; real-meme performance is the harder bar and should be the primary evaluation surface.","A testable extension is to replace OCR with human-verified transcripts and to add inter-annotator agreement scores; if victim F1 jumps substantially under cleaner labels, part of the reported difficulty is data noise rather than model reasoning.","The success of Qwen2.5-VL on Hinglish examples appears to draw on cultural knowledge about public figures such as MS Dhoni; a benchmark that varies whether that background knowledge is supplied would directly test whether the bottleneck is cultural grounding or multimodal fusion."],"forward_implications":["The best fine-tuned multilingual transformer (DeBERTa-v3-large, macro F1 0.54 on HVVMemes) still beats the best zero-shot LLMs and multimodal models, so model scale alone does not close the gap.","Victim detection is the recurring bottleneck: across all three test sets and most model families, victim F1 rarely exceeds 0.4 and often falls below 0.1, so any deployed system should treat victim outputs as unreliable.","Code-mixed English-Hindi memes form a linguistically and culturally distinct cluster in embedding space, and most models trained on English or synthetic hate data do not transfer to them; the En-Hi test set is not just a translation of the English one.","For multimodal models, a hybrid prompt that combines role definitions with structured instructions yields small but consistent macro-F1 gains over the basic prompt, particularly for hero and villain roles."],"supporting_citations":[{"why":"Supplies the original HVVMemes dataset with hero, villain, victim, and other labels skewed toward other.","marker":"[5]"},{"why":"Introduces the HVVMemes-2 English and code-mixed English-Hindi test sets used for the balanced and multilingual evaluations.","marker":"[6]"},{"why":"Provides the synthetically curated Hateful Memes source that makes up the HVVMemes-2 English set, enabling the real-versus-synthetic lexical contrast.","marker":"[2]"},{"why":"Earlier work that characterizes entities in harmful memes as hero, villain, or victim, giving this study its role scheme and baseline framing.","marker":"[12]"},{"why":"Prior victim-target detection work that motivates the paper's special attention to the victim class.","marker":"[11]"},{"why":"Multilingual narrative-role work restricted to text, providing the contrast showing why the multimodal meme setting is an open extension.","marker":"[13]"},{"why":"Reference for DeBERTa-v3-large, the best-performing fine-tuned text-only model in the benchmark.","marker":"[47]"},{"why":"Reference for Qwen2.5-VL-7B-Instruct, the strongest multimodal model on the balanced and code-mixed test sets.","marker":"[49]"},{"why":"Reference for LLaVA-NeXT, the model used to test the four prompt-design strategies.","marker":"[50]"}],"fun_headline_variants":["Memes reveal AI blind spot: spotting victims is hard","Why AI misreads who's the victim in memes","Benchmark: even top vision-language LLMs miss meme victims","Victim role in memes: the blind spot for AI and VLMs","Code-mixed memes expose cultural gaps in AI role detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark assumes the human-annotated role labels are correct ground truth and that OCR-extracted text faithfully represents what each meme says; annotation mistakes or OCR errors would change every F1 score and every model ranking without any model changing.","fun_headline_variants_meta":{"raw":{"variants":["Memes reveal AI blind spot: spotting victims is hard","Why AI misreads who's the victim in memes","Benchmark: even top vision-language LLMs miss meme victims","Victim role in memes: the blind spot for AI and VLMs","Code-mixed memes expose cultural gaps in AI role detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000238,"raw_usage":{"total_tokens":1575,"prompt_tokens":1074,"completion_tokens":501,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":690,"completion_tokens_details":{"reasoning_tokens":414}},"tokens_in":690,"tokens_out":501,"duration_ms":6175,"temperature":1.0,"reasoning_tokens":414,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:48:27.071352+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A replication with a second round of annotations on a random sample of all three test sets, plus manual transcription of meme text instead of OCR, could settle the claim: if victim F1 jumps substantially under cleaner labels and transcripts, the reported difficulty is partly an artifact of annotation or OCR noise; if it stays near 0.2, the model failure is genuine.","supporting_citations":[{"cited_title":"Findings of the constraint 2022 shared task on detecting the hero, the villain, and the victim in memes,","cited_arxiv_id":null,"evidence_quote":"Supplies the original HVVMemes dataset with hero, villain, victim, and other labels skewed toward other."},{"cited_title":"The clef-2024 checkthat! lab: Check-worthiness, subjectivity, persuasion, roles, authori- ties, and adversarial robustness,","cited_arxiv_id":null,"evidence_quote":"Introduces the HVVMemes-2 English and code-mixed English-Hindi test sets used for the balanced and multilingual evaluations."},{"cited_title":"The hateful memes challenge: Detecting hate speech in multimodal memes,","cited_arxiv_id":null,"evidence_quote":"Provides the synthetically curated Hateful Memes source that makes up the HVVMemes-2 English set, enabling the real-versus-synthetic lexical contrast."},{"cited_title":"Characterizing the entities in harmful memes: Who is the hero, the villain, the victim?","cited_arxiv_id":null,"evidence_quote":"Earlier work that characterizes entities in harmful memes as hero, villain, or victim, giving this study its role scheme and baseline framing."},{"cited_title":"DISARM: Detecting the victims targeted by harmful memes,","cited_arxiv_id":null,"evidence_quote":"Prior victim-target detection work that motivates the paper's special attention to the victim class."},{"cited_title":"SemEval-2025 task 10: Multilingual char- acterization and extraction of narratives from online news,","cited_arxiv_id":null,"evidence_quote":"Multilingual narrative-role work restricted to text, providing the contrast showing why the multimodal meme setting is an open extension."},{"cited_title":"Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled em- bedding sharing,","cited_arxiv_id":null,"evidence_quote":"Reference for DeBERTa-v3-large, the best-performing fine-tuned text-only model in the benchmark."},{"cited_title":"Qwen2.5-vl,","cited_arxiv_id":null,"evidence_quote":"Reference for Qwen2.5-VL-7B-Instruct, the strongest multimodal model on the balanced and code-mixed test sets."},{"cited_title":"Llava-next: Improved reasoning, ocr, and world knowledge,","cited_arxiv_id":null,"evidence_quote":"Reference for LLaVA-NeXT, the model used to test the four prompt-design strategies."}],"review_version":1}