{"id":"e1652cb8-7962-4507-a876-053bcd1ed7f6","arxiv_id":"2501.15438","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Re-annotated meme datasets can substitute for and augment video data in hateful video detection, yielding modest Macro-F1 gains over video-only training.","lead":"This paper tests whether hateful meme images can stand in for or supplement videos when training AI to detect hateful videos, which are scarce and costly to label. The authors add a re-labeling step to align meme and video definitions and report small performance gains on two video benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation protocol is test-set-optimized: best epoch, few-shot N, and dataset size are chosen on the test set, so reported SOTA-beating gains may be selection noise.","rationale":"The reader's weakest_assumption was re-annotation fidelity. That concern is plausible but less decisive: by design, the human annotator adjudicates whenever the original label and model prediction disagree, so the model cannot unilaterally dominate the majority vote; the missing IAA and human-only baseline would need separate experiments. The explicit test-set peeking is more load-bearing because it is admitted in the text and directly contaminates every reported result. The final numbers are the best over 5 epochs on the test set, with the dataset size and few-shot count also chosen on the test set. Since the SOTA improvements are only 0.02-0.03 Macro-F1 and no variance is reported, the comparison could be an artifact of selection. I therefore recommend keeping the reader's CONDITIONAL verdict, but grounding it primarily in the evaluation protocol rather than the re-annotation circularity. Agreement: partial.","tokens_in":15375,"tokens_out":6330,"duration_ms":59821,"concrete_test":"Re-run Tables 3 and 4 using only the official training split for all model selection: choose epochs, LoRA hyperparameters, few-shot N, and meme dataset size on a fixed validation subset carved out of the training data, then evaluate once on the untouched official test set and report mean ± std Macro-F1 over at least 5 seeds. If Vid+RM-FT no longer exceeds Vid-FT and the SOTA-paper numbers by more than the standard deviation, the headline claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the evaluation protocol, not the re-annotation loop. Section 4.1.3 explicitly says 'The best-performing epoch on the video test data set was selected to report the results.' Section 4.2 selects the meme and video dataset sizes n by testing on the MHC test set, and Section 3.2.2 selects the few-shot demonstration count N using the MHC/HateMM test sets. The final Macro-F1 values in Tables 3 and 4 are therefore the result of optimizing epoch, dataset size, and few-shot count directly on the same test sets used for the SOTA comparison. Since each configuration is run once and no error bars or seeds are reported, the small reported gains (Vid+RM-FT 0.81/0.82 vs SOTA 0.79/0.79, and +0.01-0.03 over Vid-FT) may be selection noise rather than evidence that re-annotated memes transfer. The central claim that memes can substitute for or augment video data is not supported until the protocol is rerun with a held-out validation split.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes cross-modal transfer from hateful meme datasets to hateful video detection, motivated by the scarcity and high annotation cost of video datasets. A re-annotation pipeline aligns meme labels with video dataset definitions by majority voting among the original label, a few-shot LLaMA-3.2-11B prediction, and a human annotation. The authors fine-tune LLaMA-3.2-11B and LLaVA-NeXT-Video-7B with LoRA under five strategies (No FT, Vid-FT, OM-FT, RM-FT, Vid+RM-FT) and report that re-annotated memes can substitute for video data and that augmenting video data with re-annotated memes outperforms video-only fine-tuning and prior state-of-the-art results on MHC and HateMM. They release the re-annotated meme datasets and code.","tokens_in":15610,"tokens_out":4548,"duration_ms":38542,"significance":"If the results are valid, the approach is practically significant: meme annotation is substantially cheaper than video annotation, and a successful substitution/augmentation strategy would lower the barrier for hateful video detection. The paper includes useful strengths: public release of the re-annotated datasets and code, evaluation with two different VLM families, and an explicit acknowledgment of potential data leakage for HateMM. However, the central claims rest on an evaluation protocol that selects reporting epochs, few-shot counts, and dataset sizes on the same test sets used for comparison, and the re-annotation loop involves the same model family that is later fine-tuned. These issues must be resolved before the SOTA-beating numbers can be taken as evidence for cross-modal transfer.","major_comments":[{"comment":"The evaluation protocol selects hyperparameters and reporting choices on the test sets used for the headline comparisons. Section 4.1.3 states that 'The best-performing epoch on the video test data set was selected to report the results'; Section 4.2 chooses the meme and video dataset sizes n by testing on the MHC test set; and Section 3.2.2 chooses the few-shot count N using the MHC and HateMM test sets. Each configuration is run once and no error bars, seeds, or confidence intervals are reported. The reported improvements (e.g., Vid+RM-FT Macro-F1 0.81 on MHC vs. 0.79 from the MHC paper, and +0.01–0.03 over Vid-FT in Table 4) may therefore be selection noise rather than evidence that re-annotated memes transfer. The central substitution/augmentation claim is not supported until the protocol is re-run with a held-out validation split used for epoch, N, and n selection, and with variability reported.","section":"§4.1.3, §4.2, §3.2.2, Tables 1–4"},{"comment":"The re-annotation labels are produced by majority voting among the original meme label, a LLaMA-3.2-11B prediction, and a human annotation, and the same model family (LLaMA-3.2-11B) is later fine-tuned and evaluated on the video test sets. The paper reports no inter-annotator agreement, no count of how many memes required human adjudication, and no comparison against a purely human-reannotated dataset. Consequently, the observed RM-FT gains over OM-FT in Table 3 could partly reflect the model being trained on its own outputs rather than on genuinely aligned labels. The authors should report the distribution of majority-vote outcomes, agreement statistics, and a human-only re-annotation condition to support the claim that the pipeline improves label alignment.","section":"§3.2.1–§3.2.3"},{"comment":"The paper itself notes that 'the strong zero-shot performance could be due to potential data leakage, as LLaMA-3.2-11B was released after the HateMM dataset,' and that this possibility cannot be ruled out. Since the same model is used for few-shot demonstration selection and fine-tuning on HateMM, the HateMM results in Tables 1, 3, and 4 may be inflated. The authors should either use a model whose training data provably excludes HateMM, evaluate on a split released after the model's cutoff, or provide a quantitative bound on the potential leakage effect. The current acknowledgment is not sufficient to support the state-of-the-art claim on HateMM.","section":"§3.2.2"}],"minor_comments":[{"comment":"The sentence 'We employ Low-Rank Adaptation (LoRA) [22] adapters to fine-tune the pre-trained VLM during.' ends with the dangling word 'during'; it should read 'during fine-tuning.'","section":"§3.3"},{"comment":"Reference [43] cites 'arXiv preprint arXiv:2407.xxxxx'; the full arXiv identifier and title should be provided.","section":"References"},{"comment":"The table header uses 'O' for MHC and 'H' for HateMM; the text explains that MHC merges hateful and offensive into a single offensive class, but the table would be clearer if the label definitions were restated in the caption.","section":"Table 3"},{"comment":"The annotation-time estimates in Section 4.2 do not specify whether '0.5 minutes per meme' and '2 minutes per video' are per annotator, nor how many annotators were involved; please clarify to make the cost comparison reproducible.","section":"§4.2"},{"comment":"For LLaVA-NeXT-Video-7B, the paper replaces visual input in demonstrations with 'vision-based descriptions of video content provided by human annotators'; it is unclear whether these are the same annotators as in the majority-voting step and whether this description effort is included in the reported annotation cost.","section":"§3.2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and practically important problem, and the public release of the re-annotated datasets is a strength. The main obstacle is the evaluation protocol: the headline comparisons are selected on test data without variance reporting, and the re-annotation loop uses the same model family that is subsequently evaluated. I would encourage the editor to request a revision that adds a proper validation split, reports variability across seeds, and includes a human-only re-annotation baseline; with those additions, the claims could become substantiated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is genuinely worth a look: use meme data, after a label-alignment reannotation step, as a substitute or augmentation for scarce hateful video datasets. That specific transfer path (meme-to-video) is not in the cited literature, and the released reannotated meme datasets and code are a real community resource. The writing is clear, the problem is motivated, and the authors are upfront about possible data leakage in their zero-shot results. Credit where due: this is a sensible, cheap-data strategy, and the qualitative analysis of label discrepancies between FHM/MAMI and the video datasets is useful.\n\nThe soft spots are substantial and mostly in the evaluation protocol. The paper explicitly selects the best-performing epoch on the video test set (Section 4.1.3), chooses few-shot N using the MHC/HateMM test sets (Table 1), and picks meme/video dataset sizes by testing on MHC (Table 2). With one run per configuration and no error bars or seeds, the reported Macro-F1 gains of 0.01–0.03 over Vid-FT, and 0.81/0.82 vs SOTA 0.79/0.79, could be selection noise. The stress-test note is correct: this is the load-bearing flaw, not a side issue. The reannotation circularity is real but secondary; the model family used for majority-vote prediction is the same one later fine-tuned, and there is no human-only reannotation baseline, so part of RM-FT's advantage over OM-FT may be self-training. That concern is moderate, not fatal, because the majority vote includes the original label and a human, but the absence of inter-annotator agreement and counts of human adjudication makes it impossible to judge how much model self-bias leaked in.\n\nWho is this for? Researchers working on multimodal hate detection and dataset-efficient transfer. The idea is worth building on, and the resources are reusable. But the quantitative claims need a rerun with a held-out validation split for epoch and hyperparameter selection, multiple seeds, and a human-only reannotation comparison before I'd trust the SOTA-beating numbers. This deserves a serious referee—the topic matters and the method is plausible—but it should come back requiring major revision on the evaluation protocol.","headline":"Plausible idea and useful released resources, but the evaluation is compromised by test-set selection of epoch, few-shot N, and dataset size, so the headline gains over SOTA are not trustworthy yet.","tokens_in":16147,"tokens_out":824,"would_cite":false,"duration_ms":10133,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Meme datasets, after label re-alignment, can substitute for and augment video data in hateful-video detection training, outperforming existing benchmarks.","keywords":["cross-modal transfer","hateful video detection","meme dataset","label alignment","re-annotation","vision-language models","data scarcity","multimodal hate speech"],"falsifier":"A direct test would be to re-annotate the same meme datasets with human annotators only, following the video datasets' annotation guidelines, and compare RM-FT performance against that human-only re-annotation; if the model-inclusive majority vote does not match or beat the human-only version, the reported gains could be partly an artifact of the model fine-tuning on its own outputs. Reporting inter-annotator agreement and the fraction of memes that required human adjudication would let a reader check this directly.","tokens_in":15170,"feed_emoji":"🛡️","tokens_out":8521,"duration_ms":58897,"temperature":0.7,"pith_summary":"This paper claims that hateful meme datasets, once their labels are re-aligned to the definitions used by video hate-speech benchmarks, can stand in for scarce video data when training hateful-video detection models. The authors argue this matters because video annotation is expensive and video datasets are small, while meme datasets are larger and cheaper to label. They introduce a human-assisted re-annotation pipeline that combines the original meme label, a vision-language model's few-shot prediction, and a human annotation in a majority vote, then fine-tune vision-language models on the re-annotated memes. On the MHC and HateMM benchmarks, fine-tuning on re-annotated memes alone performs comparably to fine-tuning on the video data itself, and adding the re-annotated memes to the video training set improves Macro-F1 over both the video-only baseline and the best results reported in the original dataset papers.","feed_headline":"Meme data can replace video data for hateful video detection","feed_subtitle":"After label re-alignment, meme fine-tuning rivals video fine-tuning, and combining both beats MHC and HateMM benchmarks.","key_machinery":"The load-bearing mechanism is a model-prediction-driven, human-assisted re-annotation pipeline that harmonizes label definitions across datasets. For each meme, the pipeline takes the original dataset label, a prediction from LLaMA-3.2-11B prompted with a few video demonstration examples, and a human annotation, then assigns the label by majority vote. The re-annotated memes are then used to fine-tune a vision-language model via Low-Rank Adaptation (LoRA) adapters on the query and value layers of the attention mechanism, with video frames sampled as single images for the image-based model or as 16 augmented frames for the video-based model. This machinery is what converts a mismatched label space into a transferable training signal.","core_discovery":"The paper's central claim is that cross-modal transfer from memes to videos works once label definitions are aligned. With the re-annotated Facebook Hateful Memes (FHM) dataset and LLaMA-3.2-11B, fine-tuning on memes alone (RM-FT) reaches Macro-F1 0.80 on MHC, exceeding fine-tuning on the MHC videos themselves (Vid-FT, 0.78), and on HateMM it matches Vid-FT at 0.81. Combining re-annotated memes with video data (Vid+RM-FT) yields Macro-F1 0.81 on MHC and 0.82 on HateMM, surpassing the best results reported by the MHC paper (0.79) and the HateMM paper (0.79), as well as the video-only fine-tuning baselines. The authors interpret this as evidence that meme data can substitute for video data in resource-scarce settings and that augmentation with memes provides complementary signal beyond what video-only training captures.","pith_inferences":["If the label-alignment step is the true driver, then similar cross-modal transfer should work from other image-text datasets (e.g., cyberbullying or misogyny memes) to videos in those specific domains, where the label definitions already overlap.","The majority-vote design may partly train the model on its own outputs: when the model's prediction agrees with the original label, no human review is triggered, so the model's biases could be reinforced; measuring agreement rates per source would clarify how independent the final labels are.","The observed plateau in gains beyond roughly 3,000 memes could be a model-capacity effect; a larger base model might convert the same meme data into larger improvements, and that is directly testable.","A natural extension would be to apply the same re-annotation strategy to align the two video datasets' label definitions to each other, enabling cross-video transfer without any meme data."],"forward_implications":["In settings where hateful videos are scarce or expensive to annotate, models can be trained on re-annotated memes and then applied to video classification with little or no performance loss.","Augmenting small video datasets with re-annotated memes yields consistent Macro-F1 improvements on both MHC and HateMM, offering a cheap way to push benchmark performance.","The transfer works across two different vision-language model architectures, an image-based model and a video-based model, though the larger image-based model gains more from the meme data.","The released re-annotated meme datasets and code give other researchers a ready-made resource for training or validation in video hate speech tasks."],"supporting_citations":[{"why":"Supplies the MHC video benchmark, the state-of-the-art Macro-F1 baseline (0.79), and the YouTube/Bilibili videos used for evaluation.","marker":"[34]"},{"why":"Supplies the HateMM video benchmark, the state-of-the-art baseline (0.79), and the annotation-time estimates that make the cost comparison concrete.","marker":"[12]"},{"why":"Provides the Facebook Hateful Memes dataset, the main meme source used for substitution and augmentation after re-annotation.","marker":"[23]"},{"why":"Provides the MAMI misogyny meme dataset, the secondary meme source used to test whether domain-specific memes transfer to the video tasks.","marker":"[17]"},{"why":"The LLaMA-3.2-11B vision-language model used for few-shot re-annotation predictions and for the primary fine-tuning experiments.","marker":"[16]"},{"why":"The LLaVA-NeXT-Video-7B video-based vision-language model used to test whether the cross-modal transfer holds across model architectures.","marker":"[43]"},{"why":"Defines the LoRA parameter-efficient fine-tuning method used in all fine-tuning strategies.","marker":"[22]"}],"fun_headline_variants":["Memes can replace videos for hateful content detection","Train hateful video AI on memes alone","Meme data rivals video data for hateful video detection","Fight data scarcity with memes for video hate detection","Cross-modal transfer turns memes into video hate detectors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The re-annotation pipeline assumes that the majority vote among the original meme label, a vision-language model's few-shot prediction, and a single human annotation produces labels that faithfully match the video datasets' definitions, without reporting how often the model prediction dominates or how reliable the human adjudication is.","fun_headline_variants_meta":{"raw":{"variants":["Memes can replace videos for hateful content detection","Train hateful video AI on memes alone","Meme data rivals video data for hateful video detection","Fight data scarcity with memes for video hate detection","Cross-modal transfer turns memes into video hate detectors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00081,"raw_usage":{"total_tokens":3551,"prompt_tokens":940,"completion_tokens":2611,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":2534}},"tokens_in":556,"tokens_out":2611,"duration_ms":19379,"temperature":1.0,"reasoning_tokens":2534,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:17:09.732675+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to re-annotate the same meme datasets with human annotators only, following the video datasets' annotation guidelines, and compare RM-FT performance against that human-only re-annotation; if the model-inclusive majority vote does not match or beat the human-only version, the reported gains could be partly an artifact of the model fine-tuning on its own outputs. Reporting inter-annotator agreement and the fraction of memes that required human adjudication would let a reader check this directly.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MHC video benchmark, the state-of-the-art Macro-F1 baseline (0.79), and the YouTube/Bilibili videos used for evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the HateMM video benchmark, the state-of-the-art baseline (0.79), and the annotation-time estimates that make the cost comparison concrete."},{"cited_title":"Mohan, Ankur","cited_arxiv_id":null,"evidence_quote":"Provides the Facebook Hateful Memes dataset, the main meme source used for substitution and augmentation after re-annotation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MAMI misogyny meme dataset, the secondary meme source used to test whether domain-specific memes transfer to the video tasks."},{"cited_title":"Zhang, B","cited_arxiv_id":null,"evidence_quote":"The LLaVA-NeXT-Video-7B video-based vision-language model used to test whether the cross-modal transfer holds across model architectures."}],"review_version":1}