Pith. sign in

REVIEW 3 major objections 5 minor 48 references

Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Vision-language models with definition-guided prompts can detect hateful memes and rewrite them into non-hateful, coherent versions, without task-specific training.

desk verdict A novel and useful meme-mitigation framework with credible VLM detection results, but the headline mitigation numbers rest on an underspecified human evaluation that needs real reporting before the central claim is trustworthy. read the letter →

arxiv 2505.00150 v1 pith:INYRK775 submitted 2025-04-30 cs.CV cs.AIcs.CL

Minh-Hao Van , Xintao Wu This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords multimodalmemeshatespeechdetectionvision-languagemodelsdefinition-guidedpromptsfew-shotpromptingcontentmitigationimage-textcoherenceGPT-4o
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models, prompted with an explicit definition of hate and classification rules, can both spot hateful memes and rewrite them into benign versions without any task-specific fine-tuning. The paper's detection experiments centre on GPT-4o: with the definition-guided prompt and no OCR text it reaches 75.5% accuracy and 74.2 AUROC on the seen Hateful Memes Challenge (HMC) test set, and 76.85% accuracy and 76.25 AUROC on the unseen set, besting every compared baseline in accuracy on both sets. For mitigation, the paper proposes UnHateMeme, a modular pipeline that first decides whether hate is unimodal or multimodal, then locates the source (image, text, or both), and only then substitutes a generated text or a CLIP-matched image. Human majority vote scored 88.4% of the 631 mitigated memes as non-hateful and 84.5% as coherent, against 60.2% non-hateful for a random-substitution baseline. A sympathetic reader would take this as evidence that pretrained VLMs, guided by a carefully worded definition, are a practical zero-shot route to content moderation and content repair.

What carries the argument

The central object is the definition-guided prompt: a task instruction that pastes in a formal definition of hate speech and classification criteria, then asks the VLM to reason aloud, classify, and give a hatefulness probability. In detection this prompt is the whole method, requiring no fine-tuning or OCR; in the few-shot variant it is augmented with demonstrations retrieved by CLIP visual similarity (RICES). The mitigation side is carried by the UnHateMeme decomposition: functions $h$ and $s$ classify hate type and source, and the generation functions $g_t$ (text) and $g_v$ (image) rewrite only the offending modality, with original text erased first by an inpainting pipeline built from an open-set detector, a segmentation model, and an inpainter. This machinery makes the model's reasoning explicit at every decision point and restricts generation to the component that actually carries the hate, which is what the human-evaluation results trace their advantage to.

What would settle it

Re-score the 631 mitigated memes with a fresh panel of raters who are blind to the study's purpose, reporting per-meme vote counts and an inter-rater agreement statistic; if the non-hateful and coherence rates fall materially below 88.4% and 84.5%, or if agreement is low, the mitigation claim is not established. The same study should also collect human labels of hate type and source on the original 490 memes and compare them with the model's $h$ and $s$ outputs, since wrong routing would erase UnHateMeme's advantage over random substitution.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a precisely worded prompt changes what an off-the-shelf vision-language model can do with a meme. The definition-guided prompt inserts the Hateful Memes Challenge definition of hate speech and explicit classification criteria—hateful if either modality is hateful alone, or if the two together convey a hateful message—and asks the model to explain before classifying. With this prompt, GPT-4o detects implicit hate that a simple prompt misses (e.g., a dog labeled "Bruce Jenner's cat" as deadnaming) and produces accuracy above 75% on both HMC test sets without OCR or training. UnHateMeme then turns the same model into a mitigator: the functions $h$ (unimodal vs multimodal hate) and $s$ (image, text, or both as the source) decide which modality to replace, and GPT-4o either generates a new text or describes a substitute image that CLIP matches to a candidate collection. The paper claims that this routing, rather than the generation step alone, is what raises the human-judged non-hateful rate from 60.2% to 88.4% while keeping most edited memes coherent.

Load-bearing premise

The mitigation numbers stand on the assumption that the seven human evaluators' majority-vote answers to 'is it hateful?' and 'do image and text make sense together?' are reliable, unbiased measurements, but the paper gives no inter-rater agreement or vote distribution to support that.

Editorial extensions

If this is right

  • Definition-guided prompting turns an off-the-shelf VLM into a competitive hateful-meme detector: GPT-4o in zero-shot beats every compared baseline in accuracy on the seen and unseen HMC test sets, with no training and no OCR input.
  • The few-shot results show the prompt is model-agnostic: Gemini Pro Vision improves with CLIP-retrieved demonstrations (up to 74.9% unseen accuracy with 8 shots), and smaller VLMs like LLaVA and OpenCLIP gain from OCR text.
  • UnHateMeme's routing is the key to mitigation: substituting the identified hate source yields 88.4% human-judged non-hateful memes versus 60.2% when a random modality is substituted.
  • Most of the 631 mitigated memes stay coherent (84.5%), and 68.8% are deemed both non-hateful and coherent—enough to be shareable by the paper's criteria, versus 41.8% for the naive baseline.
  • Because the framework is modular, the same prompts and routing can be wrapped around future or alternative VLMs, so the detection and mitigation capability should scale with VLM progress.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same recipe—embed a formal definition and decision rules in a prompt—should transfer to other safety tasks with codifiable policies, such as harassment, disinformation, or policy-violating product claims, giving a zero-training way to adapt VLMs to new moderation rules.
  • The current image substitution is limited by the fixed candidate collection; replacing CLIP retrieval with a generative text-to-image model (the paper lists this as future work) should raise the coherence rate and remove the collection's coverage as a bottleneck.
  • A testable extension is to check whether GPT-4o's remaining detection errors concentrate on memes whose hate depends on external knowledge or wordplay; if so, adding a knowledge-access step to the definition-guided prompt should close that gap.
  • The human-evaluation design itself could be extended by measuring reader perception with a broader, non-expert sample, since expert majority votes on 'is it hateful?' may not equal how the memes land in a real social-media feed.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes two VLM-based contributions: a definition-guided prompting technique for zero- and few-shot hateful meme detection on the Hateful Memes Challenge (HMC) dataset, and a unified mitigation framework, UnHateMeme, that routes hateful memes through analysis of hate type and source, then substitutes hateful image or text components using GPT-4o with CLIP-based image retrieval, GroundingDINO/SAM/inpainting for text removal, and task-specific prompts. The detection experiments report that GPT-4o without OCR reaches 75.5% accuracy and 74.2 AUROC on the seen HMC test set and 76.85% accuracy and 76.25 AUROC on the unseen set. The mitigation experiments generate 631 mitigated memes from 490 hateful memes, with human majority vote judging 88.4% as non-hateful and 84.5% as coherent; a Naive Generation baseline achieves 60.2% non-hateful and 81.8% coherent. The authors provide an anonymous code/data link for reproducibility.

Significance. If the results are reliable, the paper makes a useful empirical contribution: it shows that pretrained VLMs can approach or exceed specialized hateful-meme detectors without fine-tuning, and it defines a novel mitigation task with a modular pipeline and concrete prompts. The strengths include the use of a public benchmark, direct evaluations without learned model parameters, detailed prompt disclosure, and a reproducibility link. The main significance is conditional, however, because the mitigation claim rests on a human evaluation that lacks inter-annotator agreement metrics, control items, and blinding details, and because the detection superiority claim is not supported by confidence intervals or significance tests and omits relevant prompting baselines. These gaps are fixable within the scope of a revision.

major comments (3)
  1. [§5.3, Table 2] The central mitigation claim rests entirely on the seven-expert majority-vote answers to Q1 and Q2, but the manuscript reports no inter-annotator agreement statistic (e.g., Fleiss' kappa or percent agreement), no per-item vote distribution, and no control memes. It also does not state whether the raters were blind to the framework's intent or to the original memes. Because the same hatefulness definition appears in the model prompts and in the evaluation guidelines, the 88.4% non-hateful and 84.5% coherence estimates may reflect criterion alignment rather than absolute quality. Please report agreement metrics, vote counts, and results for original hateful and non-hateful memes as calibration controls, and clarify the blinding and independence of the raters.
  2. [§5.1, Table 1] The detection comparison is not statistically grounded: all baseline numbers are taken from prior papers, there are no confidence intervals or significance tests, and the Flamingo/OpenFlamingo rows report only AUROC from [1,2] while prompting-based detection methods from [3,18] are absent. Consequently, the statement that definition-guided GPT-4o 'surpasses all baselines' on the seen test set is not established. Please provide at least bootstrap confidence intervals or paired tests on the 1,000 seen and 2,000 unseen test items, and include the missing prompting baselines in the table for a controlled comparison.
  3. [§4.2, §5.3] The framework's advantage over Naive Generation depends on correct routing by the functions h (type of hate) and s (source of hate), but these two functions are never validated against ground truth. If GPT-4o misroutes memes, the downstream substitution strategy is wrong and the reported 68.8% vs. 41.84% 'shareable' comparison cannot be confidently attributed to the hate-analysis components. Please evaluate h and s on a labeled sample or have annotators check the routing decisions for the 490 memes, and report per-category routing accuracy.
minor comments (5)
  1. [Title page] The affiliation contains a typo: 'University of Arkanasas' should be 'University of Arkansas'.
  2. [§4.2] The sentence 'The embeddings of images in the substitute collection are precomputed using the CLIP visual encoder.' appears twice in the same paragraph; one copy should be deleted.
  3. [Table 4] The table cells are formatted ambiguously, e.g., '26293.6%' and '3512.5%'; these should be rendered as '262 (93.6%)' and '35 (12.5%)' for readability.
  4. [§5.3, 'Quality of mitigated memes'] The sentence 'our UnHateMeme first identifies the original meme as unimodal hate (image)' contradicts the immediately preceding statement that the hate originates from the text; it should read 'unimodal hate (text)'.
  5. [§1, Related Work] The introduction claims this is 'among the first few works' to use prompting for hateful meme detection, while [3,18] are cited as prior prompting-based detection methods; the novelty claim should be qualified to avoid inconsistency.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: detection is scored against external HMC labels and mitigation is measured by independent human majority vote.

full rationale

The paper contains no derivation chain whose conclusion is equivalent to an input. The detection contribution (Section 3) is a zero-shot and few-shot prompting evaluation on the Hateful Memes Challenge benchmark: the definition-guided prompt supplies the HMC hatefulness definition and classification criteria, but predictions are scored against the dataset's external ground-truth labels in Table 1, so the reported accuracy and AUROC values are not constructed from the prompt itself. The mitigation contribution (Section 4) proposes the UnHateMeme pipeline with the type-of-hate function h, source-of-hate function s, and generation functions gv and gt, and evaluates the resulting memes in Section 5.3 through seven experts' majority-vote answers to Q1 and Q2. The headline numbers of 88.4% non-hateful and 84.5% coherent are direct measurements of the generated memes, not inputs to the generator, and the comparison against Naïve Generation provides an external baseline. The two self-citations ([37] and [42]) occur only in peripheral related-work or domain-motivation contexts and are never used to justify the central detection or mitigation claims. The reported evaluation limitations, such as the absence of inter-annotator agreement statistics or control items, are measurement-quality concerns rather than evidence that a prediction reduces to a fitted parameter or that a self-citation chain supplies the result. Even if the same hatefulness definition were used in the human evaluation, applying an agreed criterion to judge outputs is an intended operationalization, not a circular construction. No step can be exhibited in which the paper's own equations or fitted values force the claimed outcome, so the appropriate circularity finding is zero.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

This is an empirical systems paper, so the ledger consists of benchmark and evaluation assumptions rather than mathematical axioms or fitted parameters. The only hand-chosen numeric hyperparameters are k=4 for CLIP retrieval and the few-shot demonstration counts; no new entities or fitted constants are introduced.

free parameters (2)
  • k (number of matched substitute images) = 4
    Chosen for the CLIP retrieval step in UnHateMeme (Section 5.1). It is a hand-selected hyperparameter affecting mitigation outputs; no validation sweep is reported.
  • Few-shot demonstration count = 4 or 8
    The paper evaluates 4-shot and 8-shot detection prompts (Table 1). The count is a design choice rather than a fitted value, and results improve with count.
assumptions (5)
  • domain assumption The HMC hate speech definition from [12] is an adequate operationalization of hatefulness for detection, mitigation, and human evaluation.
    The definition is embedded verbatim in detection prompts, mitigation analysis prompts, and human evaluation criteria (Sections 3.2, 4.2, 5.3). Different definitions of hate could change all measurements.
  • domain assumption CLIP embedding similarity is a valid matching score between GPT-4o's substitute-image description and candidate non-hateful images.
    Used in the gv step to retrieve top-k candidates (Section 4.2, Figure 3). If CLIP alignment is poor, selected substitutes may mismatch the text.
  • domain assumption Majority-vote labels of seven expert raters for Q1 and Q2 are reliable gold-standard measurements.
    No inter-annotator agreement or vote distribution is reported (Section 5.3, Table 2). The central mitigation claim depends on these labels.
  • domain assumption HMC test set labels are correct ground truth for the detection evaluation.
    Standard benchmark assumption used in the detection evaluation (Section 5.1).
  • domain assumption GPT-4o's zero-shot responses are stable across API versions and safety settings.
    The paper does not snapshot model versions; Gemini safety is set to the lowest level (Appendix D), and outputs could shift with model updates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Models." pith.science (2026). https://pith.science/paper/INYRK775

@misc{pith2026250500150,
  author       = {Pith},
  title        = {Pith review of: Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/INYRK775}},
  note         = {Machine review of arXiv:2505.00150}
}
read the original abstract

The rapid evolution of social media has provided enhanced communication channels for individuals to create online content, enabling them to express their thoughts and opinions. Multimodal memes, often utilized for playful or humorous expressions with visual and textual elements, are sometimes misused to disseminate hate speech against individuals or groups. While the detection of hateful memes is well-researched, developing effective methods to transform hateful content in memes remains a significant challenge. Leveraging the powerful generation and reasoning capabilities of Vision-Language Models (VLMs), we address the tasks of detecting and mitigating hateful content. This paper presents two key contributions: first, a definition-guided prompting technique for detecting hateful memes, and second, a unified framework for mitigating hateful content in memes, named UnHateMeme, which works by replacing hateful textual and/or visual components. With our definition-guided prompts, VLMs achieve impressive performance on hateful memes detection task. Furthermore, our UnHateMeme framework, integrated with VLMs, demonstrates a strong capability to convert hateful memes into non-hateful forms that meet human-level criteria for hate speech and maintain multimodal coherence between image and text. Through empirical experiments, we show the effectiveness of state-of-the-art pretrained VLMs such as LLaVA, Gemini and GPT-4o on the proposed tasks, providing a comprehensive analysis of their respective strengths and limitations for these tasks. This paper aims to shed light on important applications of VLMs for ensuring safe and respectful online environments.

Figures

Figures reproduced from arXiv: 2505.00150 by the authors.

Figure 1
Figure 1. An example of GPT-4o failing to detect hateful meme. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Our prompts for hateful memes detection. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Overview of UnHateMeme framework for mitigating hateful content in memes. original text in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Prompts for mitigating hateful content in memes. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Examples of detecting hateful memes using GPT-4o with our definition-guided prompt [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Definition-guided few-shot prompt for detecting hateful memes. [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 15 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35: 23716–23736, 2022

    work page 2022

  2. [2]

    Openflamingo: An open- source framework for training large autoregressive vision-language models

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390, 2023

    arXiv 2023

  3. [3]

    Prompting for multimodal hateful meme classification

    Rui Cao, Roy Ka-Wei Lee, Wen-Haw Chong, and Jing Jiang. Prompting for multimodal hateful meme classification. arXiv preprint arXiv:2302.04156, 2023

    arXiv 2023

  4. [4]

    Modularized networks for few-shot hateful meme detection

    Rui Cao, Roy Ka-Wei Lee, and Jing Jiang. Modularized networks for few-shot hateful meme detection. In Proceedings of the ACM Web Conference 2024, pages 4575–4584, 2024

    work page 2024

  5. [5]

    Microsoft coco captions: Data collection and evaluation server

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015

    arXiv 2015

  6. [6]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  7. [7]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  8. [8]

    Recent advances in online hate speech moderation: Multimodality and the role of large models

    Ming Shan Hee, Shivam Sharma, Rui Cao, Palash Nandi, Preslav Nakov, Tanmoy Chakraborty, and Roy Lee. Recent advances in online hate speech moderation: Multimodality and the role of large models. Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4407–4419, 2024

    work page 2024

Show all 48 references
  1. [9]

    Openclip, 2021

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021

    2021

  2. [10]

    Capalign: Improving cross modal alignment via informative captioning for harmful meme detection

    Junhui Ji, Xuanrui Lin, and Usman Naseem. Capalign: Improving cross modal alignment via informative captioning for harmful meme detection. InProceedings of the ACM Web Conference 2024, pages 4585–4594, 2024

    2024

  3. [11]

    Supervised multimodal bitransformers for classifying images and text

    Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, and Davide Testuggine. Supervised multimodal bitransformers for classifying images and text. arXiv preprint arXiv:1909.02950, 2019

    1909 arXiv

  4. [12]

    The hateful memes challenge: Detecting hate speech in multimodal memes

    Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. Advances in neural information processing systems, 33:2611–2624, 2020

    2020

  5. [13]

    The hateful memes challenge: Competition report

    Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Casey A Fitzpatrick, Peter Bull, Greg Lipstein, Tony Nelli, Ron Zhu, et al. The hateful memes challenge: Competition report. In NeurIPS 2020 Competition and Demonstration Track, pages 344–360. PMLR, 2021. 14

    2020

  6. [14]

    Why is it hate speech? masked ratio- nale prediction for explainable hate speech detection

    Jiyun Kim, Byounghan Lee, and Kyung-Ah Sohn. Why is it hate speech? masked ratio- nale prediction for explainable hate speech detection. In Proceedings of the 29th Inter- national Conference on Computational Linguistics , pages 6644–6655, Gyeongju, Repub- lic of Korea, October...

    2022

  7. [15]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023

    2023

  8. [16]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. arXiv preprint arXiv:2306.00890, 2023

    2023 arXiv

  9. [17]

    Visualbert: A simple and performant baseline for vision and language

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019

    1908 arXiv

  10. [18]

    Towards explainable harmful meme detection through multimodal debate between large language models

    Hongzhan Lin, Ziyang Luo, Wei Gao, Jing Ma, Bo Wang, and Ruichao Yang. Towards explainable harmful meme detection through multimodal debate between large language models. In Proceedings of the ACM on Web Conference 2024, pages 2359–2370, 2024

    2024

  11. [19]

    A multimodal framework for the detection of hateful memes

    Phillip Lippe, Nithin Holla, Shantanu Chandra, Santhosh Rajamanickam, Georgios Antoniou, Ekaterina Shutova, and Helen Yannakoudakis. A multimodal framework for the detection of hateful memes. arXiv preprint arXiv:2012.12871, 2020

    2012 arXiv

  12. [20]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023

    2023 arXiv

  13. [21]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023

    2023 arXiv

  14. [22]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. arXiv preprint arXiv:1908.02265, 2019

    1908 arXiv

  15. [23]

    Hatexplain: A benchmark dataset for explainable hate speech detection

    Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. Hatexplain: A benchmark dataset for explainable hate speech detection. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 14867–14875, 2021

    2021

  16. [24]

    Ethos: an online hate speech detection dataset

    Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas. Ethos: an online hate speech detection dataset. arXiv preprint arXiv:2006.08328, 2020

    2006 arXiv

  17. [25]

    Vilio: State-of-the-art visio-linguistic models applied to hateful memes

    Niklas Muennighoff. Vilio: State-of-the-art visio-linguistic models applied to hateful memes. arXiv preprint arXiv:2012.07788, 2020

    2012 arXiv

  18. [26]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. URL https://api. semanticscholar.org/CorpusID:257532815

    2023 arXiv

  19. [27]

    Unsafe diffusion: On the generation of unsafe images and hateful memes from text-to-image models

    Yiting Qu, Xinyue Shen, Xinlei He, Michael Backes, Savvas Zannettou, and Yang Zhang. Unsafe diffusion: On the generation of unsafe images and hateful memes from text-to-image models. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, page...

    2023

  20. [28]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  21. [29]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint arXiv:1506.01497, 2015

    2015 arXiv

  22. [30]

    Detecting hateful memes using a multimodal deep ensemble

    Vlad Sandulescu. Detecting hateful memes using a multimodal deep ensemble. arXiv preprint arXiv:2012.13235, 2020

    2012 arXiv

  23. [31]

    Detecting formal thought disorder by deep contextualized word representations

    Justyna Sarzynska-Wawer, Aleksander Wawer, Aleksandra Pawlak, Julia Szymanowska, Izabela Stefaniak, Michal Jarkiewicz, and Lukasz Okruszek. Detecting formal thought disorder by deep contextualized word representations. Psychiatry Research, 304:114135, 2021. 15

    2021

  24. [32]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ...

    2018

  25. [33]

    Large language models encode clinical knowledge

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138, 2022

    2022 arXiv

  26. [34]

    Resolution-robust large mask inpainting with fourier convolutions

    Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-robust large mask inpainting with fourier convolutions. In Proceedings of the IEEE/CVF winte...

    2022

  27. [35]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  28. [36]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  29. [37]

    On large visual language models for medical imaging analysis: An empirical study

    Minh-Hao Van, Prateek Verma, and Xintao Wu. On large visual language models for medical imaging analysis: An empirical study. In 2024 IEEE/ACM Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE), pages 172–176. IEEE, 2024

    2024

  30. [38]

    Detecting hate speech in memes using multimodal deep learning approaches: Prize-winning solution to hateful memes challenge

    Riza Velioglu and Jewgeni Rose. Detecting hate speech in memes using multimodal deep learning approaches: Prize-winning solution to hateful memes challenge. arXiv preprint arXiv:2012.12975, 2020

    2012 arXiv

  31. [39]

    Memecraft: Contextual and stance-driven multimodal meme generation

    Han Wang and Roy Ka-Wei Lee. Memecraft: Contextual and stance-driven multimodal meme generation. In Proceedings of the ACM on Web Conference 2024, pages 4642–4652, 2024

    2024

  32. [40]

    Next-gpt: Any-to-any multimodal llm

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519, 2023

    2023 arXiv

  33. [41]

    Aggregated residual transformations for deep neural networks

    Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492–1500, 2017

    2017

  34. [42]

    Coded hate speech detection via contextual information

    Depeng Xu, Shuhan Yuan, Yueyang Wang, Angela Uchechukwu Nwude, Lu Zhang, Anna Zajicek, and Xintao Wu. Coded hate speech detection via contextual information. In PAKDD. Springer, 2022

    2022

  35. [43]

    An empirical study of gpt-3 for few-shot knowledge-based vqa

    Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 3081–3089, 2022

    2022

  36. [44]

    Inpaint anything: Segment anything meets image inpainting

    Tao Yu, Runseng Feng, Ruoyu Feng, Jinming Liu, Xin Jin, Wenjun Zeng, and Zhibo Chen. Inpaint anything: Segment anything meets image inpainting. arXiv preprint arXiv:2304.06790, 2023

    2023 arXiv

  37. [45]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention

    Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199, 2023

    2023 arXiv

  38. [46]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  39. [47]

    Enhance multimodal transformer with external label and in-domain pretrain: Hateful meme challenge winning solution

    Ron Zhu. Enhance multimodal transformer with external label and in-domain pretrain: Hateful meme challenge winning solution. arXiv preprint arXiv:2012.08290, 2020. 16 A Ethics Statement. Large and foundation models have garnered attention from both the research community and s...

    2012 arXiv

  40. [768]

    C.2 Multimodal: unimodal pretraining Multimodal models from unimodal pretraining typically combine the output or the features of vision and linguistic models

    BERT is a bidirectional transformer-based model for language representation and understanding that learns embeddings for subwords [6]. C.2 Multimodal: unimodal pretraining Multimodal models from unimodal pretraining typically combine the output or the features of vision and li...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.