REVIEW 6 cited by
Beyond Text: Optimizing RAG with Multimodal Inputs for Industrial Applications
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) have demonstrated impressive capabilities in answering questions, but they lack domain-specific knowledge and are prone to hallucinations. Retrieval Augmented Generation (RAG) is one approach to address these challenges, while multimodal models are emerging as promising AI assistants for processing both text and images. In this paper we describe a series of experiments aimed at determining how to best integrate multimodal models into RAG systems for the industrial domain. The purpose of the experiments is to determine whether including images alongside text from documents within the industrial domain increases RAG performance and to find the optimal configuration for such a multimodal RAG system. Our experiments include two approaches for image processing and retrieval, as well as two LLMs (GPT4-Vision and LLaVA) for answer synthesis. These image processing strategies involve the use of multimodal embeddings and the generation of textual summaries from images. We evaluate our experiments with an LLM-as-a-Judge approach. Our results reveal that multimodal RAG can outperform single-modality RAG settings, although image retrieval poses a greater challenge than text retrieval. Additionally, leveraging textual summaries from images presents a more promising approach compared to the use of multimodal embeddings, providing more opportunities for future advancements.
Forward citations
Cited by 6 Pith papers
-
RAVID: Retrieval-Augmented Visual Detection: A Knowledge-Driven Approach for AI-Generated Image Identification
RAVID detects AI-generated images by retrieving similar images from a database and feeding them to a vision-language model, reporting 93.85% average accuracy on UniversalFakeDetect.
-
mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation
A systematic empirical study finds that for multimodal RAG, EVA-CLIP retrieval, listwise LVLM reranking, and feeding only the top-ranked document works best, with a self-reflection agent adding further gains.
-
VISA: Retrieval Augmented Generation with Visual Source Attribution
VISA makes RAG output point to the exact bounding box in a document screenshot that supports the answer, and fine-tuned VLMs improve box accuracy from 1.5% to 54.2% on the new Wiki-VISA benchmark.
-
HVM-GraphRAG: Holistic-View Multimodal Graph Retrieval-Augmented Generation on Complex Document
A holistic-view multimodal GraphRAG that conflict-resolves concept indices and retrieves via compact concept anchors plus modality-grouped evidence, beating strong baselines on three complex-document QA sets.
-
Document Screenshot Retrievers are Vulnerable to Pixel Poisoning Attacks
Gradient-based pixel poisoning of a single document screenshot can rank it in the top-10 for up to 41.9% of unseen queries on DSE and 26.4% on ColPali.
-
Multi-modal Retrieval Augmented Multi-modal Generation: Datasets, Evaluation Metrics and Strong Baselines
The authors create a 1,000-query multi-modal RAG benchmark with GPT-4o-based metrics, show multi-stage generation beats single-stage, and report fine-tuned 7B-8B models beating GPT-4o only in the single-stage comparison.
Discussion (0). Continue with ORCID to comment.