Pith. sign in

REVIEW 4 cited by

OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.02592 v4 pith:IS23CIWQ submitted 2024-12-03 cs.CV

Junyuan Zhang , Qintong Zhang , Bin Wang , Linke Ouyang , Zichen Wen , Ying Li , Ka-Ho Chow , Conghui He
show 1 more author
Wentao Zhang
This is my paper · ORCID
classification cs.CV
keywords noisedataimpactknowledgeohrbenchstructuredbasesdocuments
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Retrieval-augmented Generation (RAG) enhances Large Language Models (LLMs) by integrating external knowledge to reduce hallucinations and incorporate up-to-date information without retraining. As an essential part of RAG, external knowledge bases are commonly built by extracting structured data from unstructured PDF documents using Optical Character Recognition (OCR). However, given the imperfect prediction of OCR and the inherent non-uniform representation of structured data, knowledge bases inevitably contain various OCR noises. In this paper, we introduce OHRBench, the first benchmark for understanding the cascading impact of OCR on RAG systems. OHRBench includes 8,561 carefully selected unstructured document images from seven real-world RAG application domains, along with 8,498 Q&A pairs derived from multimodal elements in documents, challenging existing OCR solutions used for RAG. To better understand OCR's impact on RAG systems, we identify two primary types of OCR noise: Semantic Noise and Formatting Noise and apply perturbation to generate a set of structured data with varying degrees of each OCR noise. Using OHRBench, we first conduct a comprehensive evaluation of current OCR solutions and reveal that none is competent for constructing high-quality knowledge bases for RAG systems. We then systematically evaluate the impact of these two noise types and demonstrate the trend relationship between the degree of OCR noise and RAG performance. Our OHRBench, including PDF documents, Q&As, and the ground truth structured data are released at: https://github.com/opendatalab/OHR-Bench

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VaRS-Doc: Interpretation-Aware Variant Representations via Latent Self-Probing for Visual Document Retrieval

    cs.CV 2026-08 conditional novelty 7.0 of 10

    VaRS-Doc improves visual document retrieval by encoding each page into multiple interpretation-specific variants using latent probing tokens, then letting each query select the best-matching variant.

  2. Structured Attention Matters to Multimodal LLMs in Document Understanding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Structured LaTeX encoding of OCR text, combined with document images, improves DocQA accuracy across four MLLMs and four benchmarks without any training.

  3. ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A new benchmark converts text-only QA datasets into text images and shows that vision-language models degrade sharply on long visually presented contexts.

  4. Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation

    cs.SE 2025-06 conditional novelty 4.0 of 10

    A field study of five real-world RAG systems evaluated by 100 users, yielding user ratings and twelve engineering lessons.

Pith tools