REVIEW 8 cited by
CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Multimodal Models (LMMs) have demonstrated impressive performance in recognizing document images with natural language instructions. However, it remains unclear to what extent capabilities in literacy with rich structure and fine-grained visual challenges. The current landscape lacks a comprehensive benchmark to effectively measure the literate capabilities of LMMs. Existing benchmarks are often limited by narrow scenarios and specified tasks. To this end, we introduce CC-OCR, a comprehensive benchmark that possesses a diverse range of scenarios, tasks, and challenges. CC-OCR comprises four OCR-centric tracks: multi-scene text reading, multilingual text reading, document parsing, and key information extraction. It includes 39 subsets with 7,058 full annotated images, of which 41% are sourced from real applications, and released for the first time. We evaluate nine prominent LMMs and reveal both the strengths and weaknesses of these models, particularly in text grounding, multi-orientation, and hallucination of repetition. CC-OCR aims to comprehensively evaluate the capabilities of LMMs on OCR-centered tasks, facilitating continued progress in this crucial area.
Forward citations
Cited by 8 Pith papers
-
Position: Reasoning After Perception Means Reasoning Without Vision
Reasoning in text space cannot recover visual information that was collapsed during perception, so multimodal models need architectures that reason within the visual representation.
-
Rethinking Multilingual Vision-Language Translation: Dataset, Evaluation, and Adaptation
The authors introduce AibTrans, a multilingual image-text translation benchmark, show that common translation metrics mislead on dense images, and find that balanced multilingual fine-tuning preserves generalization b...
-
OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoning
OCR-Reasoning, a 1,069-question benchmark with reasoning-chain annotations for text-rich images, finds that no evaluated multimodal model surpasses 50% accuracy.
-
Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model
Codec-guided sparse patch selection plus a lightweight speak/silent gate yields a 4B streaming VLM that is competitive on static tasks, stronger on video/spatial benchmarks, and much cheaper at inference.
-
Physical Plausibility Reasoning via HCM-GRPO: Empowering Compact Model for Superior Performance
A 2B multimodal model trained with HCM-GRPO, a GRPO variant with partial-credit rewards and hard-case oversampling, outperforms larger models on the authors' private physical-plausibility test set.
-
MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math Reasoning in Multimodal Large Language Models
A new benchmark of 2,000 K-12 math questions photographed in real scenes finds that multimodal LLMs perform significantly worse on authentic handheld images than on clean benchmarks.
-
MMTABREAL: Real-World Benchmark for Multimodal Table Understanding
The paper releases a 500-table, 4,021-question benchmark of real-world multimodal tables and shows that leading vision-language models drop 20-40% in accuracy relative to earlier synthetic benchmarks.
-
E-ARMOR: Edge case Assessment and Review of Multilingual Optical Character Recognition
Their custom PaddleOCR-based system achieves the best F1 (0.46), fastest latency (0.17 s/image), and lowest cost ($0.006/1k images) among seven OCR systems on a private 54-language benchmark.
Discussion (0). Sign in to comment.