REVIEW 8 cited by
UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understanding
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
In the era of Large Language Models (LLMs), tremendous strides have been made in the field of multimodal understanding. However, existing advanced algorithms are limited to effectively utilizing the immense representation capabilities and rich world knowledge inherent to these large pre-trained models, and the beneficial connections among tasks within the context of text-rich scenarios have not been sufficiently explored. In this work, we introduce UniDoc, a novel multimodal model equipped with text detection and recognition capabilities, which are deficient in existing approaches. Moreover, UniDoc capitalizes on the beneficial interactions among tasks to enhance the performance of each individual task. To implement UniDoc, we perform unified multimodal instruct tuning on the contributed large-scale instruction following datasets. Quantitative and qualitative experimental results show that UniDoc sets state-of-the-art scores across multiple challenging benchmarks. To the best of our knowledge, this is the first large multimodal model capable of simultaneous text detection, recognition, spotting, and understanding.
Forward citations
Cited by 8 Pith papers
-
DREAM: Document Reconstruction via End-to-end Autoregressive Model
A single model, DREAM, jointly predicts layout elements, coordinates, and transcriptions for document reconstruction, along with a new metric (DSM) and benchmark (DocRec1K).
-
Dolphin: Document Image Parsing via Heterogeneous Anchor Prompting
A two-stage model that analyzes page layout first, then parses text, tables, and formulas in parallel, reports state-of-the-art accuracy and speed on the benchmarks it evaluates.
-
DocOCR-Eval: A Correction-Based Framework for OCR Tool Selection Without Ground Truth
OCR tools can be ranked without ground-truth labels by measuring how much a multimodal LLM must correct each tool's output.
-
Multimodal Tabular Reasoning with Privileged Structured Information
An 8B multimodal LLM trained on 9k reasoning traces distilled from structured tables reaches state-of-the-art open-source accuracy on table-image question answering and fact verification.
-
Prolonged Reasoning Is Not All You Need: Certainty-Based Adaptive Routing for Efficient LLM/MLLM Reasoning
CAR routes each query to either a short answer or full reasoning based on the perplexity of the model's draft answer, improving accuracy and cutting token use on VQA, KIE, and math/common sense benchmarks.
-
LPCAN: Lightweight Pyramid Cross-Attention Network for Rail Surface Defect Detection Using RGB-D Data
A lightweight RGB-D cross-attention network is proposed for rail defect detection, but the SOTA accuracy and generalization claims are internally inconsistent and the implementation is not public.
-
Knowledge-Embedded and Hypernetwork-Guided Few-Shot Substation Meter Defect Image Generation Method
Fine-tuning Stable Diffusion with DreamBooth-style knowledge and hypernetwork-guided crack control maps can synthesize substation meter defect images that boost a YOLOv8 defect detector's mAP when added to the training set.
-
SETransformer: A Hybrid Attention-Based Architecture for Robust Human Activity Recognition
SETransformer combines a Transformer encoder, channel attention, and attention pooling for WISDM activity recognition, but the architecture is permutation-invariant and the reported comparison omits the model itself.
Discussion (0). Sign in to comment.