Pith. sign in

REVIEW 12 cited by

Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2206.08916 v2 pith:W75QZ2VH submitted 2022-06-17 cs.CV

Jiasen Lu , Christopher Clark , Rowan Zellers , Roozbeh Mottaghi , Aniruddha Kembhavi This is my paper
classification cs.CV
keywords tasksunified-iolanguagemodelvisionacrossdiverseestimation
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We propose Unified-IO, a model that performs a large variety of AI tasks spanning classical computer vision tasks, including pose estimation, object detection, depth estimation and image generation, vision-and-language tasks such as region captioning and referring expression, to natural language processing tasks such as question answering and paraphrasing. Developing a single unified model for such a large variety of tasks poses unique challenges due to the heterogeneous inputs and outputs pertaining to each task, including RGB images, per-pixel maps, binary masks, bounding boxes, and language. We achieve this unification by homogenizing every supported input and output into a sequence of discrete vocabulary tokens. This common representation across all tasks allows us to train a single transformer-based architecture, jointly on over 90 diverse datasets in the vision and language fields. Unified-IO is the first model capable of performing all 7 tasks on the GRIT benchmark and produces strong results across 16 diverse benchmarks like NYUv2-Depth, ImageNet, VQA2.0, OK-VQA, Swig, VizWizGround, BoolQ, and SciTail, with no task-specific fine-tuning. Code and demos for Unified-IO are available at: https://unified-io.allenai.org.

Discussion (0). Sign in to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MentalThink: Shaping Thoughts in Mental SVG World

    cs.AI 2026-07 conditional novelty 7.0 of 10

    MLLMs that generate and render SVG sketches as multi-turn intermediate reasoning steps reach 55.1% on VSIBench and 76.0% on MindCube, far above the Qwen2.5-VL-7B backbone.

  2. UniGen-AR: Unifying Visual Generation with Auto-Regressive Modeling

    cs.CV 2026-07 conditional novelty 6.0 of 10

    An MLLM-conditioned next-scale VAR decoder handles 15+ unified visual generation tasks with competitive quality and substantially lower latency than diffusion baselines.

  3. On the Role of Depth in Surgical Vision Foundation Models: An Empirical Study of RGB-D Pre-training

    cs.CV 2026-01 conditional novelty 6.0 of 10

    RGB-D pre-training with explicit cross-modal objectives (MultiMAE) improves surgical detection, segmentation, pose, and depth estimation over RGB-only pre-training, with gains persisting when fine-tuned on 25% of labe...

  4. Is Visual in-Context Learning for Compositional Medical Tasks within Reach?

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Training on synthetic compositional task sequences with sequence-level masking lets a transformer-based in-context learner follow multi-step medical imaging instructions on held-out images, but well below codebook upp...

  5. MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks

    cs.CV 2025-06 conditional novelty 6.0 of 10

    AVHaystacks is a new 3100-question benchmark for audio-visual QA across 500 videos, and the MAGNET multi-agent pipeline beats current baselines on it.

  6. Qwen-Audio-VAE Technical Report

    eess.AS 2026-07 conditional novelty 5.0 of 10

    A 12.5 Hz continuous audio VAE reconstructs speech, music, and sound well while encoding 64×30s clips in 541 ms after latency-aware encoder pruning.

  7. Open-set Cross Modal Generalization via Multimodal Unified Representation

    cs.CV 2025-07 conditional novelty 5.0 of 10

    The authors propose OSCMG, an open-set version of Cross Modal Generalization, and show their MICU method with masked contrastive learning and unified jigsaw puzzles outperforms prior methods.

  8. Bitcoin Price Direction Prediction via Regime-Aware Multi-Modal Fusion of Social Sentiment and Technical Features

    cs.LG 2026-07 reject novelty 4.0 of 10

    A regime-gated sentiment/price fusion model for hourly Bitcoin direction prediction achieves near-random accuracy (AUC 0.5084 at 3h), and its key claims are contradicted by its own results.

  9. Grounding Intelligence in Movement

    cs.AI 2025-07 conditional novelty 4.0 of 10

    Movement should be treated as a first-class AI modeling modality, and a unified, biomechanically grounded movement foundation model built from aggregated data across species and sensors is the proposed path forward.

  10. Unified Multimodal Understanding via Byte-Pair Visual Encoding

    cs.CV 2025-06 conditional novelty 4.0 of 10

    Priority-guided byte-pair encoding of quantized image patches plus curriculum training yields an 8B discrete-token MLLM competitive with continuous-embedding models on VQA and multimodal benchmarks.

  11. MADFormer: Mixed Autoregressive and Diffusion Transformers for Continuous Image Generation

    cs.CV 2025-06 conditional novelty 4.0 of 10

    An empirical testbed shows that allocating more transformer layers to autoregressive conditioning improves FID under low inference budgets, while block size should scale with image resolution.

  12. Vision Generalist Model: A Survey

    cs.CV 2025-06 conditional novelty 3.0 of 10

    A structured review of vision generalist models, classifying them into encoding-based and sequence-to-sequence frameworks and summarizing datasets, benchmarks, techniques, and open problems.

Pith tools