Pith. sign in

REVIEW 22 cited by

LaViDa: A Large Diffusion Language Model for Multimodal Understanding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2505.16839 v4 pith:2EBPGS53 submitted 2025-05-22 cs.CV

LaViDa: A Large Diffusion Language Model for Multimodal Understanding

Shufan Li , Konstantinos Kallidromitis , Hritik Bansal , Akash Gokul , Yusuke Kato , Kazuki Kozuka , Jason Kuen , Zhe Lin
show 2 more authors
Kai-Wei Chang Aditya Grover
This is my paper
classification cs.CV
keywords lavidavlmsbidirectionalinferencemodelsmultimodaltasksachieves
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Modern Vision-Language Models (VLMs) can solve a wide range of tasks requiring visual reasoning. In real-world scenarios, desirable properties for VLMs include fast inference and controllable generation (e.g., constraining outputs to adhere to a desired format). However, existing autoregressive (AR) VLMs like LLaVA struggle in these aspects. Discrete diffusion models (DMs) offer a promising alternative, enabling parallel decoding for faster inference and bidirectional context for controllable generation through text-infilling. While effective in language-only settings, DMs' potential for multimodal tasks is underexplored. We introduce LaViDa, a family of VLMs built on DMs. We build LaViDa by equipping DMs with a vision encoder and jointly fine-tune the combined parts for multimodal instruction following. To address challenges encountered, LaViDa incorporates novel techniques such as complementary masking for effective training, prefix KV cache for efficient inference, and timestep shifting for high-quality sampling. Experiments show that LaViDa achieves competitive or superior performance to AR VLMs on multi-modal benchmarks such as MMMU, while offering unique advantages of DMs, including flexible speed-quality tradeoff, controllability, and bidirectional reasoning. On COCO captioning, LaViDa surpasses Open-LLaVa-Next-8B by +4.1 CIDEr with 1.92x speedup. On bidirectional tasks, it achieves +59% improvement on Constrained Poem Completion. These results demonstrate LaViDa as a strong alternative to AR VLMs. Code and models will be released in the camera-ready version.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 22 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Discrete Langevin-Inspired Posterior Sampling

    cs.LG 2026-05 unverdicted novelty 7.0

    ΔLPS is a gradient-guided discrete posterior sampler for inverse problems that works with masked or uniform discrete diffusion priors and outperforms prior discrete methods on image restoration tasks.

  2. GPO-V: Jailbreak Diffusion Vision Language Model by Global Probability Optimization

    cs.CV 2026-05 unverdicted novelty 7.0

    GPO-V is a visual jailbreak framework that bypasses safety guardrails in diffusion VLMs by globally manipulating generative probabilities during denoising.

  3. GPO-V: Jailbreak Diffusion Vision Language Model by Global Probability Optimization

    cs.CV 2026-05 unverdicted novelty 7.0

    GPO-V jailbreaks dVLMs by globally optimizing probabilities in the denoising process to bypass refusal patterns, achieving stealthy and transferable attacks.

  4. BARD: Bridging AutoRegressive and Diffusion Vision-Language Models Via Highly Efficient Progressive Block Merging and Stage-Wise Distillation

    cs.CV 2026-04 unverdicted novelty 7.0

    BARD bridges autoregressive and diffusion VLMs with progressive block merging plus stage-wise intra-diffusion distillation, delivering 3x speedup and new SOTA on open dVLMs using under 4.4M data points.

  5. ST-Veto: Spatio-Temporal Token Veto for Diffusion MLLMs via Taylor Prediction and Visual Grounding

    cs.AI 2026-07 conditional novelty 6.0

    ST-Veto improves reasoning in diffusion MLLMs by vetoing temporally unstable tokens and tokens with weak image grounding, swapping in safer near-boundary candidates.

  6. Seeing the End at Step Zero: Accelerating Diffusion MLLMs via MLP Sparsity-Aware Truncation

    cs.AI 2026-07 conditional novelty 6.0

    A diffusion multimodal LLM's early-layer MLP sparsity at denoising step 0 pinpoints the semantic boundary, enabling one-shot suffix truncation that raises throughput up to ~31x.

  7. Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding

    cs.CL 2026-07 accept novelty 6.0

    Joint AR–diffusion training yields one tri-mode LM that switches AR, diffusion, and self-speculation, beating open AR/diffusion models on accuracy and tokens-per-forward.

  8. Nemotron-Labs-Diffusion-Image: Advancing Masked Discrete Diffusion for High-Resolution Image Synthesis

    cs.CV 2026-06 conditional novelty 6.0

    Masked discrete diffusion with token editing and grouped cross-entropy reaches strong text-to-image generation scores in an 8B decoder-only model, reporting GenEval 0.90, DPG 86.9, HPSv3 10.76.

  9. Nemotron-Labs-Diffusion-Image: Advancing Masked Discrete Diffusion for High-Resolution Image Synthesis

    cs.CV 2026-06 unverdicted novelty 6.0

    A masked discrete diffusion model adds token editing at inference and grouped cross-entropy training to reach 0.90 GenEval, 86.9 DPG, and 10.76 HPSv3 scores.

  10. dMoE: dLLMs with Learnable Block Experts

    cs.CL 2026-05 unverdicted novelty 6.0

    dMoE aggregates token expert distributions to block level in dLLMs, cutting unique experts from 69.5 to 14.6, memory by 76-80%, and latency by 1.14-1.66x while retaining 99.11% performance.

  11. dWorldEval: Scalable Robotic Policy Evaluation via Discrete Diffusion World Model

    cs.RO 2026-04 unverdicted novelty 6.0

    A discrete diffusion model tokenizes multimodal robotic data and uses a progress token to predict future states and task completion for scalable policy evaluation.

  12. LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model

    cs.CV 2026-04 unverdicted novelty 6.0

    LLaDA2.0-Uni unifies multimodal understanding and generation inside one discrete diffusion large language model with a semantic tokenizer, MoE backbone, and diffusion decoder.

  13. BARD: Bridging AutoRegressive and Diffusion Vision-Language Models Via Highly Efficient Progressive Block Merging and Stage-Wise Distillation

    cs.CV 2026-04 unverdicted novelty 6.0

    Progressive supervised block merging plus stage-wise intra-diffusion distillation converts Qwen3-VL into a large-block dVLM with claimed SOTA open-dVLM scores and up to 3× decoding speedup on ≤4.4M data.

  14. Thinking Diffusion: Penalize and Guide Visual-Grounded Reasoning in Diffusion Multimodal Language Models

    cs.AI 2026-04 unverdicted novelty 6.0

    Position and step penalty plus visual reasoning guidance fix premature answering and weak visual grounding in diffusion MLLMs, delivering up to 7.5% accuracy gains and over 3x speedup.

  15. High-Entropy Tokens as Multimodal Failure Points in Vision-Language Models

    cs.CV 2025-12 unverdicted novelty 6.0

    High-entropy tokens act as concentrated multimodal failure points in VLMs, enabling sparse Entropy-Guided Attacks that achieve 93-95% success and 30-38% harmful rates with cross-model transfer.

  16. High-Entropy Tokens as Multimodal Failure Points in Vision-Language Models

    cs.CV 2025-12 conditional novelty 6.0

    Attacking only the top 20% high-entropy token positions in vision-language models causes comparable semantic damage and more harmful outputs than global attacks, and these vulnerable tokens transfer across model archi...

  17. Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Models

    cs.CV 2025-12 conditional novelty 6.0

    Sparse-LaViDa speeds up masked diffusion LLMs by truncating masked tokens, using register tokens and a step-causal attention mask, reaching ~2-2.8x speedup over LaViDa-O with comparable generation quality.

  18. LLaDA2.0: Scaling Up Diffusion Language Models to 100B

    cs.LG 2025-12 conditional novelty 6.0

    LLaDA2.0 scales discrete diffusion language models to 100B parameters via systematic conversion from autoregressive models using a 3-phase WSD training scheme and releases open-source 16B and 100B MoE variants.

  19. A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Models

    cs.CV 2025-11 conditional novelty 6.0

    Visual token pruning causes severe loss in discrete diffusion MLLMs; only from-scratch models on long-answer tasks recover via late denoising, so redundancy is recoverability, not dispensability.

  20. Lavida-O: Elastic Large Masked Diffusion Models for Unified Multimodal Understanding and Generation

    cs.CV 2025-09 conditional novelty 6.0

    Lavida-O introduces an elastic mixture-of-transformers architecture that brings high-resolution text-to-image generation, object grounding, and image editing into a single masked diffusion model, using planning and se...

  21. Structuring The Future: Diffusion LLM Speculative Decoding via Calibrated Draft Graphs

    cs.LG 2025-09 conditional novelty 6.0

    Spiffy speeds up diffusion LLM inference up to about 3x (and up to 7.9x with parallel decoding) by verifying multiple candidate unmasked states in one batched model call, while preserving greedy output.

  22. Mask-Aware Policy Gradients for Diffusion Language Models

    cs.CL 2026-07 conditional novelty 5.0

    Optimizing both token predictions and unmasking-position probabilities in MDLM policy gradients improves RL fine-tuning, reaching 87.1% GSM8K and 53.4% MBPP.