Pith. sign in

REVIEW 3 major objections 5 minor 89 references

Diffusion multimodal LLMs reveal the end of their valid answer at the very first denoising step, and that signal enables a one-shot truncation that speeds inference up to 31x.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 01:44 UTC pith:FKWYWLBK

load-bearing objection Seer is a real systems contribution, but the Step-0 semantic-oracle claim needs trivial baselines and a fixed threshold-selection protocol before the central novelty is established. the 3 major comments →

arxiv 2607.14557 v1 pith:FKWYWLBK submitted 2026-07-16 cs.AI

Seeing the End at Step Zero: Accelerating Diffusion MLLMs via MLP Sparsity-Aware Truncation

Qicheng Zhao , Qi Sun , Zheyu Yan This is my paper
classification cs.AI
keywords diffusion multimodal large language modelsMLP activation sparsitysemantic boundary detectioninference accelerationpadding wastesignal-to-noise ratioone-shot truncationserving optimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that diffusion-based multimodal large language models (DMLLMs) reveal the length of their valid answer at the very first denoising step: early-layer MLP activation sparsity shows a sharp spike exactly where the meaningful output ends, followed by a flat plateau over the padding tokens. The authors exploit this with Seer, a training-free framework that detects the boundary once via an SNR criterion and removes the redundant suffix from all later computation. This one-shot truncation eliminates the 'curse of padding' that makes fixed-window diffusion inference waste compute on empty [EOS] tokens, yielding up to ~31x throughput on some benchmarks. The paper also reports that truncation can slightly improve accuracy on visually complex tasks, because padding tokens in bidirectional attention leak background noise back into the valid prefix.

Core claim

The central discovery is that at Step-0, before any output token has been generated, the sparsity pattern of MLP activations in early layers already encodes the semantic boundary of the final answer. Valid prefix tokens keep dense activations; sparsity jumps sharply at the first [EOT]/[EOS] token (the Semantic Jump) and then stays flat over the remaining padding (the Padding Plateau). This observation turns length prediction into a one-shot detection problem: once the boundary is found at the first step, the redundant suffix can be truncated for every subsequent layer and denoising step, eliminating the majority of the FLOPs spent on padding. The paper shows the phenomenon holds across three

What carries the argument

The key mechanism is the Step-0 MLP sparsity curve: a 1D sequence of sparsity values computed from the text-side MLP activations at Layer 3 during the first denoising step. The boundary is identified by an SNR-aware local test that looks for a jump (signal) followed by a stable plateau (low noise), with a short three-token look-ahead; a conservative one-token margin prevents over-truncation. On the systems side, a Padding Waste Ratio (PWR) drives hybrid routing among static CUDA graphs, variable-length packed graphs, and an eager path, so that theoretical FLOP savings survive batched serving.

Load-bearing premise

The load-bearing premise is that at Step-0, with the input still fully masked and no output generated, early-layer MLP sparsity reliably marks the final answer boundary; the paper offers empirical evidence but no mechanism, so a different noise initialization or architecture could destroy the signal.

What would settle it

Run a DMLLM on a fixed prompt with many different random seeds for the Step-0 noise/masking. If the sparsity jump position varies more than a couple of tokens across seeds while the final answer length stays fixed, the boundary signal is not a stable oracle and the one-shot truncation would often cut valid tokens. Conversely, if the jump tracks an artificially shortened or lengthened ground-truth answer, the oracle is real.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Padding waste in DMLLMs can be removed without retraining, directly converting FLOP savings into wall-clock speedups (up to ~31x on the tested setups).
  • Removing the suffix early also purifies bidirectional cross-modal attention, which explains and enables small accuracy gains on tasks like DocVQA.
  • The boundary signal is available at Step-0, so no partial generation or confidence thresholding is needed; a single early-layer pass suffices.
  • Seer is orthogonal to KV caching and other attention-level optimizations, so its gains stack with existing accelerations.
  • Because truncation is one-shot rather than per-step eviction, it preserves static execution graphs, avoiding CPU launch overhead that plagues other dynamic-length methods.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit: if the Step-0 sparsity is a faithful oracle, then length prediction could be trained directly as a supervised head on early-layer activations, potentially yielding a more robust detector than the SNR heuristic.
  • The noise-leakage mechanism suggests that the initialization or embedding of padding tokens is not neutral; changing the masked-token representation (e.g., adding a learned 'empty' embedding) might independently suppress contamination, testable as a controlled ablation.
  • One could test whether the same jump-to-plateau signal appears in pure language diffusion models and in autoregressive LLMs with fixed windows; if it does, the truncation idea transfers beyond the multimodal setting.
  • The PWR-based routing generalizes to any dynamic-length batch serving: the hybrid of static, packed-varlen, and eager paths is a reusable pattern for preserving CUDA graph compatibility with variable shapes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Seer, a training-free framework for accelerating Diffusion Multimodal Large Language Models (DMLLMs) by detecting the semantic boundary of the generated text at the very first denoising step (Step-0) from early-layer MLP activation sparsity. Seer uses an SNR-based detector with absolute jump threshold tau_jump and tolerance gamma, performs one-shot suffix truncation for all subsequent layers and steps, and introduces a hybrid execution routing system (static graph, varlen graph, eager) to preserve throughput under batched serving. Experiments on three DMLLM families (LaViDa-LLaDA, LaViDa-Dream, MMaDA) across nine benchmarks report up to ~31x throughput improvements with maintained or slightly improved accuracy, alongside ablations on thresholds, routing, and system overhead.

Significance. If the Step-0 sparsity phenomenon is real and general, Seer targets a genuine and underexplored bottleneck: fixed-length padding in diffusion MLLMs. The paper is among the first to identify this phenomenon and provides a plausible systems-level solution. Strengths include the training-free nature, no visual token loss, orthogonality to KV cache, explicit bounding-box-grounded attention analysis, and extensive hyperparameter and routing ablations. The main weakness is that the central oracle claim is not yet separated from trivial positional signals, and the default configuration is selected on a benchmark that also appears in the main results. The paper is therefore significant if the phenomenon survives the missing controls, but the current evidence is not yet sufficient.

major comments (3)
  1. [§4.5, Eq. (17), Table 1] The default thresholds (tau_jump=0.03, gamma=0.6) are chosen by maximizing U(tau,gamma)=A_hat+lambda log T_hat on ChartQA, and ChartQA is then included in the main results Table 1. This is a selection-on-evaluation loop for the central configuration. Moreover, lambda in Eq. (17) is never specified, so the claimed 'theoretical optimum' cannot be reproduced or audited. Please select defaults on a validation split that excludes all reported benchmarks, or report main results for several threshold choices; at minimum disclose lambda and show the knee without ChartQA.
  2. [§3.2, Observation; Table 3] The load-bearing claim that Step-0 MLP sparsity is a semantic-boundary oracle is not tested against trivial baselines. At Step-0 all answer positions are masked tokens, so the observed 'semantic jump' may coincide with the prompt-to-mask transition or with positional embeddings rather than with the eventual answer content. A control is needed: e.g., compare sparsity profiles for the same prompt with different target answers, or corrupt/replace the visual input, and compare MAE/over-truncation against predicting the dataset median answer length or prompt-prefix length plus a constant. Without such a baseline, the one-shot truncation could systematically over-truncate short answers and under-truncate long ones on distributions that deviate from the training prior.
  3. [§3.6, Eq. (10)–(16), Table 10] The accuracy-improvement narrative is presented as a 'theoretical perspective,' but the decomposition v_s = c_s + n_s and the assumption E||Delta h||^2 > 0 are asserted without derivation or independent evidence. The attention metrics in Table 10 are correlational and the authors explicitly disclaim causality for this part. Given that Table 1 shows mixed accuracy effects (e.g., MME decreases on LaViDa-LLaDA from 1705.29 to 1697.94; ChartQA decreases from 60.60 to 57.80), the claim that Seer 'even improves accuracy' should be reframed as a hypothesis, not a demonstrated benefit. This is secondary to the efficiency claim, but it appears in the abstract and contributions.
minor comments (5)
  1. [§3.3, Table 3] Define MAE and over-truncation rate precisely, and state units (tokens vs. percentage). The look-ahead window size is set to 3 with a sensitivity study in Table 12, but the main text calls it 'kept fixed' without explaining the selection rule.
  2. [§3.2 vs. §3.3 vs. Appendix F] The observation is described as holding for Layers 3–6 (Figure 3b) and later L1–L6 (Appendix F), but the detector uses 'the 1D sparsity array from Layer 3.' Please justify the choice of a single layer and report boundary quality per layer or describe the aggregation.
  3. [§4.1 vs. Table 5] The main text lists D3ToM, RedVTP, VisionZip, MMTok, DivPrune, and SparseVLM as baselines, but Appendix B and Table 5 also include Focus and Daedal. Please make the baseline list consistent across the paper.
  4. [Figure 3(b)] The sparsity curves lack axis labels and numerical scales; the 'Semantic Jump' and 'Padding Plateau' would be easier to assess with explicit y-axis units and error bars across samples.
  5. [Appendix D, Table 10] The sentence 'Table 10 reports the results...' appears before Table 10 is introduced, and the table numbering in the appendix is confusing; reorder so the reference follows the table.

Circularity Check

1 steps flagged

Core Step-0 sparsity observation is empirical and not circular, but the default detector operating point is selected on ChartQA and then ChartQA is reported as a headline result, giving a localized fit-then-predict loop.

specific steps
  1. fitted input called prediction [Sec. 4.5 (Eq. 17), Table 2, Table 1]
    "To justify our default parameters ( τ_jump = 0.03, γ = 0.6) in Table 1 and avoid arbitrary tuning, we formalize this trade-off as a multi-objective optimization problem. Let Â(τ,γ) and T̂(τ,γ) denote baseline-normalized accuracy and throughput, with joint utility U defined as: U(τ,γ)= Â(τ,γ)+λ log T̂(τ,γ), (17) ... Mapping Table 2 onto the Pareto frontier identifies the “knee” ... revealing (τ_jump=0.03, γ=0.6) as the theoretical optimum ... confirming it as a principled solution rather than mere empirical coincidence."

    Table 2 is explicitly 'Evaluated on ChartQA', and the same ChartQA task appears as a headline row in Table 1, where Seer's accuracy/throughput are reported under these defaults. Thus the detector's operating point is not independently derived or held out: it is selected by maximizing U on ChartQA, with λ unspecified, and ChartQA is then used as evidence that Seer maintains overall performance. The same ChartQA-selected thresholds are also applied to all other benchmarks, so the 'theoretical optimum' claim is a post-hoc grid-search choice rather than a prediction. This affects the detector's operating point and the reported ChartQA numbers, though it does not by itself establish the existence of the Step-0 sparsity jump.

full rationale

The central claim — that Step-0 early-layer MLP sparsity reveals a semantic boundary — is an empirical profiled correlation, documented in Sec. 3.2 and Appendix F with heatmaps and later quantified by MAE/over-truncation in Table 3. It is not derived from Seer's own truncation output, and it is tested on external benchmarks, so there is no constructional equivalence between the sparsity observation and the predicted boundary. The throughput and latency results are also independent system measurements against external baselines (D3ToM, RedVTP, VisionZip, etc.), with no load-bearing self-citation chain that reduces the argument to the authors' prior work. The concrete circularity found is narrower: the detector thresholds (τ_jump, γ) are chosen on ChartQA using Eq. 17 with an unspecified λ, and the same ChartQA benchmark is then reported as a headline result in Table 1. That is a fit-then-report loop for the detector's operating point, but it does not reduce the main sparsity phenomenon to an input definition. The mechanism behind the Step-0 sparsity signal is not proven, and the accuracy-improvement story in Sec. 3.6 is post hoc, but those are correctness-risk concerns, not circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The central claim rests on an empirical sparsity-oracle assumption plus several tuned thresholds (tau_jump, gamma, tau_pad, window size, binarization threshold), one of which is selected via a Pareto curve on ChartQA. No new physical entities are introduced.

free parameters (6)
  • tau_jump (absolute jump threshold) = 0.03 (default)
    Default chosen from the ChartQA Pareto frontier (Table 2, Fig 6) and then used in all main results.
  • gamma (tolerance/noise ratio) = 0.6 (default)
    Selected jointly with tau_jump via the Pareto utility in Eq 17 on ChartQA.
  • lambda in utility U(tau,gamma) = unspecified
    Eq 17 defines the claimed Pareto optimum but never assigns a value to lambda; without it, the 'theoretical optimum' is not fully determined.
  • tau_pad (padding waste threshold) = 0.2 example; sensitivity safe zone 0.2-0.6
    Routing threshold in Sec 3.4 that decides static vs varlen vs eager execution; affects throughput claims.
  • look-ahead window size = 3
    Local window W_i in Eq 3-4; Table 12 shows size 3 as the tuned optimum.
  • sparsity binarization threshold = not reported
    Sec 3.2 says near-zero activations are defined by an absolute threshold, but the exact value is never given; the entire sparsity signal depends on it.
axioms (5)
  • ad hoc to paper Step-0 early-layer MLP sparsity is a reliable semantic-boundary oracle
    Core empirical premise introduced in Sec 3.2/Fig 3b; supported by profiling but not mechanistically proven.
  • domain assumption Padding/[EOS] tokens are semantically vacuous and attend diffusely to irrelevant visual regions
    Used in Sec 3.6 and Fig 2 to argue suffix-mediated noise leakage; selected visualizations are treated as representative.
  • ad hoc to paper The decomposition v_s = c_s + n_s with E||Delta h||^2 > 0 describes meaningful suffix-mediated noise
    Eq 10-16; the existence of a task-irrelevant nuisance component is assumed to explain occasional accuracy gains.
  • domain assumption Truncating at L* = i* + 1 preserves the final generation distribution
    Implicit throughout; over-truncation is measured on only three benchmarks and remains nonzero (0.8%-3.8%).
  • standard math Standard factorized diffusion and softmax-attention math applies
    Eq 1-2 and Eq 7-15 rely on standard transformer and discrete-diffusion background.

pith-pipeline@v1.3.0-alltime-deepseek · 29259 in / 15927 out tokens · 160580 ms · 2026-08-02T01:44:27.793874+00:00 · methodology

0 comments
read the original abstract

Diffusion Multimodal Large Language Models (DMLLMs) are highly effective for multimodal reasoning, yet their inference efficiency is significantly hindered by fixed-length generation constraints. Since the actual output length is unknown, output sequences are padded to a predefined maximum length, resulting in substantial redundant computation over unnecessary [EOS] tokens. In this work, we discover that DMLLMs implicitly reveal their valid semantic boundary at the very first denoising step through a distinct shift in MLP activation sparsity. Leveraging this observation, we propose Seer, a training-free framework that detects this boundary using a Signal-to-Noise Ratio (SNR)-based criterion and performs one-shot truncation of the redundant suffix for all subsequent computations. To preserve these theoretical gains during batched serving, Seer incorporates a hybrid execution strategy that maximizes throughput while seamlessly accommodating dynamic sequence lengths. Experimental results demonstrate that Seer effectively eliminates padding waste, accelerating throughput by up to $\sim$31$\times$. Across 9 benchmarks, Seer robustly maintains overall performance and even improves accuracy on complex visual tasks by mitigating noise leakage (e.g., DocVQA score increases from 63.52 to 63.66), offering a highly efficient, plug-and-play solution for DMLLM acceleration.

Figures

Figures reproduced from arXiv: 2607.14557 by Qicheng Zhao, Qi Sun, Zheyu Yan.

Figure 1
Figure 1. Figure 1: An illustration of the “Curse of Padding” in DM [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Visualization of cross-modal attention showing [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Motivation for dynamic semantic truncation. (a) Ground-truth Answer Length Distributions across four representative [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overview of the Seer framework. Left (Step 0, early layers): Redundant suffix tokens waste computation and diffusely [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Pareto frontier of generation accuracy versus infer [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Effect of 𝜏𝑝𝑎𝑑 on throughput. Experiments with LaViDa-LLaDA on ChartQA show that as 𝜏𝑝𝑎𝑑 increases be￾yond a critical point, throughput drops sharply across all batch sizes. This indicates that an excessively high thresh￾old forces too many skewed sequences into static execution paths, incurring severe padding waste that negates the bene￾fits of semantic truncation. long-tailed batches are forced into stat… view at source ↗
Figure 8
Figure 8. Figure 8: Throughput Scaling on LaViDa-LLaDA. Impact of different execution strategies on inference throughput across [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Throughput Scaling on LaViDa-Dream and MMaDA. Impact of different execution strategies on inference throughput [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Pareto frontier of generation accuracy versus inference throughput. (a) and (b) illustrate the trade-off on the MMMU [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Sensitivity analysis of the padding waste threshold ( [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Step-0 MLP sparsity heatmap for an extremely short response (Generation Length = 16). The semantic boundary is [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Step-0 MLP sparsity heatmap for a short response (Generation Length = 16). The blue box highlights the sharp [PITH_FULL_IMAGE:figures/full_fig_p021_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Step-0 MLP sparsity heatmap (Generation Length = 16). The early layers consistently maintain a stable “Padding [PITH_FULL_IMAGE:figures/full_fig_p022_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Step-0 MLP sparsity heatmap (Generation Length = 16) demonstrating the contrast between the informative early [PITH_FULL_IMAGE:figures/full_fig_p023_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Step-0 MLP sparsity heatmap for a medium-length response (Generation Length = 32), showing a highly consistent [PITH_FULL_IMAGE:figures/full_fig_p024_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Step-0 MLP sparsity heatmap (Generation Length = 32). The color shift within the L1–L6 window accurately pinpoints [PITH_FULL_IMAGE:figures/full_fig_p025_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Step-0 MLP sparsity heatmap for a long response (Generation Length = 64). Even as the valid prefix extends, the [PITH_FULL_IMAGE:figures/full_fig_p025_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Step-0 MLP sparsity heatmap for an extremely long response (Generation Length = 64). The visualization confirms [PITH_FULL_IMAGE:figures/full_fig_p026_19.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

89 extracted references · 28 linked inside Pith

  1. [1]

    Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang

  2. [2]

    Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. 2025. Block diffusion: Interpolating between autoregressive and diffusion language models.arXiv preprint arXiv:2503.09573(2025)

    Pith/arXiv arXiv 2025

  3. [3]

    Changwoo Baek, Jouwon Song, Sohyeon Kim, and Kyeongbo Kong. 2026. Ag- ilePruner: An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models.arXiv preprint arXiv:2603.01236 (2026)

    arXiv 2026

  4. [4]

    Wenrui Bao, Zhiben Chen, Dan Xu, and Yuzhang Shang. 2025. Learning to parallel: Accelerating diffusion large language models via learnable parallel decoding.arXiv preprint arXiv:2509.25188(2025)

    arXiv 2025

  5. [5]

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feicht- enhofer, and Judy Hoffman. 2022. Token merging: Your vit but faster.arXiv preprint arXiv:2210.09461(2022)

    Pith/arXiv arXiv 2022

  6. [6]

    Shuochen Chang, Xiaofeng Zhang, Qingyang Liu, and Li Niu. 2025. D 3 ToM: Decider-Guided Dynamic Token Merging for Accelerating Diffusion MLLMs. arXiv preprint arXiv:2511.12280(2025)

    arXiv 2025

  7. [7]

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2024. An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models. arXiv:2403.06764 [cs.CV]

    Pith/arXiv arXiv 2024

  8. [8]

    Xiwen Chen, Wenhui Zhu, Gen Li, Xuanzhao Dong, Yujian Xiong, Hao Wang, Peijie Qiu, Qingquan Song, Zhipeng Wang, Shao Tang, et al . 2026. OTPrune: Distribution-Aligned Visual Token Pruning via Optimal Transport.arXiv preprint arXiv:2602.20205(2026)

    arXiv 2026

  9. [9]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 24185–24198

    2024

  10. [10]

    Shuang Cheng, Yuhua Jiang, Zineng Zhou, Dawei Liu, Wang Tao, Linfeng Zhang, Biqing Qi, and Bowen Zhou. 2025. Sdar-vl: Stable and efficient block-wise diffusion for vision-language understanding.arXiv preprint arXiv:2512.14068 (2025)

    arXiv 2025

  11. [11]

    Zicong Cheng, Guo-Wei Yang, Jia Li, Zhijie Deng, Meng-Hao Guo, and Shi-Min Hu. 2025. Deer: Draft with diffusion, verify with autoregressive models.arXiv preprint arXiv:2512.15176(2025)

    arXiv 2025

  12. [12]

    Mohamed Dhouib, Davide Buscaldi, Sonia Vanier, and Aymen Shabou. 2025. Pact: Pruning and clustering-based token reduction for faster visual language models. InProceedings of the Computer Vision and Pattern Recognition Conference. 14582–14592

    2025

  13. [13]

    Sixun Dong, Juhua Hu, Mian Zhang, Ming Yin, Yanjie Fu, and Qi Qian. 2025. Mmtok: Multimodal coverage maximization for efficient inference of vlms.arXiv preprint arXiv:2508.18264(2025)

    arXiv 2025

  14. [14]

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. 2023. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394(2023)

    Pith/arXiv arXiv 2023

  15. [15]

    Yefei He, Feng Chen, Jing Liu, Wenqi Shao, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. 2024. Zipvl: Efficient large vision-language models with dynamic token sparsification.arXiv preprint arXiv:2410.08584(2024)

    Pith/arXiv arXiv 2024

  16. [16]

    Jianuo Huang, Yaojie Zhang, Yicun Yang, Benhao Huang, Biqing Qi, Dongrui Liu, and Linfeng Zhang. 2025. Mask tokens as prophet: Fine-grained cache eviction for efficient dllm inference.arXiv preprint arXiv:2510.09309(2025)

    arXiv 2025

  17. [17]

    Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real- world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 6700–6709

    2019

  18. [18]

    Daniel Israel, Guy Van den Broeck, and Aditya Grover. 2025. Accelerating diffusion llms via adaptive parallel decoding.arXiv preprint arXiv:2506.00413 (2025)

    arXiv 2025

  19. [19]

    Ting Jiang, Yixiao Wang, Hancheng Ye, Zishan Shao, Jingwei Sun, Jingyang Zhang, Zekai Chen, Jianyi Zhang, Yiran Chen, and Hai Li. 2025. Sada: Stability- guided adaptive diffusion acceleration.arXiv preprint arXiv:2507.17135(2025)

    Pith/arXiv arXiv 2025

  20. [20]

    Yuchu Jiang, Yue Cai, Xiangzhong Luo, Jiale Fu, Jiarui Wang, Chonghan Liu, and Xu Yang. 2025. d2Cache: Accelerating Diffusion-Based LLMs via Dual Adaptive Caching.arXiv preprint arXiv:2509.23094(2025)

    arXiv 2025

  21. [21]

    Yutao Jiang, Qiong Wu, Wenhao Lin, Wei Yu, and Yiyi Zhou. 2025. What kind of visual tokens do we need? training-free visual token pruning for multi-modal large language models from the perspective of graph. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 4075–4083

    2025

  22. [22]

    Jaeyeon Kim, Lee Cheuk-Kit, Carles Domingo-Enrich, Yilun Du, Sham Kakade, Timothy Ngotiaoco, Sitan Chen, and Michael Albergo. 2025. Any-order flexible length masked diffusion.arXiv preprint arXiv:2509.01025(2025)

    Pith/arXiv arXiv 2025

  23. [23]

    Duo Li, Zuhao Yang, Xiaoqin Zhang, Ling Shao, and Shijian Lu. 2025. A Com- prehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Models.arXiv preprint arXiv:2511.15098(2025)

    Pith/arXiv arXiv 2025

  24. [24]

    Guanghao Li, Zhihui Fu, Min Fang, Qibin Zhao, Ming Tang, Chun Yuan, and Jun Wang. 2025. Diffuspec: Unlocking diffusion language models for speculative decoding.arXiv preprint arXiv:2510.02358(2025)

    arXiv 2025

  25. [25]

    Jinsong Li, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Jiaqi Wang, and Dahua Lin

  26. [26]

    Shufan Li, Jiuxiang Gu, Kangning Liu, Zhe Lin, Zijun Wei, Aditya Grover, and Ja- son Kuen. 2025. Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Models.arXiv preprint arXiv:2512.14008(2025)

    Pith/arXiv arXiv 2025

  27. [27]

    Beyond fixed: Training-free variable-length denoising for diffusion large language models.arXiv preprint arXiv:2508.00819(2025)

    Pith/arXiv arXiv 2025

  28. [28]

    Kaihua Liang, Xin Tan, An Zhong, Hong Xu, and Marco Canini. 2026. FO- CUS: DLLMs Know How to Tame Their Compute Bound.arXiv preprint arXiv:2601.23278(2026)

    Pith/arXiv arXiv 2026

  29. [29]

    Shufan Li, Konstantinos Kallidromitis, Hritik Bansal, Akash Gokul, Yusuke Kato, Kazuki Kozuka, Jason Kuen, Zhe Lin, Kai-Wei Chang, and Aditya Grover. 2025. Lavida: A large diffusion language model for multimodal understanding.arXiv preprint arXiv:2505.16839(2025)

    Pith/arXiv arXiv 2025

  30. [30]

    Luxi Lin, Zhihang Lin, Zhanpeng Zeng, and Rongrong Ji. 2025. Speculative decoding reimagined for multimodal large language models.arXiv preprint arXiv:2505.14260(2025)

    Pith/arXiv arXiv 2025

  31. [31]

    Haokun Lin, Haobo Xu, Yichen Wu, Ziyu Guo, Renrui Zhang, Zhichao Lu, Ying Wei, Qingfu Zhang, and Zhenan Sun. 2025. Quantization meets dllms: A sys- tematic study of post-training quantization for diffusion llms.arXiv preprint arXiv:2508.14896(2025)

    arXiv 2025

  32. [32]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 26296–26306

    2024

  33. [33]

    Fuliang Liu, Xue Li, Ketai Zhao, Yinxi Gao, Ziyan Zhou, Zhonghui Zhang, Zhibin Wang, Wanchun Dou, Sheng Zhong, and Chen Tian. 2026. DART: Diffusion-Inspired Speculative Decoding for Fast LLM Inference.arXiv preprint arXiv:2601.19278(2026)

    arXiv 2026

  34. [34]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al . 2024. Mmbench: Is your multi-modal model an all-around player?. InEuropean conference on computer vision. Springer, 216–233

    2024

  35. [35]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual in- struction tuning.Advances in neural information processing systems36 (2023), 34892–34916

    2023

  36. [36]

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255(2023)

    Pith/arXiv arXiv 2023

  37. [37]

    Zhiyuan Liu, Yicun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang, and Linfeng Zhang. 2025. dllm-cache: Accelerating diffusion large language models with adaptive caching.arXiv preprint arXiv:2506.06295(2025)

    Pith/arXiv arXiv 2025

  38. [38]

    Junwei Luo, Yingying Zhang, Xue Yang, Kang Wu, Qi Zhu, Lei Liang, Jingdong Chen, and Yansheng Li. 2025. When large vision-language model meets large remote sensing imagery: Coarse-to-fine text-guided token pruning. InProceedings of the IEEE/CVF International Conference on Computer Vision. 9206–9217

    2025

  39. [39]

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering.Ad- vances in neural information processing systems35 (2022), 2507–2521

    2022

  40. [40]

    Qiankun Ma, Ziyao Zhang, Haofei Wang, Jie Chen, Zhen Song, and Hairong Zheng. 2026. ApET: Approximation-Error Guided Token Compression for Effi- cient VLMs.arXiv preprint arXiv:2602.19870(2026)

    arXiv 2026

  41. [41]

    Ji Ma, Wei Suo, Peng Wang, and Yanning Zhang. 2025. Short-lvlm: Compressing and accelerating large vision-language models by pruning redundant layers. In Proceedings of the 33rd ACM International Conference on Multimedia. 3575–3584

    2025

  42. [42]

    Yuxin Ma, Lun Du, Lanning Wei, Kun Chen, Qian Xu, Kangyu Wang, Guofeng Feng, Guoshan Lu, Lin Liu, Xiaojing Qi, et al. 2025. dinfer: An efficient inference framework for diffusion language models.arXiv preprint arXiv:2510.08666(2025)

    arXiv 2025

  43. [43]

    Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang. 2025. dkv-cache: The cache for diffusion language models.arXiv preprint arXiv:2505.15781(2025)

    Pith/arXiv arXiv 2025

  44. [44]

    Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. 2022. Infographicvqa. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 1697–1706

    2022

  45. [45]

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque

  46. [46]

    Quan Nguyen-Tri, Mukul Ranjan, and Zhiqiang Shen. 2025. Attention is all you need for kv cache in diffusion llms.arXiv preprint arXiv:2510.14973(2025)

    arXiv 2025

  47. [47]

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. 2025. Large language diffusion models.arXiv preprint arXiv:2502.09992(2025)

    Pith/arXiv arXiv 2025

  48. [48]

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. Docvqa: A dataset for vqa on document images. InProceedings of the IEEE/CVF winter conference on applications of computer vision. 2200–2209

    2021

  49. [49]

    Yuerong Song, Xiaoran Liu, Ruixiao Li, Zhigeng Liu, Zengfeng Huang, Qipeng Guo, Ziwei He, and Xipeng Qiu. [n. d.]. Sparse-dllm: Accelerating diffusion llms with dynamic cache eviction, 2025a.URL https://arxiv. org/abs/2508.02558 ([n. d.])

    arXiv

  50. [50]

    Yizheng Sun, Yanze Xin, Hao Li, Jingyuan Sun, Chenghua Lin, and Riza Theresa Batista-Navarro. 2025. Lvpruning: An effective yet simple language-guided vision token pruning approach for multi-modal large language models. InFindings of the Association for Computational Linguistics: NAACL 2025. 4299–4308

    2025

  51. [51]

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. 2021. Dynamicvit: Efficient vision transformers with dynamic token sparsification.Advances in neural information processing systems34 (2021), 13937–13949

    2021

  52. [52]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191(2024)

    Pith/arXiv arXiv 2024

  53. [53]

    Xu Wang, Chenkai Xu, Yijie Jin, Jiachun Jin, Hao Zhang, and Zhijie Deng. 2025. Diffusion llms can do faster-than-ar inference via discrete diffusion forcing.arXiv preprint arXiv:2508.09192(2025)

    Pith/arXiv arXiv 2025

  54. [54]

    Pavan Kumar Anasosalu Vasu, Fartash Faghri, Chun-Liang Li, Cem Koc, Nate True, Albert Antony, Gokula Santhanam, James Gabriel, Peter Grasch, Oncel Tuzel, et al. 2025. Fastvlm: Efficient vision encoding for vision language models. InProceedings of the Computer Vision and Pattern Recognition Conference. 19769– 19780

    2025

  55. [55]

    Linye Wei, Wenjue Chen, Pingzhi Tang, Xiaotian Guo, Le Ye, Runsheng Wang, and Meng Li. 2025. Orchestrating Dual-Boundaries: An Arithmetic Intensity Inspired Acceleration Framework for Diffusion Language Models.arXiv preprint arXiv:2511.21759(2025)

    arXiv 2025

  56. [56]

    Zichen Wen, Yifeng Gao, Weijia Li, Conghui He, and Linfeng Zhang. 2025. Token pruning in multimodal large language models: Are we solving the right problem?. InFindings of the Association for Computational Linguistics: ACL 2025. 15537– 15549

    2025

  57. [57]

    Yixiao Wang, Ting Jiang, Zishan Shao, Hancheng Ye, Jingwei Sun, Mingyuan Ma, Jianyi Zhang, Yiran Chen, and Hai Li. 2026. Zeus: Accelerating diffusion models with only second-order predictor.arXiv preprint arXiv:2604.01552(2026)

    arXiv 2026

  58. [58]

    Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. 2025. Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding.arXiv preprint arXiv:2505.22618(2025)

    Pith/arXiv arXiv 2025

  59. [59]

    Hao Wu, Yingqi Fan, Jinyang Dai, Junlong Tong, Yunpu Ma, and Xiaoyu Shen

  60. [60]

    Important Tokens

    Zichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang, Qintong Zhang, Weijia Li, Conghui He, and Linfeng Zhang. 2025. Stop Looking for “Important Tokens” in Multimodal Language Models: Duplication Matters More. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 9972–9991

    2025

  61. [61]

    Yi Xin, Qi Qin, Siqi Luo, Kaiwen Zhu, Juncheng Yan, Yan Tai, Jiayi Lei, Yuewen Cao, Keqi Wang, Yibin Wang, et al. 2025. Lumina-dimoo: An omni diffusion large language model for multi-modal generation and understanding.arXiv preprint arXiv:2510.06308(2025)

    arXiv 2025

  62. [62]

    Chen Xu and Dawei Yang. 2025. Dllmquant: Quantizing diffusion-based large language models.arXiv preprint arXiv:2508.14090(2025)

    Pith/arXiv arXiv 2025

  63. [63]

    Jingqi Xu, Jingxi Lu, Chenghao Li, Sreetama Sarkar, Souvik Kundu, and Peter A Beerel. 2025. RedVTP: Training-Free Acceleration of Diffusion Vision-Language Models Inference via Masked Token-Guided Visual Token Pruning.arXiv preprint arXiv:2511.12428(2025)

    arXiv 2025

  64. [64]

    Qiong Wu, Wenhao Lin, Yiyi Zhou, Weihao Ye, Zhanpeng Zen, Xiaoshuai Sun, and Rongrong Ji. 2024. Accelerating multimodal large language mod- els via dynamic visual-token exit and the empirical findings.arXiv preprint arXiv:2411.19628(2024)

    Pith/arXiv arXiv 2024

  65. [65]

    Chenyu Yang, Xuan Dong, Xizhou Zhu, Weijie Su, Jiahao Wang, Hao Tian, Zhe Chen, Wenhai Wang, Lewei Lu, and Jifeng Dai. 2025. PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision- Language Models. InProceedings of the Computer Vision and Pattern Recognition Conference. 24939–24949

    2025

  66. [66]

    Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. 2025. Mmada: Multimodal large diffusion language models.arXiv preprint arXiv:2505.15809(2025)

    Pith/arXiv arXiv 2025

  67. [67]

    Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. 2025. Visionzip: Longer is better but not necessary in vision language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 19792–19802

    2025

  68. [68]

    Zhuoyan Xu, Khoi Duc Nguyen, Preeti Mukherjee, Saurabh Bagchi, Somali Chaterji, Yingyu Liang, and Yin Li. 2025. Learning to inference adaptively for multimodal large language models. InProceedings of the IEEE/CVF International Conference on Computer Vision. 3552–3563

    2025

  69. [69]

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. 2024. Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800(2024)

    Pith/arXiv arXiv 2024

  70. [70]

    Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. 2025. Dream 7b: Diffusion large language models.arXiv preprint arXiv:2508.15487(2025)

    Pith/arXiv arXiv 2025

  71. [71]

    Weihao Ye, Qiong Wu, Wenhao Lin, and Yiyi Zhou. 2025. Fit and prune: Fast and training-free visual token pruning for multi-modal large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 22128–22136

    2025

  72. [72]

    Yicun Yang, Cong Wang, Shaobo Wang, Zichen Wen, Biqing Qi, Hanlin Xu, and Linfeng Zhang. 2025. Diffusion llm with native variable generation lengths: Let [eos] lead the way.arXiv preprint arXiv:2510.24605(2025)

    arXiv 2025

  73. [73]

    Zebin You, Shen Nie, Xiaolu Zhang, Jun Hu, Jun Zhou, Zhiwu Lu, Ji-Rong Wen, and Chongxuan Li. 2025. Llada-v: Large language diffusion models with visual instruction tuning.arXiv preprint arXiv:2505.16933(2025)

    Pith/arXiv arXiv 2025

  74. [74]

    Runpeng Yu, Xinyin Ma, and Xinchao Wang. [n. d.]. Dimple: Discrete diffusion multimodal large language model with parallel decoding, 2025.URL https://arxiv. org/abs/2505.16990([n. d.])

    Pith/arXiv arXiv 2025

  75. [75]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 9556–9567

    2024

  76. [76]

    X Ye, Y Gan, Y Ge, XP Zhang, and Y Tang. [n. d.]. ATP-LLAVA: adaptive token pruning for large vision language models (2024).URL https://arxiv. org/abs/2412 447 ([n. d.])

    2024

  77. [77]

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. 2025. Lmms- eval: Reality check on the evaluation of large multimodal models. InFindings of the Association for Computational Linguistics: NAACL 2025. 881–916

    2025

  78. [78]

    Qizhe Zhang, Aosong Cheng, Ming Lu, Renrui Zhang, Zhiyong Zhuo, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. 2025. Beyond text-visual attention: Exploiting visual cues for effective token pruning in vlms. InProceedings of the IEEE/CVF International Conference on Computer Vision. 20857–20867

    2025

  79. [79]

    Tianao Zhang, Zhiteng Li, Xianglong Yan, Haotong Qin, Yong Guo, and Yu- lun Zhang. 2025. Quant-dllm: Post-training extreme low-bit quantization for diffusion large language models.arXiv preprint arXiv:2510.03274(2025)

    arXiv 2025

  80. [80]

    Weili Zeng, Ziyuan Huang, Kaixiang Ji, and Yichao Yan. 2025. Skip-vision: Efficient and scalable acceleration of vision-language models via adaptive token skipping. InProceedings of the IEEE/CVF International Conference on Computer Vision. 21384–21397

    2025

Showing first 80 references.