Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

In block diffusion LLMs, the attention computed on the fully masked block at the first denoising step already reveals which context positions matter for every later step, so sparse attention can be driven by a single exact pass.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:16 UTC pith:KOUS5TKR

load-bearing objection MAGE's core observation is real and useful, but the paper overstates its recall and speedup numbers; deserves review with revisions. the 4 major comments →

arxiv 2602.14209 v2 pith:KOUS5TKR submitted 2026-02-15 cs.LG cs.CL

MAGE: All-[MASK] Block Already Knows Where to Look in Block Diffusion LLM

Omin Kwon , Yeonjae Kim , Doyeon Kim , Minseo Kim , Yeonhong Park , Jae W. Lee This is my paper
classification cs.LG cs.CL
keywords block diffusion LLMsparse attentionKV cacheall-masked blockattention patternslong-context inferencetraining-free accelerationself-distillation fine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Block diffusion language models generate tokens in parallel by denoising a block that starts fully masked. This paper claims that the attention computed on that initial all-masked block, before any real token has been decoded, already identifies the context entries that remain important throughout the block's denoising trajectory. On that basis, MAGE runs one exact attention pass at the first step, extracts per-head top-K index sets and layer-wise budgets, and reuses them for every remaining step without further training. In LongBench and needle-in-a-haystack tests, this matches the accuracy of exact attention at small KV budgets (K=512 to 1024) while delivering end-to-end speedups that grow with context length. A lightweight fine-tuning variant, aligned with the sparse pattern, often matches or exceeds exact attention at moderate budgets.

Core claim

The discovery is that the attention scores computed on an all-masked block in a block diffusion LLM exhibit temporal consistency: the top-K KV positions selected at the first denoising step retain 84–90% oracle recall across all later steps, and the layer-wise skewness of attention scores stays roughly constant. This means the structural relevance of the context, which tokens matter to the block, is largely fixed even while the block's own tokens are being generated. MAGE exploits this by making the first step exact and all subsequent steps sparse, with indices and budgets chosen once and reused, converting the repeated full-attention cost into a one-time investment.

What carries the argument

The central mechanism is MAGE's index-selection protocol on the all-masked block: for each layer, exact attention scores are computed, a per-query top-K selection is made, and the union across all queries sharing a KV head forms a head-specific candidate set; a coverage-adjusted score (union size times one minus the log of coverage) determines each layer's budget, and the resulting index sets are cached for the block's remaining steps. The design also overlaps this index selection with attention via multi-stream execution, so the extra work is hidden.

Load-bearing premise

The load-bearing premise is that the KV positions and layer-wise budgets identified from the fully masked block remain the correct ones for every later denoising step of that block; the paper gives intuition but no proof, and its recall numbers compare oracle sets to oracle sets, not MAGE's actual selections.

What would settle it

Measure the recall of MAGE's actual selected index sets, the unions formed at step 0, against the per-step oracle top-K sets across a range of tokens-per-step and context lengths; if recall drops materially below the oracle-to-oracle overlap, or if accuracy at K=512 diverges from exact attention on a long-context task, the central claim would be falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • MAGE shows that block diffusion LLMs can run long-context inference with a fraction of the KV budget at near-lossless accuracy, making memory access no longer the dominant bottleneck at long context.
  • The speedup grows with context length: measured per-step speedups reach 6.3× over exact attention at 128K context, with the first-step overhead amortized within as few as 2–4 denoising steps.
  • MAGE's sparse indices and budgets work without fine-tuning, so pretrained block diffusion models can be accelerated immediately.
  • Fine-tuning with self-distillation toward exact attention strengthens the all-masked block's guidance and can push accuracy above the exact-attention baseline, suggesting the sparse pattern acts as an implicit regularizer.
  • Existing sparse attention methods designed for autoregressive LLMs achieve only 36–82% recall when adapted to block diffusion, while all-masked-guided selection maintains 84–90% recall.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the temporal consistency holds for larger block sizes and longer contexts, MAGE's one-pass principle could extend to other parallel-decoding schemes, including fully bidirectional diffusion models, though the paper only tests block sizes up to 32 tokens per step.
  • The oracle-to-oracle recall (84–90%) is an upper bound on MAGE's actual set quality; a tighter bound on MAGE's own union-based sets would directly quantify how much accuracy headroom remains.
  • The explanation that the training objective aligns block-average queries across steps suggests the effect could be deliberately strengthened in training; one testable extension is to add a loss term that explicitly stabilizes top-K sets across denoising steps, potentially improving the fine-tuned variant further.
  • A practical stress test: at higher tokens-per-step or with multi-block interactions, if decoded tokens reorient attention, MAGE's fixed indices would be wrong; the paper's evidence is strongest for 1–4 tokens per step.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MAGE, a training-free sparse-attention method for block-diffusion LLMs. The central idea is to run one exact attention pass on the initial All-[MASK] block of each generated block, use that pass to select per-head KV index sets and per-layer budgets, then reuse those selections for all subsequent denoising steps. A lightweight fine-tuning variant (MAGE-FT) additionally uses self-distillation to strengthen the model's dependence on the All-[MASK]-guided patterns. The method is evaluated on Fast-dLLM 1.5B/7B on LongBench and NIAH against exact attention and AR-oriented sparse baselines (Quest, Tidal), with latency breakdowns at context lengths from 16K to 128K. The paper claims near-lossless accuracy at K=512, up to 6.82x end-to-end speedup, and consistent wins over Quest/Tidal.

Significance. If the central empirical premise holds, MAGE is a meaningful advance for block-diffusion inference: it identifies a property unique to block diffusion and exploits it with a simple, training-free mechanism. The paper has real strengths: evaluation across two model scales, several standard LongBench tasks, transparent latency breakdowns, and detailed engineering details (kernel modifications, multi-stream overlap, CUDA graphs). The temporal-consistency claim is also falsifiable and clearly stated. However, the current measurements overstate the claim in two ways: the headline 84-90% recall is an oracle-to-oracle overlap, not a recall of MAGE's actual index sets, and the 32K NIAH results show much larger accuracy drops than the 'near-lossless' language suggests. The contribution is plausible but needs firmer measurement before publication.

major comments (4)
  1. [Section 3.1, Figure 1] The recall reported in Figure 1 is oracle-to-oracle temporal overlap |S_oracle(0) ∩ S_oracle(t)|/K, not the recall of MAGE's selected sets. MAGE constructs per-head unions U_h and then selects T_{l,h} = SelectTopK(U_h, K_l); these sets are not the oracle top-K sets. The 84-90% figure therefore does not directly support the claim that the indices MAGE actually reuses are valid. Please report the actual recall |T_{l,h}(0) ∩ S_oracle(t)|/K, averaged over layers/heads, across denoising progress, tokens-per-step, and context length. This is load-bearing: if union-based voting or the layer-adaptive budgets have lower temporal consistency, the main mechanism is weaker than the current figures suggest.
  2. [Table 1 (NIAH, 32K)] The abstract's 'matches exact attention at k=512' and 'near-lossless accuracy' are contradicted at 32K context. For Fast-dLLM 7B, exact NIAH is 88; MAGE achieves 45 at K=512, 61 at K=1K, and only 70 at K=4K. For Fast-dLLM 1.5B, exact is 91 and MAGE reaches at most 42 (at K=4K), with 21 at K=512. These are large, systematic drops, not near-lossless. Please either scope the claim to LongBench and short contexts, or analyze why All-[MASK] guidance degrades at 32K. This also suggests the Figure 1 property may not extend to longer contexts and should be re-measured there.
  3. [Algorithm 1, Eq. (1)-(3)] The inference algorithm is under-specified at two load-bearing points. Line 7 says 'Union via voting' and line 15 says 'based on voting scores', but no vote-count definition or selection rule is given; the paper cannot be reproduced without specifying how T_{l,h} is computed from U_h. Additionally, Eq. (1) computes coverage p_h on a union U_h that already consists of top-K query selections, so p_h is not independent of K, and Eq. (2) is an ad hoc combination of |U_h| and (1-log p_h). Please provide a formal definition, a derivation, or at least an ablation showing that these choices matter.
  4. [Appendix A.3, Table 2] The layer-adaptive budget redistribution claimed in Section 3.2 is not supported by the ablation. In Table 2, many entries are negative (e.g., 1.5B NarrativeQA -0.6 at K=512, 7B Qasper -0.9 at K=256, 7B TriviaQA -0.2 at K=256/1K), and the improvements are small and inconsistent. Calling this 'modest' understates the noise. Please report averages with standard errors or paired significance tests. If the adjusted score in Eq. (2)-(3) is not consistently beneficial, simplify the method or reposition the contribution.
minor comments (5)
  1. [Abstract vs Section 5.2] The abstract claims up to 6.82x end-to-end speedup, but Section 5.2 reports 6.3x at 128K for MAGE2-n and Figure 11 shows 6.2-6.3x. Please reconcile the numbers or define exactly what the 6.82x figure corresponds to.
  2. [Section 4.2] Typo: 'provides insufficient signal for for adapting the model' should read 'for adapting'.
  3. [Section 4.2, Stage 1] The fine-tuning stage uses Top-P on attention scores, but the Top-P threshold is not reported and no sensitivity analysis is provided. Also specify the value of Kmin used in Eq. (3), since it affects the budget distribution.
  4. [Figures 4, 8, 9] The y-axes label 'Accuracy (%)' but LongBench tasks use different metrics (F1, EM, Rouge). Please use task-specific metric names for clarity.
  5. [Algorithm 1, line 14] The floor in Eq. (3) plus the Kmin lower bound can make the sum of K_l exceed the nominal budget K*L. Specify how over-allocation is handled and whether the latency numbers respect the reported average budget.

Circularity Check

0 steps flagged

No significant circularity: MAGE's first-step guidance is an empirical observation, and its index/budget choices are not fitted to the evaluated accuracy.

full rationale

The paper's central claim is empirical, not derivational. Section 3.1 measures oracle top-K overlap between the first all-masked step and later steps (84–90%), and Section 3.2 measures stability of layer-wise attention skewness. These are measurements of the model's attention patterns, not quantities defined in terms of the final accuracy metric. MAGE's Algorithm 1 selects indices directly from the first-step exact attention matrix and allocates budgets using the heuristic score s_l = max_h |U_h|·(1−log p_h) with proportional allocation. No parameter is fitted to LongBench accuracy, and no evaluated accuracy value is fed back into the index-selection or budget formula. The comparisons against Quest, Tidal, and exact attention are external benchmarks, and the cited prior work (Fast-dLLM, Block Diffusion, Quest, Tidal) is used as background or baseline, not as a self-supporting uniqueness claim. The motivating observation is measured on the same benchmark family used for evaluation, which is a generality concern but not circularity. No equation reduces to another by construction, and no prediction is a renamed fit. Therefore the appropriate circularity score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The paper's central claim rests on the empirical stability of masked-state attention patterns; this stability is asserted from figures rather than proved. The inference method introduces several hand-chosen constants (Kmin, score formula, allocation rule), but no new physical entities.

free parameters (4)
  • Kmin (minimum per-layer budget)
    Used in Eq. (3) / Algorithm 1 line 14 to clamp each layer's allocation, but never specified in the paper; it directly controls whether actual total budget exceeds K*L.
  • Top-P threshold in fine-tuning index selection
    Section 4.2 Stage 1 selects indices 'via Top-P' without reporting P; this controls the union set during MAGE-FT training.
  • Distillation temperature tau
    Eq. (4) uses temperature tau for KL distillation; tau is never reported, affecting the MAGE-FT objective.
  • KL loss weight lambda = 0.5 (1.5B), 1.5 (7B)
    Chosen by hand; affects fine-tuning behavior but not the training-free inference core.
axioms (5)
  • domain assumption Top-K KV indices selected at the all-masked step remain valid for all later denoising steps within the block.
    Section 3.1 / Figure 1; asserted empirically without proof. This is the load-bearing premise of MAGE.
  • domain assumption Per-layer attention skewness is stable across denoising steps, so budgets computed at step 1 can be reused.
    Section 3.2 / Figure 2; horizontal bands shown for four tasks, no quantitative stability statistic or proof.
  • ad hoc to paper The adjusted score s_{l,h}=|U_h|(1-log p_h) captures a layer's KV demand better than uniform allocation.
    Eq. (2); no derivation, and the paper's own ablation (Table 2) shows layer-adaptive redistribution gives only modest, mixed improvements.
  • domain assumption All queries in a block must share a single KV index set per KV head for memory-access efficiency.
    Introduction/Algorithm 1; motivates the union formation. The paper does not compare against per-query index sets with grouped gather.
  • domain assumption Sparse-attention approximation errors do not compound across denoising steps.
    Algorithm 1 reuses fixed indices for all steps t=2..T; no analysis of error accumulation.

pith-pipeline@v1.3.0-alltime-deepseek · 15354 in / 15297 out tokens · 133922 ms · 2026-08-02T23:16:18.205806+00:00 · methodology

0 comments
read the original abstract

Block diffusion LLMs are an emerging paradigm for parallel language generation, but their KV caching makes memory access the dominant bottleneck in long-context inference. Sparse attention, which attends only to a small KV subset per query, can reduce this latency with minimal accuracy loss. In block diffusion, however, the B tokens of each block must share a single KV subset, and we show this per-block constraint degrades existing sparse KV estimators by up to 25% in recall. We address this challenge by exploiting a property that emerges from the block-diffusion training objective: it aligns the block-average query across denoising steps, so the All-[MASK] block at the first step already reveals the per-block KV subset for the entire trajectory. We exploit this in MAGE ([MASK]-Guided Sparse Attention), a training-free method that runs one exact attention pass at the first step and reuses its top-k index sets for all remaining steps within the block. Across three block-diffusion families on LongBench, MAGE matches Exact Attention at k=512 with near-lossless accuracy, achieves up to 6.82x end-to-end speedup at 128K context, and runs up to 3.35x and 2.28x faster than Quest and SparseD, designed for AR LLMs and fully bidirectional diffusion LLMs, respectively.

Figures

Figures reproduced from arXiv: 2602.14209 by Doyeon Kim, Jae W. Lee, Minseo Kim, Omin Kwon, Yeonhong Park, Yeonjae Kim.

Figure 1
Figure 1. Figure 1: Top-K recall rate across denoising steps on LongBench tasks (0: All-[MASK]block, 1: fully decoded block). The top-K indices selected from the All-[MASK]block maintain 84–90% recall throughout denoising. 2.3. Dynamic Sparse Attention in Autoregressive LLMs Dynamic sparse attention techniques are widely used in AR decoding to reduce memory access by dynamically select￾ing a small subset of the KV cache to at… view at source ↗
Figure 2
Figure 2. Figure 2: Layer-wise budget across denoising progress on four LongBench tasks. Color indicates normalized budget (darker = higher). Early layers require larger budgets, while horizontal bands indicate stable relative budgets throughout denoising. Algorithm 1 MAGE Inference Require: KV cache C, block size B, budget K, steps T Ensure: Generated block x 1: Initialize block with B [MASK] tokens 2: // Phase 1: Exact Atte… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the MAGE Fine-tuning process. The process consists of three stages: (1) Index Selection identifies important KV indices via Top-K selection without gradients; (2) Sparse Forward applies the sparse mask and computes gradients; (3) Teacher Forward provides exact reference logits for self-distillation. adjusted scores: Kℓ = max  Kmin,  P sℓ ℓ ′ sℓ ′ · K · L  (3) where Kmin ensures a minimum al… view at source ↗
Figure 4
Figure 4. Figure 4: Per-task accuracy on LongBench with Fast-dLLM 1.5B across three denoising configurations (1, 2, 4 tokens/step). MAGE and MAGE-FT consistently outperform Quest and Tidal across all tasks and budgets. MAGE-FT often surpasses exact attention at moderate budgets, while maintaining robustness at higher tokens per step. (x0 → x0) and within the noisy block (xt → xt) remains exact. To ensure training stability, t… view at source ↗
Figure 5
Figure 5. Figure 5: Per-task average denoising step latency on LongBench with Fast-dLLM 1.5B across three denoising configurations (1, 2, 4 tokens/step). fine-tuning framework. We evaluate our methods on two different scale block diffusion models: Fast-dLLM 1.5B and Fast-dLLM 7B (Wu et al., 2025a). We compare our methods against the exact attention method and two sparse attention methods: Quest (Tang et al., 2024) and Tidal (… view at source ↗
Figure 6
Figure 6. Figure 6: Denoising step latency breakdown from 16K to 128K context length (top-k=2048). For MAGE1, stacked bars separate overlapped and main-stream operations. Dashed lines indicate exact attention latency. Extended results in Appendix A.4. highest recall at low budgets (K ≤ 512) across both mod￾els. Tidal degrades most severely, dropping to just 12% at K=256 on both model sizes. These results confirm that MAGE’s [… view at source ↗
Figure 8
Figure 8. Figure 8: presents per-task accuracy results on LongBench for Fast-dLLM 7B. Similar to the 1.5B model results in the main paper, MAGE consistently outperform Quest and Tidal across all tasks and budget settings [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: presents per-task per-step latency results on LongBench for Fast-dLLM 7B. 256 512 1024 2048 4096 14 16 18 20 22 24 26 1 token/step Accuracy (%) narrativeqa 256 512 1024 2048 4096 35 40 45 50 multifieldqa_en 256 512 1024 2048 4096 20 25 30 35 40 hotpotqa 256 512 1024 2048 4096 20 25 30 35 qasper 256 512 1024 2048 4096 60 65 70 75 80 85 triviaqa 256 512 1024 2048 4096 20 21 22 23 24 25 26 27 qmsum 256 512 10… view at source ↗
Figure 10
Figure 10. Figure 10: Top-K recall rate of different sparse attention strategies adapted to block diffusion. Quest (48–82%) and Tidal (36–74%) achieve limited recall, while All-[MASK]-guided selection maintains 84–90%. A.3. Ablation Study: Layer-Adaptive Budget Redistribution Tables 2 present ablation results comparing MAGE (step reuse only) and MAGE+ (with layer-adaptive budget redistribution). Values show MAGE+ accuracy with… view at source ↗
Figure 11
Figure 11. Figure 11: Complete denoising step latency breakdown across context lengths (16K–128K) and top-k values (512, 1024, 2048, 4096). Each group compares Exact attention, Quest, TidalDecode, and MAGE (first step MAGE1 and subsequent steps MAGE2-n). For MAGE1, stacked bars show overlapped (left) and main-stream (right) operations. Dashed lines indicate exact attention latency as baseline; numbers denote relative speedup (… view at source ↗
Figure 12
Figure 12. Figure 12: Comparison of attention scheduling strategies within a single denoising step. Quest alternates between importance estimation (Est.) and sparse (Sp.) attention phases. Tidal interleaves exact and sparse attention sequentially. MAGE1 (first step) performs exact attention on the main stream while asynchronously computing Top-K indices for subsequent steps. MAGE2-n leverages precomputed indices to execute spa… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. HERALD: High-Throughput Block Diffusion LLM Serving via CPU-GPU Cooperative KV Cache Retrieval

    cs.LG 2026-06 conditional novelty 7.0

    HERALD overlaps CPU-side sparse KV-cache selection with GPU-side block-denoising to hide offloading latency, achieving up to 2.28–2.47x decode throughput over dense GPU serving at 5–10% KV budgets.

  2. HERALD: High-Throughput Block Diffusion LLM Serving via CPU-GPU Cooperative KV Cache Retrieval

    cs.LG 2026-06 unverdicted novelty 6.0

    HERALD enables near-lossless accuracy at 5-10% KV budget for block dLLMs by amortizing top-k selection across denoising steps and overlapping CPU-GPU retrieval, yielding up to 2.47x higher throughput than GPU-only inference.

Reference graph

Works this paper leans on

11 extracted references · 6 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Gqa: Training generalized multi-query transformer models from multi-head check- points

    Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y ., Lebr´on, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head check- points. InProceedings of the 2023 Conference on Empir- ical Methods in Natural Language Processing,

    2023

  2. [4]

    Sdar: A synergistic diffusion-autoregression paradigm for scalable sequence generation.arXiv preprint arXiv:2510.06303,

    Cheng, S., Bian, Y ., Liu, D., Zhang, L., Yao, Q., Tian, Z., Wang, W., Guo, Q., Chen, K., Qi, B., and Zhou, B. Sdar: A synergistic diffusion-autoregression paradigm for scalable sequence generation.arXiv preprint arXiv:2510.06303,

    arXiv

  3. [6]

    Kim, J., Shah, K., Kontonis, V ., Kakade, S., and Chen, S

    URL https://arXiv.org/ abs/2410.17891. Kim, J., Shah, K., Kontonis, V ., Kakade, S., and Chen, S. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. InProceedings of the 42nd International Conference on Machine Learning, 2025a. Kim, M., Hooper, C., Tomar, A., Xu, C., Farajtabar, M., Mahoney, M. W., Keutzer, K., and ...

    Pith/arXiv arXiv

  4. [8]

    Tang, J., Zhao, Y ., Zhu, K., Xiao, G., Kasikci, B., and Han, S

    URL https://arxiv.org/abs/2502.09992. Tang, J., Zhao, Y ., Zhu, K., Xiao, G., Kasikci, B., and Han, S. Quest: Query-aware sparsity for efficient long- context llm inference. InProceedings of the 41st Inter- national Conference on Machine Learning, pp. 47901– 47911,

    Pith/arXiv arXiv

  5. [9]

    Diffusion llms can do faster-than-ar inference via dis- crete diffusion forcing.arXiv preprint arXiv:2508.09192, 2025a

    Wang, X., Xu, C., Jin, Y ., Jin, J., Zhang, H., and Deng, Z. Diffusion llms can do faster-than-ar inference via dis- crete diffusion forcing.arXiv preprint arXiv:2508.09192, 2025a. URL https://arXiv.org/abs/2508. 09192. Wang, Z., Fang, G., Ma, X., Yang, X., and Wang, X. Sparsed: Sparse attention for diffusion language mod- els.arXiv preprint arXiv:2509.24...

    Pith/arXiv arXiv

  6. [10]

    Qwen3 technical report, 2025a

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., et al. Qwen3 technical report, 2025a. URL https://arxiv.org/abs/2505.09388. Yang, L., Zhang, Z., Chen, Z., Li, Z., and Jia, Z. Tidalde- code: A fast and accurate llm decoding with position persistent sparse attention. InInternational Conference on Learning Representations, 2025b. Yang, L.,...

    Pith/arXiv arXiv

  7. [11]

    Additional Experimental Results This appendix provides detailed experimental results that supplement the main paper

    11 MAGE : All-[MASK]Block Already Knows Where to Look in Diffusion LLM A. Additional Experimental Results This appendix provides detailed experimental results that supplement the main paper. A.1. Fast-dLLM-v2-7B Results Figure 8 presents per-task accuracy results on LongBench for Fast-dLLM 7B. Similar to the 1.5B model results in the main paper, MAGE cons...

    2048

  8. [12]

    Figure 12.Comparison of attention scheduling strategies within a single denoising step

    Since the tensor shapes remain constant across decoding steps, we exploit this property to pre-compile specialized kernels that: • Eliminate intermediate memory allocations 14 MAGE : All-[MASK]Block Already Knows Where to Look in Diffusion LLM %FOPJTJOH 4UFQ ExactExactExactExactExact ExactSp.Sp.Exact MAGE1 To p-K.BJO4USFBN"TZOD4USFBN .BJO4USFBNTo p-KTidal...

    2048

  9. [2023]

    Gong, S., Agarwal, S., Zhang, Y ., Ye, J., Zheng, L., Li, M., An, C., Zhao, P., Bi, W., Peng, H., Han, J., and Kong, L

    URL https://openreview.net/ forum?id=D8DUDJQKZg. Gong, S., Agarwal, S., Zhang, Y ., Ye, J., Zheng, L., Li, M., An, C., Zhao, P., Bi, W., Peng, H., Han, J., and Kong, L. Scaling diffusion language models via adapta- tion from autoregressive models. InProceedings of the 2025 International Conference on Learning Represen- tations (ICLR),

    2025

  10. [2024]

    Cai, Z., Zhang, Y ., Gao, B., Liu, Y ., Li, Y ., Liu, T., Lu, K., Xiong, W., Dong, Y ., Hu, J., and Xiao, W

    URL https: //arxiv.org/abs/2308.14508. Cai, Z., Zhang, Y ., Gao, B., Liu, Y ., Li, Y ., Liu, T., Lu, K., Xiong, W., Dong, Y ., Hu, J., and Xiao, W. Pyramidkv: Dynamic kv cache compression based on pyramidal in- formation funneling.arXiv preprint arXiv:2406.02069,

    Pith/arXiv arXiv

  11. [2025]

    Bai, Y ., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y ., Tang, J., and Li, J

    URL https://arxiv.org/ abs/2503.09573. Bai, Y ., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y ., Tang, J., and Li, J. Longbench: A bilingual, multitask benchmark for long context understanding,

    Pith/arXiv arXiv