Pith. sign in

REVIEW 2 major objections 5 minor 43 references

Restricting reverse Mamba to the active block makes hybrid diffusion models cacheable and faster at long context.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 06:55 UTC pith:TMOEOGKX

load-bearing objection Solid systems fix for cacheable Mamba hybrids under BDLM; quality story is fine at small scale, but the headline long-context multipliers are only measured on random-init models. the 2 major comments →

arxiv 2607.02805 v1 pith:TMOEOGKX submitted 2026-07-02 cs.LG cs.AIcs.CL

Training Hybrid Block Diffusion Language Models with Partial Bidirectionality

Pranshu Chaturvedi , Parth Shroff , Tarun Suresh , Hangoo Kang , Kaiyue Wen This is my paper
classification cs.LG cs.AIcs.CL
keywords block diffusion language modelsMamba hybridspartial bidirectionalityprefix cachinglong-context inferencemasked diffusionstate-space models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Language model generation is often limited by memory bandwidth: each new token must re-read a growing key/value cache. Block diffusion tries to fix this by generating whole blocks of tokens at once and reusing finished blocks as a fixed prefix. Hybrid models that mix attention with Mamba can further cut memory traffic, but full bidirectional Mamba scans make reverse states depend on the entire noisy sequence, so they cannot be cached as a clean prefix. This paper shows that confining the reverse Mamba scan to only the current denoising block restores exact prefix caching while keeping local bidirectionality inside the block. On an 87M-parameter sweep the resulting hybrid reaches the best validation perplexity among the compared designs; at 350M it stays competitive with pure attention block diffusion; and in long-context inference it delivers large throughput gains over both full-sequence Mamba hybrids and attention-only block diffusion. The result argues that carefully partial-reverse Mamba hybrids are a practical architecture for high-throughput long-context diffusion language models.

Core claim

A block-diffusion Mamba–attention hybrid that runs reverse Mamba only inside the active denoising block produces exact, reusable prefix caches (forward Mamba boundary states plus attention key/value tensors) that match the objects used at inference. Trained under an all-block objective that back-propagates through those same caches, the hybrid obtains the best 87M C4-en NELBO perplexity in a controlled DCLM sweep, remains competitive at 350M, and substantially outperforms full-sequence DiffuMamba and BDLM attention on long-context inference throughput.

What carries the argument

Block partial-reverse Mamba: forward scan of the clean prefix yields a reusable recurrent boundary state; reverse scan is restricted to the current noised block and then added to the forward active-block states, so the prefix cache stays independent of the reverse diffusion step.

Load-bearing premise

The large reported inference speedups are measured on randomly initialized 700M models rather than trained checkpoints, so they assume architecture and kernel cost dominate over any weight-dependent effects.

What would settle it

Train a matched pair of 350M or larger BDLM attention and BDLM Mamba-H models, measure tokens-per-second on the same A100 ladder up to 262K tokens, and check whether the Mamba hybrid still crosses over and reaches roughly 3–4× attention throughput at the longest lengths.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes BDLM Mamba-H, a hybrid block diffusion language model that interleaves attention with Mamba layers while restricting reverse-direction Mamba scans to the active denoising block (Eqs. 10–13). This yields a native, timestep-free prefix cache of forward Mamba boundary states plus attention KV that matches the all-block training objective (Eq. 14) and the sampler. On an 87M DCLM learning-rate sweep, BDLM Mamba-H attains the best C4-en NELBO perplexity (61.59 vs 76.46 for BDLM attention and higher for full-sequence baselines; Tables 1, 4). At 350M with Complete(d)P-style LR transfer it remains competitive with BDLM attention (Table 3). Inference throughput on 700M random-init models is reported as 19.7× full-sequence DiffuMamba-H at 65K and 3.7× BDLM attention at 262K (Section 5.3, Appendix B, Table 6).

Significance. If the cache-aligned partial-reverse construction is correct and the long-context speedups transfer to trained models, the work supplies a concrete architectural recipe for combining block diffusion with linear-time mixers—something prior full-sequence DiffuMamba hybrids could not do with exact prefix reuse. Strengths include a clear training–inference cache interface, a full 87M LR sweep (Table 4), Complete(d)P-motivated scale-up, and explicit AdaLN factorization for optional timestep conditioning (Appendix E). The result is of practical interest for memory-bandwidth-bound long-context diffusion generation and is a natural next step after BDLM and DiffuMamba.

major comments (2)
  1. Section 5.3 and Appendix B (Table 6): the headline long-context multipliers (19.7× vs full-sequence DiffuMamba-H at 65K; 3.7× vs BDLM attention at 262K) and the ~16K crossover are measured only on 700M random-initialized models, not on the trained 350M checkpoints whose quality is reported. For selective SSMs, trained A/B/C projections can change state magnitudes, numerical paths, and whether graph-captured fixed-shape continuations remain on the same kernel path. The paper treats architecture cost as weight-independent; without at least a trained-checkpoint throughput check (or a clear argument that BF16/kernel paths are identical), the systems claim for deployed models is only partially secured.
  2. Section 5.2 / Table 3: at 350M, BDLM Mamba-H is slightly worse than BDLM attention on C4-en NELBO PPL (38.24 vs 36.97) and Paloma-C4, with only a modest MCQA edge (43.3 vs 41.3). Quality is single-run NELBO with one Monte Carlo sample and no multi-seed bars. The joint claim of “competitive quality + production long-context multipliers” therefore rests on a thin quality margin plus random-init speedups; either multi-seed validation or a trained-model throughput measurement would make the central claim load-bearing rather than provisional.
minor comments (5)
  1. Table 8: BDLM Mamba-H uses width 832 / 16 heads vs 896 / 14 for BDLM attention to match parameter count; a short note that geometry (not only mixer type) differs would avoid misreading the 350M comparison as pure architecture swap.
  2. Appendix D / Table 10: timestep conditioning helps BDLM attention but hurts BDLM Mamba-H at fixed LR; the main-text choice of no-timestep is justified, but a one-sentence pointer in §5.1 would help readers who expect AdaLN by default.
  3. Figure 2 and Table 6: full-sequence baselines stop at 65K while BDLM extends to 262K; stating the memory/OOM reason explicitly would clarify that the 19.7× comparison is at the longest common length, not an apples-to-apples full ladder.
  4. Notation: s^ℓ_k is used for both token- and block-boundary states; a brief clarification that the cache stores the state after the last token of the completed prefix would reduce ambiguity around Eqs. (10)–(13).
  5. Limitations correctly note slower BDLM training vs full-sequence; quantifying whether all-block objective cost or hybrid kernels dominate would help future systems work.

Circularity Check

0 steps flagged

No circularity: partial-reverse Mamba is a design constraint, and quality/throughput claims are external empirical measurements, not tautologies.

full rationale

This is an empirical systems/architecture paper, not a first-principles derivation that could close on itself. The load-bearing move is a design constraint: a reusable BDLM prefix cache must be a function of the clean prefix alone (independent of reverse timestep and of active-block tokens), so reverse Mamba is restricted to the active block (Eqs. 10–13) while forward boundary states and attention KV are cached. That is a definition of the architecture, not a prediction forced by fitting or by a self-cited uniqueness theorem. Training then aligns the all-block objective with those same cache objects; again a consistency choice, not a circular reduction. Reported C4-en / Paloma NELBO PPL and MCQA are measured on held-out external corpora under a fixed protocol; the 19.7× / 3.7× throughput ratios are measured wall-clock numbers (even if only on random-init 700M models—a validity issue, not circularity). Citations to BDLM (Arriola et al.), DiffuMamba (Singh et al.), and Mamba (Gu & Dao) are prior work by other authors and supply background, not a self-citation chain that forces the result. No fitted constant is renamed as a prediction; no uniqueness theorem is imported from the present authors. Score 0 with empty steps is the honest finding.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 2 invented entities

The central claim is architectural and empirical: partial reverse makes prefix caches exact and yields competitive quality plus higher long-context throughput. It rests on standard discrete-diffusion and Mamba machinery, several design choices (block size, hybrid schedule, no-timestep objective, Complete(d)P LR transfer), and the modeling premise that local reverse mixing plus clean-prefix forward state is enough. No new physical entities; free parameters are training/design knobs, not fitted universal constants.

free parameters (6)
  • BDLM block length = 256
    Fixed at 256 tokens for all BDLM runs; controls parallelism vs. cache granularity and is not derived.
  • 87M learning rates (architecture-specific best) = 8e-3 (Mamba-H); 4e-3 (BDLM attention)
    Selected by C4-en NELBO from a five-point sweep; best values differ by architecture (8e-3 Mamba-H, 4e-3 BDLM attention).
  • Complete(d)P transfer factors 87M→350M = 0.31 (attention); 0.33 (Mamba-H)
    Hidden-weight transfer exponents applied to width/depth/batch/duration ratios; paper notes laws were not derived for dLLMs.
  • Hybrid attention layer schedule = attention at layers 0,6,... / 0,6,12
    Sparse attention at fixed layers (e.g., 0,6,12) with Mamba elsewhere; design choice inherited from DiffuMamba-style hybrids.
  • Diffusion steps / factor p = 16
    Training and reported inference use p=16 reverse steps; affects quality–compute tradeoff.
  • No-timestep vs timestep conditioning choice = no-timestep for main sweep and 350M
    Main results disable timestep modulation after a fixed-LR comparison where timestep helped attention BDLM but hurt Mamba-H; choice is empirical, not forced by theory.
axioms (5)
  • domain assumption BDLM block-factorized likelihood and masked continuous-time NELBO (Arriola et al.) are valid training/evaluation objectives for comparing full-sequence and block diffusion models.
    Used throughout Sections 3–5 and Eq. 7 for PPL_NELBO ranking.
  • domain assumption A Mamba forward boundary state (conv + selective SSM) computed on clean prefix alone can be continued into the active block without reprocessing the prefix.
    Core of Eqs. 1, 10–11 and the cache interface in Section 4.
  • domain assumption Prefix cache must be independent of reverse timestep t (and of active-block tokens) to be reusable across denoising steps.
    Stated as requirement before Eq. 9; motivates partial reverse and AdaLN factorization (Appendix E).
  • ad hoc to paper Complete(d)P-style LR transfer across width, depth, batch, and duration yields strong scale-up candidates for these dLLMs.
    Applied in Section 5.1 / Table 2; Limitations admit laws were not derived for diffusion LMs.
  • ad hoc to paper Throughput measured on random-initialized models is representative of architecture-level inference cost for trained models of the same geometry.
    Section 5.3 and Appendix B use 700M random-init models for the long-context ladder.
invented entities (2)
  • BDLM Mamba-H / block partial-reverse architecture no independent evidence
    purpose: Restrict reverse Mamba to the active block so forward prefix states remain cacheable while the active block keeps local bidirectionality.
    Primary architectural contribution (Section 4, Eqs. 10–13). It is a design pattern, not an unobserved physical object; independent_evidence is false because validation is only within this paper’s experiments.
  • Cache-aligned all-block prefix objective for recurrent states no independent evidence
    purpose: Train the same forward Mamba boundary states (and attention KV) that the sampler reuses, via downstream block losses.
    Eqs. 14–17 and Figure 1; method-level construct specific to this training recipe.

pith-pipeline@v1.1.0-grok45 · 20554 in / 3986 out tokens · 40887 ms · 2026-07-12T06:55:59.445947+00:00 · methodology

0 comments
read the original abstract

High-throughput long-context generation is one of the central challenges for large language models. Generation is typically memory-bandwidth-bound rather than compute-bound: each decoding step must stream the accumulated key/value (KV) cache from memory, so bandwidth demand grows with context length while only one token is emitted. Two parallel approaches have therefore emerged: reducing memory access with efficient attention variants and linear-time mixers such as Mamba, or increasing parallel computation by generating blocks of tokens at once. However, technical challenges arise when combining these two ideas. Earlier hybrid diffusion models such as DiffuMamba use bidirectional Mamba mixing, including a reverse-direction scan relative to causal generation. This reverse scan needs to scan the entire sequence, so its states are not prefix-only and cannot be precisely reused as a cache even when diffusion is performed block by block. We propose a BDLM Mamba--attention hybrid that addresses this challenge by restricting the reverse Mamba scan to the active denoising block, which enables exact caching across blocks. In an 87M-parameter DCLM sweep, BDLM Mamba-H achieves the best C4-en validation perplexity compared to BDLM attention and full-sequence baselines. At 350M parameters, it remains competitive with BDLM attention. For long-context inference, BDLM Mamba-H reaches 19.7x the throughput of full-sequence DiffuMamba-H at 65K tokens and 3.7x the throughput of BDLM attention at 262K, showing that Mamba hybrids are a potential long-context diffusion architecture.

Figures

Figures reproduced from arXiv: 2607.02805 by Hangoo Kang, Kaiyue Wen, Parth Shroff, Pranshu Chaturvedi, Tarun Suresh.

Figure 1
Figure 1. Figure 1: BDLM Mamba-H training. The all-block objective applies this active-block compu [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: 700M random-initialized single-A100 inference throughput. Full-sequence base [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Cacheable AdaLN factorization for block diffusion. The clean prefix path is evalu [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 6 linked inside Pith

  1. [1]

    and Ho, Jonathan and Tarlow, Daniel and van den Berg, Rianne , title =

    Austin, Jacob and Johnson, Daniel D. and Ho, Jonathan and Tarlow, Daniel and van den Berg, Rianne , title =. Advances in Neural Information Processing Systems , year =

  2. [2]

    and Rush, Alexander and Kuleshov, Volodymyr , title =

    Sahoo, Subham Sekhar and Arriola, Marianne and Gokaslan, Aaron and Chiu, Justin T. and Rush, Alexander and Kuleshov, Volodymyr , title =. Advances in Neural Information Processing Systems , year =

  3. [3]

    and Yang, Zhihan and Qi, Zhixuan and Han, Jiaqi and Sahoo, Subham Sekhar and Kuleshov, Volodymyr , title =

    Arriola, Marianne and Gokaslan, Aaron and Chiu, Justin T. and Yang, Zhihan and Qi, Zhixuan and Han, Jiaqi and Sahoo, Subham Sekhar and Kuleshov, Volodymyr , title =. International Conference on Learning Representations , year =

  4. [4]

    Conference on Language Modeling , year =

    Gu, Albert and Dao, Tri , title =. Conference on Language Modeling , year =

  5. [5]

    International Conference on Machine Learning , year =

    Dao, Tri and Gu, Albert , title =. International Conference on Machine Learning , year =

  6. [6]

    Advances in Neural Information Processing Systems , year =

    FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness , author =. Advances in Neural Information Processing Systems , year =

  7. [7]

    International Conference on Computer Vision , year =

    Peebles, William and Xie, Saining , title =. International Conference on Computer Vision , year =

  8. [8]

    International Conference on Learning Representations , year =

    Wu, Chengyue and Zhang, Hao and Xue, Shuchen and Liu, Zhijian and Diao, Shizhe and Zhu, Ligeng and Luo, Ping and Han, Song and Xie, Enze , title =. International Conference on Learning Representations , year =

  9. [9]

    arXiv preprint arXiv:2505.15781 , year =

    Ma, Xinyin and Yu, Runpeng and Fang, Gongfan and Wang, Xinchao , title =. arXiv preprint arXiv:2505.15781 , year =

    Pith/arXiv arXiv

  10. [10]

    arXiv preprint arXiv:2506.06295 , year =

    Liu, Zhiyuan and Yang, Yicun and Zhang, Yaojie and Chen, Junjie and Zou, Chang and Wei, Qingyuan and Wang, Shaobo and Zhang, Linfeng , title =. arXiv preprint arXiv:2506.06295 , year =

    Pith/arXiv arXiv

  11. [11]

    arXiv preprint , year =

    Nguyen-Tri, Quan and Ranjan, Mukul and Shen, Zhiqiang , title =. arXiv preprint , year =

  12. [12]

    arXiv preprint arXiv:2508.09192 , year =

    Wang, Xu and Xu, Chenkai and Jin, Yijie and Jin, Jiachun and Zhang, Hao and Deng, Zhijie , title =. arXiv preprint arXiv:2508.09192 , year =

    Pith/arXiv arXiv

  13. [13]

    DiffuMamba: High-Throughput Diffusion

    Singh, Vaibhav and Ostapenko, Oleksiy and Noel, Pierre-Andr. DiffuMamba: High-Throughput Diffusion. International Conference on Machine Learning , year =

  14. [14]

    Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration , journal =

    Mlodozeniec, Bruno and Ablin, Pierre and B. Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration , journal =. 2025 , eprint =. doi:10.48550/arXiv.2512.22382 , url =

    doi:10.48550/arxiv.2512.22382 2025

  15. [15]

    arXiv preprint arXiv:2510.03280 , year =

    Ni, Jinjie and Liu, Qian and Du, Chao and Dou, Longxu and Yan, Hang and Wang, Zili and Pang, Tianyu and Shieh, Michael Qizhe , title =. arXiv preprint arXiv:2510.03280 , year =

    arXiv

  16. [16]

    Scaling Behavior of Discrete Diffusion Language Models , booktitle =

    von R. Scaling Behavior of Discrete Diffusion Language Models , booktitle =. 2026 , url =

    2026

  17. [17]

    Advances in Neural Information Processing Systems , year =

    Zheng, Chenyu and Zhang, Xinyu and Wang, Rongzhen and Huang, Wei and Tian, Zhi and Huang, Weilin and Zhu, Jun and Li, Chongxuan , title =. Advances in Neural Information Processing Systems , year =

  18. [18]

    International Conference on Learning Representations , year =

    Chen, Yukang and Qian, Shengju and Tang, Haotian and Lai, Xin and Liu, Zhijian and Han, Song and Jia, Jiaya , title =. International Conference on Learning Representations , year =

  19. [19]

    International Conference on Learning Representations , year =

    Peng, Bowen and Quesnelle, Jeffrey and Fan, Honglu and Shippole, Enrico , title =. International Conference on Learning Representations , year =

  20. [20]

    Findings of the Association for Computational Linguistics: ACL , year =

    Bai, Yushi and Lv, Xin and Zhang, Jiajie and Lyu, Hongchang and Tang, Jiankai and Huang, Zhidian and Du, Zhengxiao and Liu, Xiao and Zeng, Aohan and Hou, Lei and Dong, Yuxiao and Tang, Jie and Li, Juanzi , title =. Findings of the Association for Computational Linguistics: ACL , year =

  21. [21]

    and Carmon, Yair and Dave, Achal and Schmidt, Ludwig and Shankar, Vaishaal , title =

    Li, Jeffrey and Fang, Alex and Smyrnis, Georgios and Ivgi, Maor and Jordan, Matt and Gadre, Samir Yitzhak and Bansal, Hritik and Guha, Etash and Keh, Sedrick and Arora, Kushal and Garg, Saurabh and Xin, Rui and Muennighoff, Niklas and Heckel, Reinhard and Mercat, Jean and Chen, Mayee and Gururangan, Suchin and Wortsman, Mitchell and Albalak, Alon and Bitt...

  22. [22]

    and Babuschkin, Igor and Sidor, Szymon and Liu, Xiaodong and Farhi, David and Ryder, Nick and Pachocki, Jakub and Chen, Weizhu and Gao, Jianfeng , title =

    Yang, Greg and Hu, Edward J. and Babuschkin, Igor and Sidor, Szymon and Liu, Xiaodong and Farhi, David and Ryder, Nick and Pachocki, Jakub and Chen, Weizhu and Gao, Jianfeng , title =. Advances in Neural Information Processing Systems , year =

  23. [23]

    International Conference on Learning Representations , year =

    Hendrycks, Dan and Burns, Collin and Basart, Steven and Zou, Andy and Mazeika, Mantas and Song, Dawn and Steinhardt, Jacob , title =. International Conference on Learning Representations , year =

  24. [24]

    Challenging

    Suzgun, Mirac and Scales, Nathan and Sch. Challenging. Findings of the Association for Computational Linguistics: ACL , year =

  25. [25]

    Annual Meeting of the Association for Computational Linguistics , year =

    Zellers, Rowan and Holtzman, Ari and Bisk, Yonatan and Farhadi, Ali and Choi, Yejin , title =. Annual Meeting of the Association for Computational Linguistics , year =

  26. [26]

    AAAI Conference on Artificial Intelligence , year =

    Bisk, Yonatan and Zellers, Rowan and Gao, Jianfeng and Choi, Yejin , title =. AAAI Conference on Artificial Intelligence , year =

  27. [27]

    arXiv preprint arXiv:1803.05457 , year =

    Clark, Peter and Cowhey, Isaac and Etzioni, Oren and Khot, Tushar and Sabharwal, Ashish and Schoenick, Carissa and Tafjord, Oyvind , title =. arXiv preprint arXiv:1803.05457 , year =

    Pith/arXiv arXiv

  28. [28]

    Conference of the North American Chapter of the Association for Computational Linguistics , year =

    Clark, Christopher and Lee, Kenton and Chang, Ming-Wei and Kwiatkowski, Tom and Collins, Michael and Toutanova, Kristina , title =. Conference of the North American Chapter of the Association for Computational Linguistics , year =

  29. [29]

    AAAI Conference on Artificial Intelligence , year =

    Sakaguchi, Keisuke and Bras, Ronan Le and Bhagavatula, Chandra and Choi, Yejin , title =. AAAI Conference on Artificial Intelligence , year =

  30. [30]

    Advances in Neural Information Processing Systems , year =

    Attention Is All You Need , author =. Advances in Neural Information Processing Systems , year =

  31. [31]

    Proceedings of the 41st International Conference on Machine Learning , year =

    Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attention , author =. Proceedings of the 41st International Conference on Machine Learning , year =

  32. [32]

    arXiv preprint arXiv:2405.04434 , year =

    DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model , author =. arXiv preprint arXiv:2405.04434 , year =

    Pith/arXiv arXiv

  33. [33]

    Technical report , year =

    MiniMax-01: Scaling Foundation Models with Lightning Attention , author =. Technical report , year =

  34. [34]

    arXiv preprint arXiv:2506.13585 , year =

    MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention , author =. arXiv preprint arXiv:2506.13585 , year =

    Pith/arXiv arXiv

  35. [35]

    arXiv preprint arXiv:2510.19338 , year =

    Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning , author =. arXiv preprint arXiv:2510.19338 , year =

    arXiv

  36. [36]

    Advances in Neural Information Processing Systems , year =

    Structured Denoising Diffusion Models in Discrete State-Spaces , author =. Advances in Neural Information Processing Systems , year =

  37. [37]

    Advances in Neural Information Processing Systems , year =

    Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions , author =. Advances in Neural Information Processing Systems , year =

  38. [38]

    Advances in Neural Information Processing Systems , year =

    Simple and Effective Masked Diffusion Language Models , author =. Advances in Neural Information Processing Systems , year =

  39. [39]

    Advances in Neural Information Processing Systems , year =

    Large Language Diffusion Models , author =. Advances in Neural Information Processing Systems , year =

  40. [40]

    International Conference on Learning Representations , year =

    Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models , author =. International Conference on Learning Representations , year =

  41. [41]

    Technical report , year =

    Kimi Linear: An Expressive, Efficient Attention Architecture , author =. Technical report , year =

  42. [42]

    OpenReview , year =

    Jamba: A Hybrid Transformer-Mamba Language Model , author =. OpenReview , year =

  43. [43]

    2025 , howpublished =

    Qwen3-Next: Towards Ultimate Training Efficiency , author =. 2025 , howpublished =

    2025