Pith. sign in

REVIEW 3 major objections 5 minor 23 references

This paper claims that a recurrent state can serve as an attention memory, giving a Mamba-style model attention-style recall while shrinking the inference cache to a fraction of a matched Transformer's.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 16:16 UTC pith:LSPJRVMF

load-bearing objection DART's central trick is real and the recall gains at short range are solid, but the headline retrieval claim doesn't survive 4K-token needles. the 3 major comments →

arxiv 2608.02032 v1 pith:LSPJRVMF submitted 2026-08-03 cs.LG

DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling

Yixiao Qian , Song Chen , Pengkai Wang , Jiaxu Liu , Shengze Cai , Chao Xu This is my paper
classification cs.LG
keywords DARTstate space dualityMamba-2chunked scanassociative recallKV cache compressionstate-memory attentionlong-context modeling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether the same memory representation can support both recurrent compression and attention-style retrieval. It answers yes: in the state-space-duality view, Mamba-2's recurrent state is a compressed key-value cache that already supports value readout; DART adds a token-conditioned key readout from the chunk state contributions produced by the chunked scan, then runs attention over the decoded chunk-level key-value pairs. This state-memory attention is implemented as a FlashAttention-style kernel on top of the existing chunked scan. The claimed payoff is that DART recovers much of attention's associative-recall ability while keeping a length-dependent inference cache that, at chunk size 256 and state size 128, is 25% of a matched token-level attention baseline. The paper reports improved MQAR and retrieval scores over Mamba-2 at matched scale, with Pile perplexity and zero-shot downstream accuracy essentially preserved.

Core claim

On the state-space-duality view, Mamba-2 reads values from its compressed state via C_t H_t but never decodes keys from it. DART retains each chunk's contribution ΔH[c] as a chunk state memory, and decodes both a value C_t ΔH[c] and a token-conditioned key ΔH[c] E_t from that same tensor. Attention then runs over these chunk-level key-value pairs, and the retrieved readout is added to the SSM output through a zero-initialized scalar gate. Under matched settings, the paper shows this substantially improves associative recall and real-world extraction relative to Mamba-2 while cutting the length-dependent inference cache by 75% relative to a matched attention baseline when chunk size is 256 an

What carries the argument

Chunk state contribution ΔH[c] ∈ R^{N×P}: the matrix of contributions that chunk c makes to the chunk-boundary recurrent state. It has a key axis N and a value axis P. Reading from the left with C_t yields a value; contracting the value axis with E_t yields a token-conditioned key. State-memory attention (SMA) attends over these decoded pairs with query Q_t, and the retrieved readout is gated into the SSM output.

Load-bearing premise

The chunk state contribution ΔH[c] preserves enough token-level detail that attention-style routing over chunk-level keys and values can recover fine-grained cross-chunk associations; if the recurrent compression already discarded that detail, SMA cannot restore it.

What would settle it

Run DART's retrieval benchmarks with the chunk state memories ΔH[c] replaced at inference time by random tensors of the same shape or by a constant such as the mean of all chunk memories. If SMA accuracy stays substantially above the Mamba-2-only ablation, the branch is not actually reading its stored memories; if accuracy collapses to that ablation, the mechanism is load-bearing. Alternatively, train a DART variant with the key-decoding projection E_t fixed to a random projection; near-chance recall would show token-conditioned keys are doing the work.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • At chunk size 256 and state size 128, DART's length-dependent inference cache is 25% of a matched token-level attention baseline, a 75% saving.
  • DART improves multi-query associative recall over Mamba-2 at matched width and state size; with state size 64 it reaches near-perfect MQAR accuracy across tested sequence lengths.
  • DART substantially improves SWDE and FDA extraction over Mamba-2 at 130M, 370M, and 780M scales, while Pile perplexity and most zero-shot accuracy metrics stay close to the recurrent baseline.
  • The SMA branch contributes most of the retrieval gain: removing it at evaluation time drops MQAR from near-perfect to near zero and sharply reduces real-world extraction accuracy.
  • Sharing the native Mamba-2 read vector C_t with the SMA value readout is necessary for the branch to learn at small state sizes; an independent read vector produces zero recall in the tested MQAR setting.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If chunk-state compression is the right granularity, the same decoded-attention idea could be applied to other linear-attention or delta-rule recurrent models, not just Mamba-2's chunked scan, potentially giving them attention-style retrieval without token-level caches.
  • The cache saving grows with chunk size but retrieval quality may fall; a natural extension would be to make chunk size or memory resolution adaptive to content, e.g., splitting chunks that receive high attention weight.
  • The NIAH-Single-3 results at 4096 tokens suggest SMA's token-conditioned keys lose the fine-grained identity needed to route to sparse, long-range needles; a testable fix is to augment the key readout with a residual token-level projection for a small set of most-recent chunks.
  • Because SMA reuses the chunked scan's ΔH, the mechanism could be combined with cache-quantization or state-pruning methods to push effective context length further at fixed memory.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes DART, a hybrid layer that augments Mamba-2 with state-memory attention (SMA) over chunk state contributions produced by the Mamba-2 chunked scan. For each historical chunk, DART decodes a token-conditioned key and value from the chunk-state matrix ΔH[c], performs softmax attention over the resulting chunk-level KV pairs, and adds the retrieved readout to the native SSM output through a gated residual. The authors derive an inference-cache ratio of N/(2S) relative to a token-level attention cache, report large-scale pretraining on the Pile (130M–780M scale), and evaluate on MQAR, NIAH, real-world retrieval, and zero-shot downstream tasks. They claim that DART substantially improves associative recall and retrieval over Mamba-2 while preserving language-modeling quality, and substantially reduces length-dependent cache memory versus matched attention.

Significance. The core idea is original and conceptually clean: instead of interleaving separate attention and recurrent layers, DART performs attention-style retrieval over the same compressed state representation that the recurrent branch maintains. If the empirical claims hold, this is a useful step toward unifying recurrent compression and content-based retrieval in a single block. The cache-ratio derivation in Eq. (14) is a genuine analytic contribution, and the FlashAttention-style implementation makes the approach plausible to train. The paper also ships a fairly extensive evaluation suite, including MQAR, 100B-token Pile pretraining at three scales, real-world retrieval tasks, NIAH, ablations, and kernel-level efficiency measurements. The main scientific risk is the strength of the central retrieval claim, which is partially contradicted by the hardest long-context NIAH numbers and is not yet supported by resource-matched comparisons.

major comments (3)
  1. [§5.1, Figure 4] The MQAR comparison to Mamba-2 is not matched in parameters or compute. DART adds the Q_t and E_t projections, the SMA gate, and the O(L^2 N P / S) quadratic SMA computation on top of the same backbone. Under 'matched model width and state size', DART therefore has strictly more capacity and more FLOPs than Mamba-2. The claim that 'DART consistently improves MQAR recall over Mamba-2' is not yet evidence that the SMA mechanism per se is responsible; the improvement could partly reflect the extra parameters/computation. Please add a resource-matched baseline (e.g., a Mamba-2 with a comparable parameter increase or FLOP budget) or otherwise quantify how much of the gain is attributable to the SMA branch rather than to the added capacity.
  2. [§5.2, Table 3] The abstract's unqualified claim that DART 'substantially improves ... retrieval' is not supported by the hardest long-context results. At 4096 tokens with the main S=256 setting, DART 795M scores 1.4 on NIAH-Single-2 and 2.6 on NIAH-Single-3, versus Mamba-2's 1.8 and 3.6; DART is not better, and the differences are within sampling noise of 500-example NIAH evaluations. Moreover, all models are pretrained with context length 2048 (Appendix D.2), so the 4096 evaluations are length extrapolation; the paper does not state this and does not discuss its implications. Please either retrain/evaluate with a 4096-token training context, report confidence intervals, and/or explicitly qualify the retrieval claim to the settings where DART is actually shown to improve.
  3. [§3, Eq. (8)] The SMA value and key readouts use the raw chunk contribution ΔH[c] without applying the inter-chunk decay A_{e(c)+1:t} that the SSM branch would apply when propagating this contribution to the current token. Thus V_{t,c} = C_t ΔH[c] is not the actual contribution of chunk c to the current state H_t; it is the contribution at the chunk boundary. The paper neither states this design choice nor explains why omitting decay is appropriate for retrieval. This is load-bearing for the interpretation that DART 'decodes' keys and values from the recurrent state: if the decoded values ignore the recurrent dynamics, the mechanism is better described as attention over undecayed chunk memories. Please clarify and, if possible, ablate the inclusion of the decay factor.
minor comments (5)
  1. [§4, Eq. (14)] The 'matched attention baseline' is only a sequence-module cache comparison, not a full-model comparison. The paper should state more explicitly that the 75% cache saving applies to the length-dependent portion of the cache of the SMA branch versus the KV cache of a token-level attention layer, not to the total end-to-end memory of DART versus a Transformer.
  2. [§5.2, Table 3] NIAH results are reported without any uncertainty. Since NIAH uses only 500 examples per context length, a difference of 1–2 percentage points (e.g., 1.4 vs 1.8) is not meaningful. Please include standard errors or additional seeds for the key retrieval numbers.
  3. [§5.3, Figure 5] The forward-time comparison is kernel-level: SMA versus FlashAttention-2 and SSD kernels. DART must also run the SSD branch to produce ΔH, so the plot does not directly show end-to-end training or inference speed of the full DART model. Please clarify this limitation in the caption or text.
  4. [§5.1] The Transformer++ baseline is described as 'same sequence length and model width' but not matched in parameters or training compute. A sentence noting this would help readers interpret Figure 4.
  5. [§3, Eq. (6)] The terms 'value-side readout' and 'key-side readout' are used before the full KV construction is introduced. A short intuitive explanation of why C_t reads a value and E_t composes a key would improve readability.

Circularity Check

0 steps flagged

No significant circularity: DART's efficiency analysis is arithmetic from architecture definitions and its retrieval improvements are external, trained evaluations.

full rationale

DART's derivation chain is self-contained. The core efficiency claim (75% cache reduction at S=256, N=128) is obtained by direct algebra from the defined memory shapes: Eq. (14) computes MNP versus 2LP per head, giving N/(2S); this is a definitional account of the architecture, not a fitted or self-referential prediction. The SMA equations (5)-(10) define keys and values by decoding chunk state contributions; the claim that this improves retrieval is an empirical statement tested on MQAR, NIAH, SWDE/FDA, and Pile against Mamba-2 and Pythia baselines, not a quantity derived from the fitted inputs themselves. The SSD/chunked-scan apparatus is cited to Dao & Gu (2024) and Gu & Dao (2023), who are not authors of this paper, and the retrieval-suite citation to Lahoti et al. (2026) is an external benchmark choice. No self-citation is load-bearing. The paper's own flagged limitation at 4096-token NIAH-Single-2/3 (Table 3) is a capacity concern about chunk-level compression, not a circular derivation. No step reduces to its own input by construction, and no fitted parameter is relabeled as a prediction.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

The central design relies on the SSD view of Mamba-2 and on the correctness of its chunked scan, both standard results from Dao and Gu (2024). The key domain assumption is that chunk state contributions suffice as an addressable memory, which is empirically supported but not proven. No fitted constants or invented entities are introduced beyond the learned weights of the SMA branch.

axioms (3)
  • standard math SSD representation of Mamba-2 as structured causal attention (Eq. 4), interpreting the state as a compressed KV cache.
    Borrowed from Dao and Gu (2024); used to justify the design of SMA.
  • domain assumption The Mamba-2 chunked scan correctly decomposes outputs into intra- and inter-chunk terms and yields the chunk state contributions ΔH[c].
    Relies on the correctness of the Mamba-2 scan algorithm as specified in Dao and Gu (2024).
  • ad hoc to paper Attention over a fixed set of chunk state memories is a sufficient retrieval mechanism for the tasks studied.
    Core design choice of the paper; empirically supported by experiments but not proven, and weak at 4096-token NIAH settings.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling." pith.science (2026). https://pith.science/paper/LSPJRVMF

@misc{pith2026260802032,
  author       = {Pith},
  title        = {Pith review of: DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LSPJRVMF}},
  note         = {Machine review of arXiv:2608.02032}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Modern language models are built primarily from Transformers, recurrent models, and their hybrid architectures. Transformers rely on token-level attention memories, while recurrent models such as state space models (SSMs) and linear attention maintain compact recurrent states. These architectures are typically instantiated separately or interleaved at the layer level, leaving open whether a shared memory representation can support both recurrent compression and attention-style retrieval. We study this question through the state space duality (SSD) view of Mamba-2, where the SSM state can be interpreted as a compressed associative key--value (KV) cache. We observe that Mamba-2 decodes token-conditioned values from this state but does not decode token-conditioned keys. Based on this observation, we propose DART (Decoded Attention over Recurrent sTates), which retains the chunk state contributions produced by the Mamba-2 chunked scan as chunk state memories, decodes token-conditioned keys and values from these memories, and performs state-memory attention (SMA) over the resulting KV pairs. The retrieved output is then combined with the native Mamba-2 output through a gated residual connection. DART supports practical training by reusing the Mamba-2 chunked scan and implementing SMA as a FlashAttention-style computation. Our analysis and experiments show that DART substantially reduces the length-dependent inference cache compared with a matched attention baseline (e.g., $75\%$ savings when the chunk size is $S=256$ and the state size is $N=128$). Compared with Mamba-2, DART substantially improves associative recall and retrieval while preserving general language-modeling quality.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 17 linked inside Pith

  1. [1]

    Attamba: Attending to multi-token states.arXiv preprint arXiv:2411.17685,

    Yash Akhauri, Safeen Huda, and Mohamed S Abdelfattah. Attamba: Attending to multi-token states.arXiv preprint arXiv:2411.17685,

    Pith/arXiv arXiv

  2. [4]

    Think you have solved question answering? try ARC, the AI2 reasoning chal- lenge.arXiv preprint arXiv:1803.05457,

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning chal- lenge.arXiv preprint arXiv:1803.05457,

    Pith/arXiv arXiv

  3. [6]

    Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060,

    Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060,

    Pith/arXiv arXiv

  4. [9]

    Zamba: A compact 7B SSM hybrid model.arXiv preprint arXiv:2405.16712,

    Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. Zamba: A compact 7B SSM hybrid model.arXiv preprint arXiv:2405.16712,

    Pith/arXiv arXiv

  5. [10]

    Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752,

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752,

    Pith/arXiv arXiv

  6. [14]

    Repeat after me: Trans- formers are better than state space models at copying.arXiv preprint arXiv:2402.01032,

    Samy Jelassi, David Brandfonbrener, Sham M Kakade, and Eran Malach. Repeat after me: Trans- formers are better than state space models at copying.arXiv preprint arXiv:2402.01032,

    Pith/arXiv arXiv

  7. [16]

    Jamba: A hybrid transformer- Mamba language model.arXiv preprint arXiv:2403.19887,

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer- Mamba language model.arXiv preprint arXiv:2403.19887,

    Pith/arXiv arXiv

  8. [17]

    DeepSeek-V2: A strong, economical, and efficient mixture- of-experts language model.arXiv preprint arXiv:2405.04434, 2024a

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. DeepSeek-V2: A strong, economical, and efficient mixture- of-experts language model.arXiv preprint arXiv:2405.04434, 2024a. Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. ...

    Pith/arXiv arXiv 2018

  9. [19]

    Samba: Simple hybrid state space models for efficient unlimited context language modeling

    Liliang Ren, Yang Liu, Yadong Lu, Chen Liang, Weizhu Chen, et al. Samba: Simple hybrid state space models for efficient unlimited context language modeling. InInternational Conference on Learning Representations, volume 2025, pp. 53551–53575,

    2025

  10. [21]

    Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621,

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621,

    Pith/arXiv arXiv

  11. [22]

    LLaMA: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971,

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971,

    Pith/arXiv arXiv

  12. [1990]

    The Pile: An 800GB dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027,

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027,

    Pith/arXiv arXiv

  13. [1997]

    RULER: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654,

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Jia Fei, Zhang Yang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654,

    Pith/arXiv arXiv

  14. [2011]

    Gated DeltaNet-2: Decoupling erase and write in linear attention.arXiv preprint arXiv:2605.22791,

    Ali Hatamizadeh, Yejin Choi, and Jan Kautz. Gated DeltaNet-2: Decoupling erase and write in linear attention.arXiv preprint arXiv:2605.22791,

    Pith/arXiv arXiv

  15. [2016]

    RWKV: Reinventing RNNs for the transformer era

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, et al. RWKV: Reinventing RNNs for the transformer era. InFindings of the association for computational linguistics: EMNLP 2023, pp. 14048–14077,

    2023

  16. [2017]

    An empirical study of Mamba- based language models.arXiv preprint arXiv:2406.07887,

    Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, et al. An empirical study of Mamba- based language models.arXiv preprint arXiv:2406.07887,

    Pith/arXiv arXiv

  17. [2018]

    FlashAttention-2: Faster attention with better parallelism and work partitioning

    11 Preprint Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. InInter- national Conference on Learning Representations, volume 2024, pp. 35549–35562,

    2024

  18. [2019]

    Li, Berlin Chen, Caitlin Wang, Aviv Bick, J

    12 Preprint Aakash Lahoti, Kevin Y . Li, Berlin Chen, Caitlin Wang, Aviv Bick, J. Zico Kolter, Tri Dao, and Albert Gu. Mamba-3: Improved sequence modeling using state space principles.arXiv preprint arXiv:2603.15569,

    arXiv

  19. [2020]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    1901

  20. [2021]

    Simplified state space layers for sequence modeling.arXiv preprint arXiv:2208.04933,

    Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling.arXiv preprint arXiv:2208.04933,

    Pith/arXiv arXiv

  21. [2022]

    DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pape...

    2019

  22. [2023]

    Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396,

    Albert Gu, Karan Goel, and Christopher R ´e. Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396,

    Pith/arXiv arXiv

  23. [2024]

    Zoology: Measuring and improving recall in efficient language mod- els

    Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Y Zou, Atri Rudra, and Christopher R´e. Zoology: Measuring and improving recall in efficient language mod- els. InInternational conference on learning representations, volume 2024, pp. 15664–15730, 2024a. Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alber...

    Pith/arXiv arXiv 2024

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.