Pith. sign in

REVIEW 2 major objections 7 minor 39 references

Linearized 2-Simplicial Attention

T0 review · 2 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A one-mode kernelization makes 2-simplicial attention linear in sequence length, and a softmax-free KDA+LinSimp model improves 16k mean accuracy from 0.3809 to 0.3888.

desk verdict Correct kernelization and a testable architecture, but the headline accuracy claim rests on an analytic iso-FLOP budget that favors the proposed model and a single seed. read the letter →

arxiv 2608.09307 v1 pith:CGCE4C4I submitted 2026-08-10 cs.AI

Aritra Das , Dhruman Gupta , Debayan Gupta This is my paper · ORCID
classification cs.AI MSC 68T07
keywords 2-simplicialattentionlinearpositiverandomfeatureskernelapproximationlong-contextlanguagemodelingdeltaruleTritonkernelsno-softmaxmodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to make 2-simplicial attention—where a query attends to pairs of earlier tokens through a trilinear score—linear in sequence length without giving up the global reach that window-only versions lack. It rewrites the trilinear score as an inner product between a composite query and a key, so the sum over one token axis becomes an ordinary exponential kernel, which is approximated with positive random features and stored in a fixed-size recurrent state. The second token axis is kept explicit over a short rolling window of recent anchors, so each recent anchor sends a different question to the same global state. If correct, this gives a causal layer with $O(T)$ total work and a fixed inference-state size independent of $T$, and the paper's 16k-context experiments report that a no-softmax model composed with Kimi $\Delta$ Attention improves mean downstream accuracy from 0.3809 to 0.3888 while lowering LAMBADA perplexity from 715.6 to 602.6 under an analytic iso-FLOP comparison.

What carries the argument

The load-bearing object is a one-mode kernelization: the trilinear score $\langle \tau\hat q_i,\hat k_j,\hat r_c\rangle$ is rewritten as $(\tau\hat q_i\odot \hat r_c)^\top \hat k_j$. For a fixed query and anchor this is the exponential kernel $\exp(z_{ic}^\top \hat k_j)$ in the past-token index $j$, with composite query $z_{ic}=(\tau\hat q_i)\odot \hat r_c$. Positive random features $\phi(x)=m^{-1/2}\exp(\Omega x-\|x\|_2^2/2\,\mathbf{1})$ approximate that kernel; the sum over $j$ is absorbed into two recurrent prefix states, $M_i=\sum_{j\le i}\phi(\hat k_j)v_j^\top$ and $a_i=\sum_{j\le i}\phi(\hat k_j)$, while the anchor $c$ runs only over the most recent $w$ tokens. The work of these pieces is to convert the exact joint softmax over causal pairs $(j,c)$ into a linear-time, fixed-state layer that still lets every recent anchor send a different composite query to the whole prefix.

What would settle it

Give both 16k architectures the same token budget, say 2.70B tokens each, and measure the seven-task mean accuracy; if KDA+LinSimp no longer beats the KDA hybrid, the reported gain under the iso-FLOP rule is an artifact of the analytic token allocation. A cross-check is to compare the two models under matched wall-clock time or matched energy.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a one-mode kernelization of 2-simplicial attention. The trilinear score $\langle \tau\hat q_i, \hat k_j, \hat r_c\rangle$ is re-expressed as $(\tau\hat q_i \odot \hat r_c)^\top \hat k_j$, so that for a fixed query $i$ and anchor $c$, the sum over the past token $j$ is an ordinary exponential dot-product kernel. Approximating that kernel with positive random features $\phi(x)=m^{-1/2}\exp(\Omega x - \|x\|_2^2/2\,\mathbf{1})$ turns the causal sum over $j$ into two fixed-size prefix states $M_i=\sum_{j\le i}\phi(\hat k_j)v_j^\top$ and $a_i=\sum_{j\le i}\phi(\hat k_j)$, while the anchor index $c$ stays explicit only over the most recent $w=64$ tokens. The result is Theorem 1: the layer is causal, runs in total $O(T)$ work for fixed feature rank $m$ and anchor width $w$, and uses $O(mD_v+m+w(D+D_v))$ state per head independent of $T$. Composed with Kimi $\Delta$ Attention in 18 layers plus six LinSimp layers and no softmax, the model achieves the highest mean downstream accuracy among the compared architectures and, at 16k context, improves mean accuracy from 0.3809 to 0.3888 while reducing LAMBADA perplexity from 715.6 to 602.6 under an analytic iso-FLOP budget.

Load-bearing premise

The load-bearing premise is that the analytic FLOP counts in Equation (27) and Appendix D correctly represent the resource that limits training; if wall-clock time or energy is the real constraint, the reported 16k advantage could come from the larger token budget assigned to the no-softmax model (3.14B vs 2.70B) rather than from the new layer.

Editorial extensions

If this is right

  • A causal layer can include triple-token 2-simplicial interactions with total work $O(T)$, linear in sequence length for fixed feature rank and anchor width.
  • The full causal prefix remains reachable through the global state even for tokens arbitrarily far back, while the short anchor window supplies local conditioning.
  • The exact windowed 2-simplicial baseline becomes comparatively costly under a compute-matched budget because its FLOP count shrinks its assigned token budget, making the linearized layer the practical route to keeping the trilinear operator.
  • Autoregressive inference needs a fixed-size state per head that does not grow with $T$, the property that matters for long-context decoding.
  • Softmax-free models composed of delta-rule layers and LinSimp layers can improve long-context downstream accuracy and LAMBADA perplexity over a KDA hybrid under the paper's analytic iso-FLOP rule.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same one-mode rewriting could linearize the other algebraic orientation of the trilinear score or higher-order simplicial variants; the paper notes that computing both orientations doubles the contraction cost, so the choice is a capacity-cost trade-off.
  • The approximation quality depends on feature rank $m$ and anchor width $w$, and the appendix shows the kernel error falling roughly as $m^{-1/2}$ in isolation, suggesting a tunable trade-off between memory and fidelity for tasks that need exact recall.
  • The gap between the analytic iso-FLOP verdict and the measured wall-clock throughput (the KDA hybrid is faster at 16k) means the 'matched compute' conclusion is not invariant under the choice of resource metric; an energy- or latency-equated comparison would be a natural stress test.
  • The paper's mention of an $O(N)$ explicit-scan variant points to a continuum between the fixed-state compression and standard attention, which could be explored for memory-intensive tasks such as associative recall.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. The paper proposes LinSimp, a linearized form of 2-simplicial attention. The trilinear score ⟨q_i, k_j, r_c⟩ is rewritten as (τ q̂_i ⊙ r̂_c)ᵀ k̂_j, so that the sum over the token axis j becomes a softmax-like kernel sum, which is then approximated with positive random features and stored in a fixed-size recurrent state (M_i and a_i). The anchor axis c is kept explicit over a short window of w recent tokens. The authors prove (Theorem 1) that the layer is causal and has O(T) total work for fixed feature rank m and window width w, describe a custom Triton forward/backward implementation, and combine LinSimp with Kimi Delta Attention (KDA) in a 24-layer model with no softmax attention. Experiments on FineWeb-Edu and FineMath-4+ compare several architectures under equal-token and analytic iso-FLOP budgets. The headline result is that, at 16k context under an analytic iso-FLOP budget, the no-softmax KDA+LinSimp model improves mean downstream accuracy from 0.3809 to 0.3888 and lowers LAMBADA perplexity from 715.6 to 602.6 compared with a KDA hybrid.

Significance. The theoretical contribution is sound and potentially useful: the one-mode kernelization of the trilinear score is a clean idea that extends linear-attention techniques to higher-order interactions while preserving a global reach on one axis. The proof of linear complexity is straightforward and correct, and the paper includes a working Triton implementation with numerical tests against an fp32 reference. The random-feature estimator is unbiased for the kernel, and the finite-rank bias of the normalized ratio is explicitly acknowledged. However, the empirical validation is not yet at the level needed to support the headline claims: all experiments use a single seed with no variance estimates, and the 'matched compute' budget rests on an analytic FLOP model that the paper itself concedes does not measure wall-clock, energy, or system cost. The reported wall-clock speed at 16k actually favors the KDA hybrid (24.5k vs 20.2k tokens/s), undercutting the iso-FLOP assumption. If the theoretical and implementation contributions are the main value, the paper could be acceptable once the empirical claims are tempered or better supported.

major comments (2)
  1. [Section 6.3, Table 5, Eq. (27), Appendix D] The iso-FLOP rule in Eq. (27) assigns 3.14B tokens to the no-softmax KDA+LinSimp model and 2.70B to the KDA hybrid at 16k context, a 16% token advantage, because the analytic FLOP model in Appendix D counts the six softmax layers as roughly 16% more expensive per token than the LinSimp layers. However, Section 6.4 reports that the KDA hybrid is faster in wall-clock at 16k (24.5k vs 20.2k tokens/s), and Section 7 states that the FLOP counts do not measure energy, latency, or total system cost. Thus the reported +0.0079 mean accuracy gain and the LAMBADA improvement from 715.6 to 602.6 may simply reflect the larger token budget granted by the analytic rule rather than an architectural advantage of LinSimp. Since this is the central empirical claim, please provide a wall-clock-matched or energy-matched comparison, or at least a sensitivity analysis over the FLOP constants (e.g., C_KDA and C_LinSimp) showing that the conclusion is robust to plausible variations in the accounting.
  2. [Section 7 and Table 6] The paper explicitly states that experiments use a single seed and that no confidence intervals can be attached. Given that Table 6 shows per-task differences of only −0.001 to +0.020, the mean accuracy increase of 0.0079 is not statistically substantiated, and the abstract's claim of improvement 'under matched compute' is stronger than the evidence. Please provide at least three seeds or bootstrap confidence intervals, and modify the language to reflect the preliminary nature of the results.
minor comments (7)
  1. [Section 3.2] The passage beginning 'In our current formulation, computing Ni and Zi has O(1) cost...' is confusing because Ni and Zi are not defined until Section 3.3, and the discussion of increasing the cost to O(N) is not used in the proposed method; please remove or rewrite this passage to avoid ambiguity about the complexity guarantees.
  2. [Section 6.1] The sentence 'replacing three of the six attention slots with LinSimp praises mean accuracy by 0.0080' uses 'praises' where 'raises' is intended; also, 'LinSimp' should be 'LinSimp' for consistency.
  3. [Section 7] The word 'cuurrently' should be 'currently'.
  4. [Section 1] The sentence 'The 2 Simplicial Attention is expensive, a query attends to a two-dimensional plane of token pairs lead to O(n3) computation' is grammatically incorrect; please revise and use O(n^3) with proper superscript notation.
  5. [Section 6.3] The sentence 'Section 6.4 reports initial kernel measurements...' appears within Section 6.3 and refers to the following subsection; consider rephrasing to 'The following subsection reports...' or restructuring the text.
  6. [Table 1 and general formatting] The Description column in Table 1 has inconsistent punctuation and capitalization; please use a consistent style (e.g., end every entry with a period or none).
  7. [Appendix D, Eq. (41)] The expression for C_LinSimp includes the term B_C D_v without explanation; a brief comment on why the chunk size B_C appears in the per-token FLOP estimate would help the reader interpret the iso-FLOP results.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the kernel derivation, Theorem 1, and the architecture comparison are self-contained; the iso-FLOP budget rule is an acknowledged analytical modeling choice, not a circular reduction.

full rationale

The paper's derivation does not assume its conclusion. Lemma 1 proves the unbiasedness of the positive random-feature estimator using the Gaussian moment-generating function; Equation (9) is the standard exponential-kernel identity and is not equivalent to the layer output. The prefix states in Equations (10)-(11) are direct definitions, and the estimator in Equations (14)-(16) is a finite-rank approximation whose bias at finite m is explicitly acknowledged. Theorem 1's causality and O(T) claim follow by inspection of the state updates and the bounded anchor window, not by circular reference. The empirical comparison uses an analytic iso-FLOP rule (Equation 27) with counts from Appendix D; this is a resource-accounting convention, and the paper explicitly states it does not measure energy, latency, or total system cost. The wall-clock result even favors the hybrid, so the headline accuracy gain may be influenced by token-budget differences, but that is an external-validity limitation, not a circularity: no fitted parameter is renamed as a prediction and no load-bearing claim reduces to its own input. There are no self-citations by the authors. The only unusual passage is the self-referential opening of Section 6.4 ('Section 6.4 reports initial kernel measurements...'), which is stylistic and not load-bearing.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The construction rests on the standard unbiased random-feature estimate (Lemma 1), L2-normalized streams, a learned per-head temperature and gate, and the choice of feature rank m and anchor width w. The empirical comparison depends on an analytic FLOP model whose constants are chosen by hand and not validated against wall-clock time; this model changes the token budget assigned to each architecture.

free parameters (5)
  • Feature rank m = 128 for main runs, 64 for web runs
    Chosen by hand; affects approximation quality and compute. Larger m reduces random-feature bias but increases cost.
  • Anchor window width w = 64
    Chosen by hand; controls how many recent anchors condition the global state. Trade-off between expressiveness and compute.
  • Per-head temperature tau = Learned, initialized to sqrt(D)
    Learned scaling on the query; a trainable parameter that modulates the trilinear score.
  • Per-head gate gamma = Learned, initialized to -2
    Learned scalar gate on the layer output; trained with a 5x learning-rate multiplier.
  • KDA FLOP constant C_KDA = Not numerically specified
    In Appendix D, Equation (40) summarizes the KDA layer cost with a constant 'from the tested state-update implementation'. This hand-chosen constant affects the iso-FLOP token budget allocation and the reported comparison.
assumptions (5)
  • standard math Lemma 1: unbiased Gaussian random-feature kernel estimate
    Uses the Gaussian moment-generating function, a standard result from Rahimi and Recht and Choromanski et al. It is proven in the paper and is not contested.
  • domain assumption The finite-rank random-feature approximation preserves the joint normalization well enough at m=128
    The paper substitutes the exponential kernel with m-dimensional features and then forms a ratio of approximate sums. The bias of the ratio is acknowledged but not quantified for m=128; the architecture's efficacy depends on this approximation being adequate.
  • domain assumption The analytic iso-FLOP count in Appendix D is a fair proxy for training compute
    Equation (27) allocates token budgets based on hand-derived FLOP expressions. These counts are not validated against wall-clock time and, as the paper notes, do not measure energy or latency. The central 16k comparison depends on this premise.
  • ad hoc to paper An anchor window of 64 recent tokens preserves the conditioning value of the c-axis
    The paper restricts the c-axis to the most recent w tokens without a theoretical justification for w=64. The usefulness of the layer relies on this window capturing the interactions that matter.
  • domain assumption L2-normalization of q, k, r before the feature map is a benign choice
    Normalization bounds the log-feature values and follows prior 2-simplicial work, but it changes the score geometry from the original unnormalized formulation and is not derived from first principles.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Linearized 2-Simplicial Attention." pith.science (2026). https://pith.science/paper/CGCE4C4I

@misc{pith2026260809307,
  author       = {Pith},
  title        = {Pith review of: Linearized 2-Simplicial Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CGCE4C4I}},
  note         = {Machine review of arXiv:2608.09307}
}
read the original abstract

We present a linearized form of 2-simplicial attention by rewriting the trilinear score as an inner product between a composite query and a key, so that the sum over one token axis takes the same form as ordinary softmax attention. We then approximate this sum with positive random features and store the entire past in a fixed-size state, while the second axis stays explicit over a short window of recent tokens. This enables us to achieve linear cost in sequence length combined with a global reach that windowed 2-simplicial attention lacks. We implement it with custom Triton kernels and combine it with Kimi Delta Attention to build a model with no softmax attention at all. Under matched compute, this model achieves the highest mean downstream accuracy among the compared architectures, and at 16k context it improves mean accuracy over a KDA hybrid while lowering LAMBADA perplexity from 715.6 to 602.6.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 14 canonical work pages

  1. [1]

    Advances in Neural Information Processing Systems , volume =

    Attention Is All You Need , author =. Advances in Neural Information Processing Systems , volume =

  2. [2]

    International Conference on Learning Representations , year =

    Logic and the 2-Simplicial Transformer , author =. International Conference on Learning Representations , year =

    work page

  3. [3]

    arXiv preprint arXiv:2507.02754 , year =

    Fast and Simplex: 2-Simplicial Attention in Triton , author =. arXiv preprint arXiv:2507.02754 , year =

    arXiv

  4. [4]

    Advances in Neural Information Processing Systems , volume =

    Systematic Generalization with Edge Transformers , author =. Advances in Neural Information Processing Systems , volume =

    work page

  5. [5]

    arXiv preprint arXiv:2306.02896 , year =

    Representational Strengths and Limitations of Transformers , author =. arXiv preprint arXiv:2306.02896 , year =

    arXiv

  6. [6]

    arXiv preprint arXiv:2501.19215 , year =

    Strassen Attention: Unlocking Compositional Abilities in Transformers Based on a New Lower Bound Method , author =. arXiv preprint arXiv:2501.19215 , year =

    arXiv

  7. [7]

    Advances in Neural Information Processing Systems , volume =

    Random Features for Large-Scale Kernel Machines , author =. Advances in Neural Information Processing Systems , volume =

  8. [8]

    Advances in Neural Information Processing Systems , volume =

    Orthogonal Random Features , author =. Advances in Neural Information Processing Systems , volume =

    work page

Show all 39 references
  1. [9]

    International Conference on Learning Representations , year =

    Rethinking Attention with Performers , author =. International Conference on Learning Representations , year =

  2. [10]

    Proceedings of the 37th International Conference on Machine Learning , pages =

    Transformers Are RNNs: Fast Autoregressive Transformers with Linear Attention , author =. Proceedings of the 37th International Conference on Machine Learning , pages =

  3. [11]

    Proceedings of the 41st International Conference on Machine Learning , year =

    Gated Linear Attention Transformers with Hardware-Efficient Training , author =. Proceedings of the 41st International Conference on Machine Learning , year =

  4. [12]

    Advances in Neural Information Processing Systems , year =

    Parallelizing Linear Transformers with the Delta Rule over Sequence Length , author =. Advances in Neural Information Processing Systems , year =

  5. [13]

    arXiv preprint arXiv:2412.06464 , year =

    Gated Delta Networks: Improving Mamba2 with Delta Rule , author =. arXiv preprint arXiv:2412.06464 , year =

    arXiv

  6. [14]

    arXiv preprint arXiv:2510.26692 , year =

    Kimi Linear: An Expressive, Efficient Attention Architecture , author =. arXiv preprint arXiv:2510.26692 , year =

    arXiv

  7. [15]

    arXiv preprint arXiv:2312.00752 , year =

    Mamba: Linear-Time Sequence Modeling with Selective State Spaces , author =. arXiv preprint arXiv:2312.00752 , year =

    arXiv

  8. [16]

    arXiv preprint arXiv:2307.08621 , year =

    Retentive Network: A Successor to Transformer for Large Language Models , author =. arXiv preprint arXiv:2307.08621 , year =

    arXiv

  9. [17]

    Proceedings of the 40th International Conference on Machine Learning , year =

    Hyena Hierarchy: Towards Larger Convolutional Language Models , author =. Proceedings of the 40th International Conference on Machine Learning , year =

  10. [18]

    Advances in Neural Information Processing Systems , volume =

    FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness , author =. Advances in Neural Information Processing Systems , volume =

  11. [19]

    International Conference on Learning Representations , year =

    FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning , author =. International Conference on Learning Representations , year =

  12. [20]

    Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages , pages =

    Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations , author =. Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages , pages =

  13. [21]

    arXiv preprint arXiv:1805.02867 , year =

    Online Normalizer Calculation for Softmax , author =. arXiv preprint arXiv:1805.02867 , year =

    arXiv

  14. [22]

    Yang, Songlin and Zhang, Yu , year =

  15. [23]

    Advances in Neural Information Processing Systems , volume =

    Root Mean Square Layer Normalization , author =. Advances in Neural Information Processing Systems , volume =

  16. [24]

    Shazeer, Noam , journal =

  17. [25]

    arXiv preprint arXiv:2104.09864 , year =

    RoFormer: Enhanced Transformer with Rotary Position Embedding , author =. arXiv preprint arXiv:2104.09864 , year =

    arXiv

  18. [26]

    arXiv preprint arXiv:2307.09288 , year =

    Llama 2: Open Foundation and Fine-Tuned Chat Models , author =. arXiv preprint arXiv:2307.09288 , year =

    arXiv

  19. [27]

    arXiv preprint arXiv:2406.17557 , year =

    The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale , author =. arXiv preprint arXiv:2406.17557 , year =

    arXiv

  20. [28]

    arXiv preprint arXiv:2502.02737 , year =

    SmolLM2: When Smol Goes Big---Data-Centric Training of a Small Language Model , author =. arXiv preprint arXiv:2502.02737 , year =

    arXiv

  21. [29]

    arXiv preprint arXiv:2410.05192 , year =

    Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective , author =. arXiv preprint arXiv:2410.05192 , year =

    arXiv

  22. [30]

    Advances in Neural Information Processing Systems , volume =

    Training Compute-Optimal Large Language Models , author =. Advances in Neural Information Processing Systems , volume =

  23. [31]

    2023 , howpublished =

    A Framework for Few-Shot Language Model Evaluation , author =. 2023 , howpublished =

    2023

  24. [32]

    Think You Have Solved Question Answering? Try

    Clark, Peter and Cowhey, Isaac and Etzioni, Oren and Khot, Tushar and Sabharwal, Ashish and Schoenick, Carissa and Tafjord, Oyvind , journal =. Think You Have Solved Question Answering? Try

  25. [33]

    Clark, Christopher and Lee, Kenton and Chang, Ming-Wei and Kwiatkowski, Tom and Collins, Michael and Toutanova, Kristina , booktitle =

  26. [34]

    Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages =

    HellaSwag: Can a Machine Really Finish Your Sentence? , author =. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages =

  27. [35]

    Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages =

    Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering , author =. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages =

    2018

  28. [36]

    Bisk, Yonatan and Zellers, Rowan and Le bras, Ronan and Gao, Jianfeng and Choi, Yejin , booktitle =

  29. [37]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume =

    WinoGrande: An Adversarial Winograd Schema Challenge at Scale , author =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =

  30. [38]

    arXiv preprint arXiv:1609.07843 , year =

    Pointer Sentinel Mixture Models , author =. arXiv preprint arXiv:1609.07843 , year =

    arXiv

  31. [39]

    Paperno, Denis and Kruszewski, Germ. The. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics , pages =

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.