{"id":"44c72554-55a9-4b3e-91f2-65f9f3c053e0","arxiv_id":"2608.09307","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A no-softmax language model using random-feature-linearized 2-simplicial attention plus Kimi Delta Attention reports higher mean accuracy at 16k context under an analytic iso-FLOP budget.","lead":"This paper rewrites the three-way score of 2-simplicial attention as a dot product between a composite query and a key, then approximates the resulting token sum with positive random features and a fixed-size state. It reports that a no-softmax model combining this layer with Kimi Delta Attention improves mean downstream accuracy over a KDA hybrid at 16k context under an analytic iso-FLOP budget.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Iso-FLOP accounting in Eq. (27) gives the proposed model 16% more tokens than the KDA hybrid at 16k; if that analytic count is not a fair proxy for training resources, the headline accuracy gain could be a token-budget effect.","rationale":"The paper's theoretical core, including the linearization of the 2-simplicial score in Eq. (5), the positive random-feature approximation with Lemma 1, and the O(T) complexity claim in Theorem 1, is internally coherent and is not the source of my objection. The random-feature estimator is unbiased for the exponential kernel, the prefix states in Eqs. (10)-(11) are causal, and the anchor window keeps the c-mode finite. The load-bearing step is empirical: the abstract and Section 6.3 claim a matched-compute win, and the only evidence for that match is the analytic FLOP accounting of Appendix D. That accounting is explicitly not a hardware model, and the paper's own wall-clock numbers put the hybrid ahead, while granting the proposed model 16% more training tokens. Because all results are single-seed and no variance is reported, the observed +0.0079 mean-accuracy difference sits on top of an untested accounting assumption. This does not make the paper dishonest, as it is transparent about both limitations, but it means the central claim should not be accepted as a robust finding until the token-budget effect is ruled out. The proposed check with profiler-measured FLOPs or an equal-resource retrain would settle the question directly. My verdict on the reader's CONDITIONAL remains unchanged: the architecture is a plausible contribution, but the headline comparison requires that verification.","tokens_in":13259,"tokens_out":10493,"duration_ms":107844,"concrete_test":"Recompute the 16k token budgets with profiler-measured per-step FLOPs (e.g., torch.profiler or NVIDIA Nsight on the actual kernels) instead of the analytic formulas in Appendix D, then retrain both arms on FineMath-4+ at 16k context with the resulting iso-FLOP budgets and the same data order and seed. If the KDA hybrid's budget rises above 2.70B tokens and its mean accuracy reaches or exceeds 0.3888 (or its LAMBADA PPL falls to or below 602.6), the reported advantage is an artifact of the analytic FLOP count; if the proposed model still wins with the profiled budgets, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the no-softmax KDA+LinSimp model achieves the highest mean downstream accuracy under matched compute depends entirely on the analytic FLOP counts in Appendix D being a faithful proxy for the training resource that matters. In Table 5, the iso-FLOP rule (Eq. 27) assigns the proposed model 3.14B tokens and the KDA hybrid 2.70B tokens at 16k context, a 16% token advantage, because the six softmax layers in the hybrid are counted as roughly 16% more expensive per token. The paper's own Section 7 states these counts are not energy, latency, or system cost measurements, and Section 6.4 reports the hybrid is faster in wall-clock at 16k (24.5k vs 20.2k tokens/s). Thus the slower model is granted more training tokens under the analytic rule. If the analytic per-token FLOP ratio overstates the true resource difference, the reported +0.0079 mean accuracy gain and the LAMBADA improvement from 715.6 to 602.6 may simply reflect the extra 0.44B tokens consumed by the proposed model, not an architectural advantage of LinSimp. This is an external-validity gap in the central empirical claim, not an internal inconsistency: Theorem 1, the random-feature derivation, and the causality argument are internally coherent.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LinSimp, a linearized form of 2-simplicial attention. The trilinear score ⟨q_i, k_j, r_c⟩ is rewritten as (τ q̂_i ⊙ r̂_c)ᵀ k̂_j, so that the sum over the token axis j becomes a softmax-like kernel sum, which is then approximated with positive random features and stored in a fixed-size recurrent state (M_i and a_i). The anchor axis c is kept explicit over a short window of w recent tokens. The authors prove (Theorem 1) that the layer is causal and has O(T) total work for fixed feature rank m and window width w, describe a custom Triton forward/backward implementation, and combine LinSimp with Kimi Delta Attention (KDA) in a 24-layer model with no softmax attention. Experiments on FineWeb-Edu and FineMath-4+ compare several architectures under equal-token and analytic iso-FLOP budgets. The headline result is that, at 16k context under an analytic iso-FLOP budget, the no-softmax KDA+LinSimp model improves mean downstream accuracy from 0.3809 to 0.3888 and lowers LAMBADA perplexity from 715.6 to 602.6 compared with a KDA hybrid.","tokens_in":13527,"tokens_out":9811,"duration_ms":89014,"significance":"The theoretical contribution is sound and potentially useful: the one-mode kernelization of the trilinear score is a clean idea that extends linear-attention techniques to higher-order interactions while preserving a global reach on one axis. The proof of linear complexity is straightforward and correct, and the paper includes a working Triton implementation with numerical tests against an fp32 reference. The random-feature estimator is unbiased for the kernel, and the finite-rank bias of the normalized ratio is explicitly acknowledged. However, the empirical validation is not yet at the level needed to support the headline claims: all experiments use a single seed with no variance estimates, and the 'matched compute' budget rests on an analytic FLOP model that the paper itself concedes does not measure wall-clock, energy, or system cost. The reported wall-clock speed at 16k actually favors the KDA hybrid (24.5k vs 20.2k tokens/s), undercutting the iso-FLOP assumption. If the theoretical and implementation contributions are the main value, the paper could be acceptable once the empirical claims are tempered or better supported.","major_comments":[{"comment":"The iso-FLOP rule in Eq. (27) assigns 3.14B tokens to the no-softmax KDA+LinSimp model and 2.70B to the KDA hybrid at 16k context, a 16% token advantage, because the analytic FLOP model in Appendix D counts the six softmax layers as roughly 16% more expensive per token than the LinSimp layers. However, Section 6.4 reports that the KDA hybrid is faster in wall-clock at 16k (24.5k vs 20.2k tokens/s), and Section 7 states that the FLOP counts do not measure energy, latency, or total system cost. Thus the reported +0.0079 mean accuracy gain and the LAMBADA improvement from 715.6 to 602.6 may simply reflect the larger token budget granted by the analytic rule rather than an architectural advantage of LinSimp. Since this is the central empirical claim, please provide a wall-clock-matched or energy-matched comparison, or at least a sensitivity analysis over the FLOP constants (e.g., C_KDA and C_LinSimp) showing that the conclusion is robust to plausible variations in the accounting.","section":"Section 6.3, Table 5, Eq. (27), Appendix D"},{"comment":"The paper explicitly states that experiments use a single seed and that no confidence intervals can be attached. Given that Table 6 shows per-task differences of only −0.001 to +0.020, the mean accuracy increase of 0.0079 is not statistically substantiated, and the abstract's claim of improvement 'under matched compute' is stronger than the evidence. Please provide at least three seeds or bootstrap confidence intervals, and modify the language to reflect the preliminary nature of the results.","section":"Section 7 and Table 6"}],"minor_comments":[{"comment":"The passage beginning 'In our current formulation, computing Ni and Zi has O(1) cost...' is confusing because Ni and Zi are not defined until Section 3.3, and the discussion of increasing the cost to O(N) is not used in the proposed method; please remove or rewrite this passage to avoid ambiguity about the complexity guarantees.","section":"Section 3.2"},{"comment":"The sentence 'replacing three of the six attention slots with LinSimp praises mean accuracy by 0.0080' uses 'praises' where 'raises' is intended; also, 'LinSimp' should be 'LinSimp' for consistency.","section":"Section 6.1"},{"comment":"The word 'cuurrently' should be 'currently'.","section":"Section 7"},{"comment":"The sentence 'The 2 Simplicial Attention is expensive, a query attends to a two-dimensional plane of token pairs lead to O(n3) computation' is grammatically incorrect; please revise and use O(n^3) with proper superscript notation.","section":"Section 1"},{"comment":"The sentence 'Section 6.4 reports initial kernel measurements...' appears within Section 6.3 and refers to the following subsection; consider rephrasing to 'The following subsection reports...' or restructuring the text.","section":"Section 6.3"},{"comment":"The Description column in Table 1 has inconsistent punctuation and capitalization; please use a consistent style (e.g., end every entry with a period or none).","section":"Table 1 and general formatting"},{"comment":"The expression for C_LinSimp includes the term B_C D_v without explanation; a brief comment on why the chunk size B_C appears in the per-token FLOP estimate would help the reader interpret the iso-FLOP results.","section":"Appendix D, Eq. (41)"}],"recommendation":"major_revision","confidential_remarks":"The theoretical derivation is sound and the Triton implementation is a genuine engineering effort. The main weakness is the empirical validation: a single seed and an analytic FLOP model that the paper itself undermines with wall-clock measurements. A revision that either adds wall-clock/energy-matched comparisons or sensitivity analyses, provides multiple seeds or confidence intervals, and tempers the abstract's claims would be credible. Alternatively, reframing the paper as a theoretical/implementation contribution with preliminary experiments would make the current evidence more proportionate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is my read. The genuinely new piece is the one-mode kernelization in Eq. (5) — rewriting the trilinear score as an inner product between a composite query (τq_i ⊙ r_c) and k_j, so the j-sum becomes an exponential dot-product kernel that can be approximated with positive random features and a fixed-size state. Keeping the anchor axis explicit inside a 64-token window is a sensible compromise, and Theorem 1 is correct: the layer is causal and O(T) for fixed m and w. The derivation is honest about the finite-rank bias. The Triton kernels and the diagnostic test showing that QR sign correction matters (Appendix B) are real engineering work.\n\nThe soft spots are where the reader's report puts them. The 16k iso-FLOP comparison in Table 5 gives the no-softmax model 3.14B tokens versus 2.70B for the KDA hybrid, a 16% token advantage. The paper's own Section 7 says these FLOP counts are not energy, latency, or system cost, and the wall-clock numbers say the hybrid is faster (24.5k vs 20.2k tokens/s). So the headline LAMBADA improvement could be a token-budget effect. That is the main external-validity gap. Also, the abstract overstates: in the web iso-FLOP run (Table 2), the KDA hybrid has higher mean accuracy (0.3458 vs 0.3446); the proposed model wins only in the math runs. And everything is a single seed with no variance estimate, which the authors admit. At 2k math, the no-softmax model has worse validation loss than the KDA hybrid but higher mean accuracy — that inversion is not discussed.\n\nNone of this is fatal. The math is correct, the limitations are disclosed in the text, and the architecture is testable. The paper would benefit from multiple seeds, a wall-clock-controlled comparison, and released code. For someone working on efficient attention or higher-order interactions, this is worth reading and citing. I would send it to a serious venue; the empirical claims need revision, but the kernelization result is a legitimate contribution.","headline":"Correct kernelization and a testable architecture, but the headline accuracy claim rests on an analytic iso-FLOP budget that favors the proposed model and a single seed.","tokens_in":14074,"tokens_out":2624,"would_cite":false,"duration_ms":26350,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"A one-mode kernelization makes 2-simplicial attention linear in sequence length, and a softmax-free KDA+LinSimp model improves 16k mean accuracy from 0.3809 to 0.3888.","keywords":["2-simplicial attention","linear attention","positive random features","kernel approximation","long-context language modeling","delta rule attention","Triton kernels","no-softmax model"],"falsifier":"Give both 16k architectures the same token budget, say 2.70B tokens each, and measure the seven-task mean accuracy; if KDA+LinSimp no longer beats the KDA hybrid, the reported gain under the iso-FLOP rule is an artifact of the analytic token allocation. A cross-check is to compare the two models under matched wall-clock time or matched energy.","tokens_in":13021,"feed_emoji":"🧠","tokens_out":10094,"duration_ms":83952,"temperature":0.7,"pith_summary":"The paper tries to make 2-simplicial attention—where a query attends to pairs of earlier tokens through a trilinear score—linear in sequence length without giving up the global reach that window-only versions lack. It rewrites the trilinear score as an inner product between a composite query and a key, so the sum over one token axis becomes an ordinary exponential kernel, which is approximated with positive random features and stored in a fixed-size recurrent state. The second token axis is kept explicit over a short rolling window of recent anchors, so each recent anchor sends a different question to the same global state. If correct, this gives a causal layer with $O(T)$ total work and a fixed inference-state size independent of $T$, and the paper's 16k-context experiments report that a no-softmax model composed with Kimi $\\Delta$ Attention improves mean downstream accuracy from 0.3809 to 0.3888 while lowering LAMBADA perplexity from 715.6 to 602.6 under an analytic iso-FLOP comparison.","feed_headline":"Simplicial attention goes linear and beats a delta-rule hybrid at 16k","feed_subtitle":"A fixed-size global state plus a 64-token anchor window lifts mean accuracy from 0.3809 to 0.3888 under matched compute.","key_machinery":"The load-bearing object is a one-mode kernelization: the trilinear score $\\langle \\tau\\hat q_i,\\hat k_j,\\hat r_c\\rangle$ is rewritten as $(\\tau\\hat q_i\\odot \\hat r_c)^\\top \\hat k_j$. For a fixed query and anchor this is the exponential kernel $\\exp(z_{ic}^\\top \\hat k_j)$ in the past-token index $j$, with composite query $z_{ic}=(\\tau\\hat q_i)\\odot \\hat r_c$. Positive random features $\\phi(x)=m^{-1/2}\\exp(\\Omega x-\\|x\\|_2^2/2\\,\\mathbf{1})$ approximate that kernel; the sum over $j$ is absorbed into two recurrent prefix states, $M_i=\\sum_{j\\le i}\\phi(\\hat k_j)v_j^\\top$ and $a_i=\\sum_{j\\le i}\\phi(\\hat k_j)$, while the anchor $c$ runs only over the most recent $w$ tokens. The work of these pieces is to convert the exact joint softmax over causal pairs $(j,c)$ into a linear-time, fixed-state layer that still lets every recent anchor send a different composite query to the whole prefix.","core_discovery":"On the paper's own terms, the central discovery is a one-mode kernelization of 2-simplicial attention. The trilinear score $\\langle \\tau\\hat q_i, \\hat k_j, \\hat r_c\\rangle$ is re-expressed as $(\\tau\\hat q_i \\odot \\hat r_c)^\\top \\hat k_j$, so that for a fixed query $i$ and anchor $c$, the sum over the past token $j$ is an ordinary exponential dot-product kernel. Approximating that kernel with positive random features $\\phi(x)=m^{-1/2}\\exp(\\Omega x - \\|x\\|_2^2/2\\,\\mathbf{1})$ turns the causal sum over $j$ into two fixed-size prefix states $M_i=\\sum_{j\\le i}\\phi(\\hat k_j)v_j^\\top$ and $a_i=\\sum_{j\\le i}\\phi(\\hat k_j)$, while the anchor index $c$ stays explicit only over the most recent $w=64$ tokens. The result is Theorem 1: the layer is causal, runs in total $O(T)$ work for fixed feature rank $m$ and anchor width $w$, and uses $O(mD_v+m+w(D+D_v))$ state per head independent of $T$. Composed with Kimi $\\Delta$ Attention in 18 layers plus six LinSimp layers and no softmax, the model achieves the highest mean downstream accuracy among the compared architectures and, at 16k context, improves mean accuracy from 0.3809 to 0.3888 while reducing LAMBADA perplexity from 715.6 to 602.6 under an analytic iso-FLOP budget.","pith_inferences":["The same one-mode rewriting could linearize the other algebraic orientation of the trilinear score or higher-order simplicial variants; the paper notes that computing both orientations doubles the contraction cost, so the choice is a capacity-cost trade-off.","The approximation quality depends on feature rank $m$ and anchor width $w$, and the appendix shows the kernel error falling roughly as $m^{-1/2}$ in isolation, suggesting a tunable trade-off between memory and fidelity for tasks that need exact recall.","The gap between the analytic iso-FLOP verdict and the measured wall-clock throughput (the KDA hybrid is faster at 16k) means the 'matched compute' conclusion is not invariant under the choice of resource metric; an energy- or latency-equated comparison would be a natural stress test.","The paper's mention of an $O(N)$ explicit-scan variant points to a continuum between the fixed-state compression and standard attention, which could be explored for memory-intensive tasks such as associative recall."],"forward_implications":["A causal layer can include triple-token 2-simplicial interactions with total work $O(T)$, linear in sequence length for fixed feature rank and anchor width.","The full causal prefix remains reachable through the global state even for tokens arbitrarily far back, while the short anchor window supplies local conditioning.","The exact windowed 2-simplicial baseline becomes comparatively costly under a compute-matched budget because its FLOP count shrinks its assigned token budget, making the linearized layer the practical route to keeping the trilinear operator.","Autoregressive inference needs a fixed-size state per head that does not grow with $T$, the property that matters for long-context decoding.","Softmax-free models composed of delta-rule layers and LinSimp layers can improve long-context downstream accuracy and LAMBADA perplexity over a KDA hybrid under the paper's analytic iso-FLOP rule."],"supporting_citations":[{"why":"Supplies the positive orthogonal random-feature construction used to approximate the exponential dot-product kernel.","marker":"Choromanski et al., 2021"},{"why":"Defines the 2-simplicial Transformer whose trilinear score and pair-composed values this paper linearizes.","marker":"Clift et al., 2020"},{"why":"Provides the exact windowed 2-simplicial baseline and Triton kernel that motivates the global-vs-window design.","marker":"Roy et al., 2025"},{"why":"Establishes the linear-attention move of collecting the past-token sum into a recurrent state, used by $M_i$ and $a_i$.","marker":"Katharopoulos et al., 2020"},{"why":"Defines Kimi Delta Attention, the backbone that LinSimp layers are composed with in the no-softmax model.","marker":"Kimi Team et al., 2025"},{"why":"Grounds the iso-FLOP comparison convention that assigns each architecture its token budget.","marker":"Hoffmann et al., 2022"},{"why":"Introduces random features for large-scale kernel machines, the origin of the feature approximation.","marker":"Rahimi and Recht, 2007"}],"fun_headline_variants":["Linearized simplicial attention with random features beats delta-rule hybrid","Linear 2-simplicial attention: global reach, linear cost, beats hybrid","Fixed-size state with anchor window: linear simplicial attention beats hybrid","No softmax attention: linearized 2-simplicial beats KDA hybrid","Random-feature simplicial attention: global reach at linear cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the analytic FLOP counts in Equation (27) and Appendix D correctly represent the resource that limits training; if wall-clock time or energy is the real constraint, the reported 16k advantage could come from the larger token budget assigned to the no-softmax model (3.14B vs 2.70B) rather than from the new layer.","fun_headline_variants_meta":{"raw":{"variants":["Linearized simplicial attention with random features beats delta-rule hybrid","Linear 2-simplicial attention: global reach, linear cost, beats hybrid","Fixed-size state with anchor window: linear simplicial attention beats hybrid","No softmax attention: linearized 2-simplicial beats KDA hybrid","Random-feature simplicial attention: global reach at linear cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001156,"raw_usage":{"total_tokens":4850,"prompt_tokens":1067,"completion_tokens":3783,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":683,"completion_tokens_details":{"reasoning_tokens":3692}},"tokens_in":683,"tokens_out":3783,"duration_ms":25612,"temperature":1.0,"reasoning_tokens":3692,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:39:04.982196+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give both 16k architectures the same token budget, say 2.70B tokens each, and measure the seven-task mean accuracy; if KDA+LinSimp no longer beats the KDA hybrid, the reported gain under the iso-FLOP rule is an artifact of the analytic token allocation. A cross-check is to compare the two models under matched wall-clock time or matched energy.","supporting_citations":[],"review_version":1}