{"id":"68ed4e4f-ce54-415e-a2a0-c778f3195425","arxiv_id":"2605.27259","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Kan Extension Transformers re-label attention and simplicial aggregation as Kan-style weighted sums, but the reported predict-detach results are consistent with future-token leakage.","lead":"This paper proposes a categorical framework that re-describes Transformer attention, geometric mixing, and higher-order simplicial aggregation as weighted 'Kan-style' extension operators, and introduces predict-detach carriers for self-conditioning. Its predict-detach models report test perplexities as low as 1.05 on PTB, which is a red flag for gold-future leakage rather than valid causal modeling.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Predict-detach carriers are indexed by their own future position, so consuming them at an earlier target violates the paper's stated prefix condition and explains the near-perfect perplexities.","rationale":"The reader's weakest assumption is exactly the load-bearing issue: predictive carriers for future positions must be prefix-valid, but Eq. (8) defines them from h_t, which includes the current gold token. The reader's evidence—near-perfect perplexities matching explicit leakage baselines—is the right diagnostic. I see no independent support in the manuscript that overrides this: the categorical framing is explicitly hedged as 'Kan-style' rather than a proved enriched Kan extension, and the impressive predict-detach numbers are the main empirical pillar. The proposed concrete test (checking the released code and rerunning with a genuinely prefix-valid carrier) would settle whether the effect is real or an artifact. I agree with the reader's REJECT verdict, so no adjustment is needed.","tokens_in":19592,"tokens_out":5520,"duration_ms":55487,"concrete_test":"Inspect the released repository's predict-detach implementation for GT-PD and KET-PD: determine whether the carrier injected at source position s > t is computed from the teacher-forced hidden state h_s or from a prefix-valid projection such as h_t. Then rerun the same experiments with all future-position carriers replaced by the prefix-valid carrier detach(softmax(W_o h_t/T)E) (i.e., predictions made from the target's own prefix). If test perplexity jumps from ~1.05/1.59 to the strict-causal range (~120–160 on PTB/WT2), the reported predict-detach gains were caused by gold-future leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal-validity claim rests on Eq. (8) in §3.2: the predictive carrier at position t is ê_t = detach(softmax(ℓ_t/T)E) with ℓ_t = W_o h_t, where h_t is the teacher-forced hidden state at t. For any source position s > t aggregated into target t—for example the noncausal branch of Algorithm 2 in §17.2, which adds φ(e_{t+1}) built from v_{t+1}—the carrier ê_s is a function of gold tokens x≤s, including x_{t+1..s}, and is therefore not measurable from the prefix available at t. The abstract itself states the required condition: 'every carrier consumed at target t must also be measurable from the prefix available at t.' No experiment or code excerpt in the main text shows this was enforced. The reported results are consistent with leakage: GT-PD reaches 1.05 on PTB and 1.59 on WT2 (Tables 2, 5), essentially matching the explicit gold-noncausal gt_noncausal leakage diagnostic (1.05–1.05), while the small-width d=64 run lands between causal and leakage levels (16.57 on PTB), exactly the pattern expected from partial future-token contamination. The paper's own regime labels in §16.4 use a same-index predicted hint (p_t) for Regime E and a shifted prefix-valid hint (p_{t−1}) for Regime C, but the main-text GT-PD/KET-PD runs are not specified in those terms. Thus the load-bearing premise—that predict-detach permits noncausal neighborhoods without leaking gold future tokens—is unsupported and likely false as implemented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a categorical framing in which Transformer layers are weighted structured extension operators: standard attention is the singleton-neighborhood case, TopoCoend is the learned-geometric case, and the proposed Kan Extension Transformer (KET) is the simplicial case. It introduces predict-detach carriers and claims that this lets models use noncausal neighborhoods without leaking gold future tokens. Experiments compare 12 implementations on PTB, WikiText-2, and WikiText-103 across strict-causal, predict-detach, and gold-noncausal regimes. The headline results are that strict-causal quadratic KET is strongest on WT2/WT103 and that predict-detach produces large gains, with GT-PD reaching test perplexities near 1. The abstract itself qualifies the Kan-extension identity as conditional and states a prefix-measurability requirement for carriers, but the main text does not demonstrate that this requirement is satisfied in the experiments.","tokens_in":20018,"tokens_out":5941,"duration_ms":59931,"significance":"If the causal-validity claim held, the predict-detach regime would be a useful design principle, and the categorical framing could serve as a unifying language for attention, geometric mixing, and diffusion-style completion. The paper is honest about its limitations and ships detailed reproducibility artifacts, including logs and checkpoints. However, the central empirical claim is not supported: the reported predict-detach perplexities are in the same range as the paper's own leakage diagnostics, and no evidence shows that the stated prefix-measurability constraint was enforced. The categorical unification is also asserted rather than proved; Eq. (1) is a generic weighted sum, and Section 9.6 calls it 'precisely' a Kan/coend computation without supplying the required functoriality, representability, or coend conditions. As it stands, the contribution is an interpretive taxonomy plus an invalidated experimental claim, so the significance is low.","major_comments":[{"comment":"The load-bearing premise that a predict-detach carrier is prefix-valid at an earlier target is violated whenever the carrier is indexed by a future position. Eq. (8) defines ê_t from h_t, a teacher-forced hidden state, so ê_s is a function of x≤s. Algorithm 2's noncausal branch (lines 9–11) adds φ(e_{t+1}) to target t, and more generally any source s>t consumed at t carries gold tokens x_{t+1..s}. The abstract explicitly requires that 'every carrier consumed at target t must also be measurable from the prefix available at t,' but no main-text experiment or code excerpt shows that the reported GT-PD/KET-PD runs enforce this. Section 16.4 labels Regime E as a same-index predicted hint p_t, which does not match the noncausal future-position transport described in Section 3. This invalidates the predict-detach results in Tables 2, 5, 6, and 7; the d=64 intermediate perplexity is exactly the","section":"§3.2 Eq. (8); §17.2 Algorithm 2; §16.4"},{"comment":"The paper describes a leakage test—shuffling target tokens and checking whether perplexity collapses—but reports no result from it. The reported GT-PD numbers (1.05 on PTB, 1.59 on WT2, 12.84 on WT103 in Table 2) are essentially indistinguishable from the explicit gold-noncausal (Regime A) diagnostics (1.05 on PTB and 1.05 on WT2 in Tables 5–6). Table 8 makes the problem explicit: transition_gain is ≈1.000 at d=256, meaning the 'self-conditioned' run reaches the augmented-context performance level. Without the shuffling diagnostic, the claim that predict-detach 'preserves causal validity' is unsupported and the near-perfect perplexities are what leakage would look like.","section":"§17.4; Tables 2, 5–8"},{"comment":"The central theoretical claim—that the update in Eq. (1) 'is precisely a discretized enriched Kan/coend computation'—is asserted, not proved. Eq. (1) is a generic weighted sum. To identify it with an enriched left Kan extension one must specify the source category, the functorial assignment of values, representable hom-objects (or a profunctor action), and verify the coend universal property. None of these are given for attention, TopoCoend, or KET. The abstract itself hedges that the operator is 'Kan-style' when these conditions fail, which is appropriate, but the main text's stronger wording overclaims. Since the title and first claim rest on this categorical identity, the paper should either provide the functoriality/representability proof or explicitly reposition the contribution as a design language rather than a theorem.","section":"§9.6 and §2.1 Eq. (1)"}],"minor_comments":[{"comment":"Section 16.5 establishes a reporting standard with regime labels (C/E/A), but Tables 2 and 3 in the main text do not label regimes. Readers cannot tell from the tables which runs are strict-causal, predict-detach, or leakage diagnostics.","section":"§16.5 vs. §4.3"},{"comment":"The reproducibility section states that the LM figures are single-seed runs with no multi-seed error bars. Tables 1–3 should carry an explicit caveat, since the differences between some strict-causal models are small.","section":"§17.5"},{"comment":"Algorithms 1 and 2 use v_t and h_t without defining their relationship. If v_t is the value base derived from h_t, this should be stated; otherwise the incidence-restricted block's input is underspecified.","section":"§17.1–17.2"},{"comment":"The paper claims to evaluate 12 Transformer implementations, but the main text does not explicitly enumerate them. A table listing the 12 models with their regime labels would make the comparison easier to verify.","section":"§1 and §4"},{"comment":"The temperature T is used in Eq. (8) but not defined until Section 17.5. Define it when first introduced.","section":"§3.2 Eq. (8)"},{"comment":"The notation ẋ_{t+1} appears in the right-Kan completion formula but is not defined. This makes the formal regime distinction in that paragraph hard to follow.","section":"§16.2"}],"recommendation":"reject","confidential_remarks":"The stress-test concern lands squarely and is confirmed by internal evidence. Algorithm 2 consumes φ(e_{t+1}) at target t, and the reported GT-PD perplexities match the paper's own gold-noncausal leakage diagnostic. This is not a missing control; it is an implementation that violates the abstract's stated prefix condition. The central empirical claim is therefore invalid as presented. The categorical framing could be salvaged by proof or by relabeling as design language, but the experimental contribution would need to be redone with prefix-valid carriers and a reported leakage test. That is beyond a revision and warrants rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing you should know before reading: the predict-detach results, the paper's central empirical offer, look like leakage. The carrier at position t is computed from the teacher-forced hidden state at t, so any noncausal aggregation that consumes carriers from s > t is consuming values that already encode the gold future. The abstract literally says every carrier consumed at target t must be measurable from the prefix available at t, and nothing in the main text shows that was done. GT-PD hits 1.05 on PTB, matching the paper's own gold-noncausal diagnostic (1.05). That is not a self-conditioning result; that is a calibration check that failed.\n\nGive credit where it's due. The categorical framing—attention as singleton-neighborhood aggregation, geometric mixing as incidence-restricted aggregation, KET as simplicial—is a reasonable organizational language, and the paper is honest that the operator is 'Kan-style' unless specific functoriality and representability conditions hold. The information-regime taxonomy (strict-causal, gold-noncausal, predict-detach) is genuinely useful, and the leakage-diagnostic procedure in Section 17.4 is the right idea. The strict-causal comparisons are modest in scale but show quadratic KET competitive on WikiText-2/103.\n\nNow the soft spots, in proportion. The leak is load-bearing, not cosmetic. Section 16.4 distinguishes a same-index predicted hint (Regime E) from a shifted prefix-valid hint (Regime C), which shows the author understands the issue. But the headline GT-PD/KET-PD runs are not reported in those terms, and the near-perfect perplexities strongly suggest they used the same-index form. The paper should be re-run with the carrier at position t-1 or otherwise enforced prefix-measurability, and the leakage test results reported. Second: the Kan-extension claim is asserted rather than proven—Section 9.6 says a weighted sum 'is precisely a discretized enriched Kan/coend computation' without establishing functoriality or representable hom-objects. That fine as a design lens, but it is a reformulation, not a theorem. Third: single-seed runs, no error bars, and the block-denoising results are unsurprising since the model is literally given corrupted future tokens.\n\nWho gets value: people interested in categorical design patterns for transformers, and anyone working on self-conditioning or auxiliary prediction branches. It is also a good cautionary example of how easy it is to leak future information through noncausal neighborhoods.\n\nRecommendation: send it to peer review, but only with a request for major revision. The flaw is identifiable and fixable, and the question—can noncausal neighborhoods be used validly through detached predictions—is worth settling. A serious referee should see this version, but the empirical claims should not stand as-is.","headline":"Useful categorical lens, but the headline predict-detach results are almost certainly future-token leakage; the paper's own abstract states the condition the experiments don't enforce.","tokens_in":20502,"tokens_out":2137,"would_cite":false,"duration_ms":23640,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Transformer layer is a weighted structured extension over chosen neighborhoods; attention, geometric mixing, and the new KET are special cases, and predict-detach is the boundary that lets noncausal neighborhoods be used safely.","keywords":["Kan extensions","Transformers","attention","predict-detach","self-conditioning","simplicial neighborhoods","diffusion","structured completion"],"falsifier":"Run the predict-detach configurations with a strict prefix-measurability constraint (e.g., only carriers from positions ≤ t−1 or with an explicit mask) and compare perplexities to the reported values. If the strict run loses most of the reported improvement, the central causal-validity claim is falsified. Alternatively, apply the paper's shuffled-target leakage test: if a predict-detach model still collapses to near-1 perplexity, an illicit gold-future path is present.","tokens_in":19424,"feed_emoji":"🧩","tokens_out":11766,"duration_ms":99283,"temperature":0.7,"pith_summary":"The paper proposes that a Transformer layer is best understood as a weighted structured extension operator: it takes values on a chosen neighborhood system and extends them to target token positions. Standard attention is the singleton-neighborhood case; geometric incidence mixing is a sparse edge-restricted case; and the new Kan Extension Transformer (KET) is the higher-order simplicial case. The paper argues that the decisive enabler for using noncausal neighborhoods in language modeling is not the graph shape but the values transported: detached predicted carriers, not teacher-forced hidden states, turn a noncausal update into a valid self-conditioning mechanism. If this holds, attention, geometric mixing, and diffusion-style completion share one design language, and the dominant empirical lever is the information regime (what values flow) rather than the neighborhood family.","feed_headline":"One weighted extension operator unifies attention, mixing, and KET","feed_subtitle":"KET is the higher-order simplicial case of the same extension rule","key_machinery":"The central object is the weighted structured extension operator h'_t ≈ ∫^{σ∈N} W(t,σ) ⊗ X(σ), interpreted computationally as Σ_σ w(t,σ) V(σ). Different neighborhood categories N give attention (tokens), TopoCoend (learned geometric adjacency), and KET (simplices). The predict-detach carrier ê_t = detach(softmax(ℓ_t/T)E) is the second load-bearing piece: it is a prefix-derived prediction frozen before transport, and it is what makes noncausal aggregation causally legitimate.","core_discovery":"The central claim is that a Transformer layer can be viewed as a weighted left-Kan-style extension operator. Concretely, the update h'_t ≈ Σ_σ W(t,σ) V(σ) is a coend-style aggregation over a source category of neighborhoods: tokens for attention, learned geometric adjacency for TopoCoend, and simplices (tokens, edges, faces) for KET. The operator is an actual enriched left Kan extension only when source values are functorial, weights are representable hom-objects (or the specified profunctor action), and aggregation realizes the corresponding coend; otherwise 'Kan-style' denotes an interpretation. A second claim is that predict-detach, computing ê_t = detach(softmax(ℓ_t/T)E) from causal logi","pith_inferences":["The paper leaves implicit that the 'Kan-style' label is an interpretation for most practical implementations; constructing a true enriched left Kan extension (with functorial source values and representable hom-objects) could yield a different, more principled model, and the difference is testable.","The abstract's prefix-measurability requirement for predict-detach carriers is not verified in the reported experiments; a strict run that enforces it (e.g., masking future carriers) would clarify whether the large predict-detach gains are genuinely leakage-free.","The right-Kan/completion analogy suggests other partial-information objectives (infilling, masked span prediction) should show the same pattern as block denoising, which can be tested in the same harness."],"forward_implications":["If the unification holds, attention, geometric mixing, and simplicial aggregation can be engineered within one design space; the incidence-restricted KET becomes a linear-time sparse approximation to the quadratic KET.","The information-regime distinction means architecture comparisons are only fair within the same regime (strict-causal, self-conditioned, augmented-context), and cross-regime gains should be attributed to the information budget.","The paper's hierarchy places Geometric Transformers as an efficient incidence-restricted special case of KET, so improvements to geometric mixers can transfer to KET and vice versa.","Denoising-style completion (predicting a corrupted future block) is dramatically easier than direct block prediction, supporting a right-Kan/compatibility view of language modeling as structured completion rather than one-shot generation."],"fun_headline_variants":["A single coend unifies attention, mixing, and KET","Kan extensions: the categorical backbone of Transformers","KET tops strict-causal models on WikiText-2 and 103","From tokens to simplices: one extension rule"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The validity of the predict-detach regime rests on the assumption that every predictive carrier consumed at target position t is measurable from the prefix available at t; the abstract states this requirement explicitly, but the main-text experiments do not demonstrate that the released runs enforce it.","fun_headline_variants_meta":{"raw":{"variants":["A single coend unifies attention, mixing, and KET","Kan extensions: the categorical backbone of Transformers","KET tops strict-causal models on WikiText-2 and 103","From tokens to simplices: one extension rule"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00035,"raw_usage":{"total_tokens":1774,"prompt_tokens":796,"completion_tokens":978,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":909}},"tokens_in":540,"tokens_out":978,"duration_ms":11203,"temperature":1.0,"reasoning_tokens":909,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T12:58:34.838780+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the predict-detach configurations with a strict prefix-measurability constraint (e.g., only carriers from positions ≤ t−1 or with an explicit mask) and compare perplexities to the reported values. If the strict run loses most of the reported improvement, the central causal-validity claim is falsified. Alternatively, apply the paper's shuffled-target leakage test: if a predict-detach model still collapses to near-1 perplexity, an illicit gold-future path is present.","supporting_citations":[],"review_version":2}