{"id":"743071aa-fbc1-4d92-96c0-8e7e15f4ce7a","arxiv_id":"2501.00823","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper claims feed-forward networks are a special case of its proposed generalized cross-attention to a shared knowledge base, but the derivation is a tautology and no experiments are run.","lead":"This paper proposes a modular Transformer where every layer reads from a shared knowledge base using a modified cross-attention mechanism. It argues this is equivalent to a standard Transformer's feed-forward networks, but provides no experiments to show it works.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed FFN–cross-attention closure fails as written because B1l(E) is an N×|E| matrix, not a broadcast bias vector; the §5.1 derivation is dimensionally inconsistent, and the training-equivalence conclusion is a separate unsupported leap.","rationale":"The reader's REJECT is justified. The paper's central claim rests on an algebraic derivation that is internally inconsistent: the shape of B1l(E) in Eq. 8–9 does not match the rank-1 broadcast bias required for the FFN equivalence in Eq. 21–22. Even if that notation were repaired, the paper's inference from representational equivalence to identical training performance is a logical leap, as optimization trajectories can diverge for different parameterizations of the same function class. No experiments or code are provided to support the claimed benefits, and in the joint-training setting the architecture reduces, at best, to a reparameterized FFN, undermining the claimed modular advantages. The paper therefore does not meet the bar for a sound research contribution, and the REJECT verdict should stand.","tokens_in":10304,"tokens_out":9321,"duration_ms":83100,"concrete_test":"Re-derive Eq. 21 under the definition of B1l(E) in §4.2. Instantiate a minimal counterexample: N=2, |E|=1, d=1, with B1l(E)=[0;1]^T (a valid N×|E| matrix with distinct rows). Show that Cl = ReLU(Hl W + B1l(E)) W_V + b2 cannot be expressed as ReLU(Hl W1 + b1) W2 + b2 for any scalar bias b1, because the two token rows have different thresholds. If the authors intend B1l(E) to be broadcast from a |E|-dimensional vector, the test is to exhibit that broadcast operation explicitly and correct the shape in Eq. 8–9; without such a clarification, the claimed equivalence is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 equates Eq. 21, ReLU(Hl W_{(Q,K,E)} + B1l(E)) W_{(V,E)} + b2l, with the standard FFN, ReLU(Hl W1 + b1) W2 + b2 (Eq. 22). In a standard FFN, b1 is a per-hidden-unit vector broadcast across the N token rows. But §4.2 defines B1l(E) ∈ R^{N×|E|} and says it is implemented by an MLP applied to each knowledge-entry embedding, which can only produce a per-entry scalar, not an N-dependent row. If B1l(E) is literally a full matrix with distinct rows, the bias varies per token and Eq. 21 is strictly more expressive than an FFN, so the claimed closure fails for any sequence with N>1. If instead B1l(E) is meant to be a single vector tiled across rows, the proof requires an unstated broadcast/rank-1 constraint and the N×|E| notation is wrong. Thus the central 'rigorous mathematical derivation' is invalid as written. Separately, even a corrected representational equivalence would not imply 'identical performance under joint training': equivalence of function classes says nothing about optimization dynamics, local minima, or generalization, so that inference is a non-sequitur.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a modular Transformer in which the FFN sublayer is replaced by a generalized cross-attention over a globally shared knowledge base E, using ReLU sparsity, a knowledge-specific threshold B1_l(E), and an output bias b2_l. It claims to prove in Section 5 that the standard FFN is a specialized case (a 'closure') of this mechanism when E is static, and it infers that joint training of E yields functional equivalence and identical performance on any task. Empirical evaluation is deferred to future work.","tokens_in":10561,"tokens_out":8108,"duration_ms":73388,"significance":"The motivation is reasonable and the architectural idea is coherent: exposing knowledge in a shared module is a natural response to interpretability and adaptability concerns, and the complexity comparison in Table 2 is useful. The paper explicitly acknowledges its theoretical scope. If the equivalence were proven, it would formalize the existing key-value memory interpretation of FFNs. However, the central derivation is not valid as written: it contains an undischarged dimension mismatch, an unproved matrix-factorization step, and an unsupported inference from representational equivalence to training performance. The paper therefore does not currently provide the rigorous validation it claims.","major_comments":[{"comment":"Section 4.2 defines B1_l(E) as an N×|E| matrix, but the reduction to an FFN in Eq. (21) requires the bias term to be a single |E|-dimensional vector broadcast over the N token rows, as in Eq. (22). Since B1_l(E) is a function of E alone and is implemented by an MLP over knowledge entries, it cannot produce row-dependent (N-dependent) thresholds. The derivation needs either a corrected definition B1_l(E) ∈ R^{|E|} with an explicit broadcast, or an explanation of how the N dimension arises from E. As written, the FFN equivalence fails for N>1.","section":"§4.2 / §5.1 (Eqs. 8–9, 21)"},{"comment":"The statement 'By setting W1 = W_(Q,K,E)' presupposes the existence of parameters such that an arbitrary FFN first-layer weight can be written as W_Q (E W_K)^T / sqrt(dk). This requires specific rank and dimension conditions (e.g., |E| = d_ff and d_k ≥ |E|, or some factorization guarantee) and also requires a shared E to simultaneously represent all layer-specific W2^l because E is shared across layers in the modular architecture. None of these conditions are stated or proved, so the claimed closure is not established.","section":"§5.2 (Eqs. 23–26)"},{"comment":"The assertion 'we expect identical performance on any task under this joint training regime' is a non-sequitur. A representational equivalence of function classes says nothing about optimization trajectories, local minima, or generalization, and the derivation itself assumes a static E, which is inconsistent with joint training where E is updated. This claim should be removed or replaced by a precise statement about function classes, with any performance claim backed by experiments.","section":"§5.2 / §6.1"},{"comment":"The entity 'Implicit E' is never defined independently of the folding operation, so it cannot serve as the object that 'the FFN performs implicit knowledge retrieval from.' The paper should either give a concrete construction of Implicit E from a standard Transformer's FFN weights, or treat the folding only as a parameter-space identity for a fixed E.","section":"Eq. (11) / §5.3"},{"comment":"The paper contains no experiments, and the theoretical derivation is the only support for the central claim. For a cs.LG submission, this would be acceptable only if the theory were airtight; given the issues above, the lack of empirical validation is a major gap. A small-scale verification of the equivalence and of the modular architecture would be necessary in any revision.","section":"§6.2"}],"minor_comments":[{"comment":"The word 'layer-specifc' is a typo; please fix.","section":"Abstract / §1"},{"comment":"'facilitatesefficient' is missing a space; please correct.","section":"§2.2"},{"comment":"Writing B1_l(E) = B1_l(E) is tautological and does not perform a folding; consider removing or reformulating.","section":"§5.1, Eq. (18)"},{"comment":"The notation d_ff is typeset inconsistently (e.g., 'O(N ddf f)'); use a single consistent symbol throughout.","section":"§6.3 / Table 2"},{"comment":"The phrase 'IF ... THEN' logic is informal; since the paper claims rigor, please define B1_l(E) precisely, including input, output dimension, and parameterization, in a displayed equation.","section":"§4.2"}],"recommendation":"reject","confidential_remarks":"The contribution is a conceptual architecture with a theorem that is not proven. The dimension mismatch and the unproved factorization are not cosmetic; they are the core of the claimed validation. The 'identical performance' sentence would be misleading if published. I would not consider a major revision unless the authors replace the equivalence claim with a precise representational statement and add experiments, at which point the paper would be substantially different."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nHere's my read on arXiv:2501.00823.\n\nThe paper's central claim—that a standard Transformer FFN is a specialized case of the proposed generalized cross-attention—fails as written. The stress test is right: B1l(E) is defined as an N×|E| matrix in Section 4.2, but the comparison to the standard FFN in Section 5.2 requires a bias vector broadcast across the N rows. If B1l(E) is genuinely a full matrix, the per-token bias makes the cross-attention strictly more expressive than an FFN, so the closure doesn't hold for sequences longer than one. If it's meant to be a vector, the notation is wrong and the proof omits the broadcast constraint. Either way, the \"rigorous mathematical derivation\" is invalid as written.\n\nThe training-equivalence jump is also a non-sequitur. Even a correct representational equivalence says nothing about optimization dynamics, local minima, or generalization. The paper explicitly says \"we expect identical performance on any task under this joint training regime\"—that's not derived, and there are no experiments to test it.\n\nWhat the paper does well: it writes clearly, gives a fair literature review, and the complexity analysis in Table 2 is straightforward. The framing of a globally shared knowledge base with layer-specific views is a sensible research direction, and the authors are honest about having no empirical validation. They also appropriately cite Geva et al.'s key-value memory interpretation, which effectively undercuts the novelty of the FFN-as-attention result.\n\nThere's a further problem the stress test didn't mention: the folded weight Wl(Q,K,E) = WlQ(E WlK)^T / sqrt(dk) has rank at most dk. In typical Transformers, d_ff is much larger than dk (e.g., 4d vs 128), so the modular architecture cannot represent arbitrary FFN weight matrices. Sharing a single E across all layers adds another constraint that standard FFNs don't have. So even a corrected proof of single-layer equivalence wouldn't establish equivalence for the full network.\n\nWho is this for? A reader interested in the high-level idea of modular knowledge bases might skim Section 3, but the technical foundation is not sound. My recommendation is to desk reject. If the authors fix the dimension issue, address the rank limitation, and provide at least one experiment, it could be worth another look, but as it stands the paper overstates both novelty and validation.\n\nBest,\n[Your name]","headline":"The equivalence proof that the paper's whole argument rests on has a dimension mismatch in the bias term, and the training-equivalence conclusion is an unsupported leap; the paper is clearly written but not technically sound.","tokens_in":11109,"tokens_out":4691,"would_cite":false,"duration_ms":41826,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that a standard Transformer's feed-forward network is a closed form of generalized cross-attention over a static shared knowledge base, so the two architectures are functionally equivalent under joint training.","keywords":["decoder-only Transformer","feed-forward network","generalized cross-attention","shared knowledge base","knowledge retrieval","modular architecture","key-value memory","functional equivalence"],"falsifier":"Train a standard decoder-only Transformer and the modular architecture with a jointly trained shared knowledge base on the same corpus and random seed; if their training or test losses diverge materially on any task, the claimed identical performance under joint training is false. A quicker check is to inspect the learned thresholding term: if it is different for different token positions rather than a single bias applied everywhere, the folded formula ceases to match a standard feed-forward network.","tokens_in":10037,"feed_emoji":"🧩","tokens_out":11539,"duration_ms":94753,"temperature":0.7,"pith_summary":"The paper tries to establish that the feed-forward network (FFN) of a standard Transformer is not a separate nonlinear black box but the closed form of a generalized cross-attention mechanism retrieving from a fixed, implicit knowledge base. On that basis it proposes a modular architecture in which every layer queries one globally shared knowledge base through layer-specific projections, and it proves that when that base is static and trained jointly with the model, the retrieval computation folds into exactly the standard FFN equation. If the proof is correct, the modular design should match a standard Transformer's performance in joint training while making knowledge explicitly addressable, and FFNs gain a formal role as implicit knowledge-retrieval layers. The paper positions this as a theoretical foundation and defers external knowledge bases and empirical validation to future work.","feed_headline":"Every FFN layer is a folded knowledge-base lookup","feed_subtitle":"If the proof holds, knowledge and reasoning can be separated without changing what a Transformer computes.","key_machinery":"The engine of the argument is weight folding: when the knowledge base $E$ is held fixed, the generalized cross-attention update $\\mathrm{ReLU}((H_l W_Q^l)(E W_K^l)^\\top / \\sqrt{d_k} + B_1^l(E)) (E W_V^l) + b_2^l$ collapses into a two-layer affine map with a nonlinearity, structurally identical to a standard FFN. The paper calls this a closure because the cross-attention, with $E$ fixed, becomes a function of the query $H_l$ alone. The generalized attention itself uses ReLU instead of softmax for sparse retrieval, a knowledge-specific threshold $B_1^l(E)$ as an 'if' condition, and a transformation bias $b_2^l$ to connect the query and knowledge embedding spaces.","core_discovery":"The central claim is the closure identity $\\mathrm{FFN}(H_l) = \\mathrm{CrossAttention}(H_l, E_{\\mathrm{implicit}})$. With $E$ fixed, the key and value projections fold into $W_{(K,E)}^l = E W_K^l$ and $W_{(V,E)}^l = E W_V^l$, the query-key product folds into $W_{(Q,K,E)}^l = W_Q^l (W_{(K,E)}^l)^\\top / \\sqrt{d_k}$, and the thresholding term becomes a bias, producing $\\mathrm{ReLU}(H_l W_1^l + b_1^l) W_2^l + b_2^l$, the standard FFN. Because a static knowledge base turns generalized cross-attention into the FFN, the paper concludes that under joint training its modular architecture is functionally equivalent to a standard Transformer and should perform identically on any task. This equivalence is offered as rigorous validation of the modular design and as a formal basis for interpreting FFNs as performing implicit knowledge retrieval.","pith_inferences":["A concrete recipe the paper does not pursue: factor the FFN weights of a pretrained Transformer into a shared embedding matrix and layer-specific projections, then update only the embedding matrix to inject new knowledge without touching the reasoning layers.","Because the knowledge base is shared across layers while standard FFN weights are not, the equivalence requires the shared store to be expressive enough for every layer; a testable prediction is that the minimal store size grows with depth and task diversity.","The equivalence is about expressive power, not optimization: an empirical check would initialize the modular model from a standard Transformer by the folding construction and compare fine-tuning trajectories, which the paper does not attempt."],"forward_implications":["Under joint training with a static knowledge base, the modular architecture should match a standard Transformer's performance on any task, because the FFN is a closure of the generalized cross-attention.","Feed-forward networks should be understood as performing implicit, context-dependent knowledge retrieval from a compressed store rather than as arbitrary nonlinear transformations.","The folded weights of each FFN layer correspond to a layer-specific view of one shared knowledge source, giving a formal explanation for why different layers encode different levels of abstraction.","The closure argument gives a formal basis for the key-value memory interpretation of FFNs, connecting that line of work to cross-attention-based retrieval.","The size of the implicit knowledge store explains part of the parameter burden of Transformers, motivating externalizing knowledge into a separate module to scale knowledge without scaling reasoning."],"supporting_citations":[{"why":"defines the standard Transformer block and the FFN equation that the derivation targets.","marker":"[17]"},{"why":"proposes the key-value memory interpretation of FFNs that this paper claims to formalize.","marker":"[8]"},{"why":"describes retrieval-augmented generation, the baseline for the adaptability limitations the modular design addresses.","marker":"[12]"},{"why":"introduces a shared knowledge base approach that satisfies global sharing but not layer-specific views.","marker":"[4]"},{"why":"tokenizes layer-specific weights as compressed knowledge, violating global sharing and serving as a contrast architecture.","marker":"[13]"},{"why":"provides empirical evidence of layer-specific encoding that the equivalence is said to align with.","marker":"[9]"}],"fun_headline_variants":["FFN layers are cross-attention with a static knowledge base","New proof: every FFN is a special case of cross-attention","Modular Transformers match standard ones, and here's the math","Decoupling knowledge and reasoning: FFN is just attention in disguise","One equation shows FFN is equivalent to cross-attention"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The equivalence depends on the thresholding term before the activation being identical for every token in the sequence so it can fold into a per-layer bias, and on the further assumption that two architectures able to compute the same functions will also learn to the same performance.","fun_headline_variants_meta":{"raw":{"variants":["FFN layers are cross-attention with a static knowledge base","New proof: every FFN is a special case of cross-attention","Modular Transformers match standard ones, and here's the math","Decoupling knowledge and reasoning: FFN is just attention in disguise","One equation shows FFN is equivalent to cross-attention"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001159,"raw_usage":{"total_tokens":4776,"prompt_tokens":895,"completion_tokens":3881,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":3792}},"tokens_in":511,"tokens_out":3881,"duration_ms":28907,"temperature":1.0,"reasoning_tokens":3792,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:41:42.556461+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a standard decoder-only Transformer and the modular architecture with a jointly trained shared knowledge base on the same corpus and random seed; if their training or test losses diverge materially on any task, the claimed identical performance under joint training is false. A quicker check is to inspect the learned thresholding term: if it is different for different token positions rather than a single bias applied everywhere, the folded formula ceases to match a standard feed-forward network.","supporting_citations":[{"cited_title":"Retrieval-augmented generation for knowledge-intensive nlp tasks","cited_arxiv_id":null,"evidence_quote":"describes retrieval-augmented generation, the baseline for the adaptability limitations the modular design addresses."},{"cited_title":"Decouple knowledge from parameters for plug-and-play language modeling","cited_arxiv_id":null,"evidence_quote":"introduces a shared knowledge base approach that satisfies global sharing but not layer-specific views."}],"review_version":1}