{"id":"7b3a3805-65f4-43b5-bc67-e69d45161f86","arxiv_id":"2605.24330","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":1,"one_line_summary":"Interdomain Attention integrates SSMs into attention via finite feature maps and basis projections to enable query-conditioned attention over fixed states, showing gains over SSM baselines and matching softmax at 1.3B scale with length-flat scaling.","lead":"The paper proposes Interdomain Attention, which uses kernel approximations to let queries attend to a compressed fixed-size state maintained by an SSM recurrence. This hybrid aims to combine attention's query conditioning with SSM efficiency for long-context language modeling.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"No significant objection identified","rationale":"Reader correctly flags the derivation-to-implementation gap as the weakest assumption but notes the review was abstract-only. With no full text equations or tables accessible here to inspect further, no additional load-bearing concern can be substantiated.","tokens_in":1789,"tokens_out":201,"duration_ms":23499,"concrete_test":"Reproduce the 125M-scale run from the paper using the exact (non-relaxed) feature-map construction described in the derivation; if validation perplexity matches the reported relaxed version within 1%, the relaxation does not materially alter the result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The provided abstract and reader's summary describe a kernel-derived construction whose scalable form is explicitly a learned relaxation. Without the full equations, implementation details, or ablations from the manuscript, no internal inconsistency or unsupported assumption can be isolated that would falsify the central empirical claim at matched state budget.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript proposes Interdomain Attention to integrate state-space models into attention layers via kernel methods: an attention kernel is approximated by a finite feature map, keys and values are projected onto basis functions maintained by a single SSM recurrence, and each query attends over the resulting compressed coefficients through its own feature map. The scalable implementation is presented as a learned relaxation of this exact construction. In autoregressive language-modeling experiments on FineWeb-Edu with models ranging from 125M to 1.3B parameters at matched recurrent-state budget, the method is reported to improve over an SSM token mixer at every scale, to surpass a same-recipe softmax baseline at 1.3B on validation perplexity and an eight-task commonsense suite, and to retain the length-flat extrapolation behavior of its fixed-state core out to 3.5\times the training context. Ablations are said to attribute the gains primarily to the query-conditioned projection.","tokens_in":1850,"tokens_out":505,"duration_ms":52740,"significance":"If the empirical results prove robust, the work supplies a kernel-derived route for injecting content-based addressing into fixed-size recurrent states, offering a hybrid motif that could improve both efficiency and extrapolation in long-context modeling. The explicit separation between the exact derivation and its learned relaxation, together with the reported length-flat behavior, would constitute a concrete architectural contribution if the supporting evidence is made reproducible.","major_comments":[{"comment":"Abstract (empirical study paragraph): the central performance claims—outperformance of the SSM mixer at all scales and of the softmax baseline at 1.3B—are presented without any description of training procedures, hyperparameter search protocol, statistical significance testing, or data filtering. Because these claims constitute the primary evidence for the method’s utility, the absence of such details renders the results uninspectable and load-bearing for the manuscript’s conclusions.","section":"Abstract"},{"comment":"Abstract (derivation and relaxation sentence): the text states that the scalable layer is a learned relaxation of the kernel-derived construction, yet provides neither the exact equations for the finite-feature-map projection onto the SSM basis nor the precise form of the relaxation. Without these, it is impossible to determine how closely the implemented layer follows the claimed query-conditioned attention over compressed coefficients, which is the mechanistic justification for the reported gains.","section":"Abstract"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the detailed and constructive report. The two major comments identify important gaps in the abstract's presentation of experimental details and the derivation. We address each below and will revise the manuscript accordingly to improve clarity and inspectability while preserving the core contribution.","responses":[{"response":"We agree that the abstract's empirical claims would benefit from greater transparency. In the revised manuscript we will expand the experimental setup section (currently Section 5) to explicitly detail the FineWeb-Edu preprocessing and filtering steps, the hyperparameter search protocol (including learning rate schedules, batch sizes, and optimizer choices), the number of random seeds used, and any statistical testing performed. Because abstracts have strict length limits, we will add a concise pointer sentence in the abstract directing readers to this expanded section and will include a short methods summary paragraph immediately after the abstract in the camera-ready version.","revision_made":"yes","referee_comment":"[Abstract] Abstract (empirical study paragraph): the central performance claims—outperformance of the SSM mixer at all scales and of the softmax baseline at 1.3B—are presented without any description of training procedures, hyperparameter search protocol, statistical significance testing, or data filtering. Because these claims constitute the primary evidence for the method’s utility, the absence of such details renders the results uninspectable and load-bearing for the manuscript’s conclusions."},{"response":"The exact finite-feature-map construction, the projection of keys/values onto the SSM-maintained basis functions, and the learned relaxation (including the specific parameterization that replaces the exact kernel with a trainable module) are derived in Section 3 and implemented in Section 4. We will revise the abstract sentence to include a parenthetical reference to these sections and, space permitting, a one-line high-level equation sketch. This will make the connection between the kernel derivation and the deployed layer explicit without altering the abstract's length substantially.","revision_made":"partial","referee_comment":"[Abstract] Abstract (derivation and relaxation sentence): the text states that the scalable layer is a learned relaxation of the kernel-derived construction, yet provides neither the exact equations for the finite-feature-map projection onto the SSM basis nor the precise form of the relaxation. Without these, it is impossible to determine how closely the implemented layer follows the claimed query-conditioned attention over compressed coefficients, which is the mechanistic justification for the reported gains."}],"tokens_in":1523,"tokens_out":514,"duration_ms":24157,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is a kernel-based route that projects attention features through an SSM recurrence so each query can attend over compressed coefficients rather than a growing KV cache. The dual feature map step plus the learned relaxation looks like a fresh construction not covered in the cited priors.\n\nThe work runs a 125M-to-1.3B autoregressive study on FineWeb-Edu at matched state budget and claims better validation perplexity than both an SSM mixer and a same-recipe softmax baseline at the top scale, plus gains on the commonsense suite. It also keeps the length-flat behavior out to 3.5 times training context. The ablations that flag the query-conditioned projection as the main driver are a useful check.\n\nThe soft spot is that the abstract supplies no training procedure, hyperparameter search, statistical tests, or data filtering details, so the performance numbers sit on uninspectable evidence. The central claim is explicitly a learned relaxation of the derivation, which makes it harder to judge how tightly the theory guides the results. If the full paper supplies those missing pieces and the numbers hold, the argument is coherent and the circularity burden stays low.\n\nThis is aimed at people working on hybrid attention-SSM layers who need fixed memory with some query dependence. A reader focused on efficient long-context modeling would get concrete value from the construction and the scaling data. It deserves a serious referee to examine the implementation and verify the claims.","headline":"The paper gives a kernel construction for query-conditioned attention inside fixed SSM states, with scaling gains reported at 1.3B but thin supporting details.","tokens_in":2327,"tokens_out":365,"would_cite":false,"duration_ms":28751,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Interdomain Attention recovers query-conditioned attention over a fixed recurrent state by projecting keys and values onto SSM basis functions.","keywords":["Interdomain Attention","state space models","kernel methods","attention mechanisms","recurrent state","feature maps","language modeling","fixed-size state"],"falsifier":"A 1.3B-parameter run with the same training recipe in which Interdomain Attention fails to beat the softmax baseline on validation perplexity or on the eight-task commonsense suite.","tokens_in":2712,"feed_emoji":"","tokens_out":731,"duration_ms":46150,"temperature":0.7,"pith_summary":"The paper tries to establish a hybrid attention mechanism that lets queries perform content-based matching over a compressed context without a growing KV cache. It approximates the attention kernel with a finite feature map, projects the resulting keys and values onto basis functions kept by one SSM recurrence, and lets each query recover attention to those coefficients through its own feature map. A sympathetic reader would care because the construction keeps the fixed-state length independence of SSMs while adding query-specific addressing that standard SSM mixers lack. Experiments at 125M to 1.3B parameters on FineWeb-Edu show consistent gains over pure SSM mixers and, at the largest scale, over a matched softmax baseline on both perplexity and commonsense tasks. Ablations point to the query-conditioned projection step as the main driver.","feed_headline":"Interdomain Attention beats softmax at 1.3B with fixed SSM state","feed_subtitle":"Projecting keys and values onto SSM basis functions recovers query-conditioned attention over compressed context while keeping length-flat p","key_machinery":"Interdomain Attention, the mechanism that projects finite-feature-map key and value representations onto SSM-maintained basis functions so that queries can attend to the resulting compressed coefficients.","core_discovery":"Interdomain Attention integrates an SSM into an attention module through kernel methods: an attention kernel is approximated by a finite feature map, the resulting key features and values are projected onto a shared set of basis functions maintained by a single SSM recurrence, and each query attends to the compressed coefficients through its own feature map, recovering query-conditioned attention over a fixed-size state. The scalable layer is a learned relaxation of this derivation.","pith_inferences":["The same projection idea could be applied to other kernel approximations or to recurrent cores besides the SSM used here.","Hybrid layers that mix standard attention blocks with Interdomain Attention blocks might further improve long-context efficiency.","The fixed-state design removes the need for a growing KV cache while retaining a form of content-based addressing."],"forward_implications":["At matched recurrent-state budget, Interdomain Attention improves on an SSM token mixer at every scale from 125M to 1.3B in autoregressive language modeling on FineWeb-Edu.","At 1.3B parameters it surpasses a same-recipe softmax baseline on validation perplexity and on the eight-task commonsense suite.","The model inherits the length-flat behavior of its fixed-state core out to 3.5 times the training context.","Ablations show that the query-conditioned projection step accounts for most of the observed gain."],"fun_headline_variants":["Interdomain Attention approximates kernels to project KV onto SSM bases","Queries attend via feature maps to SSM-compressed attention coefficients","Interdomain Attention achieves fixed-state query attention with SSM recurrence","Learned relaxation of kernel SSM attention scales length-flat at 1.3B"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"An attention kernel approximated by a finite feature map, when its keys and values are projected onto SSM basis functions, still lets each query recover query-conditioned attention to those compressed coefficients.","fun_headline_variants_meta":{"raw":{"variants":["Interdomain Attention approximates kernels to project KV onto SSM bases","Queries attend via feature maps to SSM-compressed attention coefficients","Interdomain Attention achieves fixed-state query attention with SSM recurrence","Learned relaxation of kernel SSM attention scales length-flat at 1.3B"]},"model":"grok-4.3","cost_usd":0.006311,"raw_usage":{"total_tokens":2979,"prompt_tokens":694,"num_sources_used":0,"completion_tokens":70,"cost_in_usd_ticks":63112000,"prompt_tokens_details":{"text_tokens":694,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2215,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":694,"tokens_out":70,"duration_ms":31972,"temperature":1.0,"reasoning_tokens":2215,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-30T13:56:14.389164+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A 1.3B-parameter run with the same training recipe in which Interdomain Attention fails to beat the softmax baseline on validation perplexity or on the eight-task commonsense suite.","supporting_citations":[],"review_version":1}