{"id":"79cb7f1c-debd-4f4e-9acb-5c18b8d3fb2a","arxiv_id":"2508.17734","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Concentrating all feed-forward-network capacity into the middle 70% of a Transformer's layers, at fixed total parameter count, outperforms the standard uniform layout across model sizes and language tasks.","lead":"This paper tests whether certain 'middle' parts of a language model matter more than the parts at the edges, by moving all of a model's feed-forward processing power into the middle 70% of its layers while keeping the total number of parameters unchanged. Across models from 285 million to 1.2 billion parameters trained from scratch, the concentrated layout beats the standard uniform layout on a range of language tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"70% middle configuration may be an in-sample selection; without held-out tasks and error bars, 'consistently outperforms' is not established.","rationale":"The reader's verdict was UNVERDICTED due to missing full text, with two fragilities: the optimization-dynamics confound and the potential in-sample selection of the 70% ratio. I agree that the optimization-dynamics confound is real and limits the 'importance' interpretation, but I judge the selection-bias concern to be the single most load-bearing threat to the abstract's specific empirical claim. The claim names a precise configuration (70% middle) and asserts consistent outperformance; if that configuration was chosen by peeking at the evaluation tasks, the claim is circular and the exact percentage is likely overfit. The proposed check—disclosing the selection protocol and, if needed, re-evaluating on a true hold-out set—would settle this. Without such evidence, the paper should not be accepted as established; the verdict should be CONDITIONAL on providing this robustness evidence. I do not move to REJECT because the abstract's approach is plausible and the concern is addressable; the full text may already include the necessary controls. Thus conditional acceptance, rather than outright rejection or unverified, is the appropriate stance.","tokens_in":877,"tokens_out":4296,"duration_ms":51807,"concrete_test":"Read the full methods to determine whether the 70% consecutive-middle configuration was pre-registered/fixed a priori or chosen after exploratory runs on the same downstream tasks. If it was chosen in-sample, require a strict hold-out: re-run the chosen configuration and the uniform baseline on a set of evaluation tasks never used during any configuration search, and verify the ordering holds. Independently, run both configurations with at least 5 random seeds per model size and report mean and standard deviation on each task. If the 70% middle model does not beat uniform by more than one standard deviation on a majority of tasks, or if the ordering flips on truly held-out tasks, the central claim is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that concentrating FFNs in the middle 70% of consecutive layers consistently outperforms standard configurations across model sizes while fixing total parameter count. The most load-bearing risk is that the 70% figure itself may have been selected after inspecting the same downstream tasks used for evaluation. The abstract reports no selection protocol, no error bars, no seed counts, and no held-out task split. If 70% and the 'consecutive middle' placement were chosen by comparing several alternatives on the evaluation tasks, then the reported superiority is a selected maximum, not a principled architectural finding. This is a direct threat to the empirical ordering, not merely an interpretive caveat. A secondary, distinct limitation is that allocating all FFN capacity to some layers and removing it from others changes optimization dynamics (gradient flow, activation statistics, effective depth), so the comparison may partly measure trainability rather than layerwise contribution to task performance. However, the selection-bias concern is more load-bearing because it undermines the specific numeric recipe (70% middle) asserted in the abstract.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes to study the layerwise importance of feed-forward networks (FFNs) in Transformer language models by an architectural intervention: while keeping the total parameter count fixed, it increases the FFN dimensions in some layers and completely removes FFNs from other layers. Models are trained from scratch across three sizes (285M, 570M, 1.2B parameters) and three depths (12, 24, 40 layers). The central claim is that concentrating FFNs in 70% of the consecutive middle layers consistently outperforms standard configurations on multiple downstream tasks.","tokens_in":1033,"tokens_out":1904,"duration_ms":24540,"significance":"If substantiated, the claim would be a practical and conceptual contribution: it would show that the placement of FFN capacity is a first-order architectural decision, not merely a uniform default. Training from scratch is a strength: it avoids confounding the analysis with properties of pre-trained checkpoints. The parameter-matched comparison across sizes and depths is also a sensible design. However, the current evidence as presented is insufficient to establish the headline claim: no statistical variability, seed counts, significance tests, or selection protocol are reported. The significance therefore remains conditional on additional experimental evidence.","major_comments":[{"comment":"The abstract states that \"concentrating FFNs in 70% of the consecutive middle layers consistently outperforms standard configurations,\" but reports no number of seeds, no error bars, and no significance tests. For a claim of consistent superiority across three model sizes and multiple tasks, at least per-condition mean and variance over multiple training runs, plus a statistical comparison (e.g., paired test per task) are needed. Without these, the observed ordering could be within run-to-run noise.","section":"Abstract"},{"comment":"The selection of the 70% concentration is not disclosed. If the 70% ratio and the \"consecutive middle\" placement were chosen after evaluating several alternatives on the same downstream tasks used in the final comparison, then the reported result is a selected maximum, not a principled architectural finding. Please report the selection protocol: whether 70% was pre-registered, determined on a development set separate from the evaluation tasks, or found through a search. If a search was used, the evaluation tasks must be treated as held out, or the reported numbers must be corrected for selection.","section":"Abstract (method description)"},{"comment":"Moving all FFN parameters out of some layers and into others changes optimization dynamics in addition to FFN capacity placement. Gradient flow, activation statistics, and effective representational bottlenecks in FFN-free layers differ from the uniform baseline. The comparison may therefore partly measure trainability or optimization difficulty, not solely the layerwise importance of FFNs. The paper should include diagnostics or ablations that separate these effects, for example by comparing the proposed layouts with matched parameter counts but different optimization curricula, or by reporting gradient/activation statistics across layers.","section":"Abstract (experimental design)"}],"minor_comments":[{"comment":"The phrase \"consecutive middle layers\" is ambiguous for even layer counts (e.g., 12 or 24 layers). Please define precisely whether \"middle\" is centered, rounded, and how contiguous intervals are chosen.","section":"Abstract"},{"comment":"The downstream tasks are not named. Listing the tasks and the evaluation metric would help readers judge whether the claim of \"multiple downstream tasks\" is broad enough to support the architectural conclusion.","section":"Abstract"},{"comment":"The paper would benefit from stating the pretraining data size, token budget, and hyperparameter settings to confirm that the three model sizes are trained under comparable conditions.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript as currently visible is essentially an extended abstract: the full text is not available in the provided material, so my assessment is necessarily based on the abstract and the reported claims. The central suggestion is interesting and worth pursuing, but the empirical support is not yet convincing. The main editorial risk is that the 70% result may be an artifact of in-sample selection. I would like to see the full experimental protocol before making a final recommendation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"We only have the abstract here—the full text didn't come through—so this is a letter about the abstract, not the paper. The experimental idea is genuinely good: hold total parameter count fixed, move all FFN capacity into a middle block of layers, remove FFNs elsewhere, and train from scratch. That cleanly separates 'where capacity sits' from pretrained-checkpoint artifacts, and it's a real improvement over existing importance analyses that just probe off-the-shelf models. If the claimed ordering holds up, it's a cheap and actionable architectural rule.\n\nThe soft spots are statistical and methodological. The abstract says the 70%-middle setup 'consistently outperforms' across three sizes and multiple tasks, but there are no error bars, no seed counts, no significance tests, and no description of how 70% was chosen. That's not a pedantic complaint: if 70% was picked after trying a few ratios on the same downstream tasks, the result is a selected maximum, not a finding. The abstract gives no selection protocol, and the stress-test note is right that this is load-bearing. A second, distinct caveat is that removing FFNs from some layers changes optimization dynamics beyond just capacity placement—gradient flow, activations, effective depth—so the comparison might partly measure trainability rather than layerwise importance. That doesn't sink the design, but it should be discussed.\n\nNone of this means the paper is wrong. A full methods section with held-out tasks and multiple seeds could clear up the statistical questions. As it stands, the abstract is an intriguing pilot result, not a verified claim. The direction is worth a serious referee.\n\nRecommendation: send it to peer review. The design is novel enough and the potential payoff big enough that it deserves referee time. For my own work, I wouldn't cite the 70% claim yet.","headline":"Clever experimental design that deserves a proper look, but the headline 70% figure is unsubstantiated without error bars and selection disclosure.","tokens_in":557,"tokens_out":545,"would_cite":false,"duration_ms":22373,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that placing all feed-forward network capacity in a contiguous middle block covering 70% of layers, at a fixed total parameter count, consistently beats the standard uniform Transformer configuration across model sizes and","keywords":["layerwise importance","feed-forward networks","Transformer architecture","parameter allocation","pretraining from scratch","downstream task evaluation","middle layers","language models"],"falsifier":"Run the same reallocation with the 70% FFN block placed in the earliest layers instead of the middle; if that configuration matches the middle-block result on the same downstream tasks, the claim that middle layers are the important ones fails. Alternatively, sweep the concentration ratio (e.g., 0%, 50%, 70%, 90%, 100%) on a held-out task; the claim requires 70% to be at or near the optimum.","tokens_in":727,"feed_emoji":"🧠","tokens_out":3577,"duration_ms":39722,"temperature":0.7,"pith_summary":"The paper asks whether feed-forward networks in a Transformer matter equally at every layer, and answers no. Keeping total parameter count fixed, the authors train models from scratch with FFNs removed from some layers and enlarged in others, across three model sizes and three depths. They report that placing all FFN capacity in a contiguous block covering 70% of the middle layers beats the standard uniform configuration on multiple downstream tasks. If true, where FFN capacity sits is an architectural decision in its own right, not just how much capacity exists.","feed_headline":"Middle 70% FFN placement outperforms uniform Transformers","feed_subtitle":"Same parameter count, trained from scratch, three model sizes: the middle-heavy layout consistently wins downstream tasks.","key_machinery":"The experimental intervention is layerwise FFN reallocation: each layer is either FFN-free (attention and residual connections only) or has its FFN hidden dimension enlarged, with total parameter count held constant. This isolates the position of FFN capacity as the independent variable, and the paper's index of interest is downstream task performance as a function of where the enlarged FFN block is placed.","core_discovery":"The central claim is that layerwise FFN importance is non-uniform and peaks in the middle of the network. The evidence is a controlled parameter-allocation experiment: at a fixed total parameter budget, removing FFNs entirely from the first and last layers while increasing FFN dimension in the middle 70% of layers improves downstream task performance compared with the standard Transformer. This holds for models of 285M, 570M, and 1.2B parameters with 12, 24, and 40 layers, all trained from scratch, and is the paper's central discovery.","pith_inferences":["We would test whether contiguity matters by spreading the same 70% of layers non-contiguously; the paper's design does not separate a 'middle' effect from a 'consecutive' effect.","We would also test whether the 70% ratio is stable across task families; if different ratios win on different tasks, the claim may point to a robust middle region rather than an exact figure.","A practical corollary is that FFN-free layers cost less compute per token, so the configuration may improve throughput as well as accuracy at equal parameter count.","Because the paper trains from scratch, the ranking may be specific to pretraining dynamics and not directly transfer to fine-tuned or instruction-tuned models."],"forward_implications":["If correct, Transformer designs can reallocate FFN parameters to the middle layers and get better performance at the same parameter count.","Early and late layers appear to function adequately without FFNs, so MLP computation can be concentrated away from the input and output ends.","The advantage of the 70% middle configuration means layerwise importance should be a factor in scaling studies, not just total depth and width.","The result gives a simple default for pretraining architectures: place roughly 70% of contiguous mid-layer capacity as FFNs."],"supporting_citations":[],"fun_headline_variants":["Middle 70% FFN concentration beats uniform Transformers","Put 70% of FFN capacity in the middle: wins","Layerwise shift: FFNs in middle outperform spread-out","Train from scratch: middle FFN layout is best","Consecutive middle FFNs top standard Transformer"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"Reallocating all FFN parameters out of some layers and into others leaves everything else about how the model trains unchanged, so the observed gains reflect where FFNs sit rather than how the altered architecture learns.","fun_headline_variants_meta":{"raw":{"variants":["Middle 70% FFN concentration beats uniform Transformers","Put 70% of FFN capacity in the middle: wins","Layerwise shift: FFNs in middle outperform spread-out","Train from scratch: middle FFN layout is best","Consecutive middle FFNs top standard Transformer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000255,"raw_usage":{"total_tokens":1356,"prompt_tokens":642,"completion_tokens":714,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":386,"completion_tokens_details":{"reasoning_tokens":634}},"tokens_in":386,"tokens_out":714,"duration_ms":8312,"temperature":1.0,"reasoning_tokens":634,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:44:28.795941+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same reallocation with the 70% FFN block placed in the earliest layers instead of the middle; if that configuration matches the middle-block result on the same downstream tasks, the claim that middle layers are the important ones fails. Alternatively, sweep the concentration ratio (e.g., 0%, 50%, 70%, 90%, 100%) on a held-out task; the claim requires 70% to be at or near the optimum.","supporting_citations":[],"review_version":1}