{"id":"2240f7fe-eaf8-45ea-8bcf-3cd2e2d5a3c7","arxiv_id":"2507.17728","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A 3B-parameter Mixture-of-Experts model with experts shared across layers matches or approaches larger models on several benchmarks at much lower storage cost.","lead":"Megrez2-Preview is a language model with 3 billion active and 7.5 billion stored parameters that performs competitively on math, code, and instruction benchmarks. It shares expert modules across layers to shrink the model for deployment on phones and other limited devices.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-layer expert sharing is never isolated: all benchmark comparisons use models with different data and post-training, so the observed results cannot establish that sharing itself preserves capacity.","rationale":"The reader's weakest-assumption analysis correctly identifies the central gap: the paper never tests whether cross-layer expert sharing preserves capacity. The strongest claim is not merely that Megrez2-Preview scores well, but that its architecture achieves those scores efficiently; that requires isolating the sharing mechanism from data scale, SFT, and RLVR. I read Table 1 in good faith and do not dispute the absolute numbers, but they cannot discriminate between the architecture and the training recipe. The proposed n=1 control is a direct, decisive test: it holds data, activated parameters, and post-training fixed while removing only the sharing. Because this missing control is a condition the authors could satisfy rather than a demonstrated falsehood, the reader's CONDITIONAL verdict remains appropriate. I did not make parameter-arithmetic ambiguity the primary concern because Section 3.4 is too underspecified to resolve it independently, though the report would benefit from a released configuration to verify the 7.5B stored-parameter count.","tokens_in":8469,"tokens_out":8858,"duration_ms":95603,"concrete_test":"Train a control model with the same Megrez2-Preview recipe (same data, sequence lengths, SFT, and RLVR) but with group size n=1 in Section 3.2, i.e., each of the 30 MoE layers has its own dedicated pool of 64 experts plus 4 shared experts per layer, keeping the number of activated experts per token unchanged. Compare Table 1 averages against Megrez2-Preview. If the n=1 control matches or exceeds Megrez2-Preview, the cross-layer sharing mechanism is not responsible for the reported results; if it is substantially worse, the sharing claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central architectural claim is that reusing one 64-expert pool across every three layers (Section 3.2, Section 3.4) reduces stored parameters while 'maintaining most of the model's capacity.' The only evidence offered is Table 1, which compares Megrez2-Preview against Qwen2.5, Qwen3, Gemma-3, Phi-4-mini, and GPT-4o-mini. Each of those baselines has its own architecture, pretraining corpus (Megrez2 uses 5T tokens plus a multi-stage recipe), SFT data, and RLVR procedure. None of them is a non-sharing MoE trained on the same data with the same activated-parameter budget. Consequently, the benchmark scores are consistent with the hypothesis that the 5-trillion-token corpus, the post-training pipeline, or the dense-layer-first design drives performance, and that cross-layer sharing contributes little or nothing. The report also provides no ablation over the sharing group size n, no n=1 control, and no comparison against a same-size MoE without parameter reuse. Without such a control, the unique contribution of the Megrez2 architecture is not established, even if the model's absolute benchmark numbers are genuine.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Megrez2, a Mixture-of-Experts (MoE) language model architecture whose main novelty is cross-layer expert sharing: every group of n consecutive transformer layers reuses a single pool of M experts while each layer keeps its own gating network and projection weights, and a pre-gated routing mechanism lets the router of layer i-1 select experts for layer i. The authors instantiate this architecture as Megrez2-Preview, a model with 3B activated and 7.5B stored parameters, pretrained on 5 trillion tokens and then refined with supervised fine-tuning and reinforcement learning with verifiable rewards. Evaluation in Section 5 compares Megrez2-Preview against several dense and MoE baselines on C-EVAL, MMLU-Pro, IFEval, MATH-500, GSM8K, HumanEval, and MBPP. The paper claims that cross-layer expert sharing reduces stored parameters while maintaining capacity, and that pre-gated routing enables memory-efficient expert loading and faster inference, making the model suitable for device-native deployment.","tokens_in":8740,"tokens_out":5452,"duration_ms":53551,"significance":"If the central empirical claim held, a 3B-activated, 7.5B-stored MoE that is competitive with 7B-8B dense models while reusing experts across layers would be a practically valuable result for on-device deployment. The architecture description in Eqs. (1)-(3) is simple, internally consistent, and clearly stated, and the evaluation uses external benchmarks with no evidence that hyperparameters were fitted to those benchmarks. However, the evidence as presented does not establish the unique contribution of cross-layer expert sharing: every comparison is against models with different architectures, training data, and post-training procedures, and the deployment-efficiency claims (latency, memory, throughput) are never measured. The core idea is plausible, but the report currently reads as a model card with architectural ambition rather than a validated architecture study.","major_comments":[{"comment":"The central claim that cross-layer expert sharing 'maintains most of the model's capacity' is not tested in isolation. All Table 1 baselines differ in architecture, pretraining corpus, and post-training recipe; none is a non-sharing MoE trained on the same data with the same activated-parameter budget. The reported results are therefore equally consistent with the hypothesis that the 5-trillion-token corpus, the three-stage pretraining, the SFT/RLVR recipe, or the dense-layer-first design drives performance, and that sharing itself contributes little. An ablation with group size n=1 (per-layer experts) or a same-data comparison against a standard MoE without parameter reuse is necessary to support the architectural attribution currently made throughout the abstract and conclusion.","section":"Sections 3.2 and 3.4, Eq. (2)"},{"comment":"The evaluation protocol is not reported. The paper does not state the sampling temperature, number of runs, prompt templates, or the exact computation of Pass@1 for HumanEval and MBPP; no error bars or variance estimates are given; and many baseline entries are missing (e.g., Qwen3-4B on MMLU-Pro, GSM8K, and HumanEval; Gemma-3-4B on C-EVAL and MMLU-Pro; GPT-4o-mini on MMLU-Pro and MBPP). This makes the 'competitive or superior' claim difficult to verify, and the presence of missing entries allows selective comparison across benchmarks. The authors should either complete the table or clearly state which baseline numbers were unavailable and why.","section":"Section 5 and Table 1"},{"comment":"The summary statements overreach the data. Megrez2-Preview trails Qwen2.5-7B on GSM8K (83.6 vs 91.6) and HumanEval (74.4 vs 84.8), and trails Qwen3-8B on IFEval (80.2 vs 83.0), GSM8K (83.6 vs 93.2), and HumanEval (74.4 vs 85.9). The claim that the model 'matches or even exceeds the performance of substantially larger models' is therefore not supported as a global statement; it should be restricted to the specific benchmarks and baselines where it actually holds, with the trade-offs stated plainly.","section":"Section 5, prose around Table 1"},{"comment":"The second headline advantage, pre-gated routing for 'memory-efficient expert loading and faster inference', is never evaluated. There are no latency, peak-memory, or throughput measurements, no device-deployment results, and no comparison against a non-pre-gated variant of the same model. Since deployment efficiency is a core motivation of the paper (the 'Impossible Triangle' in Section 1), the absence of any system-level evaluation is a substantive gap that should be filled with at least basic inference-cost measurements.","section":"Sections 3.3 and 5"}],"minor_comments":[{"comment":"The table header uses 'Megrez2' rather than 'Megrez2-Preview', which is the name used elsewhere in the paper; please make the naming consistent.","section":"Table 1"},{"comment":"Equation (3) evaluates the previous layer's router G_{i-1} at h_i, which is the input to layer i (i.e., the output of layer i-1 after gating). If pre-gating means the router of layer i-1 decides for layer i, the argument to G_{i-1} should be specified precisely (e.g., the hidden state before layer i-1's gating, or h_{i-1}), because as written the notation is ambiguous and could be read as a chicken-and-egg dependency.","section":"Section 3.3, Eq. (3)"},{"comment":"The role of the 4 shared experts is ambiguous: the paper does not state whether the same 4 shared experts are reused across all layers in a group or are per-layer, and Eq. (2) omits them entirely. Please clarify how the shared experts are counted in the 7.5B stored parameter total.","section":"Section 3.4"},{"comment":"The RL description would benefit from more detail: the paper says it uses a 'modified version of GRPO' with a PPO loss and GAE, but does not give the PPO clip range, advantage normalization, rollout count, or how the GRPO sampling framework and the PPO loss interact. These details are needed for reproducibility.","section":"Section 4.2.2"},{"comment":"There are several minor text and formatting issues: 'C-EVAL' is typeset inconsistently ('C-EVAL', 'C-EV AL', 'C-E VAL'), the phrase 'device-model trilemma' appears mid-sentence without definition, and 'Pass@1' is used without a definition or citation of the standard evaluation methodology.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"This is a technical report from a group that previously released the Megrez-3B series, and the tokenizer and training recipe are explicitly inherited from that series. That is normal for a model release, but the paper should disclose more explicitly how much of the pretraining data overlaps with the baselines' training data and whether the reported benchmark numbers were produced by the authors or taken from public leaderboards. The main risk is not circularity—the benchmarks are external and no constants appear to have been fitted to them—but attribution: the architecture's unique contribution is confounded by data and training recipe. The gaps are fillable (an n=1 ablation, a same-data MoE baseline, and system-level latency/memory numbers would materially strengthen the paper), so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a real model release, and the architecture story is coherent, but the paper's central scientific claim — that reusing an expert pool across layers preserves capacity — is not actually tested. The only evidence is a benchmark table comparing Megrez2-Preview against other models with different architectures, data, and post-training pipelines. No ablation over group size n, no non-sharing MoE baseline trained on the same 5T tokens, no n=1 control. So the unique contribution of the sharing mechanism is unestablished. That is the main soft spot, and the stress-test note lands.\n\nWhat is genuinely new: the specific combination of cross-layer expert sharing and pre-gated routing, instantiated in a trained 3B-activated/7.5B-stored model. The equations in Sections 3.2 and 3.3 are simple and internally consistent. The related work is honestly cited — DeepSeekMoE, Qwen1.5-MoE, and Pre-gated MoE are all acknowledged as sources. The benchmark numbers, if reproduced, are useful data points for on-device MoE design. The training recipe is described in enough detail that a competent lab could follow it.\n\nSoft spots, in proportion: first, the evaluation is a single table with missing entries (MMLU-Pro for Qwen3, GSM8K for Qwen3-4B, etc.), no error bars, and no stated evaluation protocol. That is thin even for a technical report. Second, the prose overstates the results: the abstract says 'competitive or superior', but Megrez2-Preview trails Qwen2.5-7B on GSM8K (83.6 vs 91.6) and Qwen3-8B on HumanEval, IFEval, and MATH-500. The table actually shows a mixed picture that can be read as 'competitive at smaller storage', which is a weaker but still interesting claim. Third, the 'novel' label is generous; the mechanism is a combination of existing ideas. Fourth, the report points to a GitHub URL but does not state what is released — no weights, no eval scripts, no variance estimates. That limits reproducibility.\n\nNone of these are disqualifying for a model-release document. A reader who wants a compact architecture description and a first look at a plausible on-device MoE will find value. But a reader who wants evidence that cross-layer sharing itself preserves capacity will not find it here.\n\nFor peer review: I would send this to a serious referee — it is a real model with enough novelty and relevance to deserve expert time. But I would expect the referee to demand an ablation against a same-data non-sharing MoE, a fuller evaluation protocol, and softened claims. As it stands, it is a solid technical report, not a decisive scientific result.\n\nVerdict: worth engaging, but read the table yourself before repeating the 'superior' claim.","headline":"A competent technical report with a genuinely trained model, but the key causal claim about cross-layer expert sharing is not experimentally isolated, and the evaluation is too thin to back the 'superior' language.","tokens_in":9315,"tokens_out":2522,"would_cite":false,"duration_ms":27679,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 7.5B-parameter Mixture-of-Experts model that activates only 3B parameters per token can match or beat substantially larger language models on understanding, math, and code, thanks to cross-layer expert sharing.","keywords":["Mixture-of-Experts","cross-layer expert sharing","pre-gated routing","lightweight language model","on-device deployment","parameter efficiency","reinforcement learning with verifiable rewards","Megrez2-Preview"],"falsifier":"Train a MoE with each layer owning its own expert pool (group size 1) on the same 5-trillion-token corpus with the same SFT and RLVR post-training, holding activated parameters equal; if that model matches or beats Megrez2-Preview, cross-layer sharing is not the source of the reported performance.","tokens_in":8286,"feed_emoji":"🧠","tokens_out":5889,"duration_ms":52954,"temperature":0.7,"pith_summary":"Megrez2 is an architecture for large language models that aims to resolve the trade-off among speed, accuracy, and cost for on-device deployment. Its central move is cross-layer expert sharing: the same pool of mixture-of-experts modules is reused across every three consecutive transformer layers, cutting the stored parameter count by roughly a factor of three while keeping 3B parameters active per token. The paper reports that Megrez2-Preview, the first model built on this architecture and trained on five trillion tokens, matches or beats substantially larger models on Chinese and English understanding, instruction following, mathematical reasoning, and code generation. If correct, the result shows that a small stored footprint need not cost accuracy, which would make high-capacity language models practical on memory-constrained devices.","feed_headline":"Shared experts let a 3B-active model outscore 8B rivals","feed_subtitle":"Megrez2-Preview reuses experts across layers to fit 7.5B parameters in a 3B-active model that still tops 8B rivals.","key_machinery":"The load-bearing object is the cross-layer expert-sharing group: Megrez2 partitions the transformer into groups of $n=3$ consecutive layers, and all layers in a group draw from one shared pool of 64 experts with their own gating networks, so the total parameter count falls by roughly $n$ while activated parameters per token stay constant. Pre-gated routing moves the gating computation to the preceding layer, so selected experts can be loaded into memory before they are used, and within a group an already-loaded expert can be kept in cache across layers. The dense-layer-first layout with a hidden width of 10,944 and an expert width of 1,408 sets the capacity the shared pools are asked to preserve.","core_discovery":"The paper's claim is that a mixture-of-experts language model can shed most of its stored parameters without shrinking its per-token compute by having successive layers reuse the same expert modules. Megrez2-Preview activates only 3B parameters through top-6 routing among 64 shared experts per group of layers, yet stores just 7.5B parameters total; against Qwen2.5-7B, Qwen3-8B, Gemma-3-4B, Phi-4-mini, and GPT-4o-mini it posts higher or tied scores on C-EVAL, MMLU-Pro, IFEval, MATH-500, GSM8K, HumanEval, and MBPP. The paper attributes this balance to the combination of cross-layer expert sharing, pre-gated routing that loads experts one layer ahead, a dense-layer-first 31-layer layout, and a three-stage training pipeline topped by reinforcement learning with verifiable rewards.","pith_inferences":["If the sharing mechanism is the true driver, the same principle could be applied to other reusable modules, such as attention projections, to compress models further; this is an editorial extension the report does not explore.","The reported results could instead reflect the 5-trillion-token corpus, the turn-level loss, or the RLVR recipe; an ablation isolating group size would settle which ingredient carries the gain.","A natural testable extension is to vary the group size $n$ from 1 to 6 at fixed data and compute: a performance cliff at larger $n$ would map the capacity-memory trade-off curve, which the report leaves implicit."],"forward_implications":["Device deployment becomes feasible at 7.5B stored parameters with 3B activated, a footprint that fits in a few gigabytes even before quantization.","Memory savings scale with group size: choosing larger $n$ cuts stored parameters further, at the cost of asking the same experts to serve more layers.","Pre-gated routing lets expert loading overlap with computation, which should translate to lower latency on bandwidth-limited hardware.","Competitive scores on math and code suggest the architecture retains the reasoning capacity that larger dense models pay for with far more parameters."],"supporting_citations":[{"why":"Supplies the fine-grained MoE structure and shared-expert idea that Megrez2 modifies.","marker":"[8]"},{"why":"Provides the pre-gating mechanism adopted for memory-efficient loading.","marker":"[13]"},{"why":"Source of the dense-layer-first layout used in Megrez2-Preview.","marker":"[16]"},{"why":"A head-to-head baseline for activated-parameter efficiency.","marker":"[27]"},{"why":"Provides the 3B and 7B dense baselines Megrez2-Preview is compared with.","marker":"[28]"},{"why":"A compact baseline model in the same size class.","marker":"[14]"},{"why":"Supplies the tokenizer and the data-curation approach behind pre-training.","marker":"[15]"},{"why":"Gives the policy-gradient algorithm modified for the RLVR stage.","marker":"[21]"}],"fun_headline_variants":["3B-active Megrez2 outshines 8B rivals with shared experts","Cross-layer expert reuse lets 3B-active model outscore 8B","Megrez2-Preview: 3B active, 7.5B stored, beats larger models","Reusing experts across layers shrinks model but not scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The assumption that one shared expert pool serving three consecutive layers keeps most of the model's capacity is never tested against a non-sharing MoE trained on the same data, so the benchmark wins could stem from the 5-trillion-token corpus and post-training recipe rather than from the sharing mechanism.","fun_headline_variants_meta":{"raw":{"variants":["3B-active Megrez2 outshines 8B rivals with shared experts","Cross-layer expert reuse lets 3B-active model outscore 8B","Megrez2-Preview: 3B active, 7.5B stored, beats larger models","Reusing experts across layers shrinks model but not scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000934,"raw_usage":{"total_tokens":3986,"prompt_tokens":926,"completion_tokens":3060,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":2974}},"tokens_in":542,"tokens_out":3060,"duration_ms":22896,"temperature":1.0,"reasoning_tokens":2974,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:17:39.054357+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a MoE with each layer owning its own expert pool (group size 1) on the same 5-trillion-token corpus with the same SFT and RLVR post-training, holding activated parameters equal; if that model matches or beats Megrez2-Preview, cross-layer sharing is not the source of the reported performance.","supporting_citations":[{"cited_title":"Pre-gated moe: An algorithm-system co-design for fast and scalable mixture-of-expert inference","cited_arxiv_id":null,"evidence_quote":"Provides the pre-gating mechanism adopted for memory-efficient loading."}],"review_version":1}