{"id":"3c68c0e4-2c32-4abc-bc39-ce321c256e98","arxiv_id":"2608.07814","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LorExperts compresses MoE experts by clustering them, keeping one full-precision dominant per cluster, and encoding other experts as permutation-aligned low-rank corrections, preserving the router and outperforming prior compression baselines at ~50% compression.","lead":"This paper introduces LorExperts, a way to shrink mixture-of-experts language models by keeping one full-precision expert per cluster and storing the other experts as low-rank corrections. The method leaves the original router untouched and reports better accuracy retention than pruning, merging, and single-anchor low-rank compression at about 50% expert compression.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central low-rank premise is untested: no spectra or reconstruction errors are reported for the permutation-aligned residuals R = W_m^P - W_dominant, so the claimed ~50% compression rests on an unverified assumption.","rationale":"Reader's weakest_assumption is the same concern I would single out: LorExperts's storage savings and quality both depend on the permutation-aligned residual being low-rank. The reported end-to-end gains are large (e.g., Qwen PPL 12.36 vs D2-MoE 23.58), so missing error bars is a lesser concern; the untested spectral property of aligned residuals is the load-bearing step because it is the mechanism by which the method compresses. I also considered the 'margin grows with E' scaling claim, but that is secondary to the compression mechanism and can be softened without changing the core result. The requested check is straightforward and would settle the issue. Since the paper is otherwise honest about BTExperts's limitations and the reader already conditioned acceptance on additional evidence, no verdict change is needed.","tokens_in":13313,"tokens_out":10131,"duration_ms":85871,"concrete_test":"On Qwen3-30B-A3B, for every non-dominant member in layers 5, 23, and 40, compute R = W_m^P - W_dominant after Stage 3(i), then report the mean fraction of squared Frobenius energy retained by rank-64 truncation, ||U_r Σ_r V_r^T||_F^2 / ||R||_F^2, alongside the Table 2 raw-expert value (~0.26). Also report end-to-end Wikitext-103 PPL for (a) aligned rank-64 SVD, (b) the same SVD with a random neuron permutation, and (c) no alignment (identity permutation). If the aligned-residual energy fraction is not substantially above 0.26, or if (a) does not clearly beat (b)/(c), the low-rank-residual premise fails and the compression claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 Stage 3(i)-(ii) and Eq. (3) make LorExperts's compression hinge on the assumption that after Jonker-Volgenant neuron alignment, the residual R = W_m^P - W_dominant is well approximated by a rank-64 SVD. This is the only mechanism that lets a member be stored as B_m A_m instead of a full matrix; if the aligned residual is near full-rank, LorExperts is a storage scheme with no real compression. The paper's only spectral evidence (Table 2) is for raw expert matrices, not aligned residuals: rank 64 retains just 26% of Frobenius energy on raw experts, and Sec. 2.2's Table 1 shows co-firing experts have near-zero weight cosine similarity in the up/down projections, which dominate the concatenated neuron vector used for alignment. No table reports ||R - U_r Σ_r V_r^T||_F / ||R||_F or the energy fraction of the aligned residual after Stage 3(i), nor a controlled comparison of PPL with vs. without permutation alignment. Absent that measurement, the central ~50%-compression claim rests on an untested premise; if the aligned-residual rank-64 energy fraction is close to the raw-expert 0.26, the method's stated mechanism cannot deliver the claimed compression.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes LorExperts, a router-preserving expert compression method for MoE LLMs. It clusters experts per layer (using co-activation or weight-based distances), keeps one full-precision dominant expert per cluster, and represents each other member as W_dominant + B_m A_m, where the residual is formed after aligning the member's neurons to the dominant via the Jonker-Volgenant algorithm. It also introduces BTExperts, a tree arrangement of dominants and corrections for inference-time amortization, together with a reconstruction fine-tuning procedure. The main empirical claim is that at roughly 50% expert compression on Qwen3-30B-A3B and Gemma-4-26B-A4B, LorExperts preserves perplexity and downstream accuracy better than pruning (REAP, Freq-Prune), merging (M-SMoE, HC-SMoE), and single-shared-component delta decomposition (D2-MoE), while keeping all experts and the original router intact.","tokens_in":13600,"tokens_out":5890,"duration_ms":50621,"significance":"If the empirical claim holds, the paper makes a useful contribution: it identifies a scalable alternative to single-anchor delta decomposition, provides a mechanistic explanation in terms of co-activation communities that are decoupled from weight similarity, and uses the concentration of routing mass to justify keeping full-precision dominants. The paper is also honest in its limitations, explicitly noting that the BTExperts tree is quality-neutral and that its amortization benefit is routing-dependent, and it includes a controlled tree-vs-flat ablation with seeds. However, the central low-rank premise is not directly measured, and the best variant is evaluated at an unmatched compression ratio in the headline tables; these gaps must be closed before the central claim can be accepted.","major_comments":[{"comment":"The load-bearing premise of the method is that after permutation alignment, the residual R = W_m^P - W_dominant is well approximated at rank r=64, but no evidence for this is reported. Section 3, Stage 3(i)-(ii) and Eq. (3) construct the entire member representation from this residual, while Table 2 reports spectra only for raw expert matrices (rank-64 retains only 26% of Frobenius energy). Please report, for at least the same layers/experts as Table 2, the reconstruction error or energy fraction of the aligned residual at r=64 (and a sweep over r), and add an ablation of downstream PPL with and without the permutation-alignment step. If the aligned residual is not distinctly lower-rank than the raw expert, the stated mechanism cannot deliver the claimed roughly 50% compression.","section":"Sec. 3, Stage 3(i)-(ii), Eq. (3)"},{"comment":"The headline '~50% expert compression' comparison is not made at matched compression ratios for the best variant. In Tables 4 and 5, the strongest variant (Msoft) is evaluated at 40.6%/41.4%, while all baselines are at ~50%; the matched Msoft rows at ~51%/51.6% show a large degradation on Qwen (PPL 15.37 vs 11.99, MMLU 46.0 vs 54.9). This makes the abstract's claim that LorExperts 'at ~50%' beats baselines rest on the WS-Frob/Coact-NPMI rows rather than on the best variant. Please present the main comparison at matched compression for all methods, either by evaluating baselines at ~40% or by using a ~50% operating point of the strongest variant.","section":"Tables 4-5 and Sec. 6"}],"minor_comments":[{"comment":"Tables 4 and 5 do not report variances or the number of runs; some margins between variants are small (e.g., Qwen PPL 12.36 vs 12.20 for WS-Frob vs Coact-NPMI), so error bars or seeds are needed to support ranking claims.","section":"Tables 4-5"},{"comment":"The statement that the margin over D2-MoE 'grows with expert count E' is not directly tested, because no controlled sweep over E on a single architecture is presented; the two model families differ in more than E. This claim should either be backed by such a sweep or softened.","section":"Abstract and Sec. 8"},{"comment":"Section 6.2 claims a guaranteed ~100x speedup from a closed-form factor fit, but absolute wall-clock times are omitted and the Limitations section states the closed-form solve is 'not yet the default'. Please clarify which fitting procedure produced Tables 4-5 and whether the speedup claim refers to the current implementation or a future one.","section":"Sec. 6.2 and Limitations"},{"comment":"There are a few presentation issues: 'V olgenant' should be 'Volgenant' (Sec. 3, Stage 3(i)); the reference 'Sec. 6, Gate H' in the Limitations section appears to be a stale cross-reference; and Fig. 5 would be easier to read if the operating points of Table 4 were marked on the curves.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the journal's scope and the central idea is interesting. The two missing pieces (aligned-residual low-rank evidence and matched comparison at the headline compression ratio) are addressable with additional experiments, so I do not recommend rejection. I would also encourage the authors to ensure consistency among Sec. 4, Sec. 6.2, and the Limitations regarding the closed-form fitting procedure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my take on arXiv:2608.07814.\n\nThe new idea is a local-anchor variant of delta decomposition: instead of one shared component for all experts, LorExperts clusters experts by co-activation, keeps a full-precision dominant per cluster, and encodes each remaining member as a permutation-aligned low-rank correction to that dominant. The router and all experts stay untouched. The empirical results are genuinely good: on Qwen3-30B-A3B, the best LorExperts variant reaches PPL ~12 vs D2-MoE's 23.6 at similar compression, and MMLU 54.9 vs 45.7. The paper also deserves credit for what it does not claim: BTExperts is honestly labeled as quality-neutral, and the co-activation analysis in Sec 2 is a useful measurement.\n\nThe main soft spot is that the paper never directly measures the thing its compression depends on: the spectral decay of the aligned residual R = W_m^P - W_dominant. Table 2 reports spectra for raw expert matrices only, where rank 64 retains just 26% of Frobenius energy. If the aligned residual has similar flatness, the method's stated mechanism cannot deliver the claimed 50% compression; the storage saving would still be real, but the quality would rely entirely on the reconstruction fine-tuning, which is a different story. This is easy to fix with one table: report the aligned-residual energy fraction or reconstruction error. Without it, the central claim rests on an unverified assumption.\n\nSecondary issues: Tables 4-5 have no error bars or seeds; the best variant (Msoft) is reported at 40.6% and 41.4% compression in the headlines while baselines are at ~50%, so the headline margin is somewhat unfair (though 51% variants are also shown and still beat D2-MoE). And the 'margin grows with E' claim is not backed by a controlled experiment varying E. These are minor, but the first is load-bearing.\n\nAll in all, I'd send this to a serious referee. The method is well-motivated, the results are strong enough to matter, and the missing analysis is a gap a referee can close. I'd ask for the aligned-residual spectra and error bars before acceptance. This is worth a reading group slot, too, because the gap between mechanism and evidence is instructive.","headline":"Local-anchor low-rank expert compression with strong empirical results, but the central low-rank premise is never directly measured.","tokens_in":14180,"tokens_out":3069,"would_cite":true,"duration_ms":26064,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Local expert anchors preserve MoE accuracy at 50% expert compression without retraining the router.","keywords":["Mixture-of-Experts","expert compression","low-rank decomposition","router-preserving compression","co-activation clustering","neuron permutation alignment","MoE inference","BTExperts"],"falsifier":"Record the singular-value spectrum of the aligned residual $W_m^P - W_d$ for a sample of member-dominant pairs. If the fraction of energy retained by a rank-64 truncation of these residuals is close to the flat-spectrum baseline (about 26% on these expert shapes, per the paper's own Table 2), then alignment is not making the residuals low-rank, and LorExperts is merely re-encoding near-full-rank differences. A complementary check: on a held-out domain that the calibration set underrepresents, such as math or code, if LorExperts at ~50% compression no longer beats D2-MoE at matched compression, the routing-aware clustering claim is domain-limited.","tokens_in":13118,"feed_emoji":"🧩","tokens_out":10555,"duration_ms":79092,"temperature":0.7,"pith_summary":"This paper tries to solve a deployment problem: Mixture-of-Experts language models are cheap per token only if the expert weights fit in memory, and expert weights are growing faster than the memory budget. The authors argue that the usual fixes—pruning experts, merging similar ones, or representing every expert as one shared component plus a small delta—all break down, because expert weights are nearly orthogonal to one another yet still form functional co-activation communities. LorExperts instead clusters experts, keeps one full-precision 'dominant' per cluster, and writes every other member as a low-rank correction to its local dominant after permuting the member's neurons to align with the dominant. At roughly 50% expert compression on Qwen3-30B-A3B and Gemma-4-26B-A4B, it preserves downstream accuracy and perplexity better than pruning, merging, or single-shared-component decomposition on most tasks, and its margin over the single-anchor method grows with the number of experts. If right, this means large MoE models can be served at half the expert memory with no router retraining and no lost experts.","feed_headline":"Local anchors halve MoE experts, router untouched","feed_subtitle":"Clustering experts around a full-precision dominant plus low-rank corrections beats pruning and merging at ~50% compression.","key_machinery":"The load-bearing object is the per-cluster dominant with permutation-aligned low-rank residuals. For each cluster, a full-precision dominant expert $W_d$ is kept; for every other member $W_m$, the method solves for a permutation $P$ of the member's neurons that minimizes the total squared distance between concatenated gate/up/down columns (Eq. 1–2, exact via Jonker-Volgenant), then writes the aligned member as $W_m^P \\approx W_d + B_m A_m$ with $B_m, A_m$ from a rank-$r$ truncated SVD of the residual (Eq. 3–4). The insight that lets this work is the co-activation structure of the router: experts cluster into functional communities that are largely decoupled from weight similarity, so clustering by co-activation groups experts whose residual against a local dominant is actually small, and a frequency-seeded k-medoids step chooses dominants the router actually uses.","core_discovery":"LorExperts claims that MoE experts, despite being near-orthogonal in weight space, can be compressed at scale by replacing one global shared component with many local ones. The paper partitions experts into clusters using either weight distance or routing co-activation, keeps the highest-firing expert in each cluster as a full-precision dominant, aligns each remaining member's neurons to the dominant by solving a minimum-cost assignment over concatenated gate/up/down columns, and stores the member as the dominant plus a rank-64 SVD correction. Because every expert and the original router remain addressable, no router retraining is needed. The measured consequence is that at ~50% expert compression on Qwen3-30B-A3B and Gemma-4-26B-A4B, LorExperts preserves perplexity and knowledge/reasoning accuracy better than the baselines on most tasks, and its advantage over single-shared-component decomposition (D2-MoE) widens as the number of experts grows. The paper also introduces BTExperts, a binary-tree organization of dominants and corrections that does not improve quality but can amortize shared computation when the router co-selects experts in the same cluster.","pith_inferences":["This raises a testable extension: check whether the same co-activation communities persist after compression; if they do, routing statistics alone could guide per-layer compression ratios, skipping the weight-space analysis.","The permutation-alignment step could be lifted into other compression schemes: quantizing experts in a shared aligned basis may create redundancy that reduces quantization error, a direction the paper does not explore.","The paper's ongoing Hessian-weighted anchor suggests a stronger general result: the optimal shared component is a traffic-weighted combination of the cluster rather than any existing expert, which would carry a guarantee that using an anchor never hurts; this remains outside the paper's reported experiments.","BTExperts' speedup depends on the router co-selecting experts from the same cluster; a routing regularizer that nudges co-selection could increase amortization, at the cost of altering the learned routing."],"forward_implications":["At ~50% expert compression on 128-expert models, knowledge- and reasoning-heavy metrics (MMLU, ARC-C, HellaSwag) hold up markedly better than with pruning or merging, so memory-bound deployment of large MoE models becomes feasible without router retraining.","Because the advantage over a single shared component grows with the expert count, the method targets the regime MoE designs are moving toward, E=128 and beyond.","Routing-aware clustering (co-activation NPMI or soft consensus) beats weight-similarity clustering, implying grouping should use functional routing statistics rather than raw weights.","The compression ratio is a tunable knob with a smooth quality frontier, so a practitioner can choose an operating point to hit a quality target instead of accepting a fixed degradation.","BTExperts can cut per-token FLOPs and memory traffic by roughly 1.9x when co-selected experts share a dominant, because the dominant is computed once and broadcast to the group."],"supporting_citations":[{"why":"supplies the neuron permutation and weight-matching machinery used to align member experts to their dominant.","marker":"Ainsworth et al., 2023"},{"why":"provides the M-SMoE merging baseline and the precedent of applying permutation alignment to experts before merging.","marker":"Li et al., 2024"},{"why":"defines D2-MoE, the single-shared-component delta decomposition that LorExperts must outperform and whose failure mode motivates multiple local anchors.","marker":"Gu et al., 2025"},{"why":"defines REAP, the pruning baseline that frames the router-retraining cost and the accuracy trade-off LorExperts avoids.","marker":"Lasby et al., 2026"},{"why":"introduces the low-rank factor form B_m A_m that LorExperts reuses for per-member corrections.","marker":"Hu et al., 2022"},{"why":"presents SD-MoE, the spectral shared-component alternative that the paper groups with single-anchor methods.","marker":"Huang et al., 2026"},{"why":"documents Qwen3, the 128-expert model on which the co-activation analysis and main compression results are measured.","marker":"Yang et al., 2025"}],"fun_headline_variants":["Local anchors beat global shared component in MoE compression","Cluster experts, keep one full-precision anchor, router untouched","Local cluster anchors beat global component as MoE grows","LorExperts: local anchors preserve accuracy at half the experts","No router retraining: cluster MoE experts around local dominants"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that after permuting a member expert's neurons to match its dominant, the leftover difference $W_m^P - W_d$ can be captured by a small set of 64 correction patterns; if the aligned residuals stay nearly as complex as the original experts, LorExperts stops being real compression and becomes a storage scheme that moves the same information into different files.","fun_headline_variants_meta":{"raw":{"variants":["Local anchors beat global shared component in MoE compression","Cluster experts, keep one full-precision anchor, router untouched","Local cluster anchors beat global component as MoE grows","LorExperts: local anchors preserve accuracy at half the experts","No router retraining: cluster MoE experts around local dominants"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001151,"raw_usage":{"total_tokens":4828,"prompt_tokens":1059,"completion_tokens":3769,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":675,"completion_tokens_details":{"reasoning_tokens":3686}},"tokens_in":675,"tokens_out":3769,"duration_ms":24585,"temperature":1.0,"reasoning_tokens":3686,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:13:12.628276+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record the singular-value spectrum of the aligned residual $W_m^P - W_d$ for a sample of member-dominant pairs. If the fraction of energy retained by a rank-64 truncation of these residuals is close to the flat-spectrum baseline (about 26% on these expert shapes, per the paper's own Table 2), then alignment is not making the residuals low-rank, and LorExperts is merely re-encoding near-full-rank differences. A complementary check: on a held-out domain that the calibration set underrepresents, such as math or code, if LorExperts at ~50% compression no longer beats D2-MoE at matched compression, the routing-aware clustering claim is domain-limited.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines REAP, the pruning baseline that frames the router-retraining cost and the accuracy trade-off LorExperts avoids."}],"review_version":1}