{"id":"35e034c9-7d9c-47e1-86d4-12637c4d8330","arxiv_id":"2607.28670","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"A hierarchical copula over Gumbel routing noise changes joint expert choices across tokens while preserving every token's ordered top-K law, mixture weights, and inclusion probabilities.","lead":"H-CGA correlates the random Gumbel noise used by a frozen mixture-of-experts router, making related tokens pick similar experts and paired groups pick opposite ones, without changing any single token's routing probabilities. It is a new inference-side dial for expert locality and load variance that can be tuned by a tiny controller, though practical gains are not yet shown.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified","rationale":"The reader's weakest assumption — pre-routing measurability of grouping and dials — is indeed the most fragile operational condition, but it is not a hidden assumption or a flaw in the argument. The paper states it, motivates it, and the proposed controller respects it. I verified the core proof steps: the Gaussian-to-uniform-to-Gumbel pipeline preserves i.i.d. Gumbel marginals; the independence across expert coordinates is maintained; the top-K inclusion indicator is monotone in the transformed latent coordinates, so the association inequality applies to both parts of Proposition 1. The negative pairing relation W_g' = -α W_g + sqrt(1-α²) V_g' is coordinatewise valid and gives the required nonincreasing conditional mean. The pilot is explicitly labeled as mechanism evidence, not a task-level claim, so its minimal nature does not undermine the theoretical central claim. Hence ACCEPT stands.","tokens_in":12171,"tokens_out":13049,"duration_ms":125545,"concrete_test":"Run a Monte Carlo verification of Proposition 1 on random logits and group sizes over a grid of ρ ∈ {0, 0.3, 0.6, 0.9} and α ∈ {0, 0.5, 1}: for each cell, estimate Var(N_e) under independent routing, flat coupling, and paired coupling, and confirm Var_independent ≤ Var_flat and Var_paired ≤ Var_flat. If any cell violates the inequalities, the monotonicity argument in the Appendix needs re-examination.","verdict_should_be":"UNCHANGED","load_bearing_attack":"No significant objection identified. Theorem 1 is correct: for each token, y_te is standard normal with independent expert coordinates, so the Gumbel vector remains i.i.d. Gumbel; every downstream routing statistic is therefore preserved conditionally. Proposition 1's signed variance bounds also check out: the top-K inclusion indicator is monotone in each transformed latent coordinate, and the association inequality yields the claimed covariance signs. The one place where the argument could break is the stated pre-routing measurability assumption: if group membership, pairing, or dials depended on realized routing noise, the marginals would no longer be preserved. But the paper states this assumption explicitly, motivates it, and the proposed fixed-window / frozen-feature controller satisfies it. The pilot is admittedly minimal and is not used as evidence beyond mechanism validation. No internal inconsistency or hidden circular step was found.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies stochastic Gumbel-Top-K routing in mixture-of-experts (MoE) models and asks how the joint distribution of routing choices across tokens can be modified while keeping every token's marginal routing law exactly fixed. It proposes a hierarchical construction, H-CGA: within a group of tokens, exchangeable Gaussian latents create positive correlation among the Gumbel perturbations at each expert coordinate; across paired groups, an antithetic latent relation introduces tunable negative dependence. The main theoretical result (Theorem 1, Corollaries 1 and 2) is that each token's Gumbel noise vector remains i.i.d. Gumbel under both within-group coupling and cross-group opposition, so the ordered Top-K list, selected set, mixture weights, and conditional expected inclusion counts are unchanged relative to independent Gumbel-Top-K routing. Proposition 1 then gives signed variance comparisons: positive within-group coupling can only increase the conditional variance of realized per-expert loads relative to independent routing, while cross-group opposition can only decrease variance relative to flat coupling at the same within-group strength. The paper also describes a score-function training route for a small controller that sets the coupling dials from frozen features, and reports a deliberately small pilot that checks the exact sampler, the routing-law invariance, and the effect of the dials without claiming downstream task gains.","tokens_in":12256,"tokens_out":20692,"duration_ms":180686,"significance":"If the results hold, the paper identifies and rigorously analyzes a useful new degree of freedom in MoE routing: cross-token dependence can be adjusted without altering any single token's routing law. The central invariance theorem is simple but exactly stated, and the proof is correct: the construction preserves standard normal marginals and independence across expert coordinates, so each token's Gumbel vector is unchanged in distribution. Proposition 1's association-inequality argument is also sound; the sign-change construction correctly makes the inclusion indicators coordinatewise monotone in a common independent latent vector. The paper is explicitly honest about its scope: it is layer-local, requires pre-routing measurability of group membership and controller outputs, and the pilot is presented as mechanism validation, not as evidence of task-level gains. These explicit limitation statements are a strength. The paper would be a useful contribution to the stochastic-routing literature, giving a provable construction with two complementary dials and a clear variance trade-off.","major_comments":[],"minor_comments":[{"comment":"In the proof of Proposition 1(ii), the definition of the transformed vector V_{g'} is implicit. The recurrence W_{g'} = -alpha W_g + sqrt(1-alpha^2) V_{g'} requires V_{g' e} = eta_{g' e} for the target expert coordinate e and V_{g' e'} = -eta_{g' e'} for all e' != e. State this sign convention explicitly; as written, the reader must infer it.","section":"Appendix / Proposition 1"},{"comment":"Minor formatting: in the Router-LoRA row, the trainable-parameter count and validation cross-entropy are run together (\"1,5843.12439\") without a separator. Also the reference to \"V olker Tresp\" has an errant space.","section":"Section 5 / Table 1"},{"comment":"The text says Equation (10) is \"exactly unbiased\" for a baseline that does not depend on the scored sample's noise, then notes that the batch-mean baseline induces a (1-1/B) scaling. Rephrase to avoid the appearance of a contradiction, e.g., \"unbiased for a baseline independent of the scored sample; with the batch-mean baseline, the estimator is a scaled version...\".","section":"Section 3.5"},{"comment":"The phrase \"which joint distributions ... are reachable\" suggests a complete characterization, whereas the paper gives a sufficient two-parameter construction and a trade-off result. Suggest rephrasing to \"we construct a family of joint distributions\" or \"we exhibit one two-sided family\" to match the actual claims.","section":"Abstract / Introduction"},{"comment":"The statement that at alpha=1 the partner group receives \"the opposite push\" should be qualified: the shared latent is opposite, but the effect on token scores is scaled by each group's sqrt(rho_g). When rho_g and rho_g' differ, the magnitudes of the pushes on the two groups' tokens are not equal.","section":"Section 3.4 / Figure 2"}],"recommendation":"minor_revision","confidential_remarks":"No concerns about citation, novelty, or scope. The theoretical contribution is sound and the paper is appropriately cautious about the pilot's limited empirical scope. The minor issues are presentation-level; the core invariance and variance-bound results are correct and should be publishable after the small clarifications noted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does one thing genuinely new: it shows you can couple the Gumbel noises of different tokens in a frozen stochastic MoE router, preserving every token's marginal Top-K distribution exactly, and gives you two dials—positive within-group correlation and negative between-group opposition—that move joint routing statistics in opposite directions. Theorem 1 and Corollary 2 are correct; I checked the construction and the proof is airtight given the independence across expert coordinates. The association inequality argument in Proposition 1 also works; the appendix gives enough detail, though it's compressed.\n\nThe paper is unusually honest about the limits of what it proves. The invariance is layer-local, conditional on logits, and applies only to the stochastic Gumbel-Top-K router, not deterministic top-K, capacity clipping, or token dropping. That is not a flaw; it is the right scope. The pre-routing measurability assumption—group membership, pairing, and dials must be fixed before drawing noise—is stated explicitly and motivated, and the fixed-window/frozen-feature controller satisfies it. No hidden circularity or fitting-to-conclusion.\n\nThe main soft spot is not in the math but in the empirical support. The pilot is deliberately small and the paper says so; it is a mechanism check, not evidence of task improvement. The learned controller shows no validation cross-entropy gain, the learned strengths are tiny, and the aggregate load CV barely moves. The paper also does not directly test Proposition 1's variance bounds, and the pilot's load CV summary does not measure conditional load variance. There is no code or data, which makes reproducibility harder, though the exact equations are there.\n\nMinor point: the proof sketch of Proposition 1 in the main text leans on the appendix. The appendix is sound, but a reader who skips it has to take the sign-change argument on faith. That is fine for a theory paper.\n\nThe paper deserves a serious referee. The theoretical contribution is solid, the writing is clear, and the claims are calibrated. The experiments are honest enough to publish as a pilot, but a revised version should either extend them or make their purpose even plainer. I would send this to peer review, and I'd expect an accept after reasonable revisions. I'd cite the invariance theorem in my own MoE work; the construction is a genuinely useful addition to the routing toolbox.\n\nFor the record: I agree with the reader's ACCEPT verdict and with the stress-test note. There is no significant objection here.","headline":"A correct and clearly-scoped invariance theorem for a two-sided dependence dial in frozen MoE routing, with a pilot that honestly doesn't overclaim.","tokens_in":12796,"tokens_out":1398,"would_cite":true,"duration_ms":14721,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proves that a hierarchical copula on Gumbel routing noise can couple or oppose different tokens' expert choices while every token's full routing distribution stays exactly unchanged.","keywords":["hierarchical copula","Gumbel-Top-K","mixture-of-experts routing","cross-token dependence","routing-law invariance","load variance","score-function estimator","frozen MoE"],"falsifier":"Route one token many times with fixed logits through the H-CGA sampler at a high rho (for example 0.8) and compare the empirical frequencies of every ordered top-2 list with the frequencies under independent Gumbel-Top-K; since Theorem 1 says the two distributions are exactly equal, any difference beyond Monte Carlo error falsifies the invariance claim. The same check should pass for every alpha in [0,1].","tokens_in":11969,"feed_emoji":"🎲","tokens_out":7622,"duration_ms":63395,"temperature":0.7,"pith_summary":"Mixture-of-experts routers give each token a distribution over experts; this paper explores a separate dial: how the random choices of different tokens are coupled. It proposes H-CGA, a hierarchical copula on the Gumbel noise that makes related tokens positively correlated in which experts they pick, and paired groups negatively correlated through an antithetic signal. The paper's central claim is that both operations leave every token's full routing law—ordered top-K list, mixture weights, inclusion probabilities—exactly as if routing were independent, so conditional expected expert loads are unchanged. It also proves a signed trade-off: within-group coupling can only increase realized load variance, while cross-group opposition can only reduce it relative to flat coupling at the same strength. If true, this gives a frozen-model-compatible way to steer expert coherence and load dispersion without touching router logits or experts.","feed_headline":"Couple or oppose expert picks—per-token routing odds stay fixed.","feed_subtitle":"Within-group coupling raises expert sharing; cross-group opposition can trim load variance—per-token laws stay fixed.","key_machinery":"The key mechanism is a hierarchical Gaussian copula applied to the Gumbel noise of a frozen stochastic router. Within a group, token perturbations at each expert coordinate are formed as y = sqrt(rho) zeta_g + sqrt(1-rho) epsilon, then mapped through the normal CDF and the inverse Gumbel CDF, so the shared latent induces positive dependence while the marginal per-token noise stays standard Gumbel. Between paired groups, zeta_g' = -alpha zeta_g + sqrt(1-alpha^2) eta injects a tunable negative dependence. Independence across expert coordinates is what keeps one token's full noise vector i.i.d. Gumbel; that single property carries the entire invariance argument.","core_discovery":"The paper's central discovery is Theorem 1: in the Hierarchical Copula-Gumbel-Top-K construction, each token's noise vector is exactly i.i.d. Gumbel, even though related tokens share a correlated latent at every expert coordinate and paired groups receive an antithetic shared signal. Because the copula acts on the Gumbel perturbations coordinate-by-coordinate and each token belongs to exactly one group, the ordered top-K list, the selected expert set, and the gate-based mixture weights of every token have exactly the same conditional distribution as under independent Gumbel-Top-K routing. A corollary preserves conditional expected per-expert inclusion counts. The companion Proposition 1 sign","pith_inferences":["Beyond the paper: if later MoE layers can exploit the changed joint routing, end-to-end task effects may be stronger than the layer-local guarantees suggest; the paper explicitly leaves multi-layer end-to-end invariance unproven.","Beyond the paper: the antithetic paired-latent trick sidesteps the standard negative-correlation lower bound for many groups and could be reused as a general method for negatively coupling Plackett-Luce samples without changing marginals.","Beyond the paper: a direct test of the alpha dial's practical value would measure capacity-overflow rate under hard capacity clipping, since positive coupling's variance inflation is exactly the quantity that causes overflow—the pilot does not measure this.","Beyond the paper: because per-token cross-entropy alone cannot identify the controller's parameters, the method's usefulness probably depends on joint objectives or multi-layer compounding; the learned scalar strengths being small and patternless in the pilot is consistent with this, but the paper does not establish it."],"forward_implications":["A frozen stochastic Gumbel-Top-K MoE can be re-randomized jointly: tokens in a group will draw more similar expert sets, and matched groups will draw more opposed sets, without any change to a token's own ranked-list probabilities.","Conditional expected expert loads at a routing layer stay identical to independent routing, so the method changes the distribution of traffic around a fixed mean, not the mean itself.","Within-group coupling is always variance-increasing for realized loads, so any use of it for coherence must pay the price of burstier per-expert counts.","Cross-group opposition gives a counter-dial that can only lower realized-load variance relative to flat coupling at the same within-group strength, though not necessarily below the independent baseline.","The entire base model can stay frozen while a small controller sets the dependence strengths, trained with a score-function estimator that only evaluates the base network forward."],"fun_headline_variants":["Copula Gumbel-Top-K: couple or oppose expert picks, per-token odds fixed","Two-sided dependence control for frozen MoE routing at fixed per-token laws","MoE routing: couple within-group, oppose cross-group, keep per-token laws fixed","Frozen MoE gets dependence dials: within-group coupling, cross-group opposition"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Group membership, pairing, and both coupling strengths must be fixed before any routing noise is drawn and depend only on frozen pre-routing features; if they are adjusted after seeing routing outcomes, the per-token Gumbel marginals need not survive and Theorem 1 fails.","fun_headline_variants_meta":{"raw":{"variants":["Copula Gumbel-Top-K: couple or oppose expert picks, per-token odds fixed","Two-sided dependence control for frozen MoE routing at fixed per-token laws","MoE routing: couple within-group, oppose cross-group, keep per-token laws fixed","Frozen MoE gets dependence dials: within-group coupling, cross-group opposition"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001641,"raw_usage":{"total_tokens":6429,"prompt_tokens":885,"completion_tokens":5544,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":5461}},"tokens_in":629,"tokens_out":5544,"duration_ms":32221,"temperature":1.0,"reasoning_tokens":5461,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T00:43:07.303512+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Route one token many times with fixed logits through the H-CGA sampler at a high rho (for example 0.8) and compare the empirical frequencies of every ordered top-2 list with the frequencies under independent Gumbel-Top-K; since Theorem 1 says the two distributions are exactly equal, any difference beyond Monte Carlo error falsifies the invariance claim. The same check should pass for every alpha in [0,1].","supporting_citations":[],"review_version":1}