{"id":"3d873a3f-3a1b-455c-ae6c-b5969c5bc42d","arxiv_id":"2505.13273","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Expert disagreement inside pretrained MoE diffusion models, measured as latent variance at the first denoising step, gives a training-free prompt uncertainty signal that correlates with text-image alignment across languages.","lead":"EMoE measures uncertainty in text-to-image diffusion models by splitting a mixture-of-experts model into separate paths and checking how much their internal representations disagree, without any extra training. It can rank prompts by expected image quality and reveal language biases, but the evidence is preliminary and no code or data is released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EU(y) is confounded with prompt length: Table 2 shows longer prompts have lower uncertainty, so the quartile-CLIP trend may not establish epistemic uncertainty rather than a length proxy.","rationale":"The reader's weakest assumption concerned the theoretical justification of treating community fine-tunes as i.i.d. posterior samples. I agree that Appendix D does not establish this, but the more empirically pressing threat is the prompt-length confound that appears in the paper's own results. Table 2 is not a peripheral check: it shows a monotonic, statistically significant relationship between lower EU and longer prompts, and longer captions are a known correlate of generation quality. Because the paper never reports a prompt-length-only baseline or a partial-correlation analysis, the headline quartile result in Table 1 and Figure 3 could in principle be reproduced by ranking prompts by word count. This is directly testable from data the authors already have, unlike the abstract Gaussian-process question. If the stratification test shows the EU-CLIP trend persists within length bins, the concern would be resolved and the conditional verdict could move toward acceptance; if it disappears, the central 'epistemic uncertainty' interpretation fails. The style-diversity issue reinforces the concern but is secondary: style differences are visible in the model card (Realistic Vision, Dreamlike Anime, DreamShaper, Dreamlike Diffusion), and the GP proof does nothing to rule out that what EMoE measures is style sensitivity. I therefore keep the reader's CONDITIONAL verdict: the method may work as a practical ranker, but the epistemic-uncertainty claim needs this specific control before it can be accepted.","tokens_in":17819,"tokens_out":6842,"duration_ms":69303,"concrete_test":"Using the released or re-generated 40k English COCO data, stratify prompts into word-count bins (e.g., <=8, 9-10, 11-12, >=13 words). Within each bin, recompute the mean CLIP score per EMoE uncertainty quartile and the Jonckheere-Terpstra trend. Also compute the partial Spearman correlation between EU and CLIP controlling for word count. If the monotonic trend largely disappears or the partial correlation is near zero, EU is a length proxy rather than a distinct epistemic-uncertainty signal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that Var_i(mpost_T) in Eq. (2) estimates epistemic uncertainty about the prompt. The paper's own Table 2 shows that EU is strongly associated with prompt length: mean character count falls from 53.14 in Q1 to 51.93 in Q4, and word count from 10.58 to 10.34, with a Jonckheere-Terpstra p-value of 2.13e-12. Since longer, more descriptive prompts also tend to produce higher CLIP scores, the observed quartile trend in Table 1 may be driven by length rather than by expert disagreement as an epistemic signal. The four experts are community fine-tunes with distinct style descriptors (photorealistic, anime, fantasy, dreamlike), so disagreement may also reflect systematic style preference rather than posterior uncertainty; Appendix D's Gaussian-process argument does not establish that these checkpoints are i.i.d. posterior samples. If EU is primarily a length or style proxy, the paper's interpretation as epistemic uncertainty, and its bias-detection conclusions, are not supported, even though the ranking property might partially survive.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EMoE, a training-free method for estimating epistemic uncertainty in text-to-image diffusion models that contain mixture-of-experts (MoE) layers. The method separates the first sparse MoE layer into multiple expert-specific computation paths, keeps the same initial noise across paths, runs the first denoising step, and computes EU(y) = E_dmid[Var_i(mpost_T)] from the mid-block latents. The authors claim that this quantity ranks prompts by text-image alignment quality on COCO and CC3M more consistently than the DECU baseline, detects out-of-distribution Finnish prompts, reveals language-dependent bias across 25 languages, and can be computed early enough to halt generation. Supporting experiments include quartile-based CLIP/Aesthetic/Image Reward comparisons, multilingual correlations, and ablations over ensemble size, denoising step, latent space, and a second MoE architecture.","tokens_in":18022,"tokens_out":5568,"duration_ms":55381,"significance":"If its central claim holds, EMoE would be a practical and inexpensive diagnostic: it requires no training, no auxiliary networks, and only one early denoising step to produce a prompt-level uncertainty signal, with potential uses in risk assessment and bias auditing of black-box generative models. The paper has real strengths: experiments span COCO and CC3M with multiple metrics, the multilingual analysis is broad, ablations cover the main design choices, and the compute and memory requirements are reported in detail. The central empirical trend is plausible and replicated across datasets and metrics. However, the evidence as presented does not yet separate the proposed signal from a prompt-length proxy, and the interpretation of inter-expert variance as epistemic uncertainty rests on an assumption, not a demonstration.","major_comments":[{"comment":"A second, related issue is the size of the effect. The CLIP difference between Q1 and Q4 is 31.578 vs 31.217, with reported standard deviations of about 0.15–0.16; the Aesthetic and Image Reward gaps are similarly small. With 40,000 prompts, a Jonckheere-Terpstra p-value of 3.34e-19 can be driven by a tiny effect that has little practical utility. The paper should report effect sizes (e.g., rank-biserial correlation, Cohen's d) and, ideally, the per-prompt Spearman correlation between EU and CLIP score, not just quartile means, so that readers can judge whether the relationship is strong enough to support a 'reliable uncertainty-aware prompt signal'.","section":"§4.1, Tables 1–2, Appendix G"},{"comment":"The core interpretation of Var_i(mpost_T) as epistemic uncertainty is not established. Appendix D argues that experts can be viewed as i.i.d. samples from a posterior distribution over functions and invokes a Gaussian-process convergence argument, but the proof only shows that an ensemble of i.i.d. functions converges to a GP; it does not show that the four community fine-tunes in the SegMoE model card are i.i.d. draws from a posterior. Those experts are explicitly associated with different style descriptors (photorealistic, anime, fantasy, dreamlike), so inter-expert variance may reflect systematic style preferences or routing artifacts rather than epistemic uncertainty about the prompt. The authors should provide a concrete test that distinguishes these explanations: for example, compare EU against (a) variance across different initial noise seeds on a single expert path, (b) variance induced by style-neutral prompt perturbations of matched length, or (c) the performance of EU on a proper OOD-detection benchmark where length is controlled. If the variance is mostly style or routing signal, the practical ranking claim may partially survive, but the epistemic-uncertainty labeling and the bias-detection conclusions would need to be revised.","section":"§3.2, Eq. (2), Appendix D"},{"comment":"The claim that EMoE ranks prompts 'more consistently than diffusion-specific and router-based baselines' is not backed by a quantitative comparison. The text states that DECU 'did not demonstrate this capability' and the figures annotate 'unlike DECU', but no DECU numbers, rank correlations, AUROC values, or implementation details are reported. Because this contrast appears in the abstract and is load-bearing for the claimed advantage over existing uncertainty methods, the authors should include direct quantitative comparisons: same prompts, same metrics, and a clear description of how DECU was configured. A router-only baseline (e.g., using the gating weights without expert separation) would also be valuable to show that the signal comes from disagreement rather than from routing behavior.","section":"§4.1, Appendix C, Figures 3 and 9"},{"comment":"The multilingual bias conclusions are drawn from language-level aggregates. The reported correlation r=-0.79 (CLIP) and r=-0.73 (Image Reward) is computed across language mean points; averaging within language can inflate correlation relative to the per-prompt relationship, and no confidence intervals or per-prompt correlations are given. The claim of a 'strong negative correlation' needs to be supported by per-prompt analysis or a mixed-effects model with language as a random effect. In addition, the paper states that CLIP scores for non-English prompts are evaluated using the English version of the prompt, which means the CLIP model is effectively scoring English-text-to-image alignment even for Finnish or other languages. This introduces a potential language-specific measurement bias: differences in CLIP score may reflect CLIP's own English bias rather than generation quality alone. The authors should state this limitation explicitly in the main text and consider a language-matched evaluation or a secondary metric that is less language-sensitive.","section":"§4.2 and §4.3, Tables 3 and 8, Figures 6 and 8"}],"minor_comments":[{"comment":"The text says the CC3M results are 'presented in Figure 9 and Table 6', but Table 6 in the main text is the compute-requirements table; the reference should be to Table 9.","section":"Appendix C"},{"comment":"Equation (2) defines EU(y) as E_dmid[Var_i(mpost_T)], but Appendix A states that the reported uncertainty is sqrt(dmidsize) × EU(y). Please reconcile the scaling and define which quantity is used in all tables and figures.","section":"Equation (2) and Appendix A"},{"comment":"The description of the separated paths is ambiguous: after the first layer is separated, later sparse MoE layers are said to aggregate outputs 'within each computational path according to Equation 1'. Please clarify how the gate weights and selected expert sets are computed when the first-layer routing has been replaced by an explicit separation, and whether the same gate weights are used across all paths.","section":"§3.1"},{"comment":"The abstract promises comparison against 'diffusion-specific and router-based baselines', but the only baseline discussed in the evaluation is DECU. Either add a router-based baseline or soften the claim to match what is actually compared.","section":"Abstract and §4"},{"comment":"The ablation on ensemble size says ensembles of sizes 2 and 3 are 'sub-optimal' because Q1 yields a lower CLIP score than Q2, but no statistical significance or confidence intervals are provided. Please add error bars or significance tests, or phrase the conclusion more cautiously.","section":"Figure 7a"},{"comment":"The text says uncertainty is computed 'after the first denoising step', while Algorithm 1 extracts mpost_T for t = T at the initial step. Please make the notation consistent so that the reported step index matches the algorithm.","section":"§3.2 and Algorithm 1"},{"comment":"The example reports uncertainty values of 0.38 and 0.83, while Table 3 reports mean uncertainties of 0.37 ± 0.14 and 0.48 ± 0.19 for English and Finnish. The Finnish example is far above the mean; please note that it is a selected extreme example rather than a typical case.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The practical diagnostic contribution is potentially useful, but the epistemic-uncertainty framing is stronger than the current evidence supports. If the authors can supply the length-control analysis, a quantitative DECU comparison, and a demonstration that the disagreement signal is not primarily a style or routing artifact, the paper could become a solid contribution to uncertainty-aware generation. If those controls cannot be provided, the claims should be downgraded from 'estimates epistemic uncertainty' to 'correlates with text-image alignment quality in a prompt-length-dependent manner.'"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, EMoE is a genuinely new, zero-training trick: separate the expert paths in a pre-trained MoE diffusion U-Net at the first cross-attention layer, run the first denoising step down each path, and compute variance in the mid-block latent. That is a cheap, practical uncertainty signal that requires no training and can be computed before full generation. Second, the paper's central interpretation — that this variance is epistemic uncertainty about the prompt — is not established. The four experts are community fine-tunes with distinct aesthetic styles (photorealistic, anime, fantasy, dreamlike), so their disagreement may be style divergence rather than posterior uncertainty.\n\nCredit where due: the empirical work is honest and fairly thorough. They validate on COCO and CC3M, report quartile trends for CLIP, Aesthetic, and ImageReward, include ablations for ensemble size, denoising step, latent space, and a second MoE, and they used an off-the-shelf MoE to avoid selection bias. The multilingual result, a −0.79 correlation between uncertainty and CLIP across 25 languages, is the strongest finding and the most interesting for bias auditing. They also flag CLIP's own biases in an appendix and include a limitations section.\n\nThe load-bearing conceptual assumption is the weak point. Appendix D tries to justify the ensemble-as-posterior view with a GP argument, but the proof essentially assumes the conclusion: a pointwise CLT over hypothetical i.i.d. weight samples does not yield a Gaussian process, and these experts are not i.i.d. posterior samples. That appendix should be deleted or replaced with a statement that expert divergence is used as a heuristic. The empirical ranking claim can survive this, but the epistemic-uncertainty language should be softened.\n\nOn the stress-test's length concern: it is real but minor. Table 2 shows Q1 prompts are only ~1.2 characters and ~0.2 words longer than Q4 on average. That tiny shift is unlikely to drive a 0.36 CLIP-point gap, but the authors should add a length-matched or length-regressed baseline to close the question. The DECU comparison is qualitative only—no numbers—and the CLIP differences between quartiles are small, though the authors argue that is expected for in-distribution prompts. All of this is addressable.\n\nWho should read it: anyone working on uncertainty in generative models or auditing text-to-image bias. It deserves a real peer review. I would send it out with a recommendation of major revision, asking for a length-controlled experiment, quantitative DECU baseline, released code, and cleanup of the GP appendix.","headline":"Training-free MoE expert-variance uncertainty is a genuinely new and practical tool, and the empirical trend is real, but the epistemic-uncertainty interpretation is overclaimed and a prompt-length confound remains untested.","tokens_in":18552,"tokens_out":2926,"would_cite":true,"duration_ms":27738,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free variance signal inside diffusion MoEs predicts when a prompt will generate poorly.","keywords":["epistemic uncertainty","mixture of experts","text-to-image diffusion","training-free uncertainty estimation","expert disagreement","prompt risk","multilingual bias","latent-space variance"],"falsifier":"Translate a matched set of Finnish prompts that differ only in including a common English loanword such as 'pizza', generate images, and compare EMoE uncertainty with human-rated or ImageReward alignment; if uncertainty drops sharply for loanword prompts while alignment does not improve, the signal is tracking token familiarity rather than prompt-level epistemic uncertainty.","tokens_in":17611,"feed_emoji":"🎨","tokens_out":11340,"duration_ms":103541,"temperature":0.7,"pith_summary":"This paper claims that the disagreement among the expert components of a pre-trained mixture-of-experts text-to-image diffusion model is a usable, training-free signal for how much the model knows about a given prompt. EMoE separates the experts' computation paths from the first denoising step onward, keeps the same initial noise across all paths, and measures the variance of their mid-block latent representations. On COCO and CC3M English prompts, prompts in the lowest uncertainty quartile produce higher text-image alignment scores than prompts in the highest quartile, and the ordering also holds for aesthetic and human-preference metrics. Translated prompts in lower-resource languages produce higher uncertainty and worse alignment, with a correlation of $r = -0.79$ across 25 languages. If the claim is right, one early denoising step can flag risky or under-covered prompts before the expensive generation runs, and can expose language-related training bias.","feed_headline":"Expert disagreement in one denoising step flags weak prompts","feed_subtitle":"Screen any text-to-image prompt before generation by measuring variance across pre-trained experts' first-step latents.","key_machinery":"The central object is the separated expert path ending in the mid-block latent $m^{\\mathrm{post}}_T$. At the first sparse MoE cross-attention layer, EMoE gives each of the $M$ experts its own copy of the noisy latent and the prompt, routes each copy through the rest of the network with the MoE's normal aggregation inside each path, and reads off the $1280 \\times 8 \\times 8$ mid-block latent after one denoising step. The variance across those $M$ latents, averaged over latent dimensions, is the uncertainty score $EU(y) = \\mathbb{E}_{d_{\\mathrm{mid}}}\\left[\\mathrm{Var}_{i \\in M}\\left(m^{\\mathrm{post}}_T\\right)\\right]$. This mechanism converts a routed mixture into a shared-noise ensemble: same initial noise and same prompt, only expert-specific weights differ, so their divergence is interpreted as disagreement about the prompt. Because the score is available after one step, it can halt the denoising process for high-uncertainty prompts.","core_discovery":"The paper's central discovery is that the mean over latent dimensions of the variance across experts at the mid-block of the first denoising step orders prompts by expected text-image alignment. For a prompt $y$, after running the $M$ separated expert paths through one denoising step, the score is $EU(y) = \\mathbb{E}_{d_{\\mathrm{mid}}}\\left[\\mathrm{Var}_{i \\in M}\\left(m^{\\mathrm{post}}_T\\right)\\right]$, where $m^{\\mathrm{post}}_T$ is the mid-block latent of path $i$ at the noisiest time step $T$ (the first reverse step). On 40,000 COCO prompts, this score puts prompts into quartiles whose CLIP scores fall from 31.578 to 31.217, with the same monotone pattern for aesthetic score and ImageReward and on CC3M. The same estimator separates English from Finnish translations and yields $r = -0.79$ between mean uncertainty and CLIP score across 25 languages, with languages having more native speakers generally showing lower uncertainty. The paper reads this as evidence that expert disagreement is an estimate of epistemic uncertainty: high variance marks prompts where the model is extrapolating from under-represented training data.","pith_inferences":["Editorial extension: if inter-expert variance really tracks epistemic uncertainty, it could be used as an acquisition function for active data collection—prompts with high variance are the ones to add to training data, though the paper only demonstrates ranking, not acquisition.","Editorial extension: the Finnish 'pizza' result suggests a boundary condition: languages that share vocabulary with English may systematically under-report uncertainty even when their syntax differs, so the score should be validated against human alignment before being used as a fairness audit.","Editorial extension: the same first-step variance could be applied to other MoE architectures beyond the two tested checkpoints, but whether it generalises to non-diffusion or non-MoE models is not established by the paper."],"forward_implications":["A text-to-image system can screen prompts before committing to full denoising: high uncertainty means the prompt is likely to produce poorly aligned images, so users can revise or the system can halt early.","The score can rank prompts by expected text-image alignment on both COCO and CC3M, so it can serve as a model-specific prompt-quality metric without access to the training data.","Model audits can use EMoE to map coverage by language and vocabulary, exposing biases such as lower uncertainty for Finnish prompts that contain the recognizable English loanword 'pizza'.","The Fast EMoE variant keeps the early uncertainty check cheap enough for routine use, at about 5.5 seconds per prompt versus 3.58 seconds for standard generation.","Because the near-duplicate-checkpoint experiment uses near-identical experts, the method does not require wildly diverse experts to produce a useful uncertainty ordering."],"supporting_citations":[{"why":"Establishes ensemble variance as an epistemic uncertainty estimator, the conceptual basis for Equation 2.","marker":"[38]"},{"why":"The diffusion-specific uncertainty baseline that EMoE compares against and claims to outperform without training.","marker":"[6]"},{"why":"Defines sparse mixture-of-experts layers, the architecture whose experts EMoE separates into paths.","marker":"[54]"},{"why":"Extends sparse MoE routing and supports treating each expert as a distinct computation path.","marker":"[16]"},{"why":"Defines the latent U-Net and denoising parameterization whose mid-block latent $m^{\\mathrm{post}}_T$ carries the uncertainty score.","marker":"[49]"},{"why":"Provides the CLIP Score used as the main text-image alignment target that EMoE ranks.","marker":"[22]"},{"why":"Supplies the COCO prompts used for the main 40,000-prompt quartile experiments.","marker":"[40]"},{"why":"Supplies the CC3M prompts used to confirm the uncertainty-quality trend on a second dataset.","marker":"[53]"},{"why":"Provides the training-free gating method used to assemble and weight the pre-trained experts.","marker":"[19]"},{"why":"Provides the ImageReward human-preference metric used to corroborate the uncertainty-quality ordering.","marker":"[64]"}],"fun_headline_variants":["First-step expert variance ranks prompt quality","No-training prompt screening with MoE disagreement","One denoising step reveals prompt uncertainty","Expert split scores prompts before image generation","Variance across diffusion experts flags weak prompts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that disagreement among pre-trained fine-tunes of the same base model behaves like independent draws from a posterior over models; if expert divergence mostly reflects style preferences, routing artifacts, or shared-vocabulary effects, the score can still rank prompts without actually measuring epistemic uncertainty.","fun_headline_variants_meta":{"raw":{"variants":["First-step expert variance ranks prompt quality","No-training prompt screening with MoE disagreement","One denoising step reveals prompt uncertainty","Expert split scores prompts before image generation","Variance across diffusion experts flags weak prompts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000241,"raw_usage":{"total_tokens":1544,"prompt_tokens":992,"completion_tokens":552,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":488}},"tokens_in":608,"tokens_out":552,"duration_ms":5728,"temperature":1.0,"reasoning_tokens":488,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:15:54.948363+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Translate a matched set of Finnish prompts that differ only in including a common English loanword such as 'pizza', generate images, and compare EMoE uncertainty with human-rated or ImageReward alignment; if uncertainty drops sharply for loanword prompts while alignment does not improve, the signal is tracking token familiarity rather than prompt-level epistemic uncertainty.","supporting_citations":[{"cited_title":"Shedding light on large generative networks: Estimating epistemic uncertainty in diffusion models","cited_arxiv_id":null,"evidence_quote":"The diffusion-specific uncertainty baseline that EMoE compares against and claims to outperform without training."}],"review_version":1}