{"id":"9f49fc39-dadd-4785-a1d8-cda58b64d108","arxiv_id":"2602.06806","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RAIGen discovers rare, encoded-but-underexpressed attributes in diffusion models by ranking MSAE neurons on a minority score combining activation rarity and semantic distinctiveness.","lead":"RAIGen finds rare visual attributes in text-to-image models (like Stable Diffusion) without needing predefined categories. It analyzes internal model activations through Matryoshka sparse autoencoders, ranks features by how rarely they fire and how distinctive they are, and surfaces attributes like 'female doctor' or 'afro-textured hair' that default generations rarely produce.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Distinctiveness term d_i (Eq. 3) is the only filter separating rare attributes from rare noise, but it is validated only by LLM self-annotation; if CLIP distance is unstable for low-frequency neurons, RAIGen's top-ranked attributes may be spurious.","rationale":"The reader's weakest assumption matches the central risk I see: Eq. (4)'s distinctiveness term is the only guard against noisy rare neurons, and it lacks ground-truth validation. The paper's own F.8 demonstrates the need for this filter but validates it circularly with the LLM labeler. I do not think this overturns the conditional verdict: the user study and qualitative results give some evidence that discovered attributes are rare and often human-recognizable, and the concern is addressable with additional experiments. Therefore I would keep the verdict CONDITIONAL/UNCHANGED.","tokens_in":19622,"tokens_out":5470,"duration_ms":56413,"concrete_test":"Run a held-out semantic validation on 10 WinoBias/COCO prompts: train MSAE per prompt as in Sec. 5.2, rank neurons by frequency-only, distinctiveness-only, and the combined score, and have three human annotators label each top-10 neuron's top-activating images and heatmaps as 'coherent attribute' or 'noise'. Also compute d_i on two random halves of each neuron's top-activating images over 20 splits and record rank correlation. If the combined score's human-labeled precision is not significantly above frequency-only, or if the split-half rank correlation of d_i is below ~0.5, the distinctiveness filter is not separating semantics from sampling noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RAIGen's minority score s_i = d_i(1 - nu_i) (Eq. 4) depends on d_i (Eq. 3) to suppress low-frequency noise. The toy validation (Sec. 5.1) validates nu only under near-perfect feature-latent alignment, and F.8 shows that frequency-only ranking in real data yields 62.5% uninterpretable neurons. Therefore d_i is load-bearing. However, d_i is never checked against ground-truth semantics: F.8 judges interpretability with the same LLM annotation pipeline used to name neurons (GPT-5.2 / Llama), so 'No identified attribute' may reflect annotator failure, not noise. Statistically, for rare neurons with small nu, the centroid mu_i in Eq. (3) is estimated from very few top-activating images; a CLIP cosine distance of a small-sample centroid has high variance, and the paper reports no bootstrap or held-out stability. The user study (Table 2) verifies only that the presented attributes are rare in outputs, not that the selected neurons are semantically coherent or that d_i separates attributes from noise. If high d_i is sampling noise, the central claim of discovering interpretable rare attributes collapses.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RAIGen, a framework for label-free discovery of rare ('minority') semantic attributes in text-to-image diffusion models. It trains Matryoshka Sparse Autoencoders (MSAEs) on intermediate diffusion representations and ranks coarse-level neurons by a minority score s = d ⊙ (1−ν), where ν is neuron activation frequency and d is the CLIP cosine distance between the neuron's activation-weighted centroid and the dataset centroid. The authors validate activation frequency as a rarity signal in a controlled toy experiment, evaluate discovered attributes on WinoBias and COCO with SD v1.4 and SDXL, include a user study on attribute rarity, demonstrate extension to FLUX, and show that prompt revision based on discovered attributes can amplify rare modes.","tokens_in":19976,"tokens_out":4499,"duration_ms":43304,"significance":"If the central claim holds, RAIGen addresses a genuinely under-explored problem: moving beyond predefined fairness categories and majority-bias identification to unsupervised discovery of attributes that are internally encoded but rarely expressed. The paper has notable strengths: the toy experiment uses known ground truth with 20 seeds; the user study provides direct human evidence that discovered attributes are rare in generated images; the method is tested across multiple architectures and includes a thoughtful limitations section. However, the main quantitative validation is partly circular, and the semantic-distinctiveness term that separates rare attributes from rare noise is not independently validated. These issues bear directly on the core claim of discovering interpretable rare attributes, so the paper needs additional controls and stability analysis before the central claim can be considered established.","major_comments":[{"comment":"The main quantitative validation is partly by construction. Neurons are selected because they have low activation frequency ν_i, and Attribute Presence is then measured for attributes derived from those same neurons; low presence is therefore expected even if the neurons are semantically meaningless. The user study (Table 2) shows that human observers rate the discovered attributes as rare in images, but it does not establish that the selected neurons are semantically coherent or that d_i separates attributes from noise. Add a control condition (e.g., random low-frequency neurons, or frequency-only neurons) with human interpretability ratings, and report precision/recall against a small manually labeled set of rare attributes.","section":"§4.2, Eq. (4); Table 1; §5.3"},{"comment":"The distinctiveness term d_i is the only mechanism that suppresses rare-but-noisy neurons, but it is validated with the same LLM annotation pipeline (GPT-5.2/Llama) used to name the neurons, so the 'uninterpretable' rate is not an independent measure. Moreover, for rare neurons with small ν_i, the activation-weighted centroid μ_i is estimated from very few top-activating images, making the CLIP cosine distance high-variance; no bootstrap or split-half stability is reported. Please provide a human inter-annotator study comparing top-10 neurons from frequency-only, distinctiveness-only, and combined ranking, and report bootstrap confidence intervals for d_i.","section":"Appendix F.8; Eq. (3)"},{"comment":"The toy experiment validates activation frequency as a rarity signal only under near-perfect feature–latent alignment (mean Spearman ρ≈0.991; Hungarian matching). The paper itself notes this is a 'near-ideal' setting, and F.8 shows that frequency-only ranking yields 62.5% uninterpretable neurons in real data. The central score's robustness therefore depends on d_i, which the toy does not test. Extend the toy to include imperfect alignment, fragmented features, and noise, and verify that the combined score (Eq. 4) still recovers rare ground-truth attributes.","section":"§5.1"}],"minor_comments":[{"comment":"Typo: 'Sherif f' should be 'Sheriff'.","section":"Figure 3"},{"comment":"OpenBias presence values are identical for SD v1.4 and SDXL (0.941/0.933). Clarify whether this is expected or a reporting error.","section":"Table 1"},{"comment":"Typo: 'atributes' should be 'attributes'. Also 'prompt-revision using RAIGen discovered atributes' reads awkwardly.","section":"Appendix F.5"},{"comment":"'lower presence indicates stronger underrepresentation' is repeated verbatim twice in the same paragraph.","section":"§5.2"},{"comment":"Specify the exact timestep used for representation extraction. The main text says 'final timestep' and the appendix reports 'timestep=49' with 50 total steps; state this explicitly.","section":"Appendix D.1"},{"comment":"The FLUX experiments use a single hook point (transformer.transformer.blocks.18) and 4-step sampling. State whether the few-step setting affects the MSAE training distribution or the interpretability of the discovered neurons.","section":"Appendix F.2"}],"recommendation":"major_revision","confidential_remarks":"The evaluation design makes it difficult to distinguish genuine discovery from selection on the ranking criterion; code and data release would substantially help. The 'first framework' claim is strong and should be moderated unless a thorough concurrent-work check is included. The identical OpenBias values in Table 1 need closer scrutiny before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nShort version: this paper is the first to tackle rare-attribute discovery in T2I models, and it has a sensible pipeline: train an MSAE on diffusion bottleneck activations, rank neurons by a minority score that multiplies semantic distinctiveness with (1 − activation frequency), then annotate the top-activating images with an LLM. The toy experiment (Sec 5.1) is solid evidence that low activation frequency tracks true rarity under near-perfect feature–latent alignment. The user study (Table 2) gives human confirmation that the discovered attributes are rare in standard generations. The cross-model auditing and FLUX experiments widen the claim in a useful way. They also state their limitations honestly.\n\nThe soft spot is the distinctiveness term d_i. Frequency alone is too noisy (their own F.8 reports 62.5% uninterpretable neurons), so d_i is the filter that separates rare semantics from rare noise. But d_i is never validated against ground truth in real data. In F.8, \"uninterpretable\" is judged by the same LLM pipeline that names the neurons, so annotator failure is conflated with noise. The centroid for a rare neuron is averaged over very few top-activating images, so the CLIP cosine distance is high-variance; no bootstrap or stability check is reported. This is load-bearing, and the appendix underscores it.\n\nSecond, Table 1 is partly circular: you select for low frequency, then confirm the corresponding attributes appear rarely. That is expected. The toy experiment and user study are the real independent evidence, and they support the core idea, but they do not validate d_i. Hyperparameters were tuned without a held-out split, code is not released, and the OpenBias comparison is a different task, not a baseline.\n\nWho this is for: people auditing diffusion models for underrepresentation beyond fixed demographic categories, and researchers in sparse autoencoder interpretability. It deserves a serious referee — the task is new, the methodology is plausible, and the toy plus user studies are real evidence. The revision should add stability analysis for d_i, a code release, and a clearer separation between selection and evaluation.\n\nRecommendation: send to peer review with a careful reviewer.","headline":"RAIGen is a plausible new method for rare-attribute discovery, but its key semantic filter is under-validated and the main quantitative table is partly circular.","tokens_in":20408,"tokens_out":3384,"would_cite":true,"duration_ms":33066,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RAIGen claims diffusion models hide rare attributes that can be found by ranking sparse, semantically distinct neurons without predefined categories.","keywords":["text-to-image diffusion","rare attribute discovery","minority attributes","sparse autoencoders","Matryoshka sparse autoencoders","bias auditing","semantic distinctiveness","prompt amplification"],"falsifier":"Take a diffusion model and a prompt where the rare attributes are known by construction—for example, injecting a controlled set of rare styles or objects into the sampling distribution—run RAIGen, and check whether the top-ranked neurons correspond to those injected attributes; if most top neurons are uninterpretable or unmatched, the distinctiveness filter is not tracking human-meaningful rarity.","tokens_in":19575,"feed_emoji":"🎨","tokens_out":5506,"duration_ms":53711,"temperature":0.7,"pith_summary":"RAIGen is a framework for finding rare attributes that a text-to-image diffusion model encodes internally but rarely expresses in its outputs. Its central claim is that such minority attributes can be discovered automatically, with no predefined fairness categories, by training a Matryoshka sparse autoencoder on the model's bottleneck representations and ranking neurons by a minority score combining low activation frequency with semantic distinctiveness. The paper shows the discovered attributes—such as female doctor, doctor in a framed portrait, and afro-curly hair—appear in only about a fifth of generated images, far below majority attributes, and that human raters see them in fewer than 3 of 10 images. If correct, this gives auditors a way to find underrepresentation beyond gender and race, and a route to amplify suppressed modes through simple prompt edits. The method is demonstrated on Stable Diffusion v1.4, SDXL, and FLUX.","feed_headline":"Sparse neurons reveal the rare attributes diffusion models hide","feed_subtitle":"Rarely firing yet semantically distinct neurons expose suppressed attributes, enabling audits and targeted amplification.","key_machinery":"The load-bearing object is the Matryoshka Sparse Autoencoder (MSAE), trained on per-pixel bottleneck features extracted from the diffusion model's denoiser, producing nested coarse-to-fine sparse codes; the framework uses the coarsest level for semantically coherent attributes. On top of it sits the minority score s_i = d_i (1 − ν_i), where ν_i is the neuron's activation frequency across generated samples and d_i is the CLIP-distance between the neuron's activation-weighted image centroid and the dataset centroid; this product separates rare-but-meaningful neurons from rare noise and from frequent coherent sub-modes. Top-activating images and spatial heatmaps serve as the interpretive check,","core_discovery":"The core claim is that rare attributes are accessible as interpretable neurons in a Matryoshka sparse autoencoder trained on diffusion bottleneck representations. RAIGen scores each neuron with s(z) = d ⊙ (1 − ν), where ν is how often the neuron fires and d is the cosine distance between the neuron's activation-weighted CLIP centroid and the dataset centroid, so top-scoring neurons are both infrequent and semantically separated from the average. At the coarsest MSAE level these neurons correspond to coherent underrepresented concepts—social, contextual, stylistic, and compositional—verified through top-activating images, spatial heatmaps, and a user study. The paper also finds that suppressi","pith_inferences":["If RAIGen's premise holds, the same minority score could be used during training or fine-tuning as a monitoring signal to detect when rare modes are collapsing, not just after deployment.","Because distinctiveness is measured in CLIP space, attributes CLIP cannot separate will likely be invisible to RAIGen even if they are visually distinct—an implication of the method's design that the paper does not test.","The paper's pipeline trains a separate MSAE per prompt; amortizing representation learning across prompts could turn RAIGen into a continuous auditing tool rather than a per-scenario analysis.","Combining RAIGen's internal grounding with external world-knowledge models could cover both encoded-but-suppressed attributes and expected-but-absent ones, yielding a fuller picture of underrepresentation than either approach alone."],"forward_implications":["Bias auditing can move beyond predefined demographic categories to rare stylistic, contextual, and cultural attributes that the model encodes but rarely expresses.","RAIGen scales to larger models: attribute presence is slightly lower in SDXL than SD v1.4, suggesting larger capacity does not automatically fix underrepresentation.","Discovered attributes can be amplified through prompt revision, reducing deviation from a balanced attribute distribution while keeping generations close to the original prompt semantics.","Cross-model audits show that underrepresentation shifts with architecture—demographic gains can trade off with stylistic or contextual diversity, so auditing needs to track all forms of rarity.","The method extends to transformer-based diffusion models like FLUX, though high-scoring neurons there are more often weakly interpretable."],"fun_headline_variants":["Rare attributes in text-to-image models revealed by sparse neurons","Sparse autoencoders surface minority features diffusion models suppress","Auditing hidden biases: neuron sparsity finds rare attributes in AI images","Targeted amplification: using sparse neurons to control rare attributes","Label-free discovery of underrepresented concepts in diffusion models"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The framework's load-bearing premise is that 'semantic distinctiveness' measured by CLIP embedding distance separates genuinely meaningful rare attributes from rare-but-noisy or fragmented neurons; the paper validates activation frequency on a toy setting but never validates the CLIP filter against ground-truth semantics on real data, and its own ablation shows frequency alone yields 62.5% uninterpretable neurons.","fun_headline_variants_meta":{"raw":{"variants":["Rare attributes in text-to-image models revealed by sparse neurons","Sparse autoencoders surface minority features diffusion models suppress","Auditing hidden biases: neuron sparsity finds rare attributes in AI images","Targeted amplification: using sparse neurons to control rare attributes","Label-free discovery of underrepresented concepts in diffusion models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00019,"raw_usage":{"total_tokens":1171,"prompt_tokens":734,"completion_tokens":437,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":354}},"tokens_in":478,"tokens_out":437,"duration_ms":5031,"temperature":1.0,"reasoning_tokens":354,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T03:46:02.092546+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a diffusion model and a prompt where the rare attributes are known by construction—for example, injecting a controlled set of rare styles or objects into the sampling distribution—run RAIGen, and check whether the top-ranked neurons correspond to those injected attributes; if most top neurons are uninterpretable or unmatched, the distinctiveness filter is not tracking human-meaningful rarity.","supporting_citations":[],"review_version":1}