{"id":"93de670d-dafa-4098-9f47-9ef453bacca5","arxiv_id":"2505.19313","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"In a controlled synthetic setup, concept reachability in diffusion models drops abruptly below about 1% training presence, and activation steering remains more reliable than prompting under scarcity, underspecification, and bias.","lead":"This paper tests when text-to-image diffusion models can produce concepts that are scarce, poorly captioned, or biased in the training data, comparing standard prompting with steering that modifies internal activations. It finds a sharp drop in reachability below about 1% concept presence, and that steering often outperforms prompting under dataset degradation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 1% phase-transition threshold is confounded with an absolute sample-count effect because total dataset size is preserved while concept frequency is varied.","rationale":"The paper is an empirical study whose headline is a phase transition: reachability remains high when a concept appears in more than about 1% of the training set and collapses below it, implying that very few samples are sufficient. This claim is the basis for the abstract's statement that only a small number of samples enables reachability, and for the practical suggestion that providers need not curate large datasets. I therefore looked for the condition that would have to be true for this claim to hold: pX(f) must be the operative variable, not the absolute count of target images or the total dataset size. The protocol in Section 4.1 and Appendix B never separates these quantities. Preserving total dataset size while lowering pX(f) necessarily lowers the number of images in [f]X, and if the reduction is applied to the subset [c1=red]X, it also lowers the number of images of the exact target combination unless special care is taken. The reader's weakest assumption was external validity, namely that the synthetic setup transfers to real text-to-image models. My concern is more basic: even within the synthetic setup, the threshold is not identified as a frequency effect rather than an absolute-count effect. The CelebA experiment uses one dataset size and so cannot resolve the confound, and the Stable Diffusion results are qualitative. This does not mean the paper's other contributions are invalid: the framework, the steering methodology, and the observation that prompt-space steering can outperform prompting under scarcity and underspecification are valuable and may well survive. But the 'phase transition at 1%' is precisely the kind of claim that should be robust to dataset scale; without a scale sweep, it remains a candidate artifact. The authors released code and the setup is inexpensive (3.7M-parameter U-net, 64x64 images), so the proposed test is feasible with modest compute. My verdict recommendation is unchanged: the reader already assigned CONDITIONAL, and this concern adds a specific new condition (scale-controlled scarcity experiments) rather than overturning the overall assessment.","tokens_in":22447,"tokens_out":7216,"duration_ms":76298,"concrete_test":"Re-run the scarcity experiment from Figure 5 (c1 = red) at total dataset sizes of 27,000 and 108,000, using identical absolute counts for the reduced subset [c1 = red] (for example, 0, 135, 270, 540, 1080, 2160, 4320, and 10800 images), and plot reachability against both pX(c1) and the absolute count. Also report the number of images of the exact target combination (red, triangle, green, square) at each setting. If the transition aligns with the same absolute counts across dataset sizes, the 1% claim is an artifact of the fixed-total-size design; if the transition consistently occurs at p≈0.01 despite a 4-fold change in dataset size, the relative-threshold interpretation is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim (Section 5.2 and the Abstract) is that reachability collapses when a concept's frequency falls below roughly 1% of the training set, implying that only a small number of samples suffices to learn a concept. The experimental design, however, does not separate frequency from absolute counts. In Section 4.1 and Appendix B, the dataset starts at 54 concept combinations with 1000 images each (54,000 total), and when the presence of a concept is reduced, the total dataset size is preserved by proportionally increasing the remaining combinations. Thus pX(f) and the absolute number of images in [f]X are varied together. Depending on how the reduction is implemented, one of two problems arises. If the target combination's count is reduced along with the subset [c1=red]X, then the abrupt drop at p≈0.01 corresponds to an absolute count of roughly 540 images for the concept and roughly 30 images for the exact target combination if the reduction is uniform; at a different total dataset size the same curve would appear at a different p, so the '1% threshold' and the 'small number of samples relative to train set size' reading are unsupported. If instead only the other combinations in the subset are reduced while the exact target combination stays at 1000 images, then the exact target combination is not scarce in absolute terms, and the experiment tests positional transfer rather than learning a concept from few examples. The CelebA check in Appendix G uses a single dataset size (16,000) and therefore cannot resolve this confound. This is a correctness issue internal to the synthetic setup, not merely a question of whether the result transfers to production-scale models.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies concept reachability in text-to-image diffusion models under three dataset limitations: concept scarcity, caption underspecification, and co-occurrence bias. The authors introduce a synthetic dataset of colored shapes (54 concept combinations, 54,000 images) with known generative factors, train small 3.7M-parameter diffusion models, and compare prompt-only generation against steering in prompt-embedding space and U-Net bottleneck h-space. They report (i) a sharp, phase-transition-like drop in reachability when concept presence falls below roughly 1% of the training data, (ii) that the layer/space where steering is applied changes reachability, and (iii) that steering maintains reachability when prompting fails. They supplement the synthetic study with qualitative Stable Diffusion examples and a small CelebA experiment.","tokens_in":22766,"tokens_out":6894,"duration_ms":45250,"significance":"If the results hold, the paper provides a useful empirical map of when concepts can be accessed by prompting versus activation steering, and it argues for user-facing control mechanisms as an alternative to dataset curation. The synthetic design is a strength: the generative factors are known, the dataset modifications are controlled, and the code is released. The authors also honestly acknowledge limitations in Section 6. However, the central quantitative claims are not yet supported at the precision required: the scarcity experiments conflate relative frequency with absolute counts, the p=0 steering results lack a described source for the steering-image set, and the phase-transition threshold is drawn without uncertainty quantification. The real-data sections are qualitative or use a max-ensemble reporting choice that makes quantitative comparison difficult.","major_comments":[{"comment":"The scarcity manipulation changes pX(f) while preserving the total dataset size (54,000 images), so the relative frequency and the absolute number of images containing the target are varied together. At the claimed threshold p ≈ 0.01, the target combination (red, triangle, green, square) has roughly 540 images of the concept and only about 30 images of the exact combination (since the subset [c1=red]X contains 18 combinations). The design therefore cannot separate a frequency effect from an absolute sample-count effect, and the statement that the model needs 'few data points, relative to the train set size' is unsupported. The authors should run control experiments in which the total dataset size is varied (e.g., scaling all combinations) while holding pX(f) fixed, or report absolute counts and show that the same threshold is obtained across different total sizes.","section":"§4.1, Appendix B, §5.2"},{"comment":"The steering method in Section 4.2 requires a collection Z of images containing the target concept combination to optimize the vector vp or vh. In Section 5.2, results are reported at pX(f)=0, where the target combination is absent from the training set, but the manuscript does not specify where Z comes from in this condition. This makes the p=0 steering results irreproducible and weakens the claim that steering can reach concepts 'even when no training images contain the reduced concept.' The authors must state the source of Z (e.g., external images, images with the concept in another position, or model-generated images) and confirm that the same Z is used consistently across conditions.","section":"§4.2 and §5.2"},{"comment":"The phase-transition claim is based on mean-over-four-seeds curves with no error bars, no per-seed plots, and a threshold marked 'by eye' with a dashed red line. Given the frequency/abundance confound raised above, the strong wording 'distinct phase transition' and 'concept-agnostic, low threshold' in the contributions is not yet justified. The authors should provide per-seed results or confidence intervals, and ideally use an automated or pre-specified procedure to estimate the threshold and its uncertainty, for both the synthetic and CelebA experiments.","section":"§5.2, Figures 5 and 14, Appendix G.3"},{"comment":"The CelebA scarcity result reports the steering curve as max(prompt space, h-space). Taking the maximum of two methods per point is a form of cherry-picking and prevents a meaningful comparison of steering against prompting, which is the stated purpose of the experiment. The authors should plot both methods separately with error bars, or justify a principled way of selecting one method ahead of time.","section":"Appendix G.3, Figure 21"},{"comment":"The Stable Diffusion experiments are presented as evidence that 'concepts often remain reliably reachable through steering' in real settings, but the evaluation is qualitative: 50 images per concept are generated and a few sample panels are shown, with no quantitative accuracy or counts reported. The text acknowledges that steering sometimes fails (e.g., Figure 18), yet the summary claim is stronger than the evidence. The authors should either report the fraction of successful/improved images for the 50-image evaluation or explicitly label this part as anecdotal and move the quantitative claim to the synthetic and CelebA results.","section":"§5.5 and Appendix F"}],"minor_comments":[{"comment":"The text says 'Reachability, as defined in Section 5', but Definition 3.1 is in Section 3.4; please correct the cross-reference.","section":"§4.3"},{"comment":"The paragraph on starting-prompt selection is ambiguous: it says the starting prompt that maximizes h-space performance is chosen, then says 'unless stated otherwise' the starting prompts describe the target concepts. Please clarify which starting prompt is used for each experiment and whether the h-space choice was made on held-out targets or on the same targets reported later.","section":"§5.1"},{"comment":"The p-axis ranges differ between the main figure (0 to 0.08) and Appendix E.3 (0 to 0.175), making visual comparison of the threshold behavior across figures harder. Please use consistent axis ranges or clearly annotate the difference.","section":"Figures 5 and 13"},{"comment":"The contribution states a 'concept-agnostic, low threshold', but only four concept values (red, green, square, triangle) are varied in the scarcity experiments. Please soften this to 'observed for the tested concepts' or provide evidence across a wider set of concepts.","section":"Contributions bullet list"},{"comment":"The CelebA figure caption says the dotted red line marks threshold 0.01, but no error bars or standard deviations are shown for the four seeds; please add them or state why they are omitted.","section":"Appendix G.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a machine-learning venue and the synthetic framework is a useful contribution. The main risk is that the '1% phase transition' will be quoted as a universal law despite the confound between relative frequency and absolute count and the lack of uncertainty quantification. I recommend the authors run additional control experiments varying total dataset size and clarify the p=0 steering setup before publication. No ethical concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful empirical framework for thinking about when concepts are reachable in text-to-image models, and the steering-vs-prompting comparison is worth engaging with. But the headline claim—a sharp phase transition at around 1% training-data presence—is not actually supported by the experiments, because the design confounds concept frequency with absolute sample count.\n\nThe good stuff first. The paper introduces a clean distinction between compositionally OOD and positionally OOD concept combinations, which maps well onto how diffusion models can fail. The synthetic setup is well-controlled: 54 concept combinations of colored shapes on a black background, captions generated from known factors, three separate dataset-level obstacles varied systematically, and a released implementation. The finding that prompt-space steering remains effective under scarcity while prompting collapses is interesting and, based on the qualitative Stable Diffusion examples, plausible. The h-space results are more fragile, but the observation that intervention location matters is a useful point. The underspecification results—that reachability tracks what is actually in the captions—are solid and align with prior work.\n\nNow the soft spots. The central quantitative claim in the abstract and Section 5.2 is the phase transition at p≈0.01. Section 4.1 states that when concept presence is reduced, the total dataset size is preserved by increasing the remaining combinations. That means the proportion p and the absolute number of images containing the concept are varied together. A drop at p=0.01 corresponds to a specific absolute count (roughly 540 images for the concept and roughly 30 for the exact target combination, if the reduction is uniform). The paper reads this as \"small number of samples relative to the train set,\" but that is not what the experiment shows. To support that claim you would need to vary the total dataset size or fix the absolute count and vary the proportion independently. The CelebA check uses a single dataset size and cannot resolve this either. This is a real confound, not a matter of transferability to production-scale models.\n\nOther issues are minor by comparison: results are means over four seeds with no error bars; the 0.01 threshold is drawn by eye; the starting prompts for h-space steering are chosen post-hoc to favor h-space; and the real-data section is qualitative. The abstract's phrase \"reliably reachable\" is stronger than the evidence.\n\nWho this is for: anyone working on controllable generation, dataset curation, or activation steering in diffusion models. The framework and the negative results about prompting are worth having, but the empirical threshold claim needs to be redone or heavily tempered.\n\nRecommendation: send it to review. A serious referee will ask for variance estimates, a disentangling of frequency from absolute counts, and softer real-data language. Even if the 1% threshold does not survive, the rest of the paper is worth publishing.","headline":"Useful framework and a plausible steering-vs-prompting story, but the headline 1% phase transition is confounded with absolute sample counts and needs to be redone.","tokens_in":23227,"tokens_out":3074,"would_cite":true,"duration_ms":29185,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper establishes a phase transition in concept reachability: below roughly 1% training frequency, concepts become unreachable, while activation steering recovers concepts that prompting cannot.","keywords":["concept reachability","diffusion models","activation steering","phase transition","dataset scarcity","caption underspecification","data bias","text-to-image generation"],"falsifier":"Measure generation accuracy on a production-scale text-to-image model across concepts whose training frequency spans 0.001% to 5%: if reachability declines smoothly with frequency rather than stepping sharply near $p_X(f) \\approx 0.01$, or if steering does not beat prompting on positionally out-of-distribution combinations, the paper's two central claims fail.","tokens_in":22284,"feed_emoji":"🎨","tokens_out":11605,"duration_ms":90947,"temperature":0.7,"pith_summary":"This paper asks when a text-to-image diffusion model can actually generate a concept it was trained on, and whether the answer depends on how the concept is requested — by text prompt or by steering the model's internal activations. Working with a synthetic dataset of coloured shapes where scarcity, caption quality, and bias are varied one at a time, it claims that concept reachability undergoes a phase transition: once a concept appears in fewer than roughly 1% of training images, the model abruptly loses the ability to generate it, while above that threshold extra examples barely help. It also claims that the layer at which steering is applied decides what is reachable — modifying the text embedding is stable, while modifying the U-net's bottleneck is fragile. The practical upshot is that steering recovers concepts prompting cannot reach, such as concepts seen only in the wrong position or concepts entangled by bias, so model providers could ship steering-based controls instead of retraining. The one limitation that binds all methods is that a concept never named in captions is unreachable by any of them.","feed_headline":"Below 1% in the data, concepts slip out of reach","feed_subtitle":"Diffusion models lose a concept abruptly at a ~1% training-frequency threshold; steering recovers it when prompts fail.","key_machinery":"The carrying mechanism is the steering vector, a vector optimised on frozen model weights to minimise the denoising loss $\\mathbb{E}\\lVert \\epsilon_t - \\epsilon_\\theta(x_t, t, y_s, v)\\rVert^2$ over an auxiliary image set containing the target concept, then added either to the prompt's text embedding (prompt-space steering, $v_p$) or to the U-net bottleneck output (h-space steering, $v_h$). Reachability is scored by three trained classifiers that read off back shape, front shape, and the back-front colour pair from each generated image, with incomplete images counted as incorrect. The load-bearing phenomenon is the phase transition at $p_X(f) \\approx 0.01$: a concept-agnostic, low threshold above which more data adds little and below which reachability drops abruptly. The contrast between the two steering spaces — $v_p$ stable and prompt-matching, $v_h$ fragile with accuracy falling as more concepts are modified and correlating with the optimised vector's final norm — locates the effect at a specific stage of the transformation.","core_discovery":"The paper's central claim is that concept reachability — defined as the fraction of generated images that contain the target concept combination $(c_1, s_1, c_2, s_2)$ — is governed by the structure of the training data in a sharp, quantifiable way, and that activation steering is systematically more resilient than prompting. In the synthetic coloured-shapes setting, reachability stays high as a concept's frequency is reduced and then collapses once the concept appears in fewer than about 1% of training images, $p_X(f) \\approx 0.01$, a phase transition observed whichever of the four concept positions is thinned. When the target combination is positionally out-of-distribution — each individual concept appears in training but not in the requested position, for instance red shapes only ever appear in front but are requested in back — prompting falls to near zero while prompt-space steering reaches the combination with over 50% accuracy in one of the studied cases. When captions no longer name a concept, reachability drops for both prompting and steering, showing that captions organise the latent space and that a never-captioned concept is effectively unreachable. When two concepts are perfectly tied in the training data, steering, particularly on the prompt space, partially disentangles them, and adding independent examples of one tied concept lifts reachability of both.","pith_inferences":["If the phase transition is generic, comparable sharp thresholds should appear when steering language models toward learned concepts, and the critical frequency may shift with model scale and task complexity rather than staying at 1%.","The correlation between h-space reachability and the optimised vector's final norm suggests a cheap, image-free probe: estimate reachability from vector norms to audit which concepts a model can access.","Repeating the scarcity sweep across model sizes within one architecture would separate two explanations — a property of the learning task versus a property of model capacity — that the paper's single small model cannot distinguish.","A corollary for evaluation practice is that prompt-to-image alignment scores understate model capability, and steering-based evaluation would give a fairer measure of what a text-to-image model has actually learned."],"forward_implications":["Concepts present in at least roughly 1% of training images are reliably learnable, so data collection can stop once a concept crosses the threshold instead of chasing large counts.","Steering in the prompt space reaches concepts that prompting cannot, so model providers could ship user-facing steering controls in place of costly retraining and dataset curation.","A concept that never appears in captions is unreachable by every method tested, making caption completeness a hard lower bound on what a model can generate on demand.","Positionally out-of-distribution combinations are mostly lost to prompting but partially recovered by steering, so reachability depends on how a concept is requested, not only on whether it was learned.","The intervention layer itself determines what is reachable: text-embedding steering is stable across targets, while bottleneck steering degrades as more concepts are modified."],"supporting_citations":[{"why":"Supplies the factor-of-variation data model and the premise that train-set biases shape generative behaviour regardless of architecture.","marker":"(Zhao et al., 2018)"},{"why":"Establishes that dataset skews hinder text-to-image generalisation, the prompting-failure baseline this paper extends to steering.","marker":"(Chang et al., 2025)"},{"why":"Shows diffusion models carry semantically meaningful directions in latent space, motivating activation steering as the intervention.","marker":"(Kwon et al., 2023)"},{"why":"Provides the h-space steering method whose bottleneck vector the paper optimises and compares against prompt-space steering.","marker":"(Li et al., 2024a)"},{"why":"Documents a steerability bias in steering vectors, which the h-space reachability bias observed here is aligned with.","marker":"(Tan et al., 2024)"},{"why":"Supplies the real-data concept-frequency counts used to tie the synthetic threshold to production-model behaviour.","marker":"(Samuel et al., 2024)"},{"why":"Reports sudden transitions in when diffusion models acquire concept-generation ability, the emergence phenomenon the phase transition builds on.","marker":"(Park et al., 2024)"}],"fun_headline_variants":["Steering beats prompting when data is scarce","Diffusion models lose concepts below 1% in data","Activation steering rescues hard-to-reach concepts","Concept reachability crashes at 1% training frequency","Prompt fails, steering succeeds: concept reachability"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The synthetic coloured-shapes dataset and the small 3.7M-parameter model faithfully represent production-scale text-to-image models, so the $p_X(f) \\approx 0.01$ threshold and the steering advantage transfer to real systems.","fun_headline_variants_meta":{"raw":{"variants":["Steering beats prompting when data is scarce","Diffusion models lose concepts below 1% in data","Activation steering rescues hard-to-reach concepts","Concept reachability crashes at 1% training frequency","Prompt fails, steering succeeds: concept reachability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000151,"raw_usage":{"total_tokens":1232,"prompt_tokens":1007,"completion_tokens":225,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":623,"completion_tokens_details":{"reasoning_tokens":151}},"tokens_in":623,"tokens_out":225,"duration_ms":2514,"temperature":1.0,"reasoning_tokens":151,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:16:12.022848+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure generation accuracy on a production-scale text-to-image model across concepts whose training frequency spans 0.001% to 5%: if reachability declines smoothly with frequency rather than stepping sharply near $p_X(f) \\approx 0.01$, or if steering does not beat prompting on positionally out-of-distribution combinations, the paper's two central claims fail.","supporting_citations":[],"review_version":1}