{"id":"75f6ac0d-6c00-4219-9816-e8b9cf88a52d","arxiv_id":"2412.20651","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A scalar bias added to the latent noise of a diffusion model, tuned per target dataset, improves counterfactual medical image generation after fine-tuning.","lead":"This paper proposes Latent Drift, a tuned scalar shift added to the noise of a pre-trained diffusion model, to help it generate medical images like brain MRI and chest X-rays. The method improves image quality metrics for counterfactual tasks such as aging or adding and removing disease, but the gains rely on fitting this scalar to the target data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper does not show that grid-searched δ is a principled solution to Eq. (5), and Table 1 itself shows AUC decreases for Custom Diffusion, so the claim of consistent distribution-gap closure is not yet supported.","rationale":"The reader identified the weakest assumption as the sufficiency of a single scalar δ to implement the counterfactual objective and close the distribution gap. I agree that this is the weakest point, but I sharpen it further: the reported FID gains may be inflated by selecting δ on the evaluation distribution, and the paper's own Table 1 shows AUC dropping for Custom Diffusion with LD, which contradicts the claim of universal improvement. The proposed concrete test would settle whether the method's benefit is robust and whether the per-step mechanism matters. I keep CONDITIONAL rather than REJECT because the method is simple, reproducible, and the FID improvements are consistent in direction across four methods; the issue is whether the effect is due to the proposed mechanism or to test-time hyperparameter fitting. The reader did not emphasize the AUC inconsistency or the validation-split issue explicitly, so agreement is partial.","tokens_in":13202,"tokens_out":2490,"duration_ms":25827,"concrete_test":"Split the target dataset into a δ-selection split and a held-out evaluation split. For each fine-tuning method, choose δ by minimizing the L1 distance on the selection split only, then compute FID/KID and classification AUC on the held-out split. If the improvements over δ=0 shrink or disappear, the reported gains are due to selection on the evaluation distribution. Additionally, compare LD against a one-time shift of only the initial latent z_T by δ (keeping the reverse process otherwise unchanged); if this matches LD's performance, the per-step δ mechanism in Eq. (4) is not the active ingredient.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that a single scalar δ added to the reverse-process mean (Eq. 4) operationalizes the counterfactual objective (Eq. 5) and closes the distribution gap between pre-trained and medical domains. The derivation does not connect Eq. (5) to δ: Eq. (5) requires a classifier f and two fidelity losses, but δ is selected by minimizing an L1 distance between generated samples and the target dataset, with no classifier or counterfactual structure. The paper states this δ is found via grid search, but does not specify a validation split or cross-validation; if the same target samples used for FID evaluation are used for δ selection, the reported FID improvements could reflect test-time hyperparameter fitting rather than a genuine domain-adaptation mechanism. Moreover, Table 1 shows that on brain MR, LD reduces Custom Diffusion AUC from 0.609 to 0.544 even while improving FID, so the benefit is not consistent across metrics. Because the headline claim is that LD improves all fine-tuning methods and closes the distribution gap, this inconsistency weakens the claim. The paper's own text (Sections 3.3-3.4) asserts, rather than derives, that a constant shift is sufficient and that δ minimizes the counterfactual distance; no principled selection rule or optimality argument is provided.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Latent Drift (LD), a scalar offset δ added to the mean of the reverse diffusion transition (Eq. 4), with the goal of adapting pre-trained Stable Diffusion models to medical image domains. The authors claim that LD operationalizes a counterfactual objective (Eq. 5), can be combined with any fine-tuning method, and improves counterfactual medical image generation on brain MRI (ADNI/OASIS) and chest X-ray (CheXpert) data. The method selects δ by grid search to minimize an L1 distance between generated samples and the target dataset, and is evaluated with FID/KID/AUC, image-to-image aging and disease editing, and several ablations.","tokens_in":13521,"tokens_out":4787,"duration_ms":47500,"significance":"If the empirical claims held, the contribution would be practically attractive: a simple scalar intervention at inference time that improves FID/KID for several fine-tuning schemes, with potential applications in counterfactual medical image synthesis from small datasets. The paper's strengths are its breadth of experiments, use of three public benchmarks, and integration with multiple fine-tuning backbones. However, the formal grounding is not delivered: the connection between the counterfactual objective and δ is asserted rather than derived, the selection of δ is a grid search on the same target distribution later used for evaluation, and the headline improvement is not consistent across metrics (e.g., Custom Diffusion AUC on brain MR decreases with LD). These issues must be resolved before the empirical claims can be considered reliable.","major_comments":[{"comment":"The connection between the counterfactual objective and the scalar δ is asserted, not derived. Eq. (5) requires a classifier f and two fidelity terms ℓ_o and ℓ_in, but no classifier appears in the generation or δ-selection procedure. The statement 'ℓ_in ∝ 1/ℓ_o' is unsupported and appears to contradict the usual trade-off: increasing desired-outcome fidelity generally increases, not decreases, the input change needed for a counterfactual. The text then states that for λ>0 the minimum value of δ is found via grid search, but the grid search minimizes an L1 distance to the target dataset, not Eq. (5). The authors should either derive a concrete relation between δ and Eq. (5) or reframe LD as a heuristic distribution-shift correction and remove the counterfactual optimization formalism.","section":"Section 3.4, Eq. (5)"},{"comment":"δ is selected by minimizing an L1 distance between generated samples and the target dataset, and the same target/test data are then used for FID/KID/AUC evaluation. The paper does not report a validation split, cross-validation, or the number of target samples used for δ selection. Without this separation, the FID improvements in Table 1 may reflect fitting δ to the test distribution rather than a genuine domain-adaptation mechanism. Please specify the exact selection protocol and re-evaluate on held-out data or with nested validation.","section":"Sections 3.3–3.4 and 4.1"},{"comment":"The claim that LD improves all fine-tuning methods is contradicted by the brain MR AUC for Custom Diffusion, which drops from 0.609 to 0.544 with LD even though FID improves. Since the headline claim is consistent distribution-gap closure, the paper must reconcile this discrepancy, report per-metric consistency, or qualify the claim. In addition, all FID/KID/AUC values are point estimates without error bars or significance tests; given the small generated-sample counts (200 for brain MR, 400 for chest X-ray), the reported differences may not be reliable.","section":"Table 1"},{"comment":"The L1 norm between generated samples and target samples is used as a distance between distributions without definition or justification. Pairwise L1 distances between image tensors are not a standard divergence between distributions and can be minimized by degenerate strategies such as memorizing or averaging target samples. The authors should justify this choice, replace it with a proper distribution distance (e.g., MMD with a suitable kernel), or demonstrate on a validation set that the L1 criterion tracks FID/KID.","section":"Section 3.3"},{"comment":"The roles of λ and the 'min-max' formulation are not operationalized. The paper sets λ=1 in the counterfactual experiments but does not describe how the two terms of Eq. (5) are evaluated, and τ is ablated in the supplement without being defined in the main text. The reader cannot determine what objective is actually optimized during fine-tuning versus during inference, nor when δ is added (forward process, reverse process, or both) and over what grid. Please provide the complete algorithm, including the exact δ grid and the precise training and inference protocols.","section":"Section 3.4 and Experiments"}],"minor_comments":[{"comment":"The definition of ᾱ_t is garbled; it should be the product of α over time steps, not 'αt := 1 − βt and ¯αt = Qt s=0 αst + 1' as written.","section":"Section 3.1, Eq. (1)"},{"comment":"Custom Diffusion is cited as reference [11], but reference [11] is Textual Inversion; the correct citation for Custom Diffusion is [22].","section":"Section 2"},{"comment":"The row 'Stable Diffusion [40]' with ✗ is ambiguous; it should clarify whether this is no fine-tuning, basic fine-tuning without LD, or something else.","section":"Table 1"},{"comment":"'Structure Similarity Inced' should be 'Structural Similarity Index Measure (SSIM)'.","section":"Section 4.2.2"},{"comment":"The paper should report the exact δ values chosen for each experiment and provide a quantitative sensitivity curve of FID or AUC as a function of δ; the qualitative examples in Figure 2 do not convey the effect size.","section":"Figure 2 and Section 3.4"}],"recommendation":"major_revision","confidential_remarks":"The central formal claim (Eq. 5 to δ) is not backed by any derivation, and the δ-selection protocol appears circular with respect to the evaluation data, so the empirical results as presented cannot be taken at face value. That said, the underlying heuristic may be salvageable through a reframing as a simple distribution-shift correction with proper validation splits and error bars, and the experimental breadth is useful. I therefore do not recommend rejection at this stage, but the revision must address the load-bearing issues above."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is genuinely simple: add a scalar δ to the mean of the noise in the forward and reverse diffusion processes, and you get consistent FID/KID improvements across four fine-tuning methods on brain MRI and chest X-ray. That is a cheap, practically useful trick for small-data medical settings, and the paper shows it works with several popular fine-tuning schemes. The qualitative examples support the numbers; the generated slices look more like real anatomy with LD than without. Credit where due: the systematic evaluation across methods and datasets is the paper's real strength, and the gains are consistent for FID and KID, which makes the effect hard to dismiss as pure noise.\n\nThe soft spots are real but not disqualifying. The theory is mostly asserted: Eq. (5) is a standard counterfactual objective, but the connection to a single scalar δ is never derived. The statement \"ℓ_in ∝ 1/ℓ_o\" is unjustified, and using an L1 norm as a distribution distance is a bit hand-wavy. More importantly, δ is selected by grid search against the target dataset, and the same target distribution is used for FID/KID evaluation. That is a circularity concern: part of the gain may be fitting the hyperparameter to the test set. The paper does not report error bars or a validation split, so we cannot quantify how much. Also, Table 1 shows AUC drops for Custom Diffusion with LD on brain MR (0.609 to 0.544), so the benefit is not consistent across all metrics, even if FID looks better. The supplement is referenced but absent, which makes the method harder to reproduce than it should be.\n\nFor all that, the core empirical pattern is plausible and worth taking seriously. The mechanism is easy to reimplement, and the consistent FID/KID gains suggest there is something real here, even if the \"closes the distribution gap\" claim is overstated and the theory needs tightening. I would send this to a serious referee; with a request for error bars, a validation split for δ selection, and a more honest framing of what is being optimized, it could become a solid practical paper.\n\nBottom line: worth a proper review, not a desk reject, but the authors should be pushed on the circularity and the missing supplement.","headline":"A simple noise-mean shift that gives consistent FID/KID gains across diffusion fine-tuning methods on medical images, but with a loose theory and unsupported 'closes the distribution gap' claim.","tokens_in":14012,"tokens_out":914,"would_cite":true,"duration_ms":11015,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding a small scalar drift to the noise mean of a pretrained diffusion model lets it generate medical counterfactual images, and closes the distribution gap between general and medical domains.","keywords":["latent drift","diffusion models","counterfactual image generation","medical image synthesis","fine-tuning","distribution shift","brain MRI","chest X-ray"],"falsifier":"Re-run the four fine-tuning protocols on a held-out medical modality not in the paper and grid-search $\\delta$ by the L1 criterion; if no $\\delta$ beats the no-LD baseline on FID and KID, or if a randomly chosen $\\delta$ performs just as well, then the L1-selected drift is not what closes the distribution gap.","tokens_in":1794,"feed_emoji":"🧠","tokens_out":5794,"duration_ms":114516,"temperature":0.7,"pith_summary":"The paper introduces Latent Drift (LD), a single signed scalar added to the mean of the noise distribution at every diffusion reverse step, and to the forward target during fine-tuning, as a way to condition a pretrained diffusion model on a medical image domain without training from scratch. It frames this as a counterfactual conditioning problem: a shifted latent code exists that makes the pretrained model generate the target medical domain, and the shift is chosen by minimizing an L1 distance to the available target samples. On three longitudinal medical benchmarks, two brain MRI datasets and one chest X-ray dataset, adding LD improves FID and KID across four fine-tuning schemes and enables disease addition and removal, aging, and sex change with text-and-image conditioning. If this holds, LD is a cheap, method-agnostic way to make scarce-data medical counterfactual synthesis practical with models that were never trained on medical images.","feed_headline":"One scalar shift makes diffusion models generate medical images","feed_subtitle":"Adding a single drift value to the noise mean closes the gap between pretrained models and scarce medical data.","key_machinery":"The central object is the latent drift $\\delta$, a signed scalar injected into the diffusion process by replacing the reverse mean $\\mu_{\\theta}(x_t,t)$ with $\\mu_{\\theta}(x_t,t)+\\delta$ and adding $\\delta$ to the forward target $z_T$ during fine-tuning. Its role is to act as an extra, data-dependent conditioning signal: the paper's hypothesis is that the stochastic term $\\mathcal{N}(\\mu,\\sigma)$ does not change under text or image conditioning, so the latent prior must be shifted explicitly to reflect the target domain. The selection mechanism is a grid search over $\\delta$ minimizing the L1 distance between synthetically generated samples and the target dataset, framed as the counterfactual objective of Eq. (5). This is what carries the argument: all reported gains are attributed to a constant scalar shift in latent space.","core_discovery":"Latent Drifting is the claim that the final latent variable $z_T$ of a diffusion model should be treated as part of the conditioning, and that shifting its mean by a scalar $\\delta$ lets a pretrained model generate samples from a target distribution without access to the original training set. Concretely, the reverse transition becomes $p_{\\theta}(x_{t-1}|x_t)=\\mathcal{N}(x_{t-1};\\mu_{\\theta}(x_t,t)+\\delta,\\Sigma_{\\theta}(x_t,t))$, and during fine-tuning the same $\\delta$ is added to the forward target $z_T$. The paper connects this to the counterfactual min-max objective of Eq. (5), where $\\lambda=0$ gives standard fine-tuning and $\\lambda>0$ leads to searching $\\delta$ via grid search to minimize the L1 distance between generated and target samples. The reported outcome is consistent FID and KID improvements, plus improved downstream classifier AUC on real test sets, across all four fine-tuning methods.","pith_inferences":["Beyond the paper, the constant-$\\delta$ assumption could be tested by learning a per-sample or per-step drift; if that outperforms the grid-searched constant, the L1 selection is a crude proxy for a richer latent shift rather than the whole mechanism.","Beyond the paper, the same drift mechanism should transfer to other out-of-distribution domains beyond medical imaging, such as satellite imagery or cell microscopy, since the paper's framing depends only on a distribution gap and a small target sample set.","Beyond the paper, the interaction between $\\delta$ and classifier-free guidance or prompt diversity is unmeasured; a testable prediction is that the optimal $\\delta$ changes with guidance scale, which would connect LD to established conditioning dynamics."],"forward_implications":["Because LD is a scalar added to the sampling process, it can be layered onto any fine-tuning method that keeps the latent diffusion architecture, so future adapters should inherit the same distribution-matching benefit.","The reported AUC improvements from classifiers trained on LD-generated brain MRIs imply that synthetic counterfactual images can substitute for part of the real training data in scarce-data medical settings.","Text- and image-conditioned counterfactual edits, such as removing or adding disease, aging, and changing sex, can be produced from a pretrained general model rather than a domain-specific model trained from scratch.","Latent drift also operates at pure inference time: with identical sampled noise, varying $\\delta$ changes the identity and style of the generated image, making $\\delta$ an additional conditioning axis with no fine-tuning required."],"supporting_citations":[{"why":"Supplies the base latent diffusion architecture that Latent Drift modifies; every experiment starts from this model.","marker":"[40]"},{"why":"Defines the counterfactual min-max objective in Eq. (5) that the paper uses to justify the drift.","marker":"[47]"},{"why":"Custom Diffusion is one of the four fine-tuning schemes tested with and without LD.","marker":"[22]"},{"why":"DreamBooth is one of the four fine-tuning schemes tested with and without LD.","marker":"[41]"},{"why":"Textual Inversion is one of the four fine-tuning schemes tested with and without LD.","marker":"[11]"},{"why":"CheXpert provides the chest X-ray data for disease counterfactual generation and evaluation.","marker":"[15]"},{"why":"OASIS-3 provides longitudinal brain MRI data for the aging and Alzheimer's counterfactual experiments.","marker":"[23]"},{"why":"ADNI provides the second longitudinal brain MRI dataset for the counterfactual experiments.","marker":"[49]"},{"why":"InstructPix2Pix is the image-editing backbone used for age-conditioned manipulation with LD.","marker":"[3]"},{"why":"Pix2Pix Zero is the backbone used for disease-conditioned manipulation with LD.","marker":"[33]"}],"fun_headline_variants":["Latent drift: one scalar shifts diffusion models to medical imaging","Scalar drift makes diffusion models do counterfactual medical images","A tiny latent shift adapts diffusion models to scarce medical data","One number tweaks diffusion models for counterfactual medical imaging"],"cache_read_input_tokens":16128,"weakest_assumption_plain":"The load-bearing premise is that one constant scalar $\\delta$ added to the noise mean at every step can move a pretrained model's output distribution into the medical target domain, with $\\delta$ chosen by L1-distance grid search on a small sample set; if no single shift suffices, the method has no principled basis.","fun_headline_variants_meta":{"raw":{"variants":["Latent drift: one scalar shifts diffusion models to medical imaging","Scalar drift makes diffusion models do counterfactual medical images","A tiny latent shift adapts diffusion models to scarce medical data","One number tweaks diffusion models for counterfactual medical imaging"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000374,"raw_usage":{"total_tokens":1996,"prompt_tokens":946,"completion_tokens":1050,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":979}},"tokens_in":562,"tokens_out":1050,"duration_ms":8443,"temperature":1.0,"reasoning_tokens":979,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:15:10.174288+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the four fine-tuning protocols on a held-out medical modality not in the paper and grid-search $\\delta$ by the L1 criterion; if no $\\delta$ beats the no-LD baseline on FID and KID, or if a randomly chosen $\\delta$ performs just as well, then the L1-selected drift is not what closes the distribution gap.","supporting_citations":[{"cited_title":"Coun- terfactual explanations without opening the black box: Au- tomated decisions and the gdpr","cited_arxiv_id":null,"evidence_quote":"Defines the counterfactual min-max objective in Eq. (5) that the paper uses to justify the drift."},{"cited_title":"Multi-concept customization of text-to-image diffusion","cited_arxiv_id":null,"evidence_quote":"Custom Diffusion is one of the four fine-tuning schemes tested with and without LD."},{"cited_title":"Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation","cited_arxiv_id":null,"evidence_quote":"DreamBooth is one of the four fine-tuning schemes tested with and without LD."},{"cited_title":"An image is worth one word: Personalizing text-to-image generation using textual inversion","cited_arxiv_id":null,"evidence_quote":"Textual Inversion is one of the four fine-tuning schemes tested with and without LD."},{"cited_title":"Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison","cited_arxiv_id":null,"evidence_quote":"CheXpert provides the chest X-ray data for disease counterfactual generation and evaluation."},{"cited_title":"Oasis-3: longitudinal neuroimaging, clin- ical, and cognitive dataset for normal aging and alzheimer disease","cited_arxiv_id":null,"evidence_quote":"OASIS-3 provides longitudinal brain MRI data for the aging and Alzheimer's counterfactual experiments."},{"cited_title":"Weiner, Dallas P","cited_arxiv_id":null,"evidence_quote":"ADNI provides the second longitudinal brain MRI dataset for the counterfactual experiments."},{"cited_title":"In- structpix2pix: Learning to follow image editing instructions","cited_arxiv_id":null,"evidence_quote":"InstructPix2Pix is the image-editing backbone used for age-conditioned manipulation with LD."},{"cited_title":"Zero-shot image-to-image translation","cited_arxiv_id":null,"evidence_quote":"Pix2Pix Zero is the backbone used for disease-conditioned manipulation with LD."}],"review_version":1}