{"id":"4aea96c6-1778-42b4-af1d-72375b32c4d9","arxiv_id":"2608.12997","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Pixel artifacts in latent SDS are caused by VAE-induced pixel drift, and decoding the latent SDS lookahead provides a clean gradient direction that reduces them.","lead":"Latent score distillation can turn clean prompts into noisy 3D images because the VAE encoder leaves some pixel directions unconstrained, so noise builds up while the latent stays clean. PixSDS, a lightweight gradient repair, decodes the latent step and uses it as a clean direction, cutting artifacts without retraining.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The formal proof only shows persistence of pre-existing nullspace noise; it cannot explain the constant-initialized SDS runs where noise appears, leaving the central VAE-drift mechanism unsupported for the main experiments.","rationale":"The reader's weakest assumption is that the real SD VAE encoder has a nullspace aligned with high-frequency noise; my concern is closely related but more specific: even if such a nullspace exists, the paper's own linear analysis predicts that optimization from a constant initialization cannot enter that nullspace, so the proof cannot explain the main SDS experiments. This is a sharper mismatch than the reader stated. I do not recommend REJECT because the empirical method (PixSDS) is clearly useful and the VAE-only experiment does show a clean-latent/noisy-image phenomenon from random starts; the paper plausibly identifies a real effect. However, the title claim 'Why Latent SDS Makes Noisy Pixels' requires the mechanism to operate in the actual SDS setting with constant initialization, and that is not demonstrated. The verdict should remain CONDITIONAL, with the condition being direct measurement of nullspace drift in the SDS trajectory. My agreement is partial because the reader points to nullspace alignment, while I additionally identify the proof-to-experiment mismatch (constant vs random initialization) as the load-bearing gap.","tokens_in":13017,"tokens_out":8410,"duration_ms":88512,"concrete_test":"Re-run the 2D SDS experiment from Sec. 5.1 with the stated constant initialization, and at each step record Z_t, the SDS pixel update g_t, and the noise functional N(Z_t). Estimate the local encoder Jacobian J_t = ∂enc/∂Z at Z_t (e.g., by finite differences on a random subset of coordinates), compute the singular value decomposition of J_t, and measure the cumulative projection of g_t onto the subspace spanned by right singular vectors with singular values below a threshold (near-nullspace). If this projection and N(Z_t) grow over time while the latent z_t = enc(Z_t) stays clean, the VAE-drift mechanism is confirmed in the actual setting. If the near-nullspace projection stays negligible, the observed artifacts must be introduced by the stochastic diffusion sampling or CFG rather than by VAE underdetermination, and the title claim would need to be revised.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that latent SDS drives the optimized image into pixel-space directions weakly visible to the VAE encoder ('VAE-induced pixel drift'), making the image noisy while its latent stays clean. The formal support (Appendix A, Theorem 1 and Corollary 1) does not establish this for the paper's main experiments. For a fixed linear encoder A, each gradient-descent update lies in the row space of A, so the nullspace component of the iterate is invariant. The theorem's construction starts from x0 = s(u + αz) with u ∈ ker(A) and z chosen so that N(u + αz) < N(u); the limit x* = s u then has higher noise because the cancellations provided by z are removed. That is persistence of pre-existing nullspace noise, not creation from a clean start. The 2D SDS experiments initialize to a constant image (Sec. 5.1), whose nullspace component is zero under the paper's own assumption (the constant vector is not in ker A when the kernel has nonzero sum). A linear analysis therefore predicts no nullspace noise can appear, yet the experiments show structured noise at step 1000. The VAE-only experiment in Sec. 3 uses random initialization U(0,1), so it also demonstrates retention, not amplification. The actual SDS setting differs further: the encoder Jacobian is nonlinear and changes with Z, and the loss is stochastic (sampled timesteps and noise), so the invariant-subspace argument does not apply. The paper does not measure whether the SDS pixel updates actually have a growing component in the near-nullspace of the encoder during optimization. Without such a measurement, the proposed mechanism remains an unvalidated hypothesis for the constant-initialized SDS runs that constitute the main experimental evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper argues that structured artifacts in latent Score Distillation Sampling arise from 'VAE-induced pixel drift': the optimized image moves along pixel-space directions that are weakly visible to the VAE encoder, so the latent code remains semantically clean while the image accumulates high-frequency noise. The authors support this diagnosis with controlled experiments: latent SDS with a low-resolution latent diffusion model still produces artifacts; pixel-space SDS with a small conditional model does not; and VAE-only optimization of an image against a target latent produces structured noise while the decoded latents stay clean. They also provide a simplified linear-convolution analysis (Appendix A) intended to show that encoder-like objectives can amplify noise, and they propose PixSDS, which decodes a latent SDS look-ahead step and uses the decoded image as a per-pixel normalized clean direction to repair the SDS update. Experiments in 2D generation and in DreamGaussian/LucidDreamer 3D pipelines report artifact reduction with competitive quantitative metrics.","tokens_in":13421,"tokens_out":6264,"duration_ms":66658,"significance":"If the proposed mechanism is confirmed, this is a useful and actionable diagnosis: it shifts artifact reduction in latent SDS from heuristic gradient clipping toward enforcing VAE consistency, and it is compatible with existing SDS-style objectives. The paper has several clear strengths: controlled 2D experiments isolate the VAE mapping from tensor-shape and pixel-space optimization effects; the VAE-only optimization experiment directly demonstrates that a latent-only objective can produce clean-latent/noisy-image mismatches; the proposed repair is simple and model-agnostic; and the code is publicly released. The formal linear-convolution analysis is self-contained, though it proves a narrower retention/amplification statement than the paper's headline claim. The main unresolved technical risk is whether the real Stable Diffusion encoder actually possesses an aligned nullspace that drives drift from a clean initialization; this needs to be measured or otherwise demonstrated.","major_comments":[{"comment":"The formal result does not establish creation of nullspace noise from a clean start, yet the main claim and the 2D experiments depend on that. In Theorem 1 the constructed initial point is x0 = s(u + αz) with u ∈ ker(A), so the nullspace component is present initially and merely survives while the non-null component decays; this is persistence or amplification of pre-existing noise, not generation of noise in a constant-initialized run. Under the linear convolution model of Corollary 1 with kernel sum nonzero, the constant vector (0.5, 0.5, 0.5) used in Sec. 5.1 is not in ker(A), so the linear analysis predicts no nullspace noise should appear, yet the experiments show structured noise at step 1000. Because the real VAE encoder is nonlinear and the SDS loss is stochastic (sampled timesteps and noise), the invariant-subspace argument does not directly transfer. Please either (i) revise the theoretical claim to 'retention/amplification of existing VAE-invisible noise', (ii) measure the nullspace alignment of the real encoder's Jacobian with the observed high-frequency drift, or (iii) provide a controlled experiment showing noise creation from a clean initialization under the nonlinear VAE dynamics.","section":"§3, §5.1, Appendix A (Theorem 1, Corollary 1)"},{"comment":"The main quantitative comparison reports a single FID, CLIP Score, BRISQUE, and CLIP-IQA value per method computed on 100 images, with no error bars, no multiple seeds, and no significance testing. With FID computed on only 100 images, differences such as PixSDS+SGD at 223.0 versus 2-step-SDS at 229.8 and PixSDS+Adam at 229.6, and CLIP Score differences below one point, are plausibly within noise. Please report means and standard deviations over at least three seeds, and ideally paired statistics across the same prompts, before claiming that PixSDS achieves the best results on these metrics.","section":"§5.1, Table 1"},{"comment":"The evaluation introduces several choices that could confound the comparison: a specially designed timestep schedule ('we found this schedule to reduce oversaturation'), fp16 precision, and β = 0.1. The paper does not explicitly state that all baselines use the exact same timestep schedule, initialization, optimizer settings, and number of steps. If the schedule and β were tuned for PixSDS while baselines were run with their default schedules, the reported improvements could be partly due to the schedule rather than the VAE-consistent repair. Please state clearly that all baselines use identical settings and, ideally, ablate the timestep schedule for the strongest baseline.","section":"§5.1, §5.2"}],"minor_comments":[{"comment":"The sentence 'the optimization does not necessary converge to a less noisy solution' should read 'does not necessarily converge'.","section":"§3, toy analogy"},{"comment":"The proof uses m both for the input dimension of A ∈ R^{m×n} and as the threshold for nonzero eigenvalues; the condition '0 < γ < 2 / max_{i>m} λ_i' and the limit expression 'x* = Σ_{i=1}^m c_i e_i' should use r = dim ker(A), not m, to avoid confusion.","section":"Appendix A, Theorem 1 proof"},{"comment":"The noise functional N is defined for 1D vectors x ∈ R^n, but the text reports values for 2D color images without specifying how the functional is extended; please define the per-channel or total-variation extension used for the reported numerical values.","section":"Appendix A, Definition 1"},{"comment":"Figures 11 and 12 are subfigures of Figure 13 but are numbered independently and their captions are not self-contained; please restructure the figure numbering and make the 'specified' versus 'random' direction descriptions explicit in the main caption.","section":"Appendix B, Figures 11–13"},{"comment":"The 3D comparisons are only qualitative; please add a quantitative comparison or a small user study, or explicitly justify why the qualitative evidence is sufficient for the 3D claim.","section":"§5.2, Figures 8–9"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a real and practically important problem in latent SDS, and the proposed fix is simple and plausible. The main gap is the mismatch between the formal proof, which only covers persistence of pre-existing nullspace noise, and the paper's stronger claim of noise creation from a clean initialization. This is fixable within revision by either narrowing the theoretical claim or adding a direct nullspace measurement for the real VAE, so I do not recommend rejection. I also encourage the editor to ask for error bars in Table 1 and explicit confirmation that baselines share the same evaluation protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear X,\n\nThe short version: this paper names a real problem—latent SDS produces noisy images while the latents look clean—and offers a simple, plausible fix (decode the latent lookahead and use it as a clean direction) that seems to work in 2D and 3D. The diagnosis has good circumstantial support, but the formal proof is narrower than the paper claims, and the mechanism is not directly measured. Still, I'd send it to review; it deserves a serious referee.\n\nWhat's actually good: the paper isolates the VAE mapping as a necessary ingredient by comparing latent SDS (artifacts) to pixel-space SDS (clean), and by showing that even optimizing only through the VAE encoder creates noisy images with clean decoded latents. That's a clean, controlled experimental chain. PixSDS is lightweight, doesn't touch the diffusion model or renderer, and the ablations show each component matters. The qualitative 3D results look improved.\n\nWhere it's soft: the main claimed failure mode is that the optimized image drifts into directions the VAE encoder can't see. The proof in Appendix A shows something real but narrower: if you start with a component already in the kernel of a linear encoder, hidden by a cleverly chosen other component, gradient descent can remove the hiding part and expose the kernel noise. It doesn't show the noise appears from a clean constant initialization, which is exactly what the main 2D experiments use. The real encoder is nonlinear and the SDS loss is stochastic, so the linear invariant argument doesn't apply. The paper also never measures whether the pixel updates actually grow along near-nullspace directions of the real encoder. As a result, the mechanism remains a hypothesis for the main experiments, not a demonstrated fact.\n\nOther soft spots are minor: FID on 100 images without error bars is weak, and beta is hand-tuned (though there's a sensitivity study). The citation pattern looks fine; the self-citation to the author's 2-step-SDS is relevant.\n\nWho gets value: anyone doing SDS-based text-to-3D, and researchers interested in why latent diffusion optimization misbehaves. The method is cheap and likely to be used as a drop-in. I'd cite it if I worked in this area.\n\nRecommendation: peer review, yes, with a request to temper the theory claim and ideally add direct nullspace measurement, or at least acknowledge the gap. The empirical contribution can stand on its own.","headline":"A plausible and useful fix for latent SDS artifacts, with a theory that overclaims—worth reviewing, not desk rejecting.","tokens_in":13884,"tokens_out":4402,"would_cite":true,"duration_ms":44356,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Latent SDS noise is VAE drift, and decoding the latent step repairs it","keywords":["score distillation sampling","latent diffusion","variational autoencoder","text-to-3D generation","artifact reduction","gradient repair","noise amplification","underconstrained inverse mapping"],"falsifier":"Measure the Jacobian of the real Stable Diffusion VAE encoder at typical SDS iterates and identify its approximate nullspace; then compute whether the structured noise that appears during vanilla SDS lies mostly in that near-null subspace (e.g., whether $\\|\\mathrm{enc}(Z+n)-\\mathrm{enc}(Z)\\|$ is small for the observed noise $n$). If the observed artifacts are not near-null directions of the actual encoder, the proposed drift mechanism does not explain them.","tokens_in":12784,"feed_emoji":"🎨","tokens_out":7530,"duration_ms":65550,"temperature":0.7,"pith_summary":"Latent score distillation sampling (SDS), the standard way to generate 3D assets from text with a pretrained diffusion prior, often produces structured color blotches and high-frequency noise. This paper argues that the cause is the underconstrained inverse mapping of the VAE: the optimized image can drift along pixel-space directions the encoder barely sees, so the latent code stays clean while the pixels accumulate artifacts. The paper supports this with controlled 2D experiments, an experiment that optimizes through only the VAE encoder and reproduces the noise, and a simplified proof that encoder-like objectives can amplify a difference-based noise measure. It then proposes PixSDS, which decodes the next latent SDS step and uses the decoded image as a clean pixel-space direction, normalizing it to the SDS gradient's per-pixel magnitude. In 2D optimization and two text-to-3D pipelines, PixSDS reduces artifacts without retraining the diffusion model or changing the renderer.","feed_headline":"Latent SDS noise is VAE drift, and decoding the latent step repairs it","feed_subtitle":"Images drift along directions the VAE can't see; decoding the next latent step gives a clean update direction.","key_machinery":"The load-bearing object is the VAE encoder–decoder pair as an underconstrained map: many visually different images map to the same latent code, so the latent objective cannot see certain pixel directions. The proof machinery is the noise functional $N(x)=\\|Dx\\|^2$ (sum of squared adjacent-pixel differences) combined with the path-graph Laplacian's eigenvalue lower bound; it shows that when the kernel of the encoder-like operator has dimension at least two and contains no constant vectors, gradient descent can increase $N$ without bound. The repair machinery is the PixSDS lookahead direction: taking one latent-space SDS step, decoding it, and using $\\mathrm{dec}(\\mathrm{enc}(Z)-\\beta g_{\\mathrm{latent\\,sds}})-Z$ as a clean direction, rescaled per pixel with $c_{\\mathrm{norm}}$ so it has the same spatial magnitude as $g_{\\mathrm{sds}}$.","core_discovery":"The paper's central claim is that latent SDS fails because the pixel-space parameters are optimized through a compression operator with a large nullspace. Writing $\\mathrm{enc}$ for the VAE encoder and $\\mathrm{dec}$ for the decoder, the SDS loss is evaluated on $\\mathrm{enc}(Z)$, not on $Z$ itself, so any direction $\\delta$ with $\\mathrm{enc}(Z+\\delta)\\approx\\mathrm{enc}(Z)$ is almost free. Gradient descent can therefore move $Z$ into visually noisy configurations while $\\mathrm{enc}(Z)$, and $\\mathrm{dec}(\\mathrm{enc}(Z))$, remain clean and semantically valid. Formally, for linear encoder-like operators $A$ with kernel dimension at least 2, the paper proves (Appendix A, Theorem 1) that gradient descent on $\\frac{1}{2}\\|Ax\\|^2$ can drive a difference-based noise functional $N(x)=\\|Dx\\|^2$ to arbitrarily large values from arbitrarily clean starting points; Corollary 1 applies this to 1D convolutions of length $m\\ge 3$ with nonzero sum. PixSDS is the constructive consequence: instead of clipping big gradients, replace part of the SDS pixel update with the direction obtained by decoding the latent after an SDS step, $g_{\\mathrm{clean}}=\\mathrm{dec}(\\mathrm{enc}(Z)-\\beta g_{\\mathrm{latent\\,sds}})-Z$, matched per pixel to the original SDS gradient magnitude. The paper reports that this reduces structured artifacts while preserving semantic content.","pith_inferences":["If the nullspace alignment is confirmed in the real VAE, one could reduce artifacts at the source by regularizing the VAE to shrink its high-frequency nullspace, or by projecting SDS updates onto the encoder's row space.","The same drift mechanism likely affects other latent-space optimizations, such as real-image inversion and latent editing, where clean latent codes may mask corrupt pixel reconstructions; PixSDS-style lookahead decoding could serve as a general corrector there.","A natural extension is multi-step latent lookahead: instead of one decoded step, unroll several SDS steps in latent space and decode the final point, which could give a cleaner direction at modest extra cost."],"forward_implications":["Artifact reduction in latent SDS should aim to keep pixel updates inside the VAE-consistent subspace, rather than only clipping large gradients.","PixSDS can be layered on top of any SDS-style objective, since it only modifies the update after the SDS direction is computed.","The failure mode should appear in any optimization that drives pixels through an underconstrained encoder, not only in text-to-3D pipelines.","The $\\beta>0$ lookahead is essential: pulling toward the current decoded latent ($\\beta=0$) removes artifacts but loses semantic placement, so the clean direction must point to the next latent step.","The method transfers across diffusion backbones, including rectified-flow-based models, without retraining."],"supporting_citations":[{"why":"introduces the SDS objective whose latent variant is the subject of the paper","marker":"[16]"},{"why":"documents high-frequency texture artifacts in Gaussian-splatting SDS and serves as the integration target for the 3D experiments","marker":"[20]"},{"why":"proposes pixel-wise gradient clipping, the main alternative remedy compared against in the 2D experiments","marker":"[15]"},{"why":"removes an undesired noise component from the SDS signal and is a baseline that PixSDS complements","marker":"[7]"},{"why":"introduces variational score distillation, a baseline in the controlled 2D comparison","marker":"[21]"},{"why":"proposes an alternative distillation direction and is one of the SDS-style baselines evaluated","marker":"[13]"},{"why":"provides a second text-to-3D pipeline used to test the repair method","marker":"[9]"},{"why":"supplies the MS-COCO captions used to build the 2D evaluation set","marker":"[10]"},{"why":"defines the BRISQUE no-reference image-quality metric used to measure artifact reduction","marker":"[14]"}],"fun_headline_variants":["Latent SDS noise traced to VAE-blind pixel drift","Decoding the SDS latent step repairs pixel artifacts","Why latent SDS yields noisy pixels: VAE nullspace","PixSDS: decode latent step to fix SDS pixel noise","Stop VAE-invisible drift in latent SDS with decoding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that the real Stable Diffusion VAE encoder has a large nullspace aligned with high-frequency noise directions, so that gradient descent on the latent objective can drift into those directions; the formal proof covers only linear convolutions and the paper does not measure the actual encoder's nullspace or its alignment with noise.","fun_headline_variants_meta":{"raw":{"variants":["Latent SDS noise traced to VAE-blind pixel drift","Decoding the SDS latent step repairs pixel artifacts","Why latent SDS yields noisy pixels: VAE nullspace","PixSDS: decode latent step to fix SDS pixel noise","Stop VAE-invisible drift in latent SDS with decoding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000271,"raw_usage":{"total_tokens":1704,"prompt_tokens":1093,"completion_tokens":611,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":709,"completion_tokens_details":{"reasoning_tokens":544}},"tokens_in":709,"tokens_out":611,"duration_ms":6607,"temperature":1.0,"reasoning_tokens":544,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:52:37.880869+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the Jacobian of the real Stable Diffusion VAE encoder at typical SDS iterates and identify its approximate nullspace; then compute whether the structured noise that appears during vanilla SDS lies mostly in that near-null subspace (e.g., whether $\\|\\mathrm{enc}(Z+n)-\\mathrm{enc}(Z)\\|$ is small for the observed noise $n$). If the observed artifacts are not near-null directions of the actual encoder, the proposed drift mechanism does not explain them.","supporting_citations":[{"cited_title":"In: International Conference on Learning Representations","cited_arxiv_id":null,"evidence_quote":"documents high-frequency texture artifacts in Gaussian-splatting SDS and serves as the integration target for the 3D experiments"},{"cited_title":"Advances in Neural Information Processing Systems37, 33779–33804 (2024)","cited_arxiv_id":null,"evidence_quote":"proposes an alternative distillation direction and is one of the SDS-style baselines evaluated"},{"cited_title":"In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","cited_arxiv_id":null,"evidence_quote":"provides a second text-to-3D pipeline used to test the repair method"}],"review_version":1}