REVIEW 3 major objections 5 minor 23 references
PixSDS: Why Latent SDS Makes Noisy Pixels
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Latent SDS noise is VAE drift, and decoding the latent step repairs it
desk verdict A plausible and useful fix for latent SDS artifacts, with a theory that overclaims—worth reviewing, not desk rejecting. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the VAE encoder–decoder pair as an underconstrained map: many visually different images map to the same latent code, so the latent objective cannot see certain pixel directions. The proof machinery is the noise functional $N(x)=\|Dx\|^2$ (sum of squared adjacent-pixel differences) combined with the path-graph Laplacian's eigenvalue lower bound; it shows that when the kernel of the encoder-like operator has dimension at least two and contains no constant vectors, gradient descent can increase $N$ without bound. The repair machinery is the PixSDS lookahead direction: taking one latent-space SDS step, decoding it, and using $\mathrm{dec}(\mathrm{enc}(Z)-\beta g_{\mathrm{latent\,sds}})-Z$ as a clean direction, rescaled per pixel with $c_{\mathrm{norm}}$ so it has the same spatial magnitude as $g_{\mathrm{sds}}$.
What would settle it
Measure the Jacobian of the real Stable Diffusion VAE encoder at typical SDS iterates and identify its approximate nullspace; then compute whether the structured noise that appears during vanilla SDS lies mostly in that near-null subspace (e.g., whether $\|\mathrm{enc}(Z+n)-\mathrm{enc}(Z)\|$ is small for the observed noise $n$). If the observed artifacts are not near-null directions of the actual encoder, the proposed drift mechanism does not explain them.
Extended reading notes
Core claim
The paper's central claim is that latent SDS fails because the pixel-space parameters are optimized through a compression operator with a large nullspace. Writing $\mathrm{enc}$ for the VAE encoder and $\mathrm{dec}$ for the decoder, the SDS loss is evaluated on $\mathrm{enc}(Z)$, not on $Z$ itself, so any direction $\delta$ with $\mathrm{enc}(Z+\delta)\approx\mathrm{enc}(Z)$ is almost free. Gradient descent can therefore move $Z$ into visually noisy configurations while $\mathrm{enc}(Z)$, and $\mathrm{dec}(\mathrm{enc}(Z))$, remain clean and semantically valid. Formally, for linear encoder-like operators $A$ with kernel dimension at least 2, the paper proves (Appendix A, Theorem 1) that gradient descent on $\frac{1}{2}\|Ax\|^2$ can drive a difference-based noise functional $N(x)=\|Dx\|^2$ to arbitrarily large values from arbitrarily clean starting points; Corollary 1 applies this to 1D convolutions of length $m\ge 3$ with nonzero sum. PixSDS is the constructive consequence: instead of clipping big gradients, replace part of the SDS pixel update with the direction obtained by decoding the latent after an SDS step, $g_{\mathrm{clean}}=\mathrm{dec}(\mathrm{enc}(Z)-\beta g_{\mathrm{latent\,sds}})-Z$, matched per pixel to the original SDS gradient magnitude. The paper reports that this reduces structured artifacts while preserving semantic content.
Load-bearing premise
The argument assumes that the real Stable Diffusion VAE encoder has a large nullspace aligned with high-frequency noise directions, so that gradient descent on the latent objective can drift into those directions; the formal proof covers only linear convolutions and the paper does not measure the actual encoder's nullspace or its alignment with noise.
Editorial extensions
If this is right
- Artifact reduction in latent SDS should aim to keep pixel updates inside the VAE-consistent subspace, rather than only clipping large gradients.
- PixSDS can be layered on top of any SDS-style objective, since it only modifies the update after the SDS direction is computed.
- The failure mode should appear in any optimization that drives pixels through an underconstrained encoder, not only in text-to-3D pipelines.
- The $\beta>0$ lookahead is essential: pulling toward the current decoded latent ($\beta=0$) removes artifacts but loses semantic placement, so the clean direction must point to the next latent step.
- The method transfers across diffusion backbones, including rectified-flow-based models, without retraining.
Reading between the lines
- If the nullspace alignment is confirmed in the real VAE, one could reduce artifacts at the source by regularizing the VAE to shrink its high-frequency nullspace, or by projecting SDS updates onto the encoder's row space.
- The same drift mechanism likely affects other latent-space optimizations, such as real-image inversion and latent editing, where clean latent codes may mask corrupt pixel reconstructions; PixSDS-style lookahead decoding could serve as a general corrector there.
- A natural extension is multi-step latent lookahead: instead of one decoded step, unroll several SDS steps in latent space and decode the final point, which could give a cleaner direction at modest extra cost.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper argues that structured artifacts in latent Score Distillation Sampling arise from 'VAE-induced pixel drift': the optimized image moves along pixel-space directions that are weakly visible to the VAE encoder, so the latent code remains semantically clean while the image accumulates high-frequency noise. The authors support this diagnosis with controlled experiments: latent SDS with a low-resolution latent diffusion model still produces artifacts; pixel-space SDS with a small conditional model does not; and VAE-only optimization of an image against a target latent produces structured noise while the decoded latents stay clean. They also provide a simplified linear-convolution analysis (Appendix A) intended to show that encoder-like objectives can amplify noise, and they propose PixSDS, which decodes a latent SDS look-ahead step and uses the decoded image as a per-pixel normalized clean direction to repair the SDS update. Experiments in 2D generation and in DreamGaussian/LucidDreamer 3D pipelines report artifact reduction with competitive quantitative metrics.
Significance. If the proposed mechanism is confirmed, this is a useful and actionable diagnosis: it shifts artifact reduction in latent SDS from heuristic gradient clipping toward enforcing VAE consistency, and it is compatible with existing SDS-style objectives. The paper has several clear strengths: controlled 2D experiments isolate the VAE mapping from tensor-shape and pixel-space optimization effects; the VAE-only optimization experiment directly demonstrates that a latent-only objective can produce clean-latent/noisy-image mismatches; the proposed repair is simple and model-agnostic; and the code is publicly released. The formal linear-convolution analysis is self-contained, though it proves a narrower retention/amplification statement than the paper's headline claim. The main unresolved technical risk is whether the real Stable Diffusion encoder actually possesses an aligned nullspace that drives drift from a clean initialization; this needs to be measured or otherwise demonstrated.
major comments (3)
- [§3, §5.1, Appendix A (Theorem 1, Corollary 1)] The formal result does not establish creation of nullspace noise from a clean start, yet the main claim and the 2D experiments depend on that. In Theorem 1 the constructed initial point is x0 = s(u + αz) with u ∈ ker(A), so the nullspace component is present initially and merely survives while the non-null component decays; this is persistence or amplification of pre-existing noise, not generation of noise in a constant-initialized run. Under the linear convolution model of Corollary 1 with kernel sum nonzero, the constant vector (0.5, 0.5, 0.5) used in Sec. 5.1 is not in ker(A), so the linear analysis predicts no nullspace noise should appear, yet the experiments show structured noise at step 1000. Because the real VAE encoder is nonlinear and the SDS loss is stochastic (sampled timesteps and noise), the invariant-subspace argument does not directly transfer. Please either (i) revise the theoretical claim to 'retention/amplification of existing VAE-invisible noise', (ii) measure the nullspace alignment of the real encoder's Jacobian with the observed high-frequency drift, or (iii) provide a controlled experiment showing noise creation from a clean initialization under the nonlinear VAE dynamics.
- [§5.1, Table 1] The main quantitative comparison reports a single FID, CLIP Score, BRISQUE, and CLIP-IQA value per method computed on 100 images, with no error bars, no multiple seeds, and no significance testing. With FID computed on only 100 images, differences such as PixSDS+SGD at 223.0 versus 2-step-SDS at 229.8 and PixSDS+Adam at 229.6, and CLIP Score differences below one point, are plausibly within noise. Please report means and standard deviations over at least three seeds, and ideally paired statistics across the same prompts, before claiming that PixSDS achieves the best results on these metrics.
- [§5.1, §5.2] The evaluation introduces several choices that could confound the comparison: a specially designed timestep schedule ('we found this schedule to reduce oversaturation'), fp16 precision, and β = 0.1. The paper does not explicitly state that all baselines use the exact same timestep schedule, initialization, optimizer settings, and number of steps. If the schedule and β were tuned for PixSDS while baselines were run with their default schedules, the reported improvements could be partly due to the schedule rather than the VAE-consistent repair. Please state clearly that all baselines use identical settings and, ideally, ablate the timestep schedule for the strongest baseline.
minor comments (5)
- [§3, toy analogy] The sentence 'the optimization does not necessary converge to a less noisy solution' should read 'does not necessarily converge'.
- [Appendix A, Theorem 1 proof] The proof uses m both for the input dimension of A ∈ R^{m×n} and as the threshold for nonzero eigenvalues; the condition '0 < γ < 2 / max_{i>m} λ_i' and the limit expression 'x* = Σ_{i=1}^m c_i e_i' should use r = dim ker(A), not m, to avoid confusion.
- [Appendix A, Definition 1] The noise functional N is defined for 1D vectors x ∈ R^n, but the text reports values for 2D color images without specifying how the functional is extended; please define the per-channel or total-variation extension used for the reported numerical values.
- [Appendix B, Figures 11–13] Figures 11 and 12 are subfigures of Figure 13 but are numbered independently and their captions are not self-contained; please restructure the figure numbering and make the 'specified' versus 'random' direction descriptions explicit in the main caption.
- [§5.2, Figures 8–9] The 3D comparisons are only qualitative; please add a quantitative comparison or a small user study, or explicitly justify why the qualitative evidence is sufficient for the 3D claim.
Circularity Check
No circularity found; the derivation is self-contained and the one self-citation is not load-bearing.
full rationale
The paper's central claim — that latent SDS can produce noisy pixel images while their VAE latents remain clean — is supported by controlled experiments (VAE-only optimization, 2D SDS, and pixel-space SDS comparisons) and by a self-contained mathematical analysis in Appendix A. The proof in Theorem 1 and Corollary 1 is an existence result about gradient descent on linear encoder-like objectives; it does not fit any parameter to the observed artifacts, and it does not invoke the authors' prior work. The PixSDS repair uses the decoded latent look-ahead as a clean direction; this is a design choice motivated by the observed clean latents, not a disguised refit of the target quantity. Evaluation relies on external metrics (FID, BRISQUE, CLIP Score, CLIP-IQA) and external baselines, so the improvement is not forced by construction. The only self-citation is reference [18] (2-step-SDS), used solely as a comparison baseline in Table 1 and the qualitative figures; it is not load-bearing for the mechanism or the proof. The concern that the Appendix proof establishes persistence of pre-existing nullspace noise rather than creation from a constant initialization is a validity/scope issue about whether the simplified model applies to the real VAE and to the paper's main experiments; it is not circularity, because the paper does not reduce its main claim to that theorem by definition. Overall, no derivation step reduces to its own inputs.
Assumptions & free parameters
free parameters (2)
- beta (look-ahead step size) =
0.1 (2D), 100 x learning_rate (LucidDreamer)
- timestep annealing schedule =
t = 1000 * clip(1 - step/N, 0.4, 1.0)
assumptions (3)
- domain assumption The real VAE encoder is sufficiently linear and underdetermined for the linear noise-amplification analysis to be indicative of actual behavior.
- domain assumption The noise functional N(x) = ||Dx||^2 is a valid proxy for perceived artifact noise.
- standard math Eigenvalues of the path graph Laplacian are 2 - 2 cos(pi(k-1)/n).
Cite this review
Pith. "Pith review of PixSDS: Why Latent SDS Makes Noisy Pixels." pith.science (2026). https://pith.science/paper/6STXBZ5Q
@misc{pith2026260812997,
author = {Pith},
title = {Pith review of: PixSDS: Why Latent SDS Makes Noisy Pixels},
year = {2026},
howpublished = {\url{https://pith.science/paper/6STXBZ5Q}},
note = {Machine review of arXiv:2608.12997}
}
read the original abstract
Score Distillation Sampling (SDS) enables text-to-3D generation by optimizing rendered images with a pretrained diffusion prior, but latent SDS often produces structured color artifacts and high-frequency texture noise. We identify a failure mode of latent SDS caused by VAE-induced pixel drift: the optimized image can move along pixel-space directions that are weakly constrained by the VAE encoder, so its latent representation remains clean and semantically meaningful while the image itself accumulates visible artifacts. We support this diagnosis with controlled 2D SDS experiments, VAE-only optimization, and a simplified analysis showing that encoder-like latent objectives can amplify image-space noise when the inverse mapping to pixels is underconstrained. Motivated by this observation, we propose PixSDS, a lightweight VAE-consistent gradient repair method. PixSDS decodes a latent SDS lookahead step and uses the decoded image as a clean direction for pixel-space optimization, reducing motion in VAE-inconsistent directions without retraining the diffusion model, changing the renderer, or replacing the SDS objective. Experiments in 2D optimization and text-to-3D generation show that PixSDS substantially reduces structured artifacts while preserving semantic content. Code is publicly available at https://sevashasla.github.io/pixsds-webpage/.
Figures
Figures from the paper (11 more)
Reference graph
Works this paper leans on
-
[1]
Camboulin, C., Doimo, D., Glielmo, A.: Understanding variational autoencoders with intrinsic dimension and information imbalance (2024),https://arxiv.org/ abs/2411.01978
work page Pith review arXiv 2024
-
[2]
In: Proceedings of the IEEE/CVF international conference on computer vision
Chen, R., Chen, Y., Jiao, N., Jia, K.: Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 22246–22256 (2023)
2023
-
[3]
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., Podell, D., Dockhorn, T., English, Z., Lacey, K., Goodwin, A., Marek, Y., Rombach, R.: Scaling rectified flow transformers for high-resolution image synthesis (2024),https://arxiv.org/abs/2403.03206
arXiv 2024
-
[4]
Guisard, B.: Stable diffusion nano 2.1.https://huggingface.co/bguisard/stable- diffusion-nano-2-1(2023), hugging Face model card. Accessed: 2026-05-24
work page 2023
-
[5]
In: Proceedings of the 2021 conference on empirical methods in natural language processing
Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., Choi, Y.: Clipscore: A reference- free evaluation metric for image captioning. In: Proceedings of the 2021 conference on empirical methods in natural language processing. pp. 7514–7528 (2021)
2021
-
[6]
Advances in neural information processing systems30(2017)
Heusel,M.,Ramsauer,H.,Unterthiner,T.,Nessler,B.,Hochreiter,S.:Ganstrained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems30(2017)
2017
-
[7]
Katzir, O., Patashnik, O., Cohen-Or, D., Lischinski, D.: Noise-free score distillation (2023),https://arxiv.org/abs/2310.17590
arXiv 2023
-
[8]
Krizhevsky, A., Nair, V., Hinton, G.: Learning multiple layers of features from tiny images. Tech. rep., University of Toronto (2009)
work page 2009
Show all 23 references
-
[9]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Liang, Y., Yang, X., Lin, J., Li, H., Xu, X., Chen, Y.: Luciddreamer: Towards high- fidelity text-to-3d generation via interval score matching. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 6517–6526 (2024)
2024
-
[10]
In: European conference on computer vision
Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014)
2014
-
[11]
Liu, X., Gong, C., Liu, Q.: Flow straight and fast: Learning to generate and transfer data with rectified flow (2022),https://arxiv.org/abs/2209.03003
2022 arXiv
-
[12]
Advances in Neural Information Processing Systems37, 26011–26044 (2024)
Lukoianov, A., de Ocáriz Borde, H.S., Greenewald, K., Guizilini, V.C., Bagaut- dinov, T., Sitzmann, V., Solomon, J.: Score distillation via reparametrized ddim. Advances in Neural Information Processing Systems37, 26011–26044 (2024)
2024
-
[13]
Advances in Neural Information Processing Systems37, 33779–33804 (2024)
McAllister, D., Ge, S., Huang, J.B., Jacobs, D.W., Efros, A.A., Holynski, A., Kanazawa, A.: Rethinking score distillation as a bridge between image distribu- tions. Advances in Neural Information Processing Systems37, 33779–33804 (2024)
2024
-
[14]
IEEE Transactions on Image Processing21(12), 4695–4708 (2012).https://doi.org/10.1109/TIP.2012.2214050
Mittal, A., Moorthy, A.K., Bovik, A.C.: No-reference image quality assessment in the spatial domain. IEEE Transactions on Image Processing21(12), 4695–4708 (2012).https://doi.org/10.1109/TIP.2012.2214050
2012
-
[15]
Pan, Z., Lu, J., Zhu, X., Zhang, L.: Enhancing high-resolution 3d generation through pixel-wise gradient clipping (2024),https://arxiv.org/abs/2310.12474
2024 arXiv
-
[16]
arXiv preprint arXiv:2209.14988 (2022)
Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 (2022)
2022 arXiv
-
[17]
Pope, P., Zhu, C., Abdelkader, A., Goldblum, M., Goldstein, T.: The intrinsic dimension of images and its impact on learning (2021),https://arxiv.org/abs/ 2104.08894 PixSDS: Why Latent SDS Makes Noisy Pixels 15
2021 arXiv
-
[18]
arXiv preprint arXiv:2503.06740 (2025)
Skorokhodov, V., Durasov, N., Fua, P.: Diffusion models are secretly zero-shot 3dgs harmonizers. arXiv preprint arXiv:2503.06740 (2025)
2025 arXiv
-
[19]
Spielman, D.A.: Spectral graph theory lecture 2: The laplacian (Sep 2009),https: //www.cs.yale.edu/homes/spielman/561/2009/lect02-09.pdf, lecture notes
2009
-
[20]
In: International Conference on Learning Representations
Tang, J., Ren, J., Zhou, H., Liu, Z., Zeng, G.: Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In: International Conference on Learning Representations. vol. 2024, pp. 33879–33896 (2024)
2024
-
[21]
Advances in neural information processing systems36, 8406–8441 (2023)
Wang, Z., Lu, C., Wang, Y., Bao, F., Li, C., Su, H., Zhu, J.: Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in neural information processing systems36, 8406–8441 (2023)
2023
-
[22]
arXiv preprint arXiv:1910.03771 (2019)
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al.: Huggingface’s transformers: State-of- the-art natural language processing. arXiv preprint arXiv:1910.03771 (2019)
2019 arXiv
-
[23]
Since there is no universally accepted def- inition for measuring the amount of noise in an image, we introduce a simple proxy for this purpose
Zhu, J., Zhuang, P., Koyejo, S.: Hifa: High-fidelity text-to-3d generation with ad- vanced diffusion guidance (2024),https://arxiv.org/abs/2305.18766 A Noise Proof Wefirstproveamoregeneralstatement,fromwhichtheresultforone-dimensional convolutions follows as a special case. Si...
2024 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.