REVIEW 5 major objections 6 minor
Stochastic Dimension Zeroth-Order Estimator: Stable and Memory-Efficient Training of PINNs
T0 review · 5 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read A backprop-free optimizer can train 10-million-dimensional physics-informed neural networks on a single GPU by locking the spatial random samples across the two perturbed forward passes.
desk verdict CRNS is a genuinely clever fix for the O(1/ε²) variance deadlock; the missing 10M experiments and the O(P/ε) convergence theorem make the headline dimension-free claim unsupported as written. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Two mechanisms carry the argument. First, CRNS: for each finite-difference gradient step, the two loss evaluations at theta plus eps*P*z and theta minus eps*P*z reuse the exact same random subset of spatial derivative terms and collocation points, so the stochastic part of the loss is identical in both evaluations. The symmetric difference then cancels the random base value and the second-order stochastic curvature term, leaving a gradient estimate whose variance no longer contains the 1/eps^2 factor. Second, the implicit matrix-free subspace projection: the parameter perturbation is built from a block-diagonal Kronecker product P = bdiag(V_l ⊗ U_l) of orthonormal low-rank factors, applied t
What would settle it
Run SDZE on a moderate-dimensional stiff PDE, such as the Allen-Cahn equation used in the paper, and every few hundred steps compute the exact full-batch gradient by backpropagation. If the projected gradient norm shrinks while the full gradient norm stays far from zero over many subspace refreshes, the subspace-alignment premise is falsified. A second check: rerun a large benchmark with the perturbation step size epsilon increased tenfold; if CRNS truly cancels the 1/epsilon^2 singularity, gradient variance should stay bounded and training should remain stable.
Extended reading notes
Core claim
The central claim is that the two stochasticities in randomized-operator zeroth-order PINN training—the random sampling of spatial derivatives and the random Gaussian parameter perturbation—can be made exactly variance-cancelling. By enforcing CRNS, the paper proves that the O(1/epsilon^2) variance singularity of naive finite differences disappears: the spatial truncation noise and its stochastic Hessian term cancel in the symmetric difference, leaving only a benign first-order cross term plus a small remainder. The implicit matrix-free subspace projection then bounds the parameter-exploration second moment by (q+2) times the squared norm of the projected gradient, compressing variance from
Load-bearing premise
The load-bearing premise is that the randomly refreshed low-rank subspaces keep capturing the dominant gradient directions: the main-text convergence theorem bounds the projected gradient norm, so if the true gradient repeatedly points outside the current subspace, SDZE can stall at a point that looks converged in projection but is not a stationary point of the full loss.
Editorial extensions
If this is right
- Backpropagation can be eliminated from PINN training entirely, removing the O(P) memory of reverse-mode AD and the compilation bottleneck of nesting AD over randomized spatial operators.
- Fully dense, non-weight-sharing networks can scale to parameter counts around 10^7 on a single GPU, preserving the expressive power that weight-shared architectures sacrifice.
- The same estimator applies to arbitrary high-order differential operators expressible as unbiased randomized spatial oracles, including mixed-derivative operators, not just Laplacians.
- Randomized spatial estimators and zeroth-order optimizers become composable: existing forward-only PDE solvers can be upgraded to backprop-free training without the variance blow-up that naive combinations suffer.
- Because only forward passes are used, SDZE extends PINN training to objectives that are non-differentiable or where automatic differentiation is unavailable.
Reading between the lines
- A cheap empirical check on a mid-sized PDE would track the full exact-gradient norm alongside SDZE's projected estimate; if the full norm fails to shrink over many subspace refreshes, the projected-gradient convergence bound would not reflect true stationarity.
- CRNS is a general recipe: any setting that takes finite differences between two Monte Carlo estimates of a stochastic objective could reuse the same random seed to cancel common noise, suggesting extensions to randomized smoothing, stochastic simulators, or value-gradient estimation.
- The claimed O(1) optimizer memory is asymptotic in the parameter count but hides factors of batch size and subspace rank; the practical ceiling is set by activation memory, so the strongest benchmark is a wall-clock and memory measurement at the reported 10-million-dimension scale.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SDZE, a backprop-free optimizer for high-dimensional PINNs that combines randomized spatial estimators with zeroth-order finite differences. Two mechanisms are introduced: CRNS, which locks the spatial random seed across the two perturbed forward evaluations to remove the O(1/ε²) variance singularity, and an implicit low-rank Kronecker-structured subspace projection that avoids materializing the P×q projection matrix. The manuscript claims dual dimension-independence in space and memory, proves variance-compression and convergence bounds, and reports experiments on high-dimensional Allen-Cahn, KdV, and related equations. The central reported results, however, are the convergence theorem T=O(P/ε) for the full gradient and the 10M-dimensional flagship experiments, the latter of which appear as [TBD] in the main tables.
Significance. If established, the CRNS idea and the implicit matrix-free subspace projection would be genuinely useful: CRNS is a simple, falsifiable mechanism for a real double-stochastic variance problem, and the Kronecker-structured projection is a clean way to avoid storing a dense projection matrix. The per-step memory argument in §4.4 is plausible, and Theorems 4–5 give crisp statements about fixed-subspace variance. However, the paper's headline claims are not established. The only full-gradient convergence guarantee, Theorem 6, scales as O(P/ε), directly contradicting the 'dimension-independent complexity' claim; Eq. (21) controls only a projected gradient whose subspace is refreshed independently of the loss landscape; and the flagship 10M-dimensional experimental result is absent from Tables 1–2. The central contribution is therefore currently unsupported, and the theoretical narrative is internally inconsistent.
major comments (5)
- [Theorem 6, §5.3] The central claim of dimension-independent complexity is contradicted by the paper's own convergence guarantee. Theorem 6 states 1/T Σ E‖∇L(θ_k)‖² ≤ ε with T=O(P/ε), where P is the parameter count. Thus the number of iterations and forward evaluations grows at least linearly in P. This is not a dimension-free rate. If the intended claim is only 'per-step space and memory are independent of P', it should be stated as such; as written, the abstract's 'dimension-independent complexity in both space and memory' is unsupported.
- [Eq. (21), §4.5; Theorem 3, §5.2] The convergence guarantee in Eq. (21) bounds only E‖P_{⌊t/F⌋}^T ∇L(θ_t)‖², the norm of the gradient projected onto the current random subspace. Since P_t is generated by QR of fresh Gaussian matrices with no dependence on the loss, a small projected gradient does not imply a small full gradient. The optimizer can stall at a point with large full gradient while the projected component is small. The label 'global convergence to a stationary point' is therefore not justified. The separate Theorem 3 in §5.2 requires an assumed bounded trajectory and converges locally to a regular minimizer, not a global full-gradient guarantee.
- [Tables 1, 2, and 4] The flagship empirical result is missing. In Tables 1 and 2 the SDZE row at 10M D is [TBD], so the abstract's claim of 'the first successful training of 10-million-dimensional fully-dense PINNs' has no reported support. Table 4, the ablation over subspace rank and update frequency, is entirely [TBD]. No code or data are provided. The empirical core of the paper is therefore not verifiable from the manuscript in its current form.
- [Theorem 6, §5.3 (perturbation scale condition)] The statement of Theorem 6 conflates the tolerance ε and the perturbation scale ϵ. The condition ϵ ≤ O(ε^{1/2}/(q^{3/2}P^{1/2}L₁^{3/2})) forces the finite-difference step to shrink with P, while zeroth-order estimators require ϵ to be bounded away from zero to avoid numerical cancellation. For P=10⁷ this condition gives an extremely small perturbation, creating an additional P-dependence in the algorithm and conflicting with the stable-convergence narrative.
- [Theorem 1, Eq. (15), §4.5] The title 'Exact Algebraic Cancellation of Spatial Variance' is stronger than what Eq. (15) proves. The right-hand side retains E‖⟨∇θηω(θ),P_t z_t⟩‖², which is not cancelled. What the calculation shows is removal of the O(1/ε²) singularity, which is a useful and credible claim; however, the remaining variance term can still depend on the spatial noise field and on q. The theorem statement should be aligned with the actual bound.
minor comments (6)
- [§4.5 vs §5] Theorem numbers are duplicated: Theorems 1–3 in §4.5 are repeated as Theorems 1–3 in §5. Cross-references in the text are ambiguous.
- [Figure 1] The text refers to panels (c) and (d) of Figure 1 ('Expressivity Collapse', 'Figure 1d'), but the figure contains only panels (a) and (b).
- [Abstract vs Conclusion] The abstract and title say 'NVIDIA A100 GPU' while the conclusion says 'NVIDIA A800 GPU'. This inconsistency should be fixed.
- [Tables 1 and 2] RS-PINN is labeled (ZO) in Table 1 but (FO) in Table 2. The labeling should be consistent and correct.
- [§6 and Appendix] Several cross-references are placeholders: 'Section ??' for experiments and 'Appendix??' for experimental setup and hyperparameters. The full setup is not present in the manuscript.
- [References] The STDE reference (Shi et al., 2024) is given as 'arXiv:2401.00000', which appears to be a placeholder and not a valid arXiv identifier.
Circularity Check
No circular derivation found; CRNS and subspace variance results are derived from Taylor/Gaussian identities, and convergence imports external random-subspace ZO theory.
full rationale
The paper's derivation chain is not circular. CRNS (Sections 4.3-4.5) is analyzed by a Taylor expansion on the same locked spatial random measure ω: Eqs. (16)-(17) show the zeroth- and second-order noise terms cancel in the symmetric difference, leaving ⟨∇L,Pz⟩+⟨∇η_ω,Pz⟩+O(ε²). This is a direct consequence of using common random numbers, not a fitted prediction or a target assumed in the assumptions. The subspace variance compression (Theorems 2, 4, 5) follows from Stein's lemma and Isserlis' theorem for Gaussian z with PᵀP=Iq, yielding E[ĝ]=PPᵀ∇L and E‖ĝ‖²=(q+2)‖Pᵀ∇L‖²; again, a standard identity rather than a restatement of the desired conclusion. Convergence results are explicitly imported from external prior work: 'The proofs for the subspace variance compression and global convergence strictly follow the framework of low-dimensional Gaussian smoothing Nozawa et al. [2025]' — a dependency, not a self-citation. No target loss value or error is used to fit CRNS or the projection rank. The main concerns with the paper are correctness/evidence issues, not circularity: Theorem 6's full-gradient rate T=O(P/ε) is dimension-dependent and contradicts the abstract's dimension-independence claim, and the 10M D flagship result is marked [TBD] in Tables 1-2. These are substantive scientific problems, but they are not circularity under the requested definition.
Assumptions & free parameters
free parameters (5)
- Perturbation scale ε =
Not fixed; Theorem 4 suggests ε=1/(q+4); main experiments do not report final value
- Subspace rank r_l =
r ∈ {32, 64, 128} in Table 4 (all [TBD]); not reported for main runs
- Lazy subspace update frequency F =
F ∈ {500, 1000, 2000} in Table 4 (all [TBD]); not reported for main runs
- Learning-rate schedule constants γ, m, p =
γ_n = γ/(n+m)^p, p∈(1/2,1]; γ and m not numerically specified
- Spatial batch sizes |B|, |I|, |J| =
Not fully reported; guidance is |I|≈|B|≈100 and |J|≈|I|
assumptions (6)
- domain assumption Assumption 1: the exact loss L has locally second-order smoothness with L2-Lipschitz Hessian; the spatial noise η_ω is locally third-order smooth with bounded variance
- domain assumption Assumption 5.1: activation σ has |σ^(k)|≤1 and 1-Lipschitz derivatives up to PDE order n; residual R is bounded
- domain assumption Assumption 5.2: each PDE operator L_i is bounded on compact input spaces
- ad hoc to paper Bounded SDZE trajectory: ||W_l^n|| ≤ M^(l) for all layers and epochs
- standard math Stein's Lemma and Isserlis' theorem, plus P^T P=I_q from QR/block-diagonal construction
- domain assumption Kawaguchi's regular-minimizer property is relevant to the PINN loss landscape
Cite this review
Pith. "Pith review of Stochastic Dimension Zeroth-Order Estimator: Stable and Memory-Efficient Training of PINNs." pith.science (2026). https://pith.science/paper/LMABHWB4
@misc{pith2026260324002,
author = {Pith},
title = {Pith review of: Stochastic Dimension Zeroth-Order Estimator: Stable and Memory-Efficient Training of PINNs},
year = {2026},
howpublished = {\url{https://pith.science/paper/LMABHWB4}},
note = {Machine review of arXiv:2603.24002}
}
abstract
Physics-Informed Neural Networks (PINNs) for high-dimensional and high-order partial differential equations (PDEs) are primarily constrained by the $\mathcal{O}(d^k)$ spatial derivative complexity and the $\mathcal{O}(P)$ memory overhead of backpropagation (BP). While randomized spatial estimators successfully reduce the spatial complexity to $\mathcal{O}(1)$, their reliance on first-order optimization still leads to prohibitive memory consumption at scale. Zeroth-order (ZO) optimization offers a BP-free alternative; however, naively combining randomized spatial operators with ZO perturbations triggers a variance explosion of $\mathcal{O}(1/\varepsilon^2)$, leading to numerical divergence. To address these challenges, we propose the \textbf{S}tochastic \textbf{D}imension-free \textbf{Z}eroth-order \textbf{E}stimator (\textbf{SDZE}), a unified framework that achieves dimension-independent complexity in both space and memory. Specifically, SDZE leverages \emph{Common Random Numbers Synchronization (CRNS)} to algebraically cancel the $\mathcal{O}(1/\varepsilon^2)$ variance by locking spatial random seeds across perturbations. Furthermore, an \emph{implicit matrix-free subspace projection} is introduced to reduce parameter exploration variance from $\mathcal{O}(P)$ to $\mathcal{O}(r)$ while maintaining an $\mathcal{O}(1)$ optimizer memory footprint. Empirical results demonstrate that SDZE enables the training of 10-million-dimensional PINNs on a single NVIDIA A100 GPU, delivering significant improvements in speed and memory efficiency over state-of-the-art baselines.
Figures
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.