{"id":"ee7d86c2-672d-48ad-a9b7-0ab8bcedee5d","arxiv_id":"2603.24002","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"SDZE claims to make PINN training memory independent of both PDE dimension and parameter count by locking random seeds across zeroth-order perturbations, but the headline 10M-dimensional result is an unfilled placeholder.","lead":"SDZE is a proposed backprop-free optimizer for physics-informed neural networks that combines randomized spatial derivative estimation with zeroth-order gradient estimates, targeting O(1) memory at extreme parameter counts. The paper's key empirical claim—training 10-million-dimensional PINNs on one GPU—is not actually reported: the central experiment table cells are still marked [TBD].","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dimension-free claim is contradicted by the paper's own Theorem 6 full-gradient rate O(P/ε), and the 10M-D flagship result is still [TBD] in Tables 1–2.","rationale":"The reader's rejection is well-founded. The paper's two supporting pillars—CRNS variance cancellation and the subspace variance compression identity—are not themselves the main problem; the second-moment computation is standard Isserlis/Stein reasoning, and the CRNS finite-difference expansion does algebraically remove the 1/ε² denominator. But these pillars only control per-step variance. The missing pillar for 'dimension-independent complexity' is an iteration-complexity guarantee that does not scale with P. The only full-gradient result, Theorem 6, gives T=O(P/ε), which is the opposite of dimension-free. Eq. (21)'s projected-norm bound is insufficient: random lazy subspaces are drawn independently of the loss, so a small projected gradient does not certify stationarity. The 10M-D experiment is absent ([TBD] in Tables 1 and 2), so the empirical half of the claim is also unverified. Since the central claim is both theoretically unsupported and empirically unprovided, the REJECT verdict stands unchanged.","tokens_in":32173,"tokens_out":9096,"duration_ms":105865,"concrete_test":"Run SDZE on the quadratic loss L(θ)=||θ||² with fixed rank r and refresh frequency F for P=10⁴ versus P=10⁶, and measure iterations required to reach E||∇L|| ≤ 1. If the iteration count scales roughly linearly with P, then Theorem 6's O(P/ε) rate is confirmed and the paper's dimension-independent complexity claim is false.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is 'dimension-independent complexity in both space and memory'. Space/memory per step may be O(q) auxiliary, but the only full-gradient convergence guarantee supplied, Theorem 6 (Section 5.3), states T = O(P/ε), where P is the parameter dimension. That is linear in P, so the required number of iterations/function evaluations is not dimension-independent. Theorem 3/Eq. (21) cannot substitute: it bounds only E||P_t^T ∇L||^2, the norm of the gradient projected onto the currently refreshed random subspace, and a small projected norm does not imply a small full gradient unless that subspace happens to contain the dominant gradient directions. The subspace P_t is generated by QR of Gaussian matrices refreshed every F steps with no dependence on the loss landscape, so there is no argument or experiment establishing this alignment. Granting CRNS and the O(P)->O(q) variance compression, this still leaves the optimizer's total cost dimension-dependent. In addition, the empirical flagship claim—successful 10-million-dimensional training—is literally absent: Table 1 and Table 2 list SDZE at 10M D as [TBD], and no code or data is provided. The central claim is therefore neither theoretically nor empirically established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SDZE, a backprop-free optimizer for high-dimensional PINNs that combines randomized spatial estimators with zeroth-order finite differences. Two mechanisms are introduced: CRNS, which locks the spatial random seed across the two perturbed forward evaluations to remove the O(1/ε²) variance singularity, and an implicit low-rank Kronecker-structured subspace projection that avoids materializing the P×q projection matrix. The manuscript claims dual dimension-independence in space and memory, proves variance-compression and convergence bounds, and reports experiments on high-dimensional Allen-Cahn, KdV, and related equations. The central reported results, however, are the convergence theorem T=O(P/ε) for the full gradient and the 10M-dimensional flagship experiments, the latter of which appear as [TBD] in the main tables.","tokens_in":32487,"tokens_out":6510,"duration_ms":73301,"significance":"If established, the CRNS idea and the implicit matrix-free subspace projection would be genuinely useful: CRNS is a simple, falsifiable mechanism for a real double-stochastic variance problem, and the Kronecker-structured projection is a clean way to avoid storing a dense projection matrix. The per-step memory argument in §4.4 is plausible, and Theorems 4–5 give crisp statements about fixed-subspace variance. However, the paper's headline claims are not established. The only full-gradient convergence guarantee, Theorem 6, scales as O(P/ε), directly contradicting the 'dimension-independent complexity' claim; Eq. (21) controls only a projected gradient whose subspace is refreshed independently of the loss landscape; and the flagship 10M-dimensional experimental result is absent from Tables 1–2. The central contribution is therefore currently unsupported, and the theoretical narrative is internally inconsistent.","major_comments":[{"comment":"The central claim of dimension-independent complexity is contradicted by the paper's own convergence guarantee. Theorem 6 states 1/T Σ E‖∇L(θ_k)‖² ≤ ε with T=O(P/ε), where P is the parameter count. Thus the number of iterations and forward evaluations grows at least linearly in P. This is not a dimension-free rate. If the intended claim is only 'per-step space and memory are independent of P', it should be stated as such; as written, the abstract's 'dimension-independent complexity in both space and memory' is unsupported.","section":"Theorem 6, §5.3"},{"comment":"The convergence guarantee in Eq. (21) bounds only E‖P_{⌊t/F⌋}^T ∇L(θ_t)‖², the norm of the gradient projected onto the current random subspace. Since P_t is generated by QR of fresh Gaussian matrices with no dependence on the loss, a small projected gradient does not imply a small full gradient. The optimizer can stall at a point with large full gradient while the projected component is small. The label 'global convergence to a stationary point' is therefore not justified. The separate Theorem 3 in §5.2 requires an assumed bounded trajectory and converges locally to a regular minimizer, not a global full-gradient guarantee.","section":"Eq. (21), §4.5; Theorem 3, §5.2"},{"comment":"The flagship empirical result is missing. In Tables 1 and 2 the SDZE row at 10M D is [TBD], so the abstract's claim of 'the first successful training of 10-million-dimensional fully-dense PINNs' has no reported support. Table 4, the ablation over subspace rank and update frequency, is entirely [TBD]. No code or data are provided. The empirical core of the paper is therefore not verifiable from the manuscript in its current form.","section":"Tables 1, 2, and 4"},{"comment":"The statement of Theorem 6 conflates the tolerance ε and the perturbation scale ϵ. The condition ϵ ≤ O(ε^{1/2}/(q^{3/2}P^{1/2}L₁^{3/2})) forces the finite-difference step to shrink with P, while zeroth-order estimators require ϵ to be bounded away from zero to avoid numerical cancellation. For P=10⁷ this condition gives an extremely small perturbation, creating an additional P-dependence in the algorithm and conflicting with the stable-convergence narrative.","section":"Theorem 6, §5.3 (perturbation scale condition)"},{"comment":"The title 'Exact Algebraic Cancellation of Spatial Variance' is stronger than what Eq. (15) proves. The right-hand side retains E‖⟨∇θηω(θ),P_t z_t⟩‖², which is not cancelled. What the calculation shows is removal of the O(1/ε²) singularity, which is a useful and credible claim; however, the remaining variance term can still depend on the spatial noise field and on q. The theorem statement should be aligned with the actual bound.","section":"Theorem 1, Eq. (15), §4.5"}],"minor_comments":[{"comment":"Theorem numbers are duplicated: Theorems 1–3 in §4.5 are repeated as Theorems 1–3 in §5. Cross-references in the text are ambiguous.","section":"§4.5 vs §5"},{"comment":"The text refers to panels (c) and (d) of Figure 1 ('Expressivity Collapse', 'Figure 1d'), but the figure contains only panels (a) and (b).","section":"Figure 1"},{"comment":"The abstract and title say 'NVIDIA A100 GPU' while the conclusion says 'NVIDIA A800 GPU'. This inconsistency should be fixed.","section":"Abstract vs Conclusion"},{"comment":"RS-PINN is labeled (ZO) in Table 1 but (FO) in Table 2. The labeling should be consistent and correct.","section":"Tables 1 and 2"},{"comment":"Several cross-references are placeholders: 'Section ??' for experiments and 'Appendix??' for experimental setup and hyperparameters. The full setup is not present in the manuscript.","section":"§6 and Appendix"},{"comment":"The STDE reference (Shi et al., 2024) is given as 'arXiv:2401.00000', which appears to be a placeholder and not a valid arXiv identifier.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper has a strong central idea but the current version is not publishable in its present form. The main issue is not stylistic: the convergence theorem contradicts the dimension-freeness claim, and the flagship experiment is missing. Even if the 10M experiment were supplied, the theoretical gap regarding projected-gradient convergence would need to be closed or the claims substantially weakened. I would recommend rejection rather than major revision because the central narrative needs rework beyond a local fix."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this paper has one genuinely good idea—locking the spatial random seed across the two ZO perturbations (CRNS) to cancel the O(1/ε²) variance—and it develops that idea carefully. The Taylor-expansion derivation in Theorem 1 is standard and plausible, and the associative forward pass that never materializes the projection matrix is a practical trick that papers in this area will want to steal. Table 3 shows the expected blow-up without CRNS. That part deserves credit.\n\nThe soft spots are load-bearing, not cosmetic. The headline claim—“first successful training of 10-million-dimensional PINNs”—is literally empty: the 10M rows of Tables 1 and 2 read [TBD], the ablation table is all [TBD], and several appendix references are missing. No code or data either. You cannot verify the central empirical promise.\n\nOn the theory side, the abstract promises “dimension-independent complexity in both space and memory,” which a reader naturally takes to cover iteration count. Theorem 6 says the full-gradient convergence time is T = O(P/ε), linear in P. That is their own equation, and it contradicts the narrative. Equation 21 only controls the projected gradient norm ‖P_tᵀ∇L‖, which is weak without an argument that the lazily-refreshed random subspace contains the dominant gradient directions. The paper provides no such argument and no experiment that would test subspace alignment. So even granting CRNS, the central “dual dimension-independence” claim is neither theoretically nor empirically established.\n\nThe paper is not sloppy in its derivations, and it is not hiding the gaps—they are right there as [TBD]. That is an incomplete paper, not a dishonest one. But the incompleteness strikes at the exact claims that motivate the work.\n\nWho is this for? Researchers working on backprop-free PINN training and on randomized ZO optimization. They will find the CRNS mechanism and the implicit subspace pass worth studying. It should go to peer review—a serious referee can sort out whether the O(P/ε) bound is loose or fatal, and whether the subspace-alignment worry is real in practice. But do not accept it in this form; ask for complete experiments, code, and a convergence theorem that matches the prose. I would hold off citing it until the missing results appear.","headline":"CRNS is a genuinely clever fix for the O(1/ε²) variance deadlock; the missing 10M experiments and the O(P/ε) convergence theorem make the headline dimension-free claim unsupported as written.","tokens_in":32981,"tokens_out":2479,"would_cite":false,"duration_ms":27367,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","65N35","90C56"],"pacs":[],"model":"deepseek-v4-flash","headline":"A backprop-free optimizer can train 10-million-dimensional physics-informed neural networks on a single GPU by locking the spatial random samples across the two perturbed forward passes.","keywords":["physics-informed neural networks","zeroth-order optimization","randomized spatial estimators","variance cancellation","common random numbers","low-rank subspace projection","high-dimensional PDEs","backprop-free training"],"falsifier":"Run SDZE on a moderate-dimensional stiff PDE, such as the Allen-Cahn equation used in the paper, and every few hundred steps compute the exact full-batch gradient by backpropagation. If the projected gradient norm shrinks while the full gradient norm stays far from zero over many subspace refreshes, the subspace-alignment premise is falsified. A second check: rerun a large benchmark with the perturbation step size epsilon increased tenfold; if CRNS truly cancels the 1/epsilon^2 singularity, gradient variance should stay bounded and training should remain stable.","tokens_in":32016,"feed_emoji":"🧮","tokens_out":7287,"duration_ms":71296,"temperature":0.7,"pith_summary":"Training physics-informed neural networks on high-dimensional, high-order PDEs usually hits two walls: evaluating high-order spatial derivatives costs O(d^k), and backpropagation for parameter gradients consumes memory that grows with the number of parameters. The paper argues that SDZE, a fully forward-only optimizer, removes both walls at once. Its key move is CRNS: the same spatial random sample is reused for the positive and negative perturbed forward passes, so the Monte Carlo noise in the PDE residual cancels algebraically instead of being amplified by 1/epsilon^2. On the parameter side, an implicit low-rank subspace projection keeps exploration variance proportional to a small subspace rank rather than the parameter count, without ever materializing a projection matrix. If the claims hold, SDZE would be the first PINN framework whose spatial cost and optimizer memory are both independent of dimension, demonstrated by training a 10-million-dimensional fully dense network on one GPU.","feed_headline":"Zero-backprop optimizer trains 10M-dimensional PINNs","feed_subtitle":"Reusing spatial random samples across perturbed forward passes cancels the variance that sinks naive zeroth-order PINN training.","key_machinery":"Two mechanisms carry the argument. First, CRNS: for each finite-difference gradient step, the two loss evaluations at theta plus eps*P*z and theta minus eps*P*z reuse the exact same random subset of spatial derivative terms and collocation points, so the stochastic part of the loss is identical in both evaluations. The symmetric difference then cancels the random base value and the second-order stochastic curvature term, leaving a gradient estimate whose variance no longer contains the 1/eps^2 factor. Second, the implicit matrix-free subspace projection: the parameter perturbation is built from a block-diagonal Kronecker product P = bdiag(V_l ⊗ U_l) of orthonormal low-rank factors, applied t","core_discovery":"The central claim is that the two stochasticities in randomized-operator zeroth-order PINN training—the random sampling of spatial derivatives and the random Gaussian parameter perturbation—can be made exactly variance-cancelling. By enforcing CRNS, the paper proves that the O(1/epsilon^2) variance singularity of naive finite differences disappears: the spatial truncation noise and its stochastic Hessian term cancel in the symmetric difference, leaving only a benign first-order cross term plus a small remainder. The implicit matrix-free subspace projection then bounds the parameter-exploration second moment by (q+2) times the squared norm of the projected gradient, compressing variance from","pith_inferences":["A cheap empirical check on a mid-sized PDE would track the full exact-gradient norm alongside SDZE's projected estimate; if the full norm fails to shrink over many subspace refreshes, the projected-gradient convergence bound would not reflect true stationarity.","CRNS is a general recipe: any setting that takes finite differences between two Monte Carlo estimates of a stochastic objective could reuse the same random seed to cancel common noise, suggesting extensions to randomized smoothing, stochastic simulators, or value-gradient estimation.","The claimed O(1) optimizer memory is asymptotic in the parameter count but hides factors of batch size and subspace rank; the practical ceiling is set by activation memory, so the strongest benchmark is a wall-clock and memory measurement at the reported 10-million-dimension scale."],"forward_implications":["Backpropagation can be eliminated from PINN training entirely, removing the O(P) memory of reverse-mode AD and the compilation bottleneck of nesting AD over randomized spatial operators.","Fully dense, non-weight-sharing networks can scale to parameter counts around 10^7 on a single GPU, preserving the expressive power that weight-shared architectures sacrifice.","The same estimator applies to arbitrary high-order differential operators expressible as unbiased randomized spatial oracles, including mixed-derivative operators, not just Laplacians.","Randomized spatial estimators and zeroth-order optimizers become composable: existing forward-only PDE solvers can be upgraded to backprop-free training without the variance blow-up that naive combinations suffer.","Because only forward passes are used, SDZE extends PINN training to objectives that are non-differentiable or where automatic differentiation is unavailable."],"fun_headline_variants":["Variance-cancelling seeds shrink PINN memory to O(1)","Zero-order optimizer tames PINN variance, fits 10M dims","Common random numbers slash PINN memory, hit 10M params","Dimension-free zeroth-order method trains 10M PINNs","Subspace projection plus seed locking scales PINNs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the randomly refreshed low-rank subspaces keep capturing the dominant gradient directions: the main-text convergence theorem bounds the projected gradient norm, so if the true gradient repeatedly points outside the current subspace, SDZE can stall at a point that looks converged in projection but is not a stationary point of the full loss.","fun_headline_variants_meta":{"raw":{"variants":["Variance-cancelling seeds shrink PINN memory to O(1)","Zero-order optimizer tames PINN variance, fits 10M dims","Common random numbers slash PINN memory, hit 10M params","Dimension-free zeroth-order method trains 10M PINNs","Subspace projection plus seed locking scales PINNs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1193,"prompt_tokens":812,"completion_tokens":381,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":292}},"tokens_in":556,"tokens_out":381,"duration_ms":4398,"temperature":1.0,"reasoning_tokens":292,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T17:31:00.404657+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SDZE on a moderate-dimensional stiff PDE, such as the Allen-Cahn equation used in the paper, and every few hundred steps compute the exact full-batch gradient by backpropagation. If the projected gradient norm shrinks while the full gradient norm stays far from zero over many subspace refreshes, the subspace-alignment premise is falsified. A second check: rerun a large benchmark with the perturbation step size epsilon increased tenfold; if CRNS truly cancels the 1/epsilon^2 singularity, gradient variance should stay bounded and training should remain stable.","supporting_citations":[],"review_version":2}