{"id":"c2477a0c-3de0-440a-ae4d-f5a31d82644d","arxiv_id":"2501.03264","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SI-NP applies self-normalized importance sampling within an EM surrogate objective, yielding consistent log-likelihood gains over NP, CNP, and ML-NP on regression and image completion benchmarks.","lead":"This paper proposes a new training objective for neural processes, called SI-NP, that uses expectation maximization with self-normalized importance sampling to improve how these models fit functions. The authors show SI-NP matches or beats earlier neural process objectives on synthetic regression and image completion, and with attention it reaches state-of-the-art performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 2's improvement guarantee is proved for the exact EM surrogate (Eq. 10), but Algorithm 1 optimizes a biased finite-B SNIS estimate (Eq. 12); with B=8/16 the guarantee is not established for the implemented SI-NP.","rationale":"The reader's weakest assumption pinpoints the exact gap: the paper proves an improvement guarantee for the idealized EM surrogate, while the implemented objective is a finite-sample self-normalized importance sampling estimate. I checked the proof chain in Appendix E.1.2 (Eqs. 23-24) and the algorithm description in Sec. 4.1 and 4.2. The exact-EM argument is standard and appears correct: L(ϑ;ϑ_k) is a valid surrogate, and exact EM iterations monotonically increase the log-likelihood. However, Algorithm 1 never computes the exact posterior; it draws particles from the functional prior and uses SNIS weights. At finite B the SNIS estimator is biased, and no bound on the bias or on the resulting log-likelihood drift is provided. The paper itself acknowledges the bias in Appendix A but appeals only to asymptotic consistency, which does not establish the stated guarantee for B=8/16. This is the most load-bearing concern because the central theoretical contribution is precisely the improvement guarantee; without it, SI-NP is an empirically motivated heuristic objective with asymptotic justification. I do not see a second concern of comparable weight: the Proposition 3 proof has some algebra issues in the L'Hopital derivation, but the conclusion (one-sample SI-NP collapses toward CNP) is qualitatively consistent with the trace plots and does not affect the main claim. The experimental setup is reasonable, the code is released, and the comparisons to NP/CNP/ML-NP are fair for the stated purpose. The empirical results support qualitative improvement, so the paper remains acceptable if the theoretical claims are rephrased or the finite-B behavior is analyzed. Since this is exactly the reader's condition, I leave the verdict unchanged.","tokens_in":30873,"tokens_out":6962,"duration_ms":67535,"concrete_test":"Take a low-dimensional synthetic task (e.g., 1D RBF GP regression, latent dim d=2) and run Algorithm 1 with B=8 as in experiments. After each EM iteration, compute L(ϑ_k)=sum_τ ln ∫ p(DTτ|z;ϑ_k)p(z|DCτ;ϑ_k) dz by dense numerical quadrature (or by an independent B=10^5 MC estimate). If the resulting sequence is not monotone non-decreasing in any of 5 seeds, then the finite-B implementation violates the Proposition 2 guarantee; the paper should then be revised to claim only an asymptotic improvement guarantee.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Proposition 2 (Sec. 4.1.2) and Appendix E.1.2 show that the sequence L(ϑ_k) is non-decreasing when the E-step uses the exact posterior p(z|DT;ϑ_k) and the M-step maximizes L(ϑ;ϑ_k) in Eq. (10). Algorithm 1 does neither: line 2 only 'resets' q_phi to the intractable posterior; the implemented objective is LSI-NP in Eq. (12), where expectations are replaced by self-normalized importance sampling with the functional prior as proposal (line 4) and B=8/16 particles. SNIS is biased at finite B, and the paper's only response is an asymptotic consistency citation (Appendix A, Owen 2013, Ch. 9). No finite-B bound or monotonicity argument is given, so the central improvement-guarantee claim does not apply to the method as trained. The optional proposal update (E-step #2, Eq. 29) is skipped, so the proposal is not adapted to the posterior and the effective sample size can be much smaller than B; in the limit the objective approaches the B=1 collapse analyzed in Proposition 3. The empirical results are credible, but the theoretical headline is overspecified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies neural processes (NPs) from the perspective of the optimization objective. It argues that the standard NP approximate ELBO, which uses an approximate functional prior in the KL regularizer, is not a valid variational objective and can fail to improve the marginal log-likelihood. The authors propose SI-NP, a training objective derived from a variational expectation-maximization surrogate of the meta-dataset log-likelihood. In the exact formulation, the E-step uses the posterior under the current parameters and the M-step maximizes an expected joint log-likelihood; Propositions 1 and 2 claim that this exact EM procedure is a surrogate with a monotonic improvement guarantee. Since the exact posterior is intractable, the implemented algorithm uses self-normalized importance sampling with the functional prior as the proposal, with B=8 or 16 particles. The paper reports experiments on synthetic GP regression, image completion, and Sim2Real benchmarks, finding that SI-NP is competitive with or better than NP, CNP, and ML-NP, with additional gains when augmented with attention. The paper also includes a one-sample analysis (Proposition 3) claiming equivalence of SI-NP with CNP via prior collapse, and extensive appendix ablations.","tokens_in":31125,"tokens_out":10782,"duration_ms":105837,"significance":"If the theoretical guarantee held for the implemented algorithm, this would be a valuable contribution: it would place NP training on a principled EM footing, connect one-sample SI-NP to CNP, and explain observed underfitting of vanilla NPs. The exact-surrogate EM derivation is standard and correct, and the paper ships code plus extensive ablations over particle count, latent dimension, and attention augmentation, which strengthen the empirical story. The empirical results are credible and show consistent gains across multiple benchmarks. However, the advertised improvement guarantee is proved only for the exact EM surrogate, whereas the objective actually optimized is a finite-B self-normalized importance sampling estimate that is biased and is not a lower bound on the target log-likelihood. This gap is load-bearing for the paper's central theoretical claim, so the significance of the theoretical contribution is contingent on either repairing the analysis or substantially qualifying the claim.","major_comments":[{"comment":"Proposition 2 proves monotonic improvement of the meta-dataset log-likelihood for the exact surrogate in Eq. (10), where the E-step uses the exact posterior p(z|DT;ϑ_k) and the M-step maximizes the exact expectation in Eq. (11). The implemented SI-NP instead maximizes the self-normalized importance sampling estimate in Eq. (12) with B=8 or 16 particles, using the functional prior as the proposal and skipping the optional proposal update (Algorithm 1, lines 2-8; §4.2). For finite B, SNIS is biased and is not a lower bound on the log-likelihood, so the guarantee of Proposition 2 does not transfer to the objective actually optimized. The only mitigation offered is asymptotic consistency (Appendix A, citing Owen 2013, Ch. 9), which does not establish monotonic improvement at finite B. The sentence in §4.2 that the method can 'theoretically find at least a local optimal' based on Proposition 2 is therefore too strong. Please either prove a finite-B bias bound or a monotonicity statement, or add training-time measurements of the true log-likelihood showing that L(ϑ_k) is non-decreasing in the implemented setting, and qualify Proposition 2 accordingly.","section":"§4.1.2, Eq. (12), Algorithm 1, and §4.2"},{"comment":"The proof of prior collapse for B=1 is not correct as written. In Eqs. (35)-(36) the sample is reparameterized as \\hat z_i = μ_i + ε_i σ_i, so the quantity (μ_i - \\hat z_i)^2/(2σ_i^2) equals ε_i^2/2 and is not of the form κ_i/σ_i^2 with κ_i independent of σ_i. The L'Hôpital calculation in Eq. (37) treats κ_i as σ-independent and applies L'Hôpital's rule to a ratio whose numerator tends to κ_i rather than to zero; for κ_i>0 the displayed limit is -∞, not +∞. The conclusion of Proposition 3 may still be true under a correctly specified argument in which the M-step optimizes a fixed sample from the old prior, but the derivation as presented does not establish it. Please rewrite the proof with an explicit distinction between the sampling distribution (old parameters) and the optimized parameters, and correct the limit analysis.","section":"Appendix F.1, Proof of Proposition 3"}],"minor_comments":[{"comment":"The phrase 'reset the variational posterior q_ϕ(z) = p(z|DT;ϑ_k)' is misleading because this posterior is intractable and the implemented algorithm never constructs q_ϕ; the actual computation is the SNIS reweighting in Eq. (12). Please reword the algorithm description to make clear that the E-step is implemented by importance weighting rather than by exact posterior computation.","section":"Algorithm 1, line 2"},{"comment":"The result labeled 'L'Hôpital's Rule' is stated as an algebraic identity for differences rather than the standard rule for ratios; this makes the subsequent derivation hard to follow and should be rewritten.","section":"Appendix F.1, Theorem 1"},{"comment":"In the image completion experiments, the training particle count differs across methods (B=8 for SI-NP and B=16 for ML-NP). Please state explicitly whether the conclusions are robust to using the same B for all methods, or justify the asymmetric choice in the main text.","section":"Appendix G.2 and Table 2"},{"comment":"There are several typographical and nomenclature issues: 'de Finneti' should be 'de Finetti' (Section 2), 'Marten' should be 'Matérn' throughout, and 'the the denominator' appears in Appendix E.3.1.","section":"General"},{"comment":"The appendix reports that the learnable proposal update is unstable and that no results are shown for it, yet Algorithm 1 presents E-step #2 as an optional component of the method. Since all main experiments skip this step, the algorithm description should clearly mark the proposal update as a non-evaluated extension rather than part of the recommended procedure.","section":"Appendix H.1"}],"recommendation":"major_revision","confidential_remarks":"This is a conference paper posted on arXiv; the submission appears to be an extended version of the ICLR 2023 paper. The main risk is the mismatch between the theoretical improvement guarantee and the finite-B SNIS objective that is actually trained. This is an internal correctness issue rather than a novelty dispute, and it is fixable by either adding finite-B analysis and monotonicity measurements or by carefully qualifying the theoretical claims. The empirical study is reasonably thorough, and I would not reject on the basis of the finite-B concern alone, but the authors should be required to address it head-on in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this paper. The useful part is the diagnosis: the vanilla NP objective is not a valid ELBO. They show the KL term compares the approximate posterior to a learned approximate prior, and the gap to the true functional prior has an unknown sign, so optimizing it has no guaranteed relationship to the marginal log-likelihood. That alone is worth a citation for people working on NP objectives. The proposed fix, SI-NP, wraps EM around a self-normalized importance sampling objective and is a sensible extension of reweighted wake-sleep to learned-functional-prior settings. The one-sample equivalence with CNP (Proposition 3) is a nice observation, and their experiments show consistent, if modest, improvements over NP/CNP/ML-NP across synthetic and image benchmarks. They release code and are reasonably open about limitations.\n\nThe soft spot is the central theoretical claim. Proposition 2 gives a genuine improvement guarantee, but it is proved for the exact posterior EM surrogate in Eq. (10). The implemented Algorithm 1 replaces the posterior with self-normalized importance sampling using the functional prior as proposal, with B=8 or 16 particles, and the optional proposal update is skipped. SNIS is biased at finite B and no finite-B bound is supplied; the paper only cites asymptotic consistency. So the monotone-improvement claim does not cover the method as trained. This is an overclaim in the abstract, even though the appendix does acknowledge the bias. A revision should rephrase the guarantee as asymptotic (or for the exact EM iterations) and preferably add a finite-B analysis or at least a diagnostic. Secondary issue: the proof of Proposition 3 in Appendix F.1 is suspect. Because z is reparameterized as mu + epsilon sigma, the term (mu - zhat)^2/sigma^2 is O(1), not kappa/sigma^2 with fixed kappa, so the L'Hopital computation looks misapplied. The conclusion that the prior collapses at one sample may still be true via the -ln sigma term, but the written proof needs correction.\n\nFor a reader, the paper delivers: the invalid-ELBO diagnosis is new and useful, the objective is a reasonable competitor, and the experiments support the qualitative claims. The guarantee gap is real but not disqualifying if the claims are softened. I would send this to peer review; it deserves a serious referee who insists on fixing the Proposition 3 proof and rephrasing the guarantee. Bring it to reading group if your group does meta-learning or variational inference.","headline":"Useful NP objective and a correct diagnosis of the vanilla NP's invalid ELBO, but the advertised log-likelihood improvement guarantee only holds for the exact EM surrogate, not the biased finite-particle SNIS algorithm they actually run.","tokens_in":31673,"tokens_out":5549,"would_cite":true,"duration_ms":53445,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Neural processes underfit because their training objective is an invalid ELBO, and a variational-EM surrogate with self-normalized importance sampling is proven to increase the true log-likelihood to a local optimum.","keywords":["neural processes","expectation maximization","self-normalized importance sampling","functional prior","variational inference","meta-learning","prior collapse"],"falsifier":"On a low-dimensional latent space, run Algorithm 1 with its default small particle budget (8 for image completion) and, after every EM iteration, evaluate the exact meta-dataset log-likelihood by numerical integration or an orders-of-magnitude larger particle count; any consecutive pair of iterations in which the exact log-likelihood decreases falsifies the improvement guarantee as implemented.","tokens_in":30657,"feed_emoji":"📈","tokens_out":11754,"duration_ms":97535,"temperature":0.7,"pith_summary":"The paper argues that neural processes underfit because their training objective is an invalid ELBO (evidence lower bound), not because the model family is weak. It shows that the standard NP objective — a generative-likelihood term minus a KL divergence between an approximate posterior and an approximate prior — carries a 'prior approximation gap' of unknown sign, so maximizing it can push the true meta-dataset log-likelihood down. The fix is a variational expectation-maximization surrogate whose E-step averages against the exact posterior and whose M-step maximizes the joint log-likelihood; optimizing this surrogate by alternating steps is proven to increase the true log-likelihood monotonically and to reach at least a local optimum. Because the exact posterior average is intractable, the paper approximates it with self-normalized importance sampling, yielding the SI-NP objective it actually trains. If the guarantee holds, practitioners get an architecture-agnostic objective that improves on NP, CNP, and ML-NP baselines across synthetic regression, image completion, and Sim2Real benchmarks.","feed_headline":"EM objective gives neural processes a monotone improvement guarantee","feed_subtitle":"Replacing the invalid NP objective with importance-weighted EM lifts performance on regression and image benchmarks.","key_machinery":"The carrying object is the variational-EM surrogate function $L(\\vartheta; \\vartheta_k)$ of Eq. (10), a minorize-maximization surrogate of the meta-dataset log-likelihood: it is everywhere below $L(\\vartheta)$ and touches it at $\\vartheta = \\vartheta_k$, so every M-step that improves the surrogate cannot decrease the true log-likelihood. What makes it tractable is the replacement of its posterior expectation by the self-normalized importance-sampling estimator in Eq. (12), where latent particles $z^{(b)}$ are drawn from a proposal $q_\\eta(z|D_T)$ and weighted by normalized weights $\\hat{\\omega}^{(b)}$ proportional to $p(D_T|z^{(b)}; \\vartheta_k) p(z^{(b)}|D_C; \\vartheta_k) / q_\\eta(z^{(b)}|D_T)$; these weights reallocate gradient toward particles that explain the target data well, which is the mechanism by which the functional prior adapts.","core_discovery":"The paper's central claim is that the vanilla NP objective $L_{\\mathrm{NP}}(\\vartheta, \\phi) = \\mathbb{E}_{q_\\phi(z)}[\\ln p(D_T|z; \\vartheta)] - D_{\\mathrm{KL}}(q_\\phi(z) \\| q_\\phi(z|D_C))$ is not a valid lower bound on the meta-dataset log-likelihood $L(\\vartheta) = \\ln p(D_T|D_C; \\vartheta)$: substituting the approximate prior $q_\\phi(z|D_C)$ for the learned functional prior $p(z|D_C; \\vartheta)$ adds a prior-approximation gap of undetermined sign, so optimizing this 'consistent regularizer' can move the true objective the wrong way. In its place, the paper constructs the surrogate $L(\\vartheta; \\vartheta_k) = \\mathbb{E}_{p(z|D_T;\\vartheta_k)}[\\ln p(D_T, z|D_C; \\vartheta) - \\ln p(z|D_T; \\vartheta_k)]$, proves it minorizes the log-likelihood, and shows the EM update $\\vartheta_{k+1} = \\arg\\max_\\vartheta L(\\vartheta; \\vartheta_k)$ gives $L(\\vartheta_{k+1}) \\ge L(\\vartheta_k)$ with convergence to at least a local optimum (Propositions 1-2). The tractable version replaces the intractable posterior expectation with a self-normalized importance-sampling estimate over particles $z^{(b)}$ drawn from a proposal $q_\\eta(z|D_T)$, defaulting to the functional prior as the proposal; with one particle, the objective collapses the functional prior to a Dirac delta, making SI-NP exactly CNP (Proposition 3). The paper reports that SI-NP outperforms NP, CNP, and ML-NP objectives on GP-kernel regression, image completion, and Sim2Real benchmarks, with attention-augmented SI-ANP reaching the best results among the compared baselines.","pith_inferences":["A direct test of the guarantee would track the exact (large-particle) meta-dataset log-likelihood across EM iterations with B = 8 or 16; if it ever decreases, the implemented finite-sample objective, not the exact surrogate, is what fails to satisfy Proposition 2.","The optional proposal update (E-step #2) was skipped because joint training was unstable; finding a stable schedule for it is the most promising route to larger effective sample sizes and a tighter match between the implemented objective and the proven surrogate.","Because the objective is architecture-agnostic, the same EM surrogate could be ported to other learned-prior stochastic-process models such as deep Gaussian processes, where the same invalid-ELBO diagnosis would apply.","The empirically observed prior collapse on MNIST suggests a regime map: on low-semantic-complexity tasks SI-NP will behave like a deterministic CNP, so the multi-particle objective is primarily buying performance where function families are semantically rich."],"forward_implications":["The same encoder-decoder architectures used by NP, CNP, and ML-NP can be kept unchanged; switching to the SI-NP objective is what aligns training with the meta-dataset log-likelihood, so the gain is orthogonal to architecture and compounds with structural inductive biases.","With one Monte Carlo particle, SI-NP provably collapses to CNP, which explains why deterministic conditional models are competitive on simple tasks while multi-particle SI-NP retains functional-prior uncertainty on complex tasks.","Because SI-NP is an importance-weighted objective, its evaluation log-likelihood rises with the number of particles, so the 32-particle numbers reported are lower bounds on its performance in the particle limit.","The learned functional prior's covariance trace tracks the semantic complexity of the task family and shrinks as context points grow, giving a quantitative diagnostic for when latent randomness matters."],"supporting_citations":[{"why":"Defines the vanilla NP generative model and its approximate-ELBO objective that the paper diagnoses as an invalid ELBO and sets out to improve.","marker":"(Garnelo et al., 2018b)"},{"why":"Defines the CNP objective and the conditional baseline that SI-NP provably reduces to in the one-particle limit.","marker":"(Garnelo et al., 2018a)"},{"why":"Supplies the ML-NP Monte-Carlo maximum-likelihood objective and the experimental protocol the paper compares against.","marker":"(Foong et al., 2020)"},{"why":"Supplies the self-normalized importance-sampling estimator that makes the posterior expectation tractable.","marker":"(Tokdar & Kass, 2010)"},{"why":"Provides the asymptotic convergence guarantee for self-normalized importance sampling that the paper cites to bridge the finite-particle estimator to the exact expectation.","marker":"(Owen, 2013)"},{"why":"Defines the minorize-maximization surrogate-function framework used in the proof of Proposition 1.","marker":"(Hunter & Lange, 2004)"},{"why":"Supplies the expectation-maximization framework on which the variational EM algorithm is built.","marker":"(Bishop & Nasrabadi, 2006)"},{"why":"Provides the attention module used to augment SI-NP in the extended experiments reporting top results.","marker":"(Kim et al., 2019)"}],"fun_headline_variants":["EM gives neural processes a valid lower bound and monotone gains","Self-normalized importance sampling fixes NP underfitting","Neural processes get a theoretical boost from EM updates","From invalid objective to guaranteed improvement: EM for NPs","SI-NP: EM-based training with provable log-likelihood gains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The monotone-improvement guarantee is proved for the exact posterior expectation, while the implemented algorithm replaces that expectation with a self-normalized importance-sampling estimate using 8 or 16 particles; the guarantee transfers to the implemented method only if that finite-particle bias is negligible.","fun_headline_variants_meta":{"raw":{"variants":["EM gives neural processes a valid lower bound and monotone gains","Self-normalized importance sampling fixes NP underfitting","Neural processes get a theoretical boost from EM updates","From invalid objective to guaranteed improvement: EM for NPs","SI-NP: EM-based training with provable log-likelihood gains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000718,"raw_usage":{"total_tokens":3329,"prompt_tokens":1152,"completion_tokens":2177,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":768,"completion_tokens_details":{"reasoning_tokens":2095}},"tokens_in":768,"tokens_out":2177,"duration_ms":16092,"temperature":1.0,"reasoning_tokens":2095,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:14:12.311364+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a low-dimensional latent space, run Algorithm 1 with its default small particle budget (8 for image completion) and, after every EM iteration, evaluate the exact meta-dataset log-likelihood by numerical integration or an orders-of-magnitude larger particle count; any consecutive pair of iterations in which the exact log-likelihood decreases falsifies the improvement guarantee as implemented.","supporting_citations":[],"review_version":1}