Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Guaranteed prediction sets for functional surrogate models

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper constructs conformal prediction sets for function-valued surrogates as nested zonotopes, preserving the coverage guarantee when SVD dimensions are truncated.

desk verdict Solid new conformal machinery for functional surrogates, but the SVD-truncation coverage claim in Section 4.1 is unproven and needs fixing before the headline guarantee is trustworthy. read the letter →

arxiv 2501.18426 v2 pith:UXEVRWUW submitted 2025-01-30 cs.LG cs.AI

Ander Gray , Vignesh Gopakumar , Sylvain Rousseau , Sébastien Destercke This is my paper · ORCID
classification cs.LGcs.AI
keywords conformalpredictionsetszonotopesfunctionalsurrogatemodelsneuraloperatorsuncertaintyquantificationPDEemulatorsSVDdimensionreduction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's goal is to give pre-trained functional surrogate models—maps between function spaces, such as neural operators that emulate PDE solvers—a set-valued prediction with a finite-sample statistical guarantee: for a new input, the true output function lies inside the predicted set with probability at least $1-\alpha$. The construction works on the surrogate's calibration errors: it reduces them by SVD, encloses the projected errors in a zonotope, calibrates a nested family of zonotopes with conformal prediction, and propagates the sets back through the linear SVD map. A Cartesian product with a bounding box over the discarded SVD modes is introduced so the guarantee survives the dimension reduction. If correct, this gives PDE emulators and other scientific ML models a practical, model-agnostic uncertainty quantification with a formal coverage guarantee, at a small computational cost once calibration is done.

What carries the argument

The load-bearing object is the nested zonotope family $Z^\alpha_{p_Z} = \langle c_Z(1-\alpha) + p_Z\alpha,\; G_Z(1-\alpha)\rangle$, where $c_Z$ is a centre, $G_Z$ a generator matrix, and $p_Z$ a data-depth core inside the enclosing zonotope. As $\alpha$ goes from 0 to 1 the sets shrink and contract toward $p_Z$, and the paper proves $Z^{\alpha_1} \supseteq Z^{\alpha_2}$ for $\alpha_1 \le \alpha_2$ via half-space inclusion. Calibration turns the abstract parameter $\alpha$ into a conformal quantile: each calibration point is scored by the largest $\alpha$ at which it remains inside the family, and the prediction level is the appropriate empirical quantile of these scores. Zonotopes matter because they are closed under Cartesian products and pass exactly through linear maps, which lets the method append the truncation bounding box and then project the whole set back to the output space in one matrix operation.

What would settle it

Generate many exchangeable test points and check the truncated-coordinate indicator $I_E(x_{n+1})$ directly: if $E$ is the min-max box of $n$ calibration points in $k$ truncated dimensions, the frequency of $x_{n+1} \in E$ is approximately $(1-2/(n+1))^k$; for $k$ large relative to $n$, this empirical coverage drops below $1-\alpha$, and the final prediction set $C_\alpha$ will miss the true function even though the low-dimensional zonotope contains its projection.

Watch

Extended reading notes

Core claim

The central claim is that one can construct a prediction set $C_\alpha = \hat{f}(X_{n+1}) + U\Sigma(Z^\alpha_{p_Z} \times E)$ that satisfies $P(F_{n+1} \in C_\alpha) \ge 1-\alpha$ under exchangeability. Here $U\Sigma$ is the SVD reconstruction map, $Z^\alpha_{p_Z}$ is a nested family of zonotopes enclosing the projected calibration errors, and $E$ is a bounding box on the truncated SVD coordinates. The paper proves that the zonotope family is nested, calibrates it by ranking calibration points according to the largest membership level at which each point stays inside the family (the conformal score), and argues that the Cartesian product with $E$ does not disturb calibration because $E$ contains all calibration data in the truncated dimensions and remains fixed as $\alpha$ changes. Numerical experiments on Burgers, wave, and Navier-Stokes surrogates show empirical coverage close to the nominal level with tighter sets than a supremum-based baseline.

Load-bearing premise

The guarantee in the truncated SVD dimensions rests on the assumption that a new point's discarded coordinates, not just the calibration points' coordinates, fall inside the min-max bounding box $E$; with many truncated dimensions this becomes substantially less likely than the nominal $1-\alpha$.

Editorial extensions

If this is right

  • Functional surrogates such as Fourier neural operators can report a set of plausible solution fields, not just a point prediction, with a user-chosen coverage level.
  • The calibration cost is paid once (75 seconds in the Burger's example), while a new prediction set is computed in about 0.02 seconds, making the method usable as an online wrapper.
  • Because the construction is model agnostic and needs only a held-out calibration set, the same procedure applies to any surrogate whose output is a finely discretised function.
  • The SVD truncation bound means users do not need to keep every output dimension in the expensive set-calibration step; a modest number of modes plus a bounding box preserves the formal guarantee.
  • Compared with supremum-based functional conformal bands, the zonotope sets are tighter on the PDE benchmarks at similar empirical coverage levels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The truncated-dimension guarantee could be strengthened by replacing the empirical min-max box $E$ with a separately calibrated set, for instance a family of boxes with its own conformal quantile; the paper currently uses $E$ only through its containing the calibration data and staying fixed as $\alpha$ changes.
  • Since calibration relies only on membership scores, the same pipeline works with any parametric nested family closed under the propagation map, so ellipsoidal or non-symmetric families are a natural comparison.
  • A testable extension is an input-dependent calibration step that conditions the enclosing zonotope or the core $p_Z$ on $X$, which would improve conditional coverage at the cost of a more complex calibration procedure.
  • A users could choose the SVD truncation rank by balancing the tightness gained from keeping more modes against the extra volume contributed by the bounding box in the truncated coordinates.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper proposes a conformal prediction method for functional surrogate models. The authors map prediction residuals to a low-dimensional SVD space, fit an enclosing zonotope with a data-depth core, define a nested family of zonotopes, calibrate the nesting parameter by rank/membership scores, and propagate the calibrated sets back through the SVD to obtain a functional prediction set. To compensate for SVD truncation, they take a Cartesian product with the min/max box of the truncated residuals and claim the conformal guarantee is preserved. The main positive contribution is the use of zonotopes for exact linear propagation and the nested-set calibration. The central validity claim, however, is not established: the Section 4.1 truncation argument is incorrect, and the fit/calibration protocol needs clarification.

Significance. Zonotope-based conformal sets for high-dimensional functional outputs would be a useful contribution if the validity gap is repaired: the method is model-agnostic, the set family is nested, propagation is linear and exact, and the experiments are run on standard PDE benchmarks and compared against functional conformal baselines. The paper's strengths are the explicit Proposition 1 family, the use of conformal quantiles rather than fitted parameters to set alpha, and the practical computational pipeline. The advertised guarantee-preserving truncation bound is not currently proven, and the empirical coverage in Table 1 does not by itself establish the finite-sample claim.

major comments (3)
  1. [Section 4.1] The argument that the Cartesian product with the bounding box E preserves the conformal guarantee is incorrect. The statement that IE always returns true during calibration, and thus would not change the scoring in Eq. (4), only holds for the calibration points; a new exchangeable point can fall outside E in the truncated coordinates. For continuous data, the per-coordinate probability of lying inside the calibration min/max interval is (n-1)/(n+1), so with d truncated coordinates the probability of lying inside E is ((n-1)/(n+1))^d under independence, which can be far below 1. The test membership score for R_alpha = Z^alpha_pZ x E then has extra mass at zero compared with the calibration scores, and the conformal quantile in Eq. (3) does not imply P(F_{n+1} in C_alpha) >= 1-alpha. A separate calibration of E (for example, per-dimension conformal intervals with a Bonferroni split of alpha) is needed to restore the guarantee.
  2. [Sections 3.2 and 4] Eq. (3) is asserted with the usual exchangeability argument, but the paper does not state whether the same data are used to fit the enclosing zonotope Z and core pZ and to compute the calibration scores alpha_i. As written, Section 3.1 fits Z on some data and Section 3.2 immediately calibrates on that same data, which is a double use: the score function then depends on the very calibration scores to which it is compared, breaking the exchangeability required for Eq. (3). The experiments in Section 4.2 mention a separate 1025-point set used to train the SVD, but the method section contains no such split and no validity proof for any split. The authors should either prove validity for the data-dependent family (e.g., by a full-conformal argument that includes the test point in the construction) or explicitly adopt a split-conformal protocol with distinct fitting and calibration data and prove Eq. (3) for that protocol.
  3. [Table 1] The empirical validation should not be taken as establishing the guarantee. At alpha = 0.1 the rotated-zonotope coverage is 88.57 for Burgers (FNO) and 87.33 for Navier Stokes (FNO), and for validation sizes on the order of 500 to 1000 the latter is several standard errors below the nominal 90 percent. While finite-sample fluctuation is possible, this pattern is consistent with the Section 4.1 defect, and the paper should provide error bars and a discussion before claiming empirical support.
minor comments (5)
  1. [Section 3.2, Eq. (3)] Please clarify the ordering used in sorted alpha and replace alpha_{ceil(epsilon n)} by the finite-sample quantile with n+1, since the standard split-conformal quantile uses ceil((1-alpha)(n+1)); as written, the notation is ambiguous.
  2. [Appendix A.2] The half-space containment condition a1 <= lambda a2 should be a1 = lambda a2 with lambda >= 0; coordinatewise inequality is neither necessary nor sufficient for parallelism.
  3. [Section 4.1] The phrase the data is totally bounded (and remains so as alpha changes) in the extra dimensions is tautological for a finite calibration set and says nothing about a future draw; it should be replaced by an explicit probabilistic statement.
  4. [Figure 6] The claim that the guarantee holds irrespective of the grid discretisation is supported only by a Gaussian example; please provide the argument in the text and state the data-generation protocol for the 200 million coverage points.
  5. [Table 2] The column header Calibration data dimensions is ambiguous: for Burgers the entry 2048 1024 does not match the 500 calibration points stated in Section 4.2, so the table should state which quantities are training size, calibration size, and output dimension.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the conformal calibration is data-driven and the truncation-box concern in Section 4.1 is a soundness gap, not a circular derivation.

full rationale

The derivation of the coverage guarantee is self-contained in the sense that matters for circularity. The nested zonotope family Z^alpha in Proposition 1 is parameterized by alpha and calibrated by assigning each calibration point its largest membership level (Eq. 4), then taking a quantile of the resulting scores (Eq. 3). This is standard split-conformal reasoning: the scores are computed from calibration data, the quantile is chosen from those scores, and the exchangeability assumption supplies the probability bound. No parameter is fitted to the test set, and the claimed P(F_{n+1} in C_alpha) >= 1 - alpha is not assumed as an input; it follows from the conformal quantile argument. The SVD truncation bounding box in Section 4.1 does raise a correctness concern: E is the min-max box of the calibration errors, so I_E = 1 for every calibration point by construction, but a new exchangeable error can leave E, so the test-time coverage of R_alpha = Z_alpha x E is not established by the two stated reasons in the paper. That is a soundness gap, not circularity; it is not a case of a prediction being equivalent to its inputs or of a fitted parameter being renamed as a prediction. Self-citations such as Gopakumar et al. 2024a supply external surrogate models and benchmark data; they are not load-bearing for the conformal validity argument. The paper also honestly states its limitations, including exchangeability and marginal coverage, which further indicates the authors are not hiding the assumptions behind a circular chain.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method introduces no exotic new objects; zonotopes and SVD are standard. The free parameters are modeling choices affecting set tightness, not constants tuned to force the claimed coverage. The main caveat is the unverified assumption about the truncated-coordinate bounding box.

free parameters (3)
  • Number of retained SVD components k = 32 for PDE examples; 2-18 for MLP benchmarks
    Chosen to capture ~99% of error variance. Affects the number of truncated dimensions and thus the size of the bounding box E and the tightness of the final set.
  • Choice of data-depth method for core pZ = Mahalanobis depth in main examples
    Affects the shape of the nested sets but not the conformal validity. In the paper it is chosen by the authors, not fitted to any target.
  • Zonotope fitting method = Rotated hyperrectangle (PCA) in main examples
    Generates the enclosing zonotope. The convex-hull method is used for small dimensions.
assumptions (3)
  • domain assumption Exchangeability of calibration and test data (same joint distribution, no drift).
    Required for the conformal guarantee in Section 2 and used throughout.
  • domain assumption The SVD dimension reduction and zonotope fitting are performed on a dataset separate from the conformal calibration set.
    Necessary for the calibration scores to be exchangeable with a test score, as described in Section 4.1 and the experimental setup.
  • ad hoc to paper The truncated dimensions' coordinates of a new point have probability near 1 of lying inside the data-min/max box E.
    This is the load-bearing but unproved premise behind the claim in Section 4.1 that the Cartesian product with E preserves the coverage guarantee.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Guaranteed prediction sets for functional surrogate models." pith.science (2026). https://pith.science/paper/UXEVRWUW

@misc{pith2026250118426,
  author       = {Pith},
  title        = {Pith review of: Guaranteed prediction sets for functional surrogate models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UXEVRWUW}},
  note         = {Machine review of arXiv:2501.18426}
}
read the original abstract

We propose a method for obtaining statistically guaranteed prediction sets for functional machine learning methods: surrogate models which map between function spaces, motivated by the need to build reliable PDE emulators. The method constructs nested prediction sets on a low-dimensional representation (an SVD) of the surrogate model's error, and then maps these sets to the prediction space using set-propagation techniques. This results in prediction sets for functional surrogate models with conformal prediction coverage guarantees. We use zonotopes as basis of the set construction, which allow an exact linear propagation and are closed under Cartesian products, making them well-suited to this high-dimensional problem. The method is model agnostic and can thus be applied to complex Sci-ML models, including Neural Operators, but also in simpler settings. We also introduce a technique to capture the truncation error of the SVD, preserving the guarantees of the method.

Figures

Figures reproduced from arXiv: 2501.18426 by the authors.

Figure 1
Figure 1. Example zonotopes in blue, and their associated [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Examples of nested zonotopes families, with [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. One may interpret s(α) as the cumulative proba￾bility distribution (cdf) of α, which when sampled yields a random zonotope. Z s(α) pZ marginal X1 s(α) [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Illustration of fitting a zonotope using the principal [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Illustration of fitting using the convex hull. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Effect of grid size in membership computation [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Shows three examples of calibration using confor [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Examples of constructed prediction sets of the FNO for Burger’s equation, predicted on test data set. The method [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: gives a visual description of how the nested sets Z α pZ are constructed. Given a zonotope Z and a point pZ ∈ Z, the centres of the parametric zonotopes move along the line defined by cZ (1 − α) + pZα, and additionally the generators are contracted by (1 − α). Thus, th…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Epistemic Artificial Intelligence is Essential for Machine Learning Models to Truly 'Know When They Do Not Know'

    cs.AI 2025-05 conditional novelty 3.0 of 10

    Machine learning should use second-order uncertainty measures, such as credal sets and random sets, so models can explicitly represent ignorance and avoid overconfident predictions on unfamiliar data.

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    Take two sets L and K composed as the intersection of n sets L = Tn i=1 Ai and K = Tn i=1 Bi, if each Bi ⊆ Ai, then K ⊆ L

    work page

  2. [2]

    Since a zonotope can be considered as an intersection of half-spaces (H-representation), it suffices to study the subset-hood of the individual half-spaces composing Z α1 pZ and Z α2 pZ

    work page

  3. [3]

    A zonotope Z = ⟨cZ , GZ ⟩ can be characterised as the image of the box × n 1 [−1, 1] in Rn, where n is the number generators, by an affine transformation defined by the rotation and stretching with matrix GZ and translation with vector cZ

    work page

  4. [4]

    Equivalently it can be defined as the intersection of the same affine transformation of the half-spaces defining the box × n 1 [−1, 1]

    work page

  5. [5]

    (b) b1 ≤ λb2, where λ the same as (a)

    A half-space H1 = {x ∈ Rn|a⊤ 1 x ≤ b1} is a subset of another half-space H2 = {x ∈ Rn|a⊤ 2 x ≤ b2}, H1 ⊆ H2 if: (a) a1 and a2 are positively aligned, i.e., they are parallel and have the same direction: a1 ≤ λa2 for λ ≥ 0. (b) b1 ≤ λb2, where λ the same as (a). Condition (a) is straightforward to prove: Z α1 pZ transforms any half-space normal vectors a a...

    work page 2021

  6. [8]

    Lifting: A fully local, point-wise operation that projects the input domain to a higher dimensional latent representation a ∈ Rda → ν0 ∈ Rdν0 ,

    work page

  7. [9]

    Iterative Kernel Integration: Expressed as a sum of a local linear operator, and a non-local integral kernel operator, that iterates νi → νi+1 for several layers: νi+1 = σ(W νi(x) + κ(a; ϕ)νi(x)), where W is the learnable linear components and σ is a non-linear activation (as in traditional networks). The kernel κ (with learnable parameters ϕ) characteris...

    work page

  8. [10]

    Projection Similar to lifting, but with reversed dimensions νn ∈ Rdνn → u ∈ Rdu, where n is the total number of layers. A Fourier Neural Operator is a specific class of the above, which defines κ with Fourier convolutions: (κ(a; ϕ)νi)(x) = F −1(F (Rϕ) · F(νi)), where F and F −1 are the Fourier and inverse Fourier transform, and Rϕ is a learnable complex-v...

    work page

Show all 16 references
  1. [11]

    Input layer: Rdin → R256

  2. [12]

    Hidden layers: 3 layers of R256 → R256 transformations with tanh activation

  3. [13]

    Training proceeded for 500 epochs using mini-batches of size 50 and mean squared error loss

    Output layer: R256 → Rdout The network was trained using the Adam optimizer with an initial learning rate of 5 × 10−3, which was reduced by a factor of 0.5 every 50 epochs. Training proceeded for 500 epochs using mini-batches of size 50 and mean squared error loss. Input featu...

  4. [14]

    Lifting layer: a full connected layer R2 → R64 (192 parameters)

  5. [15]

    Fourier layers: 4 Fourier layers (69696 parameters each), each consisting of: (a) a full connected later R64 → R64 (4160 parameters) (b) Fourier operator kernel R64 → R64, truncated to 16 modes (65536 parameters)

  6. [16]

    The FNO was trained for 500 epochs using the Adams optimiser with stepsize of 10−3 and a weight decay of 10−4, on an L2 loss, and is timed at around 8 minutes on an NVIDIA A100 GPU

    Projection layer: two layer fully connected network R64 → R125 → R1, with Gelu (Gaussian Error Linear Unit) activation (8449 paramaters). The FNO was trained for 500 epochs using the Adams optimiser with stepsize of 10−3 and a weight decay of 10−4, on an L2 loss, and is timed ...

  7. [7825]

    URL https://www.sciencedirect.com/ science/article/pii/S0045782522004595

    doi: https://doi.org/10.1016/j.cma.2022.115399. URL https://www.sciencedirect.com/ science/article/pii/S0045782522004595. Fang Zhou, Q Claire, and Ross D King. Predicting the geographical origin of music. In 2014 IEEE International Conference on Data Mining , pages 1115–1120. ...

    2022

  8. [8856]

    Ryan Martin and Chuanhai Liu

    URL https://openreview.net/forum? id=cGpegxy12T. Ryan Martin and Chuanhai Liu. Inferential models: reason- ing with uncertainty. CRC Press, 2015. Soundouss Messoudi, Sébastien Destercke, and Sylvain Rousseau. Copula-based conformal prediction for multi- target regression. Patt...

    2015

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.