The reviewed record of science sign in
Pith

arxiv: 2606.07771 · v1 · pith:WBSTQWGZ · submitted 2026-06-05 · astro-ph.IM · astro-ph.GA· cs.AI

Beyond Point Estimates: Benchmarking Uncertainty Quantification Methods on the AION-1 Astronomical Foundation Model

Karla Tame-Narvaez , Aleksandra \'Ciprijanovi\'c , Shubhendu Trivedi This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-27 20:39 UTCgrok-4.3pith:WBSTQWGZrecord.jsonopen to challenge →

classification astro-ph.IM astro-ph.GAcs.AI
keywords conformal predictionuncertainty quantificationgalaxy propertiesfoundation modelslocal validityregression calibrationastrophysics
0
0 comments X

The pith

Conformal prediction methods achieve reliable 90% coverage and local validity for galaxy property estimates from foundation model embeddings, while standard baselines do not.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper benchmarks seven uncertainty quantification approaches on regression tasks that predict galaxy redshift, stellar mass, age, metallicity, and star-formation rate from Legacy Survey photometry and DESI spectra, using frozen embeddings from an astronomical foundation model. Distribution-free conformal techniques reach marginal coverage within about one percentage point of the nominal 90% target across all five properties. Non-conformal methods such as deep ensembles and Monte Carlo dropout do not calibrate reliably. Only the Locally Valid and Discriminative framework, especially when run on the foundation-model embeddings, also supplies finite-sample local validity that adapts interval width to each galaxy's individual prediction difficulty rather than relying solely on marginal guarantees.

Core claim

Distribution-free conformal methods achieve marginal coverage within ∼1 pp of the nominal 90% across all properties, while non-conformal baselines fail to calibrate reliably. Among conformal approaches, Conformalized Quantile Regression delivers the best coverage in the bin with the poorest model predictions. Only the Locally Valid and Discriminative framework—particularly when operating on the foundation-model embeddings—also provides finite-sample local validity, producing intervals that adapt to each galaxy's local prediction difficulty.

What carries the argument

The Locally Valid and Discriminative (LVD) framework, which supplies finite-sample local validity guarantees when applied to foundation-model embeddings for regression tasks.

If this is right

  • Conformalized Quantile Regression yields the tightest reliable intervals in the regions where point predictions are weakest.
  • Locally Valid and Discriminative intervals adapt their width to each galaxy's individual prediction difficulty rather than using a single marginal width.
  • Conformal prediction becomes the preferred uncertainty framework for downstream inference that uses foundation-model embeddings in astrophysics.
  • Local validity guarantees remain available even when the underlying point predictor is a frozen foundation model.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same conformal workflow could be tested on embeddings from other astronomical foundation models to check whether local validity transfers.
  • Local validity might reduce systematic errors in downstream analyses that combine many galaxy property estimates, such as population studies.
  • If the method is applied to new photometric surveys, the coverage guarantees would still hold provided the exchangeability assumption between calibration and test sets remains reasonable.

Load-bearing premise

The evaluation treats the derived labels as accurate ground truth and assumes the chosen data splits allow the conformal coverage and local validity guarantees to hold.

What would settle it

A new collection of galaxies drawn from the same distribution where the conformal prediction intervals cover the true property values at a rate materially below the nominal 90% would falsify the coverage claim.

Figures

Figures reproduced from arXiv: 2606.07771 by Aleksandra \'Ciprijanovi\'c, Karla Tame-Narvaez, Shubhendu Trivedi.

Figure 1
Figure 1. Figure 1: Predicted vs. True log Z with 90% prediction intervals for CQR (left) and LCNet (right). Both achieve similar marginal coverage (∼ 90%), but CQR produces more uniform intervals, while LCNet adapts interval widths to local prediction difficulty. A small number (10 cases) of LCNet intervals are unbounded (which is actually a desirable behaviour, as it can be used to discover outliers and anomalies). 5 Result… view at source ↗
read the original abstract

Foundation models for astronomical surveys offer powerful learned representations that can be transferred to downstream regression tasks such as galaxy property estimation. However, point predictions alone are insufficient for scientific inference; reliable uncertainty quantification (UQ) is essential. We compare seven UQ methods on galaxy property regression using frozen AION-1 foundation-model embeddings, predicting redshift, stellar mass, stellar-population age, gas-phase metallicity, and specific star-formation rate, from Legacy Survey photometry/imaging and DESI spectra, with PROVABGS-derived labels. Distribution-free conformal methods achieve marginal coverage within $\sim$1\,pp of the nominal 90\% across all properties, while non-conformal baselines (Deep Ensembles, MC~Dropout) fail to calibrate reliably. Among conformal approaches, Conformalized Quantile Regression (CQR) delivers the best coverage in the bin with the poorest model predictions. More importantly, only the Locally Valid and Discriminative (LVD) framework -- particularly when operating on AION-1 embeddings -- also provides finite-sample \emph{local validity}, producing intervals that adapt to each galaxy's local prediction difficulty rather than relying on marginal guarantees alone. These results establish conformal prediction, and LVD in particular, as the preferred UQ framework for uncertainty-aware inference on foundation-model embeddings in astrophysics.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript benchmarks seven uncertainty quantification methods for regressing galaxy properties (redshift, stellar mass, age, metallicity, sSFR) from Legacy Survey photometry/DESI spectra using frozen AION-1 foundation-model embeddings and PROVABGS-derived labels. It claims that distribution-free conformal methods achieve marginal coverage within ~1 pp of the nominal 90% level across properties, that non-conformal baselines (Deep Ensembles, MC Dropout) fail to calibrate, that CQR performs best in poorly predicted bins, and that only the Locally Valid and Discriminative (LVD) framework—especially on AION-1 embeddings—delivers finite-sample local validity in addition to marginal coverage.

Significance. If the central empirical claims hold after addressing label noise, the work would be significant for astro-ph.IM: it supplies a concrete, multi-property comparison of UQ methods on foundation-model embeddings and identifies LVD as providing both marginal and local validity guarantees. The reproducible experimental setup on public survey data and the explicit contrast between marginal and local validity constitute strengths that could guide adoption of conformal methods for uncertainty-aware downstream inference.

major comments (2)
  1. [Abstract and Results] The evaluation computes all coverage and local-validity statistics against PROVABGS-derived labels treated as exact ground truth (Abstract and Results). These labels are themselves posterior summaries from an SED-fitting pipeline subject to modeling assumptions, parameter degeneracies, and noise; no propagation of PROVABGS uncertainties or sensitivity analysis on label noise is reported. Because conformal guarantees are with respect to the observed label distribution, this directly affects whether the reported intervals can be interpreted as reliable for the underlying physical quantities, which is load-bearing for the claim that LVD provides “uncertainty-aware inference” suitable for scientific use.
  2. [Abstract and Methods] The abstract states that LVD “provides finite-sample local validity” when operating on AION-1 embeddings, yet the manuscript supplies no explicit definition of the local-validity metric, the binning or conditioning procedure used to verify it, or the precise implementation details that distinguish it from standard conformal methods. Without these, it is impossible to confirm that the reported local-validity advantage is not an artifact of the chosen evaluation protocol or data splits.
minor comments (2)
  1. [Abstract] The abstract refers to “seven UQ methods” and “the bin with the poorest model predictions” without naming the methods or defining the binning criterion; these should be stated explicitly in the opening paragraph for clarity.
  2. [Methods] Notation for the seven methods (Deep Ensembles, MC Dropout, CQR, LVD, etc.) should be introduced consistently in a table or methods subsection rather than only in the abstract.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which help clarify the interpretation of our results and the presentation of the LVD method. We address each point below and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: [Abstract and Results] The evaluation computes all coverage and local-validity statistics against PROVABGS-derived labels treated as exact ground truth (Abstract and Results). These labels are themselves posterior summaries from an SED-fitting pipeline subject to modeling assumptions, parameter degeneracies, and noise; no propagation of PROVABGS uncertainties or sensitivity analysis on label noise is reported. Because conformal guarantees are with respect to the observed label distribution, this directly affects whether the reported intervals can be interpreted as reliable for the underlying physical quantities, which is load-bearing for the claim that LVD provides “uncertainty-aware inference” suitable for scientific use.

    Authors: We agree that PROVABGS labels are subject to SED-fitting uncertainties and that conformal coverage is formally with respect to the observed label distribution. In the revised manuscript we will add a sensitivity analysis that perturbs the labels by draws from their reported posterior uncertainties, recomputes coverage and local-validity statistics, and discusses the distinction between coverage w.r.t. the observed labels versus the underlying physical quantities. This will be presented in a new subsection of Results. revision: yes

  2. Referee: [Abstract and Methods] The abstract states that LVD “provides finite-sample local validity” when operating on AION-1 embeddings, yet the manuscript supplies no explicit definition of the local-validity metric, the binning or conditioning procedure used to verify it, or the precise implementation details that distinguish it from standard conformal methods. Without these, it is impossible to confirm that the reported local-validity advantage is not an artifact of the chosen evaluation protocol or data splits.

    Authors: We will expand the Methods section with an explicit definition of the local-validity metric (empirical coverage conditioned on local difficulty), the binning/conditioning procedure (quantiles of absolute residual or embedding-nearest-neighbor distance), and the precise algorithmic differences between LVD and standard conformal methods (including pseudocode). These additions will make the local-validity claims fully reproducible and allow direct verification against the evaluation protocol. revision: yes

Circularity Check

0 steps flagged

No circularity; empirical benchmark with external labels

full rationale

The paper reports empirical coverage and local-validity results for conformal UQ methods (CQR, LVD) versus baselines on AION-1 embeddings for five galaxy properties. All quantities are measured on held-out galaxies against PROVABGS-derived labels; no equations, fitted parameters, or predictions are defined in terms of themselves. No self-citation chain, ansatz smuggling, or renaming of known results is load-bearing for the central claims. The evaluation is a standard benchmark study whose conclusions are falsifiable against the chosen labels and splits.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

Abstract-only text supplies no explicit free parameters, axioms, or invented entities; the central claim implicitly rests on the domain assumption that the chosen labels and data splits are representative and that the foundation-model embeddings are fixed inputs.

axioms (2)
  • domain assumption PROVABGS-derived labels serve as accurate ground truth for the five galaxy properties.
    The paper evaluates all UQ methods against these labels.
  • domain assumption The Legacy Survey photometry, DESI spectra, and associated galaxies form a representative test distribution for conformal coverage guarantees.
    Conformal methods require exchangeability or similar assumptions on the data.

pith-pipeline@v0.9.1-grok · 5790 in / 1438 out tokens · 22015 ms · 2026-06-27T20:39:30.178742+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 3 canonical work pages · 1 internal anchor

  1. [1]

    The Hyper Suprime-Cam SSP survey: Overview and survey design

    Hiroaki Aihara, Nobuo Arimoto, Robert Armstrong, et al. The Hyper Suprime-Cam SSP survey: Overview and survey design. Publications of the Astronomical Society of Japan, 70(SP1):S4, 2018

    2018

  2. [2]

    Prediction of Star Formation Rates Using an Artificial Neural Network

    Ashraf Ayubinia, Jong-Hak Woo, Fatemeh Hafezianzadeh, Taehwan Kim, and Changseok Kim. Prediction of Star Formation Rates Using an Artificial Neural Network. ApJ, 980(2):177, February 2025

    2025

  3. [3]

    Constructing normalized nonconformity measures based on maximizing predictive efficiency

    Anthony Bellotti. Constructing normalized nonconformity measures based on maximizing predictive efficiency. In Alexander Gammerman, Vladimir V ovk, Zhiyuan Luo, Evgueni Smirnov, and Giovanni Cherubin, editors, Proceedings of the Ninth Symposium on Conformal and Probabilistic Prediction and Applications, volume 128 of Proceedings of Machine Learning Resear...

    2020

  4. [4]

    The DESI Experiment Part I: Science,Targeting, and Survey Design

    DESI Collaboration, Amir Aghamousa, Jessica Aguilar, Steve Ahlen, Shadab Alam, Lori E. Allen, Carlos Allende Prieto, James Annis, Stephen Bailey, et al. The DESI Experiment Part I: Science,Targeting, and Survey Design. arXiv e-prints, page arXiv:1611.00036, October 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  5. [5]

    Schlegel, Dustin Lang, Robert Blum, Kaylan Burleigh, Xiaohui Fan, Joseph R

    Arjun Dey, David J. Schlegel, Dustin Lang, Robert Blum, Kaylan Burleigh, Xiaohui Fan, Joseph R. Findlay, Doug Finkbeiner, et al. Overview of the DESI Legacy Imaging Surveys. AJ, 157(5):168, May 2019

    2019

  6. [6]

    Simul- taneous derivation of galaxy physical properties with multimodal deep learning

    Mario Gai, Mario Bove, Giovanni Bonetta, Davide Zago, and Rossella Cancelliere. Simul- taneous derivation of galaxy physical properties with multimodal deep learning. MNRAS, 532(2):1391–1401, August 2024

    2024

  7. [7]

    Prusti, J

    Gaia Collaboration, T. Prusti, J. H. J. de Bruijne, et al. The Gaia mission. Astronomy & Astrophysics, 595:A1, 2016

    2016

  8. [8]

    Dropout as a Bayesian approximation: Representing model uncertainty in deep learning

    Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning (ICML), 2016

    2016

  9. [9]

    Simple and scalable predictive uncertainty estimation using deep ensembles

    Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems 30 (NeurIPS), 2017

    2017

  10. [10]

    Tibshirani, and Larry A

    Jing Lei, Max Grazier G’Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry A. Wasser- man. Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113:1094 – 1111, 2016

    2016

  11. [11]

    Estimation of age and metallicity for galaxies based on multi-modal deep learning

    Ping Li, Li-Li Wang, Guang-Jun Yang, Jia-Bao Feng, and Yan-Ke Tang. Estimation of age and metallicity for galaxies based on multi-modal deep learning. A&A, 698:A222, June 2025

    2025

  12. [12]

    Locally valid and discriminative prediction intervals for deep learning models

    Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Locally valid and discriminative prediction intervals for deep learning models. In Neural Information Processing Systems, 2021

    2021

  13. [13]

    AstroCLIP: a cross-modal foundation model for galaxies

    Liam Parker, Francois Lanusse, Siavash Golkar, Leopoldo Sarra, Miles Cranmer, Alberto Bietti, Michael Eickenberg, Geraud Krawezik, Michael McCabe, Rudy Morel, Ruben Ohana, Mariel Pettee, Bruno Régaldo-Saint Blancard, Kyunghyun Cho, Shirley Ho, and Polymathic AI Collaboration. AstroCLIP: a cross-modal foundation model for galaxies. MNRAS, 531(4):4990– 5011...

    2024

  14. [14]

    https://arxiv.org/abs/2510.17960

    Liam Parker, Francois Lanusse, Jeff Shen, Ollie Liu, Tom Hehir, Leopoldo Sarra, Lucas Meyer, Micah Bowles, et al. AION-1: Omnimodal Foundation Model for Astronomical Sciences. arXiv e-prints, page arXiv:2510.17960, October 2025

    work page arXiv 2025

  15. [15]

    Yaniv Romano, Evan Patterson, and Emmanuel J. Candès. Conformalized quantile regression. In Advances in Neural Information Processing Systems 32 (NeurIPS), 2019

    2019

  16. [16]

    A Deep Multimodal Multi–Head Neural Network for Joint Estimation of Stellar Age, Lifetime, and Evolutionary Stage

    Jing Rou Puah and Sasa Arsovski. A Deep Multimodal Multi–Head Neural Network for Joint Estimation of Stellar Age, Lifetime, and Evolutionary Stage. arXiv e-prints , page arXiv:2511.18477, November 2025. 6

    work page arXiv 2025

  17. [17]

    Algorithmic Learning in a Random World

    Vladimir V ovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, 2005

    2005

  18. [18]

    Donald G. York, J. Adelman, John E. Anderson, et al. The Sloan Digital Sky Survey: Technical summary. The Astronomical Journal, 120(3):1579–1587, 2000. 7

    2000