Pith. sign in

REVIEW 4 major objections 7 minor 9 references

NeurIPS 2024 Ariel Data Challenge: Characterisation of Exoplanetary Atmospheres Using a Data-Centric Approach

T0 review · 4 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that calibrated, data-dependent uncertainty estimation—not a larger or more complex model—was the decisive factor in improving the Gaussian Log-Likelihood (GLL) score on the NeurIPS 2024 Ariel Data Challenge.

desk verdict A candid competition report whose practical lesson about uncertainty is plausible but not cleanly demonstrated once you unbundle the iteration steps. read the letter →

arxiv 2505.08940 v1 pith:4E7N3K3E submitted 2025-05-13 cs.LG astro-ph.IM

Jeremie Blanchard , Lisa Casino , Jordan Gierschendorf This is my paper · ORCID
classification cs.LGastro-ph.IM
keywords exoplanetatmospheresArielmissionGaussianlog-likelihoodheteroskedasticuncertaintybaggingdata-centricAItransitspectroscopyout-of-distributiongeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This competition report argues that on the NeurIPS 2024 Ariel Data Challenge, the decisive lever was not model capacity but calibrated uncertainty. The authors show that adding heteroskedastic error estimates, derived from the spread of 50 bagged Ridge regressions, raised their private Gaussian Log-Likelihood score from 45.12% to 49.93%, while the baseline sat at 38.44%. They frame the result as evidence for a data-centric philosophy: restrained feature sets, simple linear models, and honest error bars generalize better to out-of-distribution stars than rich feature engineering or complex architectures. The paper also records where that philosophy hit its limits: extra features and geometric corrections sometimes hurt test performance, and accurate wavelength prediction, not uncertainty, remains the ceiling.

What carries the argument

The load-bearing mechanism is heteroskedastic uncertainty estimation produced by bagging: 50 Ridge kernel regressors, each trained on 60–80% of the data sampled with replacement, with the standard deviation of their per-wavelength predictions used as σ_user. The GLL score couples accuracy and uncertainty, so calibrating σ_user against the actual prediction error directly improves the score; the authors show that the final iteration's σ values track model disagreement rather than being fixed.

What would settle it

Take any submission's predicted σ_user, replace it with a single optimally tuned per-wavelength uncertainty shared across all submissions, and recompute GLL: if the leaderboard order stays nearly unchanged, then the heteroskedastic uncertainty term was not what drove the score, and the paper's central claim is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that the competition's Gaussian Log-Likelihood (GLL) score is far more sensitive to the predicted uncertainty σ_user than to the accuracy gains from additional features. The final pipeline, a bagged Ridge regression with heteroskedastic σ estimated from prediction spread, reached a private GLL of 49.93%, versus 45.12% without the heteroskedastic treatment and 38.44% for the baseline. The authors attribute this jump to calibrated uncertainty rather than architectural complexity, and they note that even a perfect uncertainty estimate would only bring the cross-validation GLL to about 80%, leaving mean-spectrum prediction accuracy as the next bottleneck.

Load-bearing premise

The load-bearing premise is that the competition's simulated spectra and GLL scoring metric faithfully represent the noise behaviour and scientific goals of the real Ariel mission, so that lessons about uncertainty and feature restraint transfer to operational retrieval.

Editorial extensions

If this is right

  • On this benchmark, teams that invest in calibrated uncertainty estimation should outscore teams that only improve mean predictions, because the GLL metric penalizes overconfidence directly.
  • Restrained feature sets are safer for out-of-distribution targets: the jump from 10 to 158 features coincided with a private-score drop from 38.44% to 32.73%.
  • Simple bagged linear models with polynomial kernels provide a strong, interpretable baseline for transit-spectroscopy retrieval tasks, particularly when the test set contains unseen stellar types.
  • Geometric detrending of AIRS signals can remove real astrophysical variation; the authors observed a slight test-set degradation and suggest such corrections be validated on held-out stars before deployment.
  • Even with ideal uncertainty estimation, this pipeline's validation GLL would cap near 80%, so improving the accuracy of the mean spectrum remains the next bottleneck.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, a post-hoc recalibration test—rescaling every team's σ_user to the empirical per-wavelength error—would show how much of the private leaderboard is determined by uncertainty tuning rather than retrieval accuracy.
  • If the Ariel simulator's noise model is realistic, operational retrieval software should treat uncertainty as a first-class deliverable on par with the spectrum itself, since downstream atmospheric characterisation inherits both.
  • The overfitting pattern observed from 10 to 288 features suggests a testable feature-selection rule for transit spectroscopy: choose features that improve cross-validated GLL on held-out stars, not in-distribution R².
  • A natural extension is to keep the heteroskedastic bagging mechanism but replace the hand-built tabular features with a small network trained on raw image sequences; if the same uncertainty gain appears, the claim would extend beyond tabular modelling.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper reports the authors' participation in the NeurIPS 2024 Ariel Data Challenge, a competition to recover exoplanetary transmission spectra and uncertainty estimates from simulated Ariel mission observations. The authors describe a data-centric pipeline based on feature engineering from binned spectral signals, a Ridge kernel regression model with bagging, and heteroskedastic uncertainty quantification computed as the standard deviation of ensemble predictions. Their central claim is that uncertainty estimation plays a decisive role in the Gaussian Log-Likelihood (GLL) score, contributing gains of several percentage points, with a final private GLL score of 49.93% versus a baseline of 38.44%. The paper also reports that more complex features and a geometric correction hurt generalization, and it discusses limitations of tabular modeling and the gap to perfect uncertainty calibration.

Significance. If the central claim were cleanly established, the paper would offer a useful practical lesson for exoplanet retrieval challenges: calibrated heteroskedastic uncertainty, obtained from ensemble disagreement, can matter more than model complexity or additional features. The manuscript has strengths in its honest reporting of negative results, including the failure of geometric correction and the overfitting caused by feature proliferation, and in its explicit discussion of the GLL metric's incentives. The paper also states a self-assessed ceiling on the approach (80% cross-validation GLL even with perfect uncertainty), which is informative although not fully derived. The significance is, however, mostly retrospective and methodological rather than novel in machine learning, and it depends on whether the reported iteration history can support the causal attribution to uncertainty estimation.

major comments (4)
  1. [Abstract and Section 4, Table 1] The central claim that 'uncertainty estimation plays a crucial role in the Gaussian Log-Likelihood (GLL) score, impacting performance by several percentage points' is not supported by the evidence in Table 1, because the iteration history changes multiple components at once. From iteration 5 to 6, the private GLL improves from 39.69% to 45.12% while adding ensemble-based uncertainty quantification but also removing the geometric signal correction and leaving the feature count and bin count unchanged; from iteration 6 to 7, the improvement to 49.93% coincides with changing the number of spectral bins from 8 to 10 and 'fine-tuning sigma'. Since Table 1b lists contributions but is not an ablation, the observed gains cannot be uniquely attributed to the uncertainty estimator. The authors should provide a controlled comparison in which only the sigma estimation changes, keeping features, preprocessing, binning, and fold splits identical, or should explicitly reframe the claim as a joint effect of several simultaneous changes.
  2. [Section 3 and Figure 2] The paper states that feedback from Kaggle's public leaderboard was used to iteratively refine the pipeline, as shown by the yellow arrow in Figure 2. This makes the iteration trajectory in Table 1 a process of model selection on the public test set rather than a pre-specified comparison of hypotheses. In particular, hyperparameters such as the number of bins, the feature count, and the Ridge kernel settings in Section 5.1.6 were refined using public-leaderboard feedback, so the reported cross-validation and private scores are conditional on this selection. The authors should state how many decisions were made using the public leaderboard and whether any independent validation split was used to choose the final iteration; without this information, the magnitude of the reported gains may be inflated by selection effects.
  3. [Section 5.1.6 and Figure 6] Figure 6 is presented as a comparison between fixed uncertainty in iteration 5 and heteroskedastic uncertainty in iteration 7, with the caption implying that the better calibration in 6b is due solely to the sigma estimation scheme. However, iterations 5 and 7 also differ in the number of spectral bins, the geometric signal correction, and potentially other pipeline settings, so the figure does not isolate the effect of uncertainty estimation. This figure should either be replaced with a matched pair of settings that differ only in the sigma estimator, or be explicitly labeled as an illustrative contrast between two full pipeline variants rather than as evidence for the paper's causal claim.
  4. [Section 6] The statement that 'even if our final approach, with a cross-validation GLL score of 66%, had achieved perfect uncertainty estimation (i.e., uncertainty matching the error), the cross-validation GLL score would have only reached 80%' is used to conclude that mean prediction accuracy remains the main bottleneck. This 80% ceiling should be derived explicitly: if σ_user is set to |y − μ_user|, the GLL per observation becomes −0.5(log(2π)+1), and the resulting score depends on L_ref and L_ideal from Eq. (2). Please report the calculation or formula used, since this number is load-bearing for the paper's central limitation argument and is not reproducible from the text as written.
minor comments (7)
  1. [Abstract] The phrase 'improving the GLL score by 11%' is ambiguous and appears inconsistent with Table 1a: the private GLL rises from 38.44% to 49.93%, which is an increase of 11.49 percentage points but approximately 29.9% relative to the baseline. Please state explicitly whether the 11% refers to percentage points, relative improvement, or some other quantity.
  2. [Section 4] The sentence 'achieving a private GLL score of 50%, which was the highest score among all submissions' is misleading because the paper states earlier that the team did not rank among the top-performing models in the competition; the intended meaning is presumably the highest score among the authors' submitted iterations. Please rephrase.
  3. [Section 5.1.6 and Figure 5 caption] The Figure 5 caption refers to the 'Gaussian Log-Likelihood (GLL) score defined in the equation (2)', but the GLL is defined in Eq. (1) while Eq. (2) gives the score conversion. Please correct the cross-reference.
  4. [Table 1a] The row 'Mean uncertainty (σ_user)' should specify whether the average is taken over all wavelengths, all samples, and over the cross-validation folds or the public/private test sets, and whether it is the mean of the predicted σ across test instances; this affects interpretation of the relationship between uncertainty and GLL.
  5. [Section 5.1.6] The paper reports final hyperparameters for the Ridge kernel but does not describe how α, γ, polynomial degree, sampling fraction, or ensemble size were chosen, or whether these choices were made using cross-validation or public-leaderboard feedback. A brief tuning description would improve reproducibility.
  6. [Throughout] There are numerous typographical errors and awkward phrasings, including 'data-centric business approach' in the abstract, 'and and' in Section 5, 'the model ability to generalize' in Section 5.1.3, and 'resulting to a lower overall score' in Section 5.1.6. A careful copyedit is needed.
  7. [Section 5.1.1] The sentence 'This section is prior work done by the competition Authors and was used globally by all the competitors' should be attributed more precisely, for example by stating that the calibration procedures follow the competition's shared preprocessing notebook, with a proper citation to that notebook.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; the uncertainty conclusion is an empirical attribution, not a derivation from its own inputs.

full rationale

The paper's central claim is that heteroskedastic uncertainty estimation improves the Gaussian Log-Likelihood score. This is an empirical attribution based on the iteration table in Section 4, not a derivation whose conclusion is already embedded in its inputs. The uncertainty estimates are computed as the standard deviation of 50 bagged model predictions (Section 5.1.6), and the GLL score is the competition's externally defined metric (Equations 1 and 2). No parameter is fitted to the private test set and then renamed as a prediction, and no self-citation is invoked to justify the main result. The use of public leaderboard feedback to refine the pipeline is iterative model selection, which weakens the out-of-sample interpretation of the reported scores but is not definitional circularity. The comparison between iterations is confounded because multiple components change at once (e.g., bins, geometric correction, and sigma estimation all vary between iterations 5 and 7), but confounding is an experimental validity concern, not circularity. No step in the paper reduces by construction to its own inputs, so no circularity step meets the quoted-reduction test.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper's central claim rests on several hand-picked preprocessing and modeling choices, plus the assumption that the challenge benchmark proxies real retrieval. No new physical or model entities are introduced. The sigma prediction is an ensemble statistic, not a newly postulated quantity.

free parameters (5)
  • Ridge kernel hyperparameters (alpha, gamma, polynomial degree) = alpha=0.0001, gamma=0.0001, degree=3
    Reported in Section 5.1.6 as the final model parameters; selected through experiments with no sensitivity analysis, so performance depends on these choices.
  • Number of spectral bins n = 5, 8, or 10 across iterations
    Chosen per experiment in Section 5.1.2; binning granularity directly changes the feature representation and the final score changed when moving from 8 to 10 bins.
  • Number of engineered features = 10, 42, 158, or 288 across iterations
    Feature count was varied in Section 4 and identified as a cause of overfitting; the final 288-feature set is one specific hand-built configuration.
  • Transit segmentation breakpoints = not numerically stated; determined per signal
    Sections 5.1.3 and 5.1.5 rely on identifying ingress and egress breakpoints to segment the signal into flat and transition zones; this choice affects all extracted features.
  • Bagging sampling fraction and ensemble size = 80% sampling, 50 models
    Final configuration in Section 5.1.6; the uncertainty estimate is the standard deviation across these 50 models, so ensemble size and sampling fraction shape the predicted sigma.
assumptions (5)
  • domain assumption The competition's simulated data and GLL metric faithfully represent the real Ariel spectral retrieval problem.
    The paper generalizes conclusions to future missions and real-world systems in Sections 1 and 6, but all data are synthetic and the metric is challenge-defined.
  • domain assumption The atmospheric transmission spectrum is recoverable from hand-engineered tabular features of binned, image-derived time series.
    The entire pipeline in Section 3 depends on this; the paper itself concludes that tabular modeling has inherent limitations for this task.
  • domain assumption The standard calibration and cleaning notebook from the competition authors is correct and introduces no systematic error.
    Section 5.1.1 states this is prior work used by all competitors; the paper adopts it without re-derivation or independent validation.
  • domain assumption Linear models extrapolate to out-of-distribution data better than tree-based models.
    Section 6 cites reference [8] for this principle and uses it to justify Ridge regression over tree-based models, but the principle is not validated on this dataset.
  • domain assumption Bootstrap ensemble disagreement is a valid estimate of predictive uncertainty.
    Section 5.1.6 defines sigma as the standard deviation across 50 bagged predictions; there is no calibration against true error covariance or comparison with other uncertainty methods.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NeurIPS 2024 Ariel Data Challenge: Characterisation of Exoplanetary Atmospheres Using a Data-Centric Approach." pith.science (2026). https://pith.science/paper/4E7N3K3E

@misc{pith2026250508940,
  author       = {Pith},
  title        = {Pith review of: NeurIPS 2024 Ariel Data Challenge: Characterisation of Exoplanetary Atmospheres Using a Data-Centric Approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4E7N3K3E}},
  note         = {Machine review of arXiv:2505.08940}
}
read the original abstract

The characterization of exoplanetary atmospheres through spectral analysis is a complex challenge. The NeurIPS 2024 Ariel Data Challenge, in collaboration with the European Space Agency's (ESA) Ariel mission, provided an opportunity to explore machine learning techniques for extracting atmospheric compositions from simulated spectral data. In this work, we focus on a data-centric business approach, prioritizing generalization over competition-specific optimization. We briefly outline multiple experimental axes, including feature extraction, signal transformation, and heteroskedastic uncertainty modeling. Our experiments demonstrate that uncertainty estimation plays a crucial role in the Gaussian Log-Likelihood (GLL) score, impacting performance by several percentage points. Despite improving the GLL score by 11%, our results highlight the inherent limitations of tabular modeling and feature engineering for this task, as well as the constraints of a business-driven approach within a Kaggle-style competition framework. Our findings emphasize the trade-offs between model simplicity, interpretability, and generalization in astrophysical data analysis.

Figures

Figures reproduced from arXiv: 2505.08940 by the authors.

Figure 1
Figure 1. Illustration of the transformation applied to the raw images signal, from image domain to time domain, in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed methodology. The workflow consists of a local data-centric pipeline including [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Mean Representation and Binning of AIRS signal [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Illustration of the signal correction process. The green line represents the raw signal obtained from the AIRS [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Variation of the Gaussian Log-Likelihood (GLL) score defined in the equation (2) with respect to the [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Comparison of ground truth (y) and predictions (µuser, σuser) with fixed and heteroskedastic (i.e., data￾dependent) uncertainty approaches. The fixed σuser in the approach 6a can lead to overly large uncertainties (bottom plot) or overly small uncertainties (top plot),…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

9 extracted references · 6 canonical work pages

  1. [1]

    Mugnai, Rebecca L

    Kai Hou Yip, Lorenzo V . Mugnai, Rebecca L. Coates, Andrea Bocchieri, Andreas Papageorgiou, Orphée Faucoz, Tara Tahseen, Virginie Batista, Angèle Syty, Arun Nambiyath Govindan, Sohier Dane, Maggie Demkin, Enzo Pascale, Jean-Philippe Beaulieu, Quentin Changeat, Pierre Drossart, Billy Edwards, Paul Eccleston, Clare Jenner, Ryan King, Theresa Lueftinger, Nik...

    work page 2024

  2. [2]

    Díaz, Rachel Drummond, Mauro Focardi, Jose M

    Giovanna Tinetti, Paul Eccleston, Carole Haswell, Pierre-Olivier Lagage, Jérémy Leconte, Theresa Lüftinger, Giusi Micela, Michel Min, Göran Pilbratt, Ludovic Puig, Mark Swain, Leonardo Testi, Diego Turrini, Bart Vandenbussche, Maria Rosa Zapatero Osorio, Anna Aret, Jean-Philippe Beaulieu, Lars Buchhave, Martin Ferus, Matt Griffin, Manuel Guedel, Paul Hart...

    work page

  3. [3]

    Kirkpatrick, D

    Mark Mazumder, Colby Banbury, Xiaozhe Yao, Bojan Karlaš, William Gaviria Rojas, Sudnya Diamos, Greg Diamos, Lynn He, Alicia Parrish, Hannah Rose Kirk, Jessica Quaye, Charvi Rastogi, Douwe Kiela, David Jurado, David Kanter, Rafael Mosquera, Will Cukierski, Juan Ciro, Lora Aroyo, Bilge Acun, Lingjiao Chen, Mehul Raje, Max Bartolo, Evan Sabri Eyuboglu, Amira...

    work page 2023

  4. [4]

    Data-centric artificial intelligence, 2024

    Johannes Jakubik, Michael Vössing, Niklas Kühl, Jannis Walk, and Gerhard Satzger. Data-centric artificial intelligence, 2024. URL https://arxiv.org/abs/2212.11854

    arXiv 2024

  5. [5]

    Andrew ng, ai minimalist: The machine-learning pioneer says small is the new big

    Eliza Strickland. Andrew ng, ai minimalist: The machine-learning pioneer says small is the new big. IEEE Spectrum, 59(4):22–50, 2022. doi: 10.1109/MSPEC.2022.9754503

    work page arXiv 2022

  6. [6]

    S. Donati. Photodetectors: Devices, Circuits, and Applications . Prentice Hall PTR, 2000. ISBN 9780130203373. URL https://books.google.ca/books?id=8OEmAQAAMAAJ

    work page 2000

  7. [7]

    Bagging predictors

    Leo Breiman. Bagging predictors. Machine Learning, 24(2):123–140, 1996

    1996

  8. [8]

    Decision trees do not generalize to new variations

    Yoshua Bengio, Olivier Delalleau, and Clarence Simard. Decision trees do not generalize to new variations. Computational Intelligence, 26(4):449–467, 2010. 12

    work page 2010

Show all 9 references
  1. [2021]

    URL https://arxiv.org/abs/2104.04824

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.