Pith. sign in

REVIEW 2 major objections 2 minor 12 references

Jointly learning feature and sample relevancies through one marginal likelihood objective enables simultaneous sparsification of both the model and the data.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Joint ARD extends sparse Bayesian learning to sparsify both features and samples simultaneously through a single marginal likelihood objective while preserving conjugacy.

T0 review reviewed 2026-06-29 challenge →

load-bearing objection The paper extends ARD to joint feature-and-sample sparsification via marginal likelihood, but the conjugacy and closed-form update claims are the part that needs explicit checking. the 2 major comments →

arxiv 2605.29908 v1 pith:2WL6ICIZ submitted 2026-05-28 stat.ML cs.LG

Joint Model and Data Sparsification via the Marginal Likelihood

Alexander Timans , Thomas M\"ollenhoff , Christian A. Naesseth , Mohammad Emtiyaz Khan , Eric Nalisnick This is my paper
classification stat.ML cs.LG
keywords sparse Bayesian learningautomatic relevance determinationmarginal likelihoodmodel sparsificationdata sparsificationrobust regressionconjugate models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes extending automatic relevance determination from features alone to both features and individual samples in linear regression. This symmetric treatment lets a single Bayesian objective prune irrelevant features and downweight contaminated or outlier samples at the same time. The extension is constructed to keep the model conjugate, so standard optimization routines still have closed-form updates. Empirical tests on regression tasks show the resulting models are both sparse and more robust to data issues than standard sparse Bayesian learning.

Core claim

By placing separate relevance parameters on each feature and each data point and optimizing them jointly via the marginal likelihood, the method achieves simultaneous model and data sparsification while preserving conjugacy and admitting closed-form updates for standard procedures.

What carries the argument

Joint ARD extension that assigns per-feature and per-sample relevance parameters inside a single marginal-likelihood objective.

Load-bearing premise

The joint relevance parameters for samples can be introduced without breaking conjugacy or losing closed-form updates.

What would settle it

A controlled regression experiment in which the joint method fails to produce both lower prediction error and higher sparsity than separate feature-only ARD plus robust weighting.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • The approach yields prediction models that are sparse in both parameters and training points.
  • It aligns with robust regression and influence-function views by automatically downweighting influential samples.
  • Standard variational or EM procedures can still be applied because conjugacy is retained.
  • The same framework supports extensions to other conjugate linear models beyond basic regression.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The symmetric treatment suggests similar joint relevancy schemes could apply to other Bayesian models that already use ARD on parameters.
  • Because sample relevancies act like per-point weights, the method may offer a Bayesian route to influence-function diagnostics without explicit leave-one-out computation.
  • If sample relevancies concentrate on a small subset, the approach effectively performs Bayesian data subset selection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper proposes jointly optimizing per-feature ARD precisions α_j and per-sample relevancies γ_i within a linear Gaussian model, so that a single marginal-likelihood objective simultaneously prunes irrelevant features and down-weights contaminated samples. It asserts that this symmetric extension of standard SBL preserves conjugacy, yields closed-form hyperparameter updates, and produces robust sparse predictors.

Significance. If the conjugacy claim holds, the work supplies a principled, single-objective Bayesian mechanism for joint model-and-data sparsification that extends ARD without requiring variational approximations or separate robust-loss stages; this would be useful for high-dimensional regression with outliers.

major comments (2)
  1. [§3] §3 (Proposed method): the central claim that introducing sample relevancies γ_i preserves conjugacy and closed-form updates for the joint marginal likelihood p(y | {α_j}, {γ_i}) is asserted in the abstract and introduction but receives no explicit derivation or update equations. The effective noise covariance after reweighting by γ_i must be shown to remain diagonal (or to admit Woodbury/determinant identities) so that the evidence remains analytically tractable; without this algebra the “preserves conjugacy” guarantee cannot be verified.
  2. [§4] §4 (Experiments): the reported gains in robustness and sparsity are presented without ablation on the individual contributions of feature versus sample pruning, and without comparison against standard robust SBL baselines that already down-weight outliers via explicit noise modeling; this weakens the claim that the joint formulation is necessary.
minor comments (2)
  1. [§3] Notation for the sample relevancies γ_i is introduced without an explicit prior form or update rule; a short derivation or pseudocode block would clarify how the γ_i enter the marginal likelihood.
  2. [Figure 1] Figure 1 caption should state the exact values of the regularization parameters used for the competing methods so that the sparsity-accuracy trade-off can be reproduced.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We address each major point below.

read point-by-point responses

  1. Referee: [§3] §3 (Proposed method): the central claim that introducing sample relevancies γ_i preserves conjugacy and closed-form updates for the joint marginal likelihood p(y | {α_j}, {γ_i}) is asserted in the abstract and introduction but receives no explicit derivation or update equations. The effective noise covariance after reweighting by γ_i must be shown to remain diagonal (or to admit Woodbury/determinant identities) so that the evidence remains analytically tractable; without this algebra the “preserves conjugacy” guarantee cannot be verified.

    Authors: We agree that an explicit derivation would strengthen the paper. The sample relevancies γ_i enter as a diagonal reweighting of the noise covariance, yielding Σ = diag(σ²/γ_i). Because this matrix remains diagonal, the standard Woodbury identity and log-determinant identities for the marginal likelihood continue to hold without modification, producing closed-form EM-style updates for both the feature precisions α_j and the sample relevancies γ_i. We will insert the full algebraic derivation and the resulting update equations into the revised §3. revision: yes

  2. Referee: [§4] §4 (Experiments): the reported gains in robustness and sparsity are presented without ablation on the individual contributions of feature versus sample pruning, and without comparison against standard robust SBL baselines that already down-weight outliers via explicit noise modeling; this weakens the claim that the joint formulation is necessary.

    Authors: We acknowledge that the current experiments do not isolate the separate contributions of feature and sample pruning nor benchmark against existing robust SBL variants. In the revision we will add (i) an ablation that disables either the α_j or the γ_i updates while keeping the other active, and (ii) direct comparisons against robust SBL baselines that introduce per-sample noise variances or explicit outlier weights. These additions will clarify whether the joint single-objective formulation is required for the observed gains. revision: yes

Circularity Check

0 steps flagged

No circularity; joint ARD extension presented as algebraic extension without reduction to inputs

full rationale

The provided abstract asserts that the joint model-and-data sparsification 'preserves conjugacy, admits closed-form updates' but supplies no equations. No derivation chain is visible that reduces a claimed prediction to a fitted parameter by construction, nor any self-citation that bears the central conjugacy claim. The method is described as optimizing a single marginal-likelihood objective over both feature and sample relevancies; this is a modeling choice, not a circular redefinition of outputs as inputs. Absent explicit equations showing e.g. the evidence p(y|α,γ) collapsing to a reparameterization of the standard SBL objective, the derivation remains self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

2 free parameters · 1 axioms · 0 invented entities

Review performed on abstract only; the ledger therefore records only the assumptions explicitly stated in the abstract.

free parameters (2)
  • feature relevancies
    Optimized via marginal likelihood as in standard ARD
  • sample relevancies
    New parameters introduced for data sparsification and optimized jointly
axioms (1)
  • domain assumption The joint ARD model preserves conjugacy
    Invoked to guarantee closed-form updates for standard optimization procedures

reviewed 2026-06-29 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Joint Model and Data Sparsification via the Marginal Likelihood." pith.science (2026). https://pith.science/paper/2WL6ICIZ

@misc{pith2026260529908,
  author       = {Pith},
  title        = {Pith review of: Joint Model and Data Sparsification via the Marginal Likelihood},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2WL6ICIZ}},
  note         = {Machine review of arXiv:2605.29908}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Sparse recovery in linear systems underpins applications from signal processing to high-dimensional regression. Sparse Bayesian Learning, grounded in the principle of automatic relevance determination (ARD), offers a practical Bayesian mechanism for feature sparsity via marginal likelihood optimization. Yet, its reliance on a homoscedastic noise model renders it sensitive to data contaminations such as outliers or misspecified noise, harming model fit and predictions. Instead, we propose jointly learning individual feature and sample relevancies, enabling simultaneous model and data sparsification via a single Bayesian objective. This symmetric pruning of model and data offers a natural extension that preserves conjugacy, admits closed-form updates for standard optimization procedures, and aligns with perspectives from robust regression and influence functions. Empirical results across diverse regression tasks affirm that a joint ARD approach consistently yields both sparse and robust prediction models.

Figures

Figures reproduced from arXiv: 2605.29908 by Alexander Timans, Christian A. Naesseth, Eric Nalisnick, Mohammad Emtiyaz Khan, Thomas M\"ollenhoff.

Figure 1
Figure 1. Figure 1: An illustration of the proposed sparsification approach for polynomial regression on the mcycle dataset (Silverman, 1985) (d = 12, n = 128). Left: Model-only sparsification identifies two prunable features (marked in red), but treats every sample as equally relevant for model fit, including six known outliers (•). Right: Joint learning of per-weight (γ) and per-sample (λ) parameters via the marginal likeli… view at source ↗
Figure 2
Figure 2. Figure 2: Left: The evolution of feature (1/γ) and sample relevance (1/λ) over the optimization trajectory for the mcycle experiment in [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: We visualize results for the synthetic setting in § 5.1 (n = 500, d = 50, avg. over 10 trials). Left: Representative data and weight recovery behaviour against key design parameters, here for ℓ2-IRLS. Recovery degrades as the signal becomes increasingly sparse, or contaminated noise resembles inlier noise (m ≤ 1). Right: Relative gains in signal recovery from heteroscedastic vs. homoscedastic modelling, co… view at source ↗
Figure 4
Figure 4. Figure 4: Kernel regression performance on Boston as a function of data contamination (n = 506, 20% test split, avg. over 20 trials, ± 1σ). The RVM corresponds to ℓ2-IRLS in an RBF kernel basis with fixed scalar lengthscale. indicates joint ARD downweighs roughly the right amount, as do robust Student-t (Geweke, 1993) and Huber (Huber, 1964) baselines. In addition, joint ARD simultaneously yields meaningful feature … view at source ↗
Figure 6
Figure 6. Figure 6: Top row: We run an additional experiment complementing § 4 with original high-leverage outliers representing giant stars contained in the StarsCYG dataset. Center row: Those four samples clearly exhibit strong leverage and are able to pull the model fit towards them unless excluded a priori. Here, we run joint ARD with standard EM updates on the single original feature. Bottom row: Motivated by the connect… view at source ↗
Figure 7
Figure 7. Figure 7: We report predictive performance (MAE), outlier recovery, ESS(y), and ESS(θ) for the neural network experiment in § 5.4 across methods. Rows (top to bottom): OLS, Ridge, MacKay, and EM (avg. over 10 trials, ± 1σ). As expected, OLS and Ridge exhibit poor outlier recovery and yield ESS(y) = 1.0 (no data sparsification), while Ridge attains feature sparsity comparable to the SBL approaches. For MacKay and EM,… view at source ↗
Figure 8
Figure 8. Figure 8: We provide additional qualitative examples complementing [PITH_FULL_IMAGE:figures/full_fig_p036_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 4 canonical work pages · 2 internal anchors

  1. [1]

    and Vandenberghe, L.Convex Optimization

    16 Boyd, S. and Vandenberghe, L.Convex Optimization. Cam- bridge University Press, 2004. 21 Bühlmann, P. and Van De Geer, S.Statistics for High- dimensional Data: Methods, Theory and Applications. Springer Science & Business Media, 2011. 1 Campbell, T. and Beronov, B. Sparse Variational Inference: Bayesian Coresets from Scratch.Advances in Neural Informat...

    2004

  2. [2]

    E., Verma, P., John, S., Solin, A., and Khan, M

    30 Chang, P. E., Verma, P., John, S., Solin, A., and Khan, M. E. Memory-based Dual Gaussian Processes for Sequential Learning.International Conference on Machine Learn- ing, 2023. 9 Chartrand, R. and Yin, W. Iteratively Reweighted Algo- rithms for Compressive Sensing.IEEE International Conference on Acoustics, Speech and Signal Processing,

    2023

  3. [3]

    and Hadi, A

    3 Chatterjee, S. and Hadi, A. S. Influential Observations, High Leverage Points, and Outliers in Linear Regression. Statistical Science, 1986. 5 Chen, S. S., Donoho, D. L., and Saunders, M. A. Atomic Decomposition by Basis Pursuit.SIAM Review, 2001. 1, 3 Cook, R. D. and Weisberg, S. Characterizations of an Em- pirical Influence Function for Detecting Infl...

    1986

  4. [4]

    Shaving Weights with Occam’s Razor: Bayesian Sparsification for Neural Networks using the Marginal Likelihood.Advances in Neural Information Processing Systems, 2024

    3 Dhahri, R., Immer, A., Charpentier, B., Günnemann, S., and Fortuin, V . Shaving Weights with Occam’s Razor: Bayesian Sparsification for Neural Networks using the Marginal Likelihood.Advances in Neural Information Processing Systems, 2024. 16 Dua, D., Graff, C., et al. UCI Machine Learning Repository,

    2024

  5. [5]

    7 Dutordoir, V ., Hensman, J., van der Wilk, M., Ek, C

    URLhttps://archive.ics.uci.edu. 7 Dutordoir, V ., Hensman, J., van der Wilk, M., Ek, C. H., Ghahramani, Z., and Durrande, N. Deep Neural Networks 10 Joint Model and Data Sparsification via the Marginal Likelihood as Point Estimates for Deep Gaussian Processes.Ad- vances in Neural Information Processing Systems, 2021. 9 Faul, A. and Tipping, M. Analysis of...

    2021

  6. [6]

    Automatic Relevance Determination For Deep Generative Models

    7 Grünwald, P., van Ommen, T., et al. Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing It.Bayesian Analysis, 2017. 30 Hampel, F. R., Ronchetti, E. M., Rousseeuw, P. J., and Stahel, W. A.Robust Statistics: The Approach Based on Influence Functions. John Wiley & Sons, 2011. 3 Harrison Jr, D. and Rubinfeld, D. ...

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  7. [7]

    3 Khan, M. E. Knowledge Adaptation as Posterior Correction. arXiv Preprint (arXiv:2506.14262), 2025. 23, 25 Khan, M. E. and Rue, H. The Bayesian Learning Rule. Journal of Machine Learning Research, 2023. 9, 16, 23, 24, 25 11 Joint Model and Data Sparsification via the Marginal Likelihood Khan, M. E. E., Immer, A., Abedi, E., and Korzepa, M. Ap- proximate ...

    work page arXiv 2025

  8. [8]

    3 Lotfi, S., Izmailov, P., Benton, G., Goldblum, M., and Wil- son, A. G. Bayesian Model Selection, the Marginal Like- lihood, and Generalization.International Conference on Machine Learning, 2022. 9 Louizos, C., Ullrich, K., and Welling, M. Bayesian Com- pression for Deep Learning.Advances in Neural Infor- mation Processing Systems, 2017. 16 Lugosi, G. an...

    2022

  9. [9]

    1, 5 MacKay, D. J. Probable Networks and Plausible Predictions: A Review of Practical Bayesian Methods for Supervised Neural Networks.Network: Computation in Neural Sys- tems, 1995. 1, 3, 5 Martin, R., Mess, R., and Walker, S. G. Empirical Bayes Posterior Concentration in Sparse High-dimensional Lin- ear Models.Bernoulli, 2017. 16 Martino, L., Elvira, V ....

    1995

  10. [10]

    Student-t Pro- cesses as Alternatives to Gaussian Processes.Interna- tional Conference on Artificial Intelligence and Statistics,

    16 Shah, A., Wilson, A., and Ghahramani, Z. Student-t Pro- cesses as Alternatives to Gaussian Processes.Interna- tional Conference on Artificial Intelligence and Statistics,

  11. [11]

    Combined modeling of sparse and dense noise for improvement of Relevance Vector Machine

    3 Shen, Y ., Daheim, N., Cong, B., Nickl, P., Marconi, G. M., Raoul, B. C. E. M., Yokota, R., Gurevych, I., Cremers, D., Khan, M. E., et al. Variational Learning is Effective for Large Deep Networks.International Conference on Machine Learning, 2024. 9, 31 Silverman, B. W. Some Aspects of the Spline Smoothing Approach to Non-parametric Regression Curve Fi...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  12. [12]

    Fromsymmetrytogeometry: Tractablenonconvexproblems,

    1, 2, 3, 4, 6, 20 Tipping, M. E. and Faul, A. C. Fast Marginal Likelihood Maximisation for Sparse Bayesian Models.International Workshop on Artificial Intelligence and Statistics, 2003. 3 Titsias, M. Variational Learning of Inducing Variables in Sparse Gaussian Processes.International Conference on Artificial Intelligence and Statistics, 2009. 3, 6 Titsia...

    work page arXiv 2003

This paper was first reviewed by grok-4.3 on June 29, 2026.