REVIEW 3 major objections 4 minor 1 cited by
Stability and List-Replicability for Agnostic Learners
T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper characterizes agnostic stable learning: a binary hypothesis class admits a learner that repeats itself with probability ρ(ε) on fresh samples iff its Littlestone dimension is finite, and the stability probability may depend on…
desk verdict Resolves two CCMY24 open problems with plausible, likely-correct theorems, but the written proof has three fixable gaps—off-by-one in the Section 3 lower bound, a loss-accounting slip in Claim 2.2, and a list-membership mismatch in Theorem 1.9(ii). read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Littlestone dimension, the largest depth of a complete binary 'mistake tree' whose every root-to-leaf path is realized by some hypothesis in H; it is the dimension that already characterizes online learnability. The lower-bound proof reduces the problem to threshold classes: a classical model-theoretic fact (Proposition 3.3) says every class with infinite Littlestone dimension contains arbitrarily large threshold classifiers. A hypergraph Ramsey argument then forces the learner's behaviour on a random threshold sample to be governed by a fixed table of output probabilities, and a boosting lemma shows those probabilities are well approximated by convex combinations from a short list of hypotheses. A volume argument on the convex hull of the list produces two samples that must label some domain point differently, contradicting the probability jump the learner must exhibit.
What would settle it
A single explicit hypothesis class with infinite Littlestone dimension but finite threshold dimension would falsify Proposition 3.3 and invalidate the lower-bound machinery; equivalently, exhibiting an L-list-replicable learner for any class of infinite Littlestone dimension would refute Theorem 3.1 and Theorem 1.4(i).
Extended reading notes
Core claim
The central claim is Theorem 1.4: a binary concept class H is agnostically ρ-global stable for some schedule ρ:(0,1)→(0,1) if and only if Ldim(H)<∞. Here ρ-global stability means that for every distribution D there is some hypothesis h with LD(h)≤LD(H)+ε that the learner outputs with probability at least ρ(ε). Part (i) shows that if Ldim(H)=∞, no positive stability schedule exists, even when the distributions are realizable; part (ii) shows that finite Littlestone dimension suffices, by observing that the realizable stable learner of a prior theorem already gives an agnostic stable learner with an explicit ρ(ε). The paper further proves that for any fixed γ>0, every class that is γ-agnostically globally stable—uniform stability probability, restricted to distributions with LD(H)≤γ—is finite, so the realizable case γ=0 is the only one admitting infinite classes.
Load-bearing premise
The lower-bound proof takes as given that every hypothesis class with infinite Littlestone dimension contains threshold classifiers over arbitrarily large finite sets; if that model-theoretic fact failed, the reduction to threshold classes—and with it the proof of Theorem 1.4(i)—would collapse.
Editorial extensions
If this is right
- If Ldim(H)<∞ then, for every ε>0, some learner outputs a hypothesis of loss at most LD(H)+ε with probability at least ρ(ε), where ρ(ε)=1/((d+1)2^{2d+1}4^{n(ε)}) and n(ε) is the sample size needed by the realizable stable learner.
- If Ldim(H)=∞ then no positive function ρ can make H ρ-globally stable; in particular the impossibility holds for realizable distributions, not only for agnostic ones.
- Combined with known equivalences, Theorem 1.4 makes agnostic ρ-global stability coincide with online learnability, approximate private learnability, and global stability.
- For every γ>0, γ-agnostic global stability holds only for finite classes, so restricting attention to distributions with small class error does not restore infinite learnable classes.
- Stability and list-replicability are equivalent in all these relaxed settings, so the same finiteness and Littlestone-dimension dichotomies hold for list-replicable learners.
Reading between the lines
- The explicit stability schedule in Lemma 4.2 decays like 1/4^{n(ε)}, where n(ε) is inherited from the realizable learner; a matching impossibility argument might show that such exponential decay is unavoidable for finite-dimensional classes.
- The proof's reliance on a non-constructive threshold-dimension theorem suggests that making that theorem constructive could yield concrete hard distributions and quantitative impossibility results for infinite-dimensional classes.
- The dichotomy is proved for binary labels; the same threshold-class reduction is likely to transfer to multiclass or real-valued prediction, though the paper does not make that extension.
- Theorem 1.6 shows the class-error dependent relaxation collapses to finiteness for any γ>0; whether allowing γ to shrink with the sample size changes the picture is a natural next question.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies two relaxations of agnostic global stability introduced by Chase, Chornomaz, Moran, and Yehudayoff (CCMY24): excess-error dependent stability, where the stability parameter may depend on the excess error, and class-error dependent stability, where the distribution is assumed to have small population loss. The main results are: (i) a complete characterization of excess-error dependent stability by the Littlestone dimension (Theorem 1.4); (ii) a proof that only finite classes are γ-agnostically globally stable for any γ>0 (Theorem 1.6); and (iii) an equivalence between these stability notions and the corresponding list-replicability notions (Theorem 1.9). The proofs use Ramsey-theoretic arguments, a boosting lemma, and a volume-based argument, together with the classical Hodges--Shelah threshold-dimension result.
Significance. If the results are correct, they resolve two open problems of CCMY24, strengthen the Bun--Livni--Moran characterization from the realizable to the agnostic excess-error-dependent setting, and connect the resulting notion to online learnability and approximate private learnability. The paper is clearly written and the high-level proof strategy is promising. The constructive upper bound in Theorem 1.4(ii) is explicit, and the lower-bound argument is a genuinely new use of list-replicability techniques. However, the written proof contains several load-bearing gaps that must be repaired before the claims can be accepted.
major comments (3)
- [Section 3, Eq. (16) and Claims 3.7/3.9] There is an off-by-one indexing error in the lower-bound proof of Theorem 3.1. After fixing t0, the paper defines p_k := p_{t0,k} and asserts in Eq. (16) that for every S ∈ Π_{t0} and x ∉ S_X, f_S(x) ∈ [p_k − δ, p_k] with k = ord_{S_X}(x). But by the definition of h^R_t in Claim 3.6, a sample with exactly t0 zeros corresponds to threshold index t0+1, not t0. Hence the probabilities that actually govern the learner's output on S1 and S2 are p_{t0+1,k}, not p_{t0,k}. As written, the jump in Claim 3.7 and the final contradiction |p_c − p_{c+1}| ≥ 1/(2n) versus |p_c − p_{c+1}| ≤ 2δ + 4√δ are not justified because the bounds |p_c − f_{S1}(x)| ≤ δ and |p_{c+1} − f_{S2}(x)| ≤ δ do not follow from Eq. (16). This is a load-bearing error; it appears fixable by an index shift (e.g., redefining p_k := p_{t0+1,k} or reindexing Π_t), but as written the proof does not establish Theorem 3.1.
- [Section 2, Claim 2.2] The proof of Theorem 1.9(ii) has two gaps that affect the claimed stability/list-replicability equivalence. First, Eq. (7) only provides uniform convergence for hypotheses in H, but Condition 3(b) is applied to an arbitrary h ∈ {0,1}^X. The displayed chain "LD(h) ≤ inf_{h'∈H} L_Q(h') + 3ε/4" omits the sampling-error term; even for h ∈ H, the correct bound is LD(h) ≤ L_Q(h)+ε/4 ≤ inf L_Q + ε ≤ LD(H)+5ε/4, so the conclusion LD(h) ≤ LD(H)+ε does not follow. Second, the agnostic list-replicability definition (Definition 1.7) requires the list to consist of hypotheses from H, while the set Λ defined in the proof may contain arbitrary functions, and the algorithm A′ is designed to output any such function. Thus the proof shows at best that A′ outputs a low-loss function from a small arbitrary list, not a hypothesis from H. This is load-bearing because Theorem 1.9 is used to transfer the lower bound in Theorem 3.1 and to resolve the CCMY24 list-replicability questions.
- [Section 4, Lemma 4.2] The proof of the finite-Littlestone-dimension upper bound contains an unjustified constant. It asserts γ := LD(h*) ≤ LD(H)+ε/2 ≤ 3/4, but LD(H)<2/3 does not imply LD(H)+ε/2 ≤ 3/4 for all ε>0. Consequently, the bound (1−γ)^n ≥ 4^{−n} used for the stability probability does not follow in general. This is repairable by a case split: if a small-loss h* with γ ≤ 3/4 exists, use the conditioning argument; otherwise a constant classifier already has loss at most 1/2 ≤ LD(H)+ε, making the stability property trivial. As written, however, the proof of Theorem 1.4(ii) is incomplete.
minor comments (4)
- [Section 3, proof of Claim 3.9] The proof's final sentence says x lies in the intersection of the (c+1)-th interval of S1 and the c-th interval of S2, while the claim statement and the subsequent application refer to the c-th interval of S1 and the (c+1)-th interval of S2. Please make the interval indices consistent.
- [Section 1, Theorem 1.6 proof] The proof uses the inequality "γ′ < 1/10" but γ′ is defined as min{γ,1/10}, so only γ′ ≤ 1/10 holds. In addition, the argument that h*(x*) ≠ b* leads to a contradiction assumes ε is small enough that 1−γ′ > γ′(1+ε); for large ε this inequality can fail. Since for large ε the loss condition is vacuous, a separate case should be stated.
- [Section 1, Definition 1.7] In the realizable definition of list-replicability the list hypotheses are not required to belong to H, but in the agnostic definition they are explicitly required to be in H. If this asymmetry is intentional, it should be explained; if not, the agnostic definition should be aligned with the realizable one or with the original definition in [CMY23].
- [Section 3, Lemma 3.5 and Eq. (12)] The symbol δ is used both for the failure probability in Eq. (12) (δ = n^{-10}) and for the generic confidence parameter in Lemma 3.5 and elsewhere. This does not create a technical problem, but renaming one of them would improve readability.
Circularity Check
No significant circularity: the derivation chain rests on external theorems and internal reductions, with no fitted input renamed as a prediction.
full rationale
I walked the paper's derivation chain. Theorem 1.4(i) is reduced to Theorem 3.1 through Theorem 1.9, which is proved self-contained in Section 2; the reduction does not assume the conclusion, since list-replicability and global stability are formally related there and then used only as a bridge. Theorem 3.1 uses Proposition 3.3 (Hodges/Shelah) as an external, parameter-free model-theoretic input, and the proof then proceeds through hypergraph Ramsey (Claim 3.6), a boosting lemma (Lemma 3.5) proved by a standard Hoeffding argument, and a volume-based approximate-rank contradiction (Claims 3.7-3.9); none of these steps defines its target into existence. Theorem 1.4(ii) invokes Theorem 4.1 from [BLM20] as an external finite-Littlestone global-stability learner and adapts that learner to agnostic distributions by conditioning on a low-loss hypothesis; no parameter is fitted to data and then reported as a prediction. The paper's references to [CCMY24], [CMY23], and [HM25] are external context or parallel independent work, not load-bearing self-citations, and no uniqueness theorem from the authors' own prior work is imported to force a choice. I also considered the off-by-one indexing concern between the samples in Pi_t0 and the probability matrix p_{t,k} in Claim 3.6/Equation (16); even if that is a genuine proof bug, it is a correctness issue, not circularity, because it does not make any claim true by construction or by self-reference. The main characterization is therefore self-contained against external benchmarks as far as circularity is concerned.
Assumptions & free parameters
assumptions (6)
- standard math Ramsey theorem for hypergraphs (used in Claim 3.6 to extract a homogeneous set of size M)
- standard math Hodges-Shelah theorem: infinite Littlestone dimension implies infinite threshold dimension (Proposition 3.3)
- standard math Uniform convergence for finite VC dimension classes (used in Claim 2.2 and in the proof of Theorem 1.9(ii))
- domain assumption Bun-Livni-Moran theorem: finite Littlestone dimension implies realizable global stability (Theorem 4.1)
- domain assumption Chase-Chornomaz-Moran-Yehudayoff theorem: agnostic global stability iff finite class (Theorem 1.3)
- domain assumption Chase-Moran-Yehudayoff equivalence between global stability and global list-replicability (Theorem 1.8)
Cite this review
Pith. "Pith review of Stability and List-Replicability for Agnostic Learners." pith.science (2026). https://pith.science/paper/WYS4EOZX
@misc{pith2026250105333,
author = {Pith},
title = {Pith review of: Stability and List-Replicability for Agnostic Learners},
year = {2026},
howpublished = {\url{https://pith.science/paper/WYS4EOZX}},
note = {Machine review of arXiv:2501.05333}
}
read the original abstract
Two seminal papers--Alon, Livni, Malliaris, Moran (STOC 2019) and Bun, Livni, and Moran (FOCS 2020)--established the equivalence between online learnability and globally stable PAC learnability in binary classification. However, Chase, Chornomaz, Moran, and Yehudayoff (STOC 2024) recently showed that this equivalence does not hold in the agnostic setting. Specifically, they proved that in the agnostic setting, only finite hypothesis classes are globally stable learnable. Therefore, agnostic global stability is too restrictive to capture interesting hypothesis classes. To address this limitation, Chase et al. introduced two relaxations of agnostic global stability. In this paper, we characterize the classes that are learnable under their proposed relaxed conditions, resolving the two open problems raised in their work. First, we prove that in the setting where the stability parameter can depend on the excess error (the gap between the learner's error and the best achievable error by the hypothesis class), agnostic stability is fully characterized by the Littlestone dimension. Consequently, as in the realizable case, this form of learnability is equivalent to online learnability. As part of the proof of this theorem, we strengthen the celebrated result of Bun et al. by showing that classes with infinite Littlestone dimension are not stably PAC learnable, even if we allow the stability parameter to depend on the excess error. For the second relaxation proposed by Chase et al., we prove that only finite hypothesis classes are globally stable learnable, even if we restrict the agnostic setting to distributions with small population loss.
Forward citations
Cited by 1 Pith paper
-
The Role of Randomness in Stability
Randomness complexity for replicability and differential privacy equals, up to one bit, the inverse log of global stability, and finite randomness complexity of PAC learning exactly matches finite Littlestone dimension.
Reference graph
Works this paper leans on
-
[1]
Noga Alon, Mark Bun, Roi Livni, Maryanthe Malliaris, and Shay Moran, Private and online learnability are equivalent, J. ACM 69 (2022), no. 4
work page 2022
-
[2]
852--860
Noga Alon, Roi Livni, Maryanthe Malliaris, and Shay Moran, Private pac learning implies finite littlestone dimension, Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, 2019, pp. 852--860
2019
-
[3]
Monya Baker, 1,500 scientists lift the lid on reproducibility, Nature 533 (2016), no. 7604, 452--454
work page 2016
-
[4]
Philip Ball, Is AI leading to a reproducibility crisis in science? , Nature 624 (2023), no. 7990, 22--25
work page 2023
-
[5]
Mark Bun, Marco Gaboardi, Max Hopkins, Russell Impagliazzo, Rex Lei, Toniann Pitassi, Satchit Sivakumar, and Jessica Sorrell, Stability is stable: Connections between replicability, privacy, and adaptive generalization, Proceedings of the 55th Annual ACM Symposium on Theory of Computing, 2023, pp. 520--527
work page 2023
- [6]
- [7]
-
[8]
Zachary Chase, Shay Moran, and Amir Yehudayoff, Stability and Replicability in Learning , 2023 IEEE 64th Annual Symposium on Foundations of Computer Science (FOCS), IEEE Computer Society, 2023, pp. 2430--2439
work page 2023
Show all 22 references
-
[9]
Eric Eaton, Marcel Hussing, Michael Kearns, and Jessica Sorrell, Replicable reinforcement learning, Advances in Neural Information Processing Systems 36 (2023), 15172--15185
2023
-
[10]
Hossein Esfandiari, Alkis Kalavasis, Amin Karbasi, Andreas Krause, Vahab Mirrokni, and Grigoris Velegkas, Replicable bandits, The Eleventh International Conference on Learning Representations, 2023
2023
-
[11]
Hossein Esfandiari, Amin Karbasi, Vahab Mirrokni, Grigoris Velegkas, and Felix Zhou, Replicable clustering, Advances in Neural Information Processing Systems 36 (2023), 39277--39320
2023
-
[12]
Badih Ghazi, Ravi Kumar, and Pasin Manurangsi, User-level differentially private learning via correlated sampling, Advances in Neural Information Processing Systems 34 (2021), 20172--20184
2021
-
[13]
Max Hopkins and Shay Moran, The role of randomness in stability, arXiv preprint arXiv:2502.08007 (2025)
2025 arXiv
-
[14]
Wilfrid Hodges, A shorter model theory, Cambridge university press, 1997
1997
-
[15]
Russell Impagliazzo, Rex Lei, Toniann Pitassi, and Jessica Sorrell, Reproducibility in learning, Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2022, 2022, p. 818–831
2022
-
[16]
22861--22878
Alkis Kalavasis, Amin Karbasi, Kasper Green Larsen, Grigoris Velegkas, and Felix Zhou, Replicable learning of large-margin halfspaces, International Conference on Machine Learning, PMLR, 2024, pp. 22861--22878
2024
-
[17]
15586--15622
Alkis Kalavasis, Amin Karbasi, Shay Moran, and Grigoris Velegkas, Statistical indistinguishability of learning algorithms, International Conference on Machine Learning, PMLR, 2023, pp. 15586--15622
2023
-
[18]
Amin Karbasi, Grigoris Velegkas, Lin Yang, and Felix Zhou, Replicability in reinforcement learning, Advances in Neural Information Processing Systems 36 (2023), 74702--74735
2023
-
[19]
Nick Littlestone, Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm, Machine learning 2 (1988), 285--318
1988
-
[20]
F. P. Ramsey, On a problem of formal logic, Proceedings of the London Mathematical Society s2-30 (1930), no. 1, 264--286
1930
-
[21]
Shelah, Classification theory and the number of nonisomorphic models, second ed., Studies in Logic and the Foundations of Mathematics, vol
S. Shelah, Classification theory and the number of nonisomorphic models, second ed., Studies in Logic and the Foundations of Mathematics, vol. 92, North-Holland Publishing Co., Amsterdam, 1990
1990
-
[22]
Shai Shalev-Shwartz and Shai Ben-David, Understanding machine learning: From theory to algorithms, Cambridge university press, 2014
2014
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.