REVIEW 2 major objections 5 minor 37 references
A Near-optimal Algorithm for Learning Margin Halfspaces with Massart Noise
T0 review · 2 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read This paper gives a computationally efficient algorithm that learns $\gamma$-margin halfspaces with Massart noise using $\tilde{O}(1/(\gamma^2\epsilon^2))$ samples, nearly matching the known lower bound for efficient learners.
desk verdict Solid near-optimal SGD learner for Massart margin halfspaces, with a real but fixable bug in the final hypothesis-selection step. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing construction is the clipped reweighted convex loss $L_{\lambda,v}(w) = \mathbb{E}_{x,y}[\ell_\lambda(w,x,y)/\max(|v\cdot x|,\gamma/2)]$, where $\ell_\lambda(w,x,y)=(1\{\mathrm{sign}(w\cdot x)\ne y\}-\lambda)|w\cdot x|$ is the LeakyReLU surrogate for the 0-1 loss. Each round fixes $v=w_t$, so the loss stays convex in $w$ while the $|w\cdot x|$ factor reweights errors toward confident points. The clipping by $\gamma/2$ is where the margin assumption enters: for inputs whose current margin is small, the target halfspace's margin $\gamma$ makes $|w^*\cdot x|/\max(|w_t\cdot x|,\gamma/2) \ge 1$, forcing the gradient's sign to point toward $w^*$. The proof splits the stochastic gradient into a signal part $G_1$ satisfying $G_1\cdot(w-w^*) \ge 2(\mathrm{err}(w)-\eta)$ and a zero-mean estimation part, then bounds the cumulative estimation error with subgaussian concentration via Hoeffding's lemma.
What would settle it
Concrete check: compute $g_1(w,x)\cdot(w-w^*)$ and compare with $2(\mathrm{err}(w,x)-\eta)$ on a fine grid of unit vectors $x$ with $|w^*\cdot x| \ge \gamma$ and weight vectors $w$ with $\|w\| \le 1$ in $d=2$ or $d=3$; a single pair $(w,x)$ where the inequality fails—especially on the region $|w\cdot x| < \gamma/2$, where the clipping and margin condition do the work—would invalidate the structural lemma (Claims 2.3 and 2.4) on which the sample bound rests.
Extended reading notes
Core claim
Theorem 2.1 is the paper's central claim: for any distribution on the unit sphere satisfying $\eta$-Massart noise with respect to an unknown halfspace $\mathrm{sign}(w^*\cdot x)$ with margin $\gamma$, Algorithm 1 returns a hypothesis $\hat{w}$ with $\mathrm{err}_D(\hat{w}) \le \eta+\epsilon$ with probability at least $1-\delta$ using $n = O(N+T) = \tilde{O}(1/(\epsilon^2\gamma^2))$ samples and $O(dNT)$ time. The algorithm is projected stochastic gradient descent with step size proportional to $\gamma^2\epsilon$, applied to the loss $\ell_\lambda(w,x,y)W(w_t\cdot x,\gamma/2)$, where $\ell_\lambda$ is the LeakyReLU proxy $(1\{\mathrm{sign}(w\cdot x)\ne y\}-\lambda)|w\cdot x|$ and $W(a,b)=1/\max(a,b)$; a final step selects, among the iterates, the one with smallest empirical 0-1 error. The proof constructs a potential $\Phi(w)=\|w-w^*\|^2$ and shows the gradient has a component that decreases this potential by an amount proportional to $(\mathrm{err}(w_t)-\eta)$ whenever $\mathrm{err}(w_t) \ge \eta+\epsilon$, while the estimation noise is subgaussian and bounded in total. The sample complexity nearly matches the known SQ and low-degree lower bounds for the problem and matches the best known bound for the special case of random classification noise.
Load-bearing premise
The load-bearing premise is the $\gamma$-margin condition—every clean example satisfies $|w^*\cdot x| \ge \gamma$, with the algorithm given $\gamma$ and the noise ceiling $\eta$ in advance; if examples can approach the separating hyperplane arbitrarily closely, the sign-reversal argument that makes each gradient point toward $w^*$ stops working and the $\tilde{O}(1/(\gamma^2\epsilon^2))$ sample bound no longer follows from the proof.
Editorial extensions
If this is right
- If Theorem 2.1 is correct, the computational sample complexity of learning margin halfspaces with Massart noise is $\tilde{\Theta}(1/(\gamma^2\epsilon^2))$ up to logarithmic factors, leaving only the gap to the information-theoretic bound $\tilde{\Theta}(1/(\gamma^2\epsilon))$ as the measurable price of computational efficiency.
- The algorithm is simple enough to implement—projected SGD with fixed step size $c\gamma^2\epsilon$ plus a final empirical-error selection—so the near-optimal sample bound is a practical guarantee rather than an existence statement.
- The result matches the sample complexity previously known only for the easier Random Classification Noise special case, indicating that the harder Massart model does not incur additional sample cost in the margin regime.
- Because every step of the proof uses the margin assumption, the method does not directly yield a dimension-optimal learner for general halfspaces; the paper states this extension is open and would require new ideas.
Reading between the lines
- Our testable prediction: on synthetic $\gamma$-margin distributions with constant Massart noise, the empirical sample complexity of Algorithm 1 with step size $c\gamma^2\epsilon$ should scale as $1/(\gamma^2\epsilon^2)$; a sweep over $\gamma$ and $\epsilon$ would confirm the theorem's stated dependence or expose hidden constants.
- Our transfer suggestion: the clipping reweighting $1/\max(|v\cdot x|,\gamma/2)$ is a general template for convexifying objectives that are nonconvex only through a denominator, and may be useful in margin-based active learning or regression with bounded noise.
- Our open-problem reading: replacing the known margin with a dimension-dependent slack in the clipping is the natural route to remove the margin assumption; the paper says this needs new ideas, but the proof structure shows where such slack could enter.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies PAC learning of gamma-margin halfspaces under Massart noise and proposes an online projected SGD algorithm with a final empirical-error selection step. The main claimed result, Theorem 2.1, is a sample complexity of O~(1/(gamma^2 epsilon^2)) and error eta+epsilon, nearly matching SQ/low-degree lower bounds. The proof decomposes the stochastic gradient into a direction that correlates with w-w* and a zero-mean estimation error, uses a potential argument on ||w_t-w*||^2 with subgaussian concentration, and then selects the best empirical candidate among T+1 iterates.
Significance. If the main theorem holds, this is a substantial improvement over the previous efficient algorithm with O~(1/(gamma^4 epsilon^3)) samples, and it matches the best known algorithmic guarantees for Random Classification Noise. The structural lemma (Lemma 2.2) and the potential-decrease argument are clean and self-contained, and the algorithm is simple and practical. The lower-bound evidence is from SQ/low-degree hardness, so 'near-optimal' is appropriately qualified. The main proof up to the identification of one good candidate is sound; the final selection step, however, contains a load-bearing error that must be corrected before the stated theorem is valid.
major comments (2)
- [Section 2, Step (5) and final paragraph of Theorem 2.1] The stated bound N = Theta(log(1/(gamma delta))/(epsilon(1-2 eta))) is not sufficient for the final selection step. The candidates w_t are only known to include a hypothesis with error at most eta+epsilon; they need not be close to the Bayes classifier, and the excess over the Bayes error can be as large as eta even when the optimal error is 0. The [MN06] fast-rate/Bernstein condition therefore does not apply when comparing an arbitrary candidate with the best candidate in the finite set. Uniformly controlling the empirical 0-1 errors of T+1 hypotheses requires N = Omega(log(T/delta)/epsilon^2) by a Hoeffding union bound. With this corrected N, the total sample count n = N+T remains O~(1/(epsilon^2 gamma^2)) because T dominates, but the stated runtime O(dNT) gains an extra factor of 1/epsilon. As written, the proof of Theorem 2.1 does not rule out selecting a hypothesis with true error above eta+epsilon.
- [Section 2, proof of Theorem 2.1, last paragraph] Even after correcting the selection sample size, the proof concludes err_D(w_hat) <= min_t err_D(w_t) + epsilon <= eta + 2 epsilon, while Theorem 2.1 promises eta + epsilon. The internal accuracy parameter must be rescaled, for example by replacing epsilon with epsilon/2 throughout the SGD target and the selection tolerance, or the theorem statement must be adjusted accordingly. This is a statement-level but local fix.
minor comments (5)
- [Definition 1.2] Condition (i) as written says y = sign(w* . x) for (x,y)~D, which is inconsistent with Massart noise; it should say that the Massart-corrupted label is generated from the clean label sign(w* . x), or that sign(w* . x) is the target concept in Definition 1.1.
- [Theorem 2.1 vs. proof of Theorem 2.1] The theorem statement gives N = Theta(log(1/(gamma delta))/(epsilon(1-2 eta))), while the proof's last paragraph uses N = O(log(T/delta)/(epsilon(1-2 eta))). These two expressions should be reconciled after the correction in the major comments.
- [Equation (4) and surrounding text] There are several formatting artifacts and typos in this passage, including a stray 'a' after 'Inequality (4)' and '/bracehtipupleft' fragments, which should be cleaned up.
- [Claims 2.3 and 2.4] In the proofs of these claims, the notation W(w . x) is sometimes written without the second argument gamma/2; the definition in Equation (3) uses W(v . x, gamma/2), so the proofs should consistently write W(w . x, gamma/2).
- [Proof of Theorem 2.1, constant in T] The argument that the final distance satisfies ||w_{T+1}-w*||^2 <= 0 should explicitly account for the initial distance ||w_0-w*||^2, which can be as large as 4; this is absorbed by choosing constants in T appropriately, but as written the equality to 0 is only up to universal constants.
Circularity Check
No significant circularity: the SGD upper bound is derived self-contained from the margin and Massart definitions.
full rationale
The central derivation is self-contained. Algorithm 1's step size lambda_t = c gamma^2 epsilon, iteration count T = Theta(log(1/delta)/(epsilon^2 gamma^2)), and validation size N are chosen analytically rather than fitted to data. The convergence argument reduces to Lemma 2.2, whose Claims 2.3 and 2.4 are algebraic consequences of the Massart noise and gamma-margin definitions, and the potential Phi(w) = ||w - w*||^2 is analyzed directly. Self-citations to [DDK+23a,b] and [DKTZ24] appear only as context; the paper explicitly states that [DKTZ24] has no implications for the sample complexity of our problem, and the lower bounds are not used as inputs to the upper-bound proof. The final selection step invokes the external [MN06] bound; whether that invocation is quantitatively correct would be a correctness concern, not a circularity, since it does not make the theorem equivalent to its own inputs.
Assumptions & free parameters
free parameters (1)
- c (step-size constant) =
sufficiently small universal constant (c <= 1/8)
assumptions (4)
- domain assumption The data distribution is supported on the unit sphere S^{d-1} and all halfspaces are homogeneous (threshold 0)
- domain assumption The distribution satisfies the gamma-margin condition: |w* . x| >= gamma for all x in the support
- domain assumption The noise is eta-Massart with a known upper bound eta < 1/2
- standard math Standard probabilistic concentration inequalities (Hoeffding's lemma, SZ07) and projection properties
Cite this review
Pith. "Pith review of A Near-optimal Algorithm for Learning Margin Halfspaces with Massart Noise." pith.science (2026). https://pith.science/paper/CO5YEQBV
@misc{pith2026250109691,
author = {Pith},
title = {Pith review of: A Near-optimal Algorithm for Learning Margin Halfspaces with Massart Noise},
year = {2026},
howpublished = {\url{https://pith.science/paper/CO5YEQBV}},
note = {Machine review of arXiv:2501.09691}
}
abstract
We study the problem of PAC learning $\gamma$-margin halfspaces in the presence of Massart noise. Without computational considerations, the sample complexity of this learning problem is known to be $\widetilde{\Theta}(1/(\gamma^2 \epsilon))$. Prior computationally efficient algorithms for the problem incur sample complexity $\tilde{O}(1/(\gamma^4 \epsilon^3))$ and achieve 0-1 error of $\eta+\epsilon$, where $\eta<1/2$ is the upper bound on the noise rate. Recent work gave evidence of an information-computation tradeoff, suggesting that a quadratic dependence on $1/\epsilon$ is required for computationally efficient algorithms. Our main result is a computationally efficient learner with sample complexity $\widetilde{\Theta}(1/(\gamma^2 \epsilon^2))$, nearly matching this lower bound. In addition, our algorithm is simple and practical, relying on online SGD on a carefully selected sequence of convex losses.
Reference graph
Works this paper leans on
-
[1]
P. Awasthi, M. F. Balcan, N. Haghtalab, and R. Urner. Efficient learning of linear separators under bounded noise. In Proceedings of The 28th Conference on Learning Theory, COLT 2015 , pages 167--190, 2015
work page 2015
-
[2]
P. Awasthi, M. F. Balcan, N. Haghtalab, and H. Zhang. Learning and 1-bit compressed sensing under asymmetric noise. In Proceedings of the 29th Conference on Learning Theory, COLT 2016 , pages 152--192, 2016
work page 2016
-
[3]
D. Angluin and P. Laird. Learning from noisy examples. Machine Learning , 2(4):343--370, 1988
work page 1988
-
[4]
A. Blum. Machine learning: My favorite results, directions, and open problems. In 44th Symposium on Foundations of Computer Science (FOCS 2003) , pages 11--14, 2003
work page 2003
-
[5]
S. Chen, F. Koehler, A. Moitra, and M. Yau. Classification under misspecification: Halfspaces, generalized linear models, and connections to evolvability. In Advances in Neural Information Processing Systems, NeurIPS , 2020
work page 2020
-
[6]
S. Chen, F. Koehler, A. Moitra, and M. Yau. Online and distribution-free robustness: Regression and contextual bandits with huber contamination. In FOCS , 2021
work page 2021
-
[7]
G. Chandrasekaran, V. Kontonis, K. Stavropoulos, and K. Tian. Learning noisy halfspaces with a margin: Massart is no harder than random. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024
work page 2024
-
[8]
I. Diakonikolas, J. Diakonikolas, D. M. Kane, P. Wang, and N. Zarifis. Information-computation tradeoffs for learning margin halfspaces with random classification noise. In COLT , 2023
work page 2023
Show all 37 references
-
[9]
Diakonikolas, J
I. Diakonikolas, J. Diakonikolas, D. M. Kane, P. Wang, and N. Zarifis. Near-optimal bounds for learning gaussian halfspaces with random classification noise. In NeurIPS , 2023
2023
-
[10]
Diakonikolas, T
I. Diakonikolas, T. Gouleakis, and C. Tzamos. Distribution-independent PAC learning of halfspaces with M assart noise. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch\' e Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 , pages...
2019
-
[11]
Diakonikolas, R
I. Diakonikolas, R. Impagliazzo, D. M. Kane, R. Lei, J. Sorrell, and C. Tzamos. Boosting in the presence of M assart noise. In Proceedings of The 34th Conference on Learning Theory, COLT , 2021
2021
-
[12]
Diakonikolas and D
I. Diakonikolas and D. Kane. Near-optimal S tatistical Q uery hardness of learning halfspaces with M assart noise. In Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 4258--4282. PMLR , 2022. Preliminary Version 2021: Arxiv eprint:...
2022 arXiv
-
[13]
Diakonikolas, D
I. Diakonikolas, D. M. Kane, V. Kontonis, C. Tzamos, and N. Zarifis. A polynomial time algorithm for learning halfspaces with T sybakov noise. arXiv , 2020
2020
-
[14]
Diakonikolas, D
I. Diakonikolas, D. M. Kane, V. Kontonis, C. Tzamos, and N. Zarifis. Efficiently learning halfspaces with T sybakov noise. STOC , 2021
2021
-
[15]
Diakonikolas, D
I. Diakonikolas, D. M. Kane, V. Kontonis, C. Tzamos, and N. Zarifis. Learning general halfspaces with general M assart noise under the gaussian distribution. In STOC '22: 54th Annual ACM SIGACT Symposium on Theory of Computing, 2022 , pages 874--885. ACM , 2022
2022
-
[16]
Diakonikolas, D
I. Diakonikolas, D. Kane, P. Manurangsi, and L. Ren. Cryptographic hardness of learning halfspaces with M assart noise. In Advances in Neural Information Processing Systems , 2022
2022
-
[17]
Diakonikolas, D
I. Diakonikolas, D. Kane, L. Ren, and Y. Sun. SQ lower bounds for learning single neurons with M assart noise. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022 , 2022
2022
-
[18]
Diakonikolas, D
I. Diakonikolas, D. Kane, and C. Tzamos. Forster decomposition and learning halfspaces with noise. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021 , pages 7732--7744, 2021
2021
-
[19]
Diakonikolas, V
I. Diakonikolas, V. Kontonis, C. Tzamos, and N. Zarifis. Learning halfspaces with M assart noise under structured distributions. In Conference on Learning Theory, COLT , 2020
2020
-
[20]
Diakonikolas, V
I. Diakonikolas, V. Kontonis, C. Tzamos, and N. Zarifis. Learning halfspaces with T sybakov noise. arXiv , 2020
2020
-
[21]
Diakonikolas, V
I. Diakonikolas, V. Kontonis, C. Tzamos, and N. Zarifis. Online Linear Classification with M assart Noise , 2024. Arxiv eprint: 2405.12958
2024 arXiv
-
[22]
Diakonikolas, J
I. Diakonikolas, J. Park, and C. Tzamos. Relu regression with M assart noise. In Advances in Neural Information Processing Systems , 2021
2021
-
[23]
Diakonikolas, C
I. Diakonikolas, C. Tzamos, and D. M. Kane. A strongly polynomial algorithm for approximate forster transforms and its application to halfspace learning. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing, STOC 2023 , pages 1741--1754. ACM , 2023
2023
-
[24]
Dunagan and S
J. Dunagan and S. Vempala. Optimal outlier removal in high-dimensional spaces. J. Computer & System Sciences , 68(2):335--373, 2004
2004
-
[25]
Johnson and J
W. Johnson and J. Lindenstrauss. Extensions of Lipshitz mapping into Hilbert space . Contemporary Mathematics , 26:189--206, 1984
1984
-
[26]
Kontonis, F
V. Kontonis, F. Iliopoulos, K. Trinh, C. Baykal, G. Menghani, and E. Vee. Slam: Student-label mixing for distillation with unlabeled examples. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023 , 2023
2023
-
[27]
Massart and E
P. Massart and E. Nedelec. Risk bounds for statistical learning. Ann. Statist. , 34(5):2326--2366, 10 2006
2006
-
[28]
Nasser and S
R. Nasser and S. Tiegel. Optimal SQ lower bounds for learning halfspaces with M assart noise. In Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 1047--1074. PMLR , 2022
2022
-
[29]
Rosenblatt
F. Rosenblatt. The P erceptron: a probabilistic model for information storage and organization in the brain. Psychological Review , 65:386--407, 1958
1958
-
[30]
R. H. Sloan. Types of noise in data for concept learning. In Proceedings of the First Annual Workshop on Computational Learning Theory , COLT '88, pages 91--96, San Francisco, CA, USA, 1988. Morgan Kaufmann Publishers Inc
1988
-
[31]
R. H. Sloan. Corrigendum to types of noise in data for concept learning. In Proceedings of the Fifth Annual ACM Conference on Computational Learning Theory, COLT 1992 , page 450, 1992
1992
-
[32]
Shalev-Shwartz and S
S. Shalev-Shwartz and S. Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014
2014
-
[33]
Smale and D
S. Smale and D. Zhou. Learning theory estimates via integral operators and their approximations. Constructive approximation , 26(2):153--172, 2007
2007
-
[34]
L. G. Valiant. A theory of the learnable. In Proc.\ 16th Annual ACM Symposium on Theory of Computing (STOC) , pages 436--445. ACM Press, 1984
1984
-
[35]
Vershynin
R. Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018
2018
-
[36]
Yan and C
S. Yan and C. Zhang. Revisiting perceptron: Efficient and label-optimal learning of halfspaces. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017 , pages 1056--1066, 2017
2017
-
[37]
Zhang, P
Y. Zhang, P. Liang, and M. Charikar. A hitting time analysis of stochastic gradient langevin dynamics. In Proceedings of the 30th Conference on Learning Theory, COLT 2017 , pages 1980--2022, 2017
2017
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.