REVIEW 3 major objections 5 minor 37 references
Investigating an Overfitting and Degeneration Phenomenon in Self-Supervised Multi-Pitch Estimation
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Adding self-supervised objectives to unlabeled audio does not help polyphonic pitch estimation; it drives the model to blank predictions on that audio while supervised performance stays intact.
desk verdict A useful cautionary finding about self-supervised multi-pitch estimation, but the key experiments change loss weighting alongside data quantity, so the central causal claim is not proven. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the set of BCE-based self-supervised losses defined over the model's output salience-gram: a timbre-invariance loss (random parabolic equalization), a percussion-invariance loss (superimposing drum audio), a geometric-equivariance loss (vertical pitch-shift and horizontal time-shift/stretch), and an energy-based loss that supervises the salience-gram with a weighted harmonic sum of the input. Because these losses compare predictions against either the model's own transformed predictions or coarse energy targets, an all-zero output can satisfy them on data that has no ground-truth, which the experiments identify as the route to degeneration.
What would settle it
Train the same model on URMP with self-supervised losses applied to held-out URMP-T2 samples, but replace the energy-based and sparsity losses with a simple cross-entropy-style per-frame activity prior (e.g., a fixed target of non-zero salience); if blank-prediction collapse disappears on T2, the original collapse is tied to the specific loss formulation rather than to self-supervision per se.
Extended reading notes
Core claim
The paper's central discovery is that combining supervised multi-pitch estimation with self-supervised invariance/equivariance objectives helps under closed conditions but backfires when the self-supervised objectives are applied to additional unlabeled data: the model's predictions on that data converge to a blank salience-gram, a trivial solution that satisfies the self-supervised losses, while performance on the supervised dataset remains intact. The effect grows with the number of unlabeled samples, appears regardless of which external dataset is used, and persists even when the unlabeled samples are held-out tracks from the same URMP corpus. Energy-based targets prevent the blank collapse but impose targets too coarse and rigid to be useful, and fine-tuning from a strong checkpoint does not escape the failure.
Load-bearing premise
The load-bearing premise is that the self-supervised objectives were implemented faithfully and fairly weighted, so a subtle implementation bug or loss-scale mismatch is not the real cause of the collapse.
Editorial extensions
If this is right
- Training on the closed URMP set with all self-supervised objectives improves MPE F1 over supervised-only training by several points across all evaluation datasets, showing the objectives capture useful pitch structure.
- Adding 16 extra unlabeled samples per batch from NSynth, MusicNet, or FMA degrades predictions to blank on those distributions, with severity increasing with the number of added samples.
- Degeneration is not explained by distribution shift, since using 10 held-out URMP tracks as the self-supervised-only data still causes a moderate performance drop on URMP and Bach10.
- Energy-based losses (harmonic-sum targets plus sparsity) prevent the trivial blank output but still yield degraded predictions, and fine-tuning the best reference checkpoint at one-fifth the learning rate does not remedy the problem.
- The paper's explanation is that the self-supervised losses, applied without corresponding supervision, exert too strong a pull toward the trivial solution on the unlabeled distribution.
Reading between the lines
- The same collapse may appear in other dense prediction tasks that combine supervised and self-supervised losses across disjoint data subsets; a quick test would be to monitor prediction entropy on the unlabeled subset during training.
- A direct ablation of the loss-averaging scheme (e.g., computing self-supervised losses only on labeled samples, or normalizing them per-subset) would test whether the dilution of the supervised signal is itself a contributor.
- Because the paper attributes monophonic SSL's stability to categorical cross-entropy and the monophonic inductive bias, a cross-entropy-style formulation for polyphonic salience—such as a per-frame one-hot content prior—is a concrete candidate fix worth testing.
- If the blank-prediction trajectory is visible in validation curves before the final checkpoint, an early-stopping or re-weighting schedule triggered by such monitoring could mitigate the failure in practice.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper extends a supervised convolutional multi-pitch estimation (MPE) framework with self-supervised objectives based on pitch-invariant and pitch-equivariant transformations, following the authors' prior SS-MPE work. Under closed training on URMP, adding these objectives improves performance substantially. The central new finding is that when additional unlabeled data are included in the batch for self-supervision only, the model continues to perform well on URMP but degenerates on the unlabeled data, producing blank pitch-salience predictions. The authors investigate this with several follow-up experiments: varying the number of extra samples, applying energy-based objectives, fine-tuning from a supervised checkpoint, and a same-distribution control in which part of URMP is held out for self-supervision only. They conclude that self-supervision without corresponding supervision pushes the model toward degeneration. The paper is clearly written and includes useful validation curves, but the central empirical claim is not fully established because the experiments vary multiple factors at once and rely on single runs.
Significance. If the reported phenomenon is robust, it is an important negative result for semi-supervised MPE: naively adding self-supervised objectives on unlabeled data can be harmful, and the paper identifies a specific collapse mode and directions for mitigation. The breadth of the investigation is a strength: multiple additional-data domains, a same-distribution control, and several mitigation attempts are considered. The paper also provides full validation curves, which is helpful for assessing the dynamics of degeneration. However, the central conclusion depends on comparisons that do not isolate the role of the missing supervision from other changes in the optimization setup, and all experiments are single runs without error bars. These issues need to be addressed before the claim can be accepted.
major comments (3)
- [§3.4, §4.2, Eqs. (2)–(4)] The experiments do not isolate the absence of supervision as the cause of degeneration. When extra unlabeled samples are added, the supervised loss is averaged over the 8 supervised samples, while each self-supervised loss is averaged over all samples in the expanded batch. A sample-count ratio argument such as "3*(8+16) vs 3*8" is not a gradient ratio because Eqs. (2)–(4) are normalized by batch size; with exchangeable per-sample gradients, the expected gradient of each SSL loss is invariant to batch size. The substantive problem is that the extra samples change the distribution over which the SSL losses are averaged, and if unlabeled samples have systematically larger per-sample SSL gradients, the effective weight of the SSL losses relative to the supervised loss changes. The paper does not measure loss scales or gradient norms, and it does not include a control condition in which the same extra data are used with supervision. The URMP-T1/T2 comparison in Sec. 4.2 is the right idea, but it still changes both the data distribution of the SSL losses and the fraction of SSL-gradient contribution coming from supervised versus unsupervised samples. To support the paper's central claim, the authors should add a condition with labeled extra samples (or otherwise hold the composition of the SSL gradient constant) and report whether degeneration occurs without the labels.
- [§3.1, §3.4] The final model in every experiment is selected as the checkpoint with maximum F1 on the URMP validation set. In the additional-data experiments, this selection criterion is on the same distribution as the supervised training data, so the stable URMP performance may be partly by construction: the checkpoint that best fits URMP is chosen, even if it has already degenerated on other datasets. The paper's claim of "simultaneously overfitting to URMP while degenerating elsewhere" should be supported by reporting results at fixed training epochs or at checkpoints selected by a criterion that does not depend on the supervised validation set. The full curves in Figs. 1, 3, and 4 are helpful, but the quantitative claims about final performance rely on this selection protocol.
- [§3.3, §3.4, Figs. 1–4] All experimental results appear to be from a single training run with no error bars, no multiple seeds, and no significance tests. Given that the central claims include comparative statements about the severity of degeneration (e.g., "more extreme" for URMP-T2 than for NSynth, or "moderate decrease" for the same-distribution control), the lack of uncertainty quantification makes these comparisons fragile. At minimum, the authors should report results over several seeds and include variance information in the figures and Table 1.
minor comments (5)
- [§3.1] There is a typo: "first100 epochs" should be "first 100 epochs", and "22, 050 Hz" contains an unusual space that should be removed.
- [§3.3] The phrase "trained with such a small of amount of audio" is a typo; it should be "such a small amount of audio".
- [Table 1] The grayed values indicating train/test overlap are not visible in the text-only version of the paper; please use an alternative notation such as asterisks or superscripts.
- [§2.3.1] The notation for applying equivariant transformations to the target salience-gram, tev(Ŷ), is used in Eq. (3) but not explicitly defined; clarifying that this means corresponding translations/stretching of the salience-gram would improve readability.
- [§4.3] The concluding sentence "self-supervision without corresponding supervision essentially pushes the model to degenerate" is too strong given the experimental scope; a more cautious phrasing would restrict the claim to the investigated architecture, objectives, and loss-averaging scheme.
Circularity Check
No circularity; the central degeneration claim is an experimental finding, not a consequence of its own definitions.
full rationale
The paper does not derive its main claim from a parameter fitted to the target quantity, nor does it import a uniqueness theorem from the authors' own prior work. Its central observation, that adding unlabeled samples for the self-supervised losses (Eqs. 2-5) degrades predictions on those samples while supervised URMP validation stays stable, is obtained from training trajectories in Secs. 3.4 and 4; the conclusion in Sec. 4.3 is an interpretation of those experiments, not a definitional identity. The self-citations ([10], [14]) supply the loss forms and backbone architecture, but the degeneration result would stand or fall on the experimental curves regardless of whether those losses are ultimately the intended SSL theory. The only candidate concern is an experimental confound noted in Sec. 3.1: the supervised loss is averaged over the supervised subset while the SSL losses are averaged over the enlarged batch, so adding unlabeled samples changes the effective SSL-to-supervised weighting as well as the data distribution. That is a validity or confounding issue for the paper's causal interpretation, not circularity: no output is equivalent to an input by construction, and no fitted quantity is relabeled as a prediction. No circular steps are therefore attested.
Assumptions & free parameters
free parameters (2)
- additional samples per batch =
16
- loss weighting (equal, unweighted) =
1.0 for all objectives
assumptions (3)
- domain assumption The self-supervised objectives from [10] correctly encode pitch-invariance and pitch-equivariance and remain valid when integrated into a supervised framework.
- domain assumption All BCE-based losses operate on approximately the same numerical scale, making equal weighting reasonable.
- domain assumption The model architecture and HCQT features are adequate for MPE; the degeneration is not an artifact of capacity limitations.
Cite this review
Pith. "Pith review of Investigating an Overfitting and Degeneration Phenomenon in Self-Supervised Multi-Pitch Estimation." pith.science (2026). https://pith.science/paper/TC7HIORI
@misc{pith2026250623371,
author = {Pith},
title = {Pith review of: Investigating an Overfitting and Degeneration Phenomenon in Self-Supervised Multi-Pitch Estimation},
year = {2026},
howpublished = {\url{https://pith.science/paper/TC7HIORI}},
note = {Machine review of arXiv:2506.23371}
}
read the original abstract
Multi-Pitch Estimation (MPE) continues to be a sought after capability of Music Information Retrieval (MIR) systems, and is critical for many applications and downstream tasks involving pitch, including music transcription. However, existing methods are largely based on supervised learning, and there are significant challenges in collecting annotated data for the task. Recently, self-supervised techniques exploiting intrinsic properties of pitch and harmonic signals have shown promise for both monophonic and polyphonic pitch estimation, but these still remain inferior to supervised methods. In this work, we extend the classic supervised MPE paradigm by incorporating several self-supervised objectives based on pitch-invariant and pitch-equivariant properties. This joint training results in a substantial improvement under closed training conditions, which naturally suggests that applying the same objectives to a broader collection of data will yield further improvements. However, in doing so we uncover a phenomenon whereby our model simultaneously overfits to the supervised data while degenerating on data used for self-supervision only. We demonstrate and investigate this and offer our insights on the underlying problem.
Reference graph
Works this paper leans on
-
[10]
MT3: Multi-task multitrack music transcrip- tion,
J. Gardner, I. Simon, E. Manilow, C. Hawthorne, and J. Engel, “MT3: Multi-task multitrack music transcrip- tion,” in Proceedings of ICLR, 2021
work page 2021
-
[1]
Investigating an Overfitting and Degeneration Phenomenon in Self-Supervised Multi-Pitch Estimation
INTRODUCTION Pitch is a perceptual attribute of sound events that produce waves or harmonics that oscillate at integer multiples of a fundamental frequency (F0) [1]. Pitch is a foundational aspect of music, and it is often useful to represent musi- cal content in terms of relationships between pitch ( i.e., melody and harmony). In Music Information Retrie...
work page Pith review arXiv 2025
-
[2]
FRAMEWORK In this section, we describe our feature extraction module, model architecture, and training objectives. Our method- ology can be viewed as the integration of self-supervised techniques for MPE [10] into a supervised framework. 2.1 Model & Features We adopt a modified version of the fully convolutional 2D autoencoder used in the Timbre-Trap fram...
-
[3]
EXPERIMENTS In this section, we detail our experimental setup and our initial investigation into the joint training paradigm. 3.1 Training & Evaluation Details We train and validate the model in each experiment on URMP [18] following the splits proposed in [3]. Train- ing is conducted on batches of 4 second excerpts using AdamW optimizer [19] with batch s...
-
[4]
DISCUSSION In this section, we investigate the phenomenon uncovered in Sec. 3.4 further and conduct several follow-up experi- ments in an effort to identify the underlying problem. 4.1 Overfitting & Degeneration In order to illustrate and characterize the problem of de- generation, Fig. 2 shows predictions for a single sample from Ref. along with predicti...
work page 2000
-
[5]
CONCLUSION We have demonstrated that self-supervised objectives can substantially improve upon the standard supervised train- ing paradigm for MPE. However, in attempting to extend self-supervised learning beyond the distribution of data that is already grounded with supervised learning, we en- counter issues whereby our model simultaneously overfits to t...
-
[6]
2222129 and synergistic activities funded by NSF grant DGE-1922591
ACKNOWLEDGMENTS This work is supported by National Science Foundation (NSF) grant No. 2222129 and synergistic activities funded by NSF grant DGE-1922591
-
[7]
Music foundation model as generic booster for music downstream tasks,
W. Liao, Y . Takida, Y . Ikemiya, Z. Zhong, C.-H. Lai, G. Fabbro, K. Shimada, K. Toyama, K. Cheuk, M. A. Martínez-Ramírez et al., “Music foundation model as generic booster for music downstream tasks,” arXiv preprint arXiv:2411.01135, 2024
arXiv 2024
Show all 37 references
-
[8]
Müller, Fundamentals of Music Processing: Audio, Analysis, Algorithms, Applications
M. Müller, Fundamentals of Music Processing: Audio, Analysis, Algorithms, Applications. Springer, 2015
2015
-
[9]
Auto- matic music transcription: An overview,
E. Benetos, S. Dixon, Z. Duan, and S. Ewert, “Auto- matic music transcription: An overview,” IEEE Signal Processing Magazine, vol. 36, no. 1, pp. 20–30, 2019
2019
-
[11]
Input audio is resampled to22, 050 Hz, and N is the number frames using a hop size of 256 samples
XH ∈ [0, 1]6×K×N with K = 440 frequency bins starting from fmin = 27.5 Hz and 5 bin per semitone reso- lution. Input audio is resampled to22, 050 Hz, and N is the number frames using a hop size of 256 samples. We main- tain the original set of harmonics H = {0.5, 1, 2, 3, 4, 5...
-
[12]
Unaligned supervi- sion for automatic music transcription in the wild,
B. Maman and A. H. Bermano, “Unaligned supervi- sion for automatic music transcription in the wild,” in Proceedings of ICML, 2022
2022
-
[13]
Scaling polyphonic transcription with mix- tures of monophonic transcriptions,
I. Simon, J. Gardner, C. Hawthorne, E. Manilow, and J. Engel, “Scaling polyphonic transcription with mix- tures of monophonic transcriptions,” in Proceedings of ISMIR, 2022
2022
-
[14]
MERT: Acoustic music understanding model with large-scale self-supervised training,
Y . Li, R. Yuan, G. Zhang, Y . Ma, X. Chen, H. Yin, C. Lin, A. Ragni, E. Benetos, N. Gyenge et al. , “MERT: Acoustic music understanding model with large-scale self-supervised training,” in Proceedings of ICLR, 2024
2024
-
[15]
SPICE: Self- supervised pitch estimation,
B. Gfeller, C. Frank, D. Roblek, M. Sharifi, M. Tagliasacchi, and M. Velimirovi ´c, “SPICE: Self- supervised pitch estimation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP) , vol. 28, pp. 1118–1128, 2020
2020
-
[16]
PESTO: Pitch estimation with self-supervised transposition-equivariant objective,
A. Riou, S. Lattner, G. Hadjeres, and G. Peeters, “PESTO: Pitch estimation with self-supervised transposition-equivariant objective,” in Proceedings of ISMIR, 2023
2023
-
[17]
Toward fully self- supervised multi-pitch estimation,
F. Cwitkowitz and Z. Duan, “Toward fully self- supervised multi-pitch estimation,” arXiv preprint arXiv:2402.15569, 2024
2024 arXiv
-
[18]
Deep salience representations for F0 estimation in polyphonic music,
R. M. Bittner, B. McFee, J. Salamon, P. Li, and J. P. Bello, “Deep salience representations for F0 estimation in polyphonic music,” in Proceedings of ISMIR, 2017
2017
-
[19]
A lightweight instrument- agnostic model for polyphonic note transcription and multipitch estimation,
R. M. Bittner, J. J. Bosch, D. Rubinstein, G. Meseguer- Brocal, and S. Ewert, “A lightweight instrument- agnostic model for polyphonic note transcription and multipitch estimation,” in Proceedings of ICASSP , 2022
2022
-
[20]
Comparing deep models and evaluation strategies for multi-pitch estimation in music recordings,
C. Weiß and G. Peeters, “Comparing deep models and evaluation strategies for multi-pitch estimation in music recordings,” IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP) , vol. 30, pp. 2814–2827, 2022
2022
-
[21]
Timbre-Trap: A low-resource framework for instrument-agnostic music transcription,
F. Cwitkowitz, K. W. Cheuk, W. Choi, M. A. Martínez- Ramírez, K. Toyama, W.-H. Liao, and Y . Mitsu- fuji, “Timbre-Trap: A low-resource framework for instrument-agnostic music transcription,” in Proceed- ings of ICASSP, 2024
2024
-
[22]
A matlab toolbox for efficient perfect reconstruction time-frequency transforms with log- frequency resolution,
C. Schörkhuber, A. Klapuri, N. Holighaus, and M. Dörfler, “A matlab toolbox for efficient perfect reconstruction time-frequency transforms with log- frequency resolution,” in Proceedings of AES, 2014
2014
-
[23]
Jazz bass transcription us- ing a U-net architecture,
J. Abeßer and M. Müller, “Jazz bass transcription us- ing a U-net architecture,” Electronics, vol. 10, no. 6, p. 670, 2021
2021
-
[24]
Im- proving perceptual quality of drum transcription with the expanded groove MIDI dataset,
L. Callender, C. Hawthorne, and J. Engel, “Im- proving perceptual quality of drum transcription with the expanded groove MIDI dataset,” arXiv preprint arXiv:2004.00188, 2020
2004 arXiv
-
[25]
Creating a multitrack classical music performance dataset for multimodal music analysis: Challenges, in- sights, and applications,
B. Li, X. Liu, K. Dinesh, Z. Duan, and G. Sharma, “Creating a multitrack classical music performance dataset for multimodal music analysis: Challenges, in- sights, and applications,” IEEE Transactions on Multi- media, vol. 21, pp. 522–535, 2018
2018
-
[26]
Decoupled weight decay regularization,
I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proceedings of ICLR, 2019
2019
-
[27]
Multiple fundamen- tal frequency estimation by modeling spectral peaks and non-peak regions,
Z. Duan, B. Pardo, and C. Zhang, “Multiple fundamen- tal frequency estimation by modeling spectral peaks and non-peak regions,” IEEE Transactions on Audio, Speech, and Language Processing (TASLP) , vol. 18, no. 8, pp. 2121–2133, 2010
2010
-
[28]
Escaping from the abyss of manual annotation: New methodology of building polyphonic datasets for automatic music transcription,
L. Su and Y .-H. Yang, “Escaping from the abyss of manual annotation: New methodology of building polyphonic datasets for automatic music transcription,” in Proceedings of CMMR, 2015
2015
-
[29]
High quality musical audio source sepa- ration,
J. Fritsch, “High quality musical audio source sepa- ration,” Master’s thesis, UPMC / IRCAM / Telécom ParisTech, 2012
2012
-
[30]
Learn- ing features of music from scratch,
J. Thickstun, Z. Harchaoui, and S. Kakade, “Learn- ing features of music from scratch,” in Proceedings of ICLR, 2017
2017
-
[31]
GuitarSet: A dataset for guitar transcription,
Q. Xi, R. M. Bittner, J. Pauwels, X. Ye, and J. P. Bello, “GuitarSet: A dataset for guitar transcription,” in Pro- ceedings of ISMIR, 2018
2018
-
[32]
mir_eval: A trans- parent implementation of common MIR metrics,
C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang, and D. P. Ellis, “mir_eval: A trans- parent implementation of common MIR metrics,” in Proceedings of ISMIR, 2014
2014
-
[33]
MedleyDB: A multitrack dataset for annotation-intensive MIR research
R. M. Bittner, J. Salamon, M. Tierney, M. Mauch, C. Cannam, and J. P. Bello, “MedleyDB: A multitrack dataset for annotation-intensive MIR research.” inPro- ceedings of ISMIR, 2014
2014
-
[34]
Improving music transcription by pre-stacking a U-Net,
F. Pedersoli, G. Tzanetakis, and K. M. Yi, “Improving music transcription by pre-stacking a U-Net,” in Pro- ceedings of ICASSP, 2020
2020
-
[35]
A framework for invertible, real-time constant-Q transforms,
N. Holighaus, M. Dörfler, G. A. Velasco, and T. Grill, “A framework for invertible, real-time constant-Q transforms,” IEEE Transactions on Audio, Speech, and Language Processing (TASLP) , vol. 21, pp. 775–785, 2012
2012
-
[36]
Neural audio synthesis of musical notes with wavenet autoencoders,
J. Engel, C. Resnick, A. Roberts, S. Dieleman, M. Norouzi, D. Eck, and K. Simonyan, “Neural audio synthesis of musical notes with wavenet autoencoders,” in Proceedings of ICML, 2017
2017
-
[37]
FMA: A dataset for music analysis,
M. Defferrard, K. Benzi, P. Vandergheynst, and X. Bresson, “FMA: A dataset for music analysis,” in Proceedings of ISMIR, 2017
2017
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.