Pith. sign in

REVIEW 4 major objections 5 minor 32 references

Detecting Music Performance Errors with Transformers

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read Polytune is an end-to-end transformer that detects music errors by comparing score and performance audio without explicit alignment.

desk verdict Solid engineering, but the headline F1 is measured entirely on synthetic test sets from the same pipeline that made the training data; the 40-point gain is real on that distribution, not yet on real performances. read the letter →

arxiv 2501.02030 v1 pith:U7NKNJFR submitted 2025-01-03 cs.SD cs.AIeess.AS

classification cs.SDcs.AIeess.AS
keywords musicerrordetectiontransformerend-to-endlearningscore-informedassessmentsyntheticdatagenerationMIDI-DDSPmulti-instrumentdynamictimewarping
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Music error detection has usually meant transcribing the performance, aligning it to the score with dynamic time warping, and then comparing the two; the paper argues that all of that can be replaced by a single end-to-end transformer. Polytune takes two spectrograms as input—one rendered from the reference score and one from the student's performance—and directly decodes an annotated score in which every note is labeled correct, missed, or extra. The implicit alignment learned in the network removes the failure mode where DTW misaligns notes when a performance contains wrong or extra notes. To get enough training data, the authors inject synthetic errors into existing MIDI datasets and render the results through a neural synthesizer, producing datasets with hundreds of hours of audio across 14 instruments. On those datasets, Polytune reaches a 64.1% average Error Detection F1, about 40 percentage points higher than the reimplemented alignment-based baselines.

What carries the argument

The mechanism is a dual-encoder transformer. Two Audio Spectrogram Transformer (AST) encoders process the two input spectrograms separately—one for score audio, one for performance audio—and their outputs are concatenated and fed through a joint encoder, with a T5-style decoder generating tokens via greedy autoregressive sampling. The token vocabulary is MIDI-like (time, on/off, pitch) with the addition of a Label token for each note, so each decoded event is explicitly Correct, Missed, or Extra. Training data come from Algorithm 1, which injects errors into MIDI files with a Poisson-selected note rate and truncated-normal offsets for pitch and time, then synthesizes audio with MIDI-DDSP; a weighted cross-entropy loss with weight 10 on error tokens handles the class imbalance. This combination lets the model learn alignment implicitly rather than through a separate DTW stage.

What would settle it

Gather recordings of real beginner students playing from the same scores, have expert musicians label every note as correct, missed, or extra, and run the trained Polytune model on those recordings without fine-tuning. If its missed-note and extra-note F1 scores fall to roughly 30% or below, or below the DTW baseline, the claim that the method generalizes beyond synthetic data would be refuted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that explicit alignment is not a necessary component of music error detection. Polytune learns to compare the score audio and performance audio in a joint latent space, and its output vocabulary is the familiar MIDI-like token sequence augmented with three label tokens: Correct, Missed, and Extra. Averaged over 14 instruments, the model reports F1 scores of 95.0% for correct notes, 49.2% for missed notes, and 48.0% for extra notes, with an overall Error Detection F1 of 64.1% that beats the reimplemented DTW-based baselines by 40 percentage points. The same model, without instrument-specific tokens, works on all tested instruments, which the authors present as evidence that a single end-to-end transformer can serve as a general performance-error annotator.

Load-bearing premise

The load-bearing assumption is that the synthetic pipeline—errors injected into MIDI and rendered with one neural synthesizer per instrument—produces performances that are representative enough of real student mistakes; if real errors differ in timbre, timing, or type, the reported F1 scores will not transfer.

Editorial extensions

If this is right

  • An end-to-end transformer can annotate correct, missed, and extra notes without an explicit alignment stage, so alignment-induced misclassifications disappear from the pipeline.
  • The synthetic error injection algorithm turns existing MIDI transcription datasets into large error-detection datasets, removing the data scarcity that previously limited the task to a handful of tracks.
  • Because the output vocabulary is instrument-agnostic, one trained model can detect errors across 14 instruments rather than requiring a separate system per instrument.
  • Since the inputs are raw spectrograms, the same architecture can be extended to timing or dynamics errors without changing the input representation.
  • The reported 40-point average F1 improvement over reimplemented DTW baselines indicates that learned implicit alignment can outperform explicit time warping on this task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the dual-encoder 'reference audio versus query audio' design is generic; the same approach could be adapted to singing assessment, pronunciation tutoring, or any task where one recording is compared to a reference recording.
  • Editorial inference: the large gap between correct-note F1 (95.0%) and missed-note F1 (49.2%) suggests the model is much better at verifying played notes than at detecting omissions; a score-aware auxiliary objective might close that gap.
  • Editorial inference: because the error-injection distribution defines what counts as an error, real student errors with different statistics (such as rhythmic anticipation or repeated notes) could require retraining with additional error types rather than transferring directly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Polytune, an end-to-end transformer model for detecting performance errors in music. The model takes two audio spectrogram streams as input—one synthesized from a reference score and one from the student performance—and outputs MIDI-like tokens annotated as Correct, Missed, or Extra. The authors introduce Algorithm 1, which injects synthetic errors (missed notes, pitch changes, timing shifts, extra notes) into existing MIDI datasets, and they render the augmented files with MIDI-DDSP to create MAESTRO-E and CocoChorales-E. They compare Polytune against a reimplemented MT3-plus-DTW baseline on 4,401 synthetic test tracks, reporting average Error Detection F1 of 64.1%, roughly 40 percentage points above the baseline. The paper claims that Polytune achieves state-of-the-art music error detection without an explicit alignment stage.

Significance. If the claims hold, the work is significant: it demonstrates a learned, implicit alignment mechanism for score-informed error detection, scales training data through synthetic error injection, and extends error detection to 14 instruments. The public release of code and datasets is a genuine strength, as is the detailed description of the architecture and data generation pipeline. However, the significance is conditional on whether the synthetic pipeline transfers to real performances. The evaluation is entirely on renderings produced by the same MIDI-DDSP pipeline used for training, and the baseline is a self-reimplementation rather than original code or published results. These two issues mean that the headline numbers currently establish performance on a synthetic distribution, not on the real student-practice setting the paper motivates. The central idea is promising and the main missing piece—real-data validation or a clear reframing of the claims—is addressable.

major comments (4)
  1. [§3.3 and §5] The evaluation is restricted to synthetic test sets generated by the same pipeline (Algorithm 1 and MIDI-DDSP synthesis) used to create the training data, and Section 5 concedes that using only one synthesizer per instrument may restrict generalization. Consequently, the claims of 'state-of-the-art music error detection' and the 40-percentage-point improvement are, as presented, claims about MIDI-DDSP renderings rather than real performances. The paper should either validate on real recordings (e.g., the small Benetos et al. dataset or newly recorded student performances with annotations) or substantially narrow the stated claims to synthetic benchmarks. This is the load-bearing point for the paper's application setting.
  2. [§3.4 and Tables 3–4] The baseline is a new reimplementation—MT3 with DTW—not the original code or results from Benetos et al. or Wang et al., and no evidence is provided that the reimplementation reproduces or approximates the published methods. Because the headline '40 percentage point improvement' is computed relative to this self-reimplementation, the comparison is weakened. Please report the baseline's hyperparameters, MT3 initialization, DTW settings, and, if possible, validation against the original published approaches on a shared dataset.
  3. [§4.1 and §4.2] No error bars, confidence intervals, or multi-seed runs are reported. The F1 differences, especially for Missed and Extra classes, could be sensitive to random dataset generation, model initialization, or decoding randomness. Please report variance across at least a few training runs or bootstrap confidence intervals on the test set, and indicate how many tracks or notes each F1 value is based on.
  4. [§3.3, Algorithm 1 and Output Data] Algorithm 1 injects pitch-change (wrong note) and timing-shift errors, but the output vocabulary in Table 2 contains only Correct, Missed, and Extra labels. The paper should explain how pitch-change and timing-shift errors are converted into these label tokens and how they are counted in the evaluation. Without this operationalization, the 'wrong note' behavior described in Figure 1 is not directly measurable from the reported metrics.
minor comments (5)
  1. [Abstract] The phrase 'alignment targets.;' contains a stray semicolon and should be corrected.
  2. [§3.3] The text reads 'two truncated normal distribution distributions' and should be 'two truncated normal distributions.'
  3. [Table 1] The row label 'Y ousician' contains an extra space; it should read 'Yousician.'
  4. [Table 1 and §3.4] Table 1 lists a 'Combined' baseline, but Section 3.4 does not define how this baseline combines Benetos et al. and Wang et al. Please add a definition or remove the entry.
  5. [Figures 3 and 5] The captions refer to 'Music note 1' and 'Music note 2' and to specific pitches such as 'A' and 'C#', but the figures do not clearly mark these notes; please add explicit labels to the figures.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Polytune's synthetic-data evaluation is a genuine held-out generalization measure, not a fitted input renamed as a prediction.

full rationale

Polytune's error detection is trained and tested on CocoChorales-E and MAESTRO-E, with both train and test examples produced by Algorithm 1. This shared generative pipeline raises external-validity concerns (the paper itself notes the use of only one synthesizer per instrument in Sec. 5), but it is not a circular derivation: the model never receives the error-injection parameters at inference, the test set is disjoint from training, and the reported F1 is a genuine generalization measure on that distribution. The baseline is an upgraded re-implementation of prior work, not an input to Polytune. No load-bearing self-citation, imported uniqueness, renamed known result, or fitted-parameter-relabeled-as-prediction appears. The central comparison is self-contained against the synthetic benchmarks, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central results rest on synthetic audio and error-generation assumptions, plus a self-built baseline. The model architecture itself uses standard transformer components, so no new physical or mathematical entities are introduced.

free parameters (5)
  • Error rate lambda = sampled from U(0.1, 0.4)
    Used in Algorithm 1 to decide which notes receive an error; chosen by hand, not fitted to real data.
  • Pitch offset distribution P = truncated normal, mean 0, SD 1
    Controls pitch changes and extra-note pitch offsets in Algorithm 1; chosen to simulate errors.
  • Timing offset distribution Q = truncated normal, mean 0, SD 0.02
    Controls timing shifts and extra-note time offsets in Algorithm 1; chosen to simulate errors.
  • Segment length = 2.145 seconds
    Non-overlapping audio segment length used for inputs; chosen without ablation.
  • Error class weight alpha = 10
    Weight for error tokens in the cross-entropy loss (Equation 1); chosen to address class imbalance.
assumptions (3)
  • domain assumption MIDI-DDSP synthesized audio adequately represents real instrument performances for error detection.
    Used to create both training and test audio; Section 3.3 and limitation in Section 5.
  • domain assumption Algorithm 1's Poisson selection and truncated normal offsets produce realistic performance errors.
    Algorithm 1 and surrounding text; no real error corpus is used for validation.
  • domain assumption The reimplemented MT3 plus DTW baseline fairly represents prior score-informed error detection systems.
    Section 3.4 replaces original components with MT3 and DTW; original implementations are not run.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Detecting Music Performance Errors with Transformers." pith.science (2026). https://pith.science/paper/U7NKNJFR

@misc{pith2026250102030,
  author       = {Pith},
  title        = {Pith review of: Detecting Music Performance Errors with Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U7NKNJFR}},
  note         = {Machine review of arXiv:2501.02030}
}
read the original abstract

Beginner musicians often struggle to identify specific errors in their performances, such as playing incorrect notes or rhythms. There are two limitations in existing tools for music error detection: (1) Existing approaches rely on automatic alignment; therefore, they are prone to errors caused by small deviations between alignment targets.; (2) There is a lack of sufficient data to train music error detection models, resulting in over-reliance on heuristics. To address (1), we propose a novel transformer model, Polytune, that takes audio inputs and outputs annotated music scores. This model can be trained end-to-end to implicitly align and compare performance audio with music scores through latent space representations. To address (2), we present a novel data generation technique capable of creating large-scale synthetic music error datasets. Our approach achieves a 64.1% average Error Detection F1 score, improving upon prior work by 40 percentage points across 14 instruments. Additionally, compared with existing transcription methods repurposed for music error detection, our model can handle multiple instruments. Our source code and datasets are available at https://github.com/ben2002chou/Polytune.

Figures

Figures reproduced from arXiv: 2501.02030 by the authors.

Figure 1
Figure 1. The score on top is the performance transcription [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of differences between (a) previous [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Deficiency of Dynamic Time Warping (DTW): DTW encounters challenges when aligning complex sequences of [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Architecture of Polytune. The diagram illustrates the process flow starting with the Score and Performance Audio inputs, each processed through dedicated AST encoders. These encoded features are concatenated and passed through a joint encoder and a decoder with cross-a…
Figure 5
Figure 5. Figure 5: Qualitative Comparison of MIDI Note Events: [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 29 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Akbari, H.; Chuang, W.-H.; Yuan, L.; Chang, S.-F.; Gong, B.; Qian, R.; and Cui, Y. 2021. VATT : Transformers for Multimodal Self - Supervised Learning from Raw Video , Audio and Text . In 35th Conference on Neural Information Processing Systems

  4. [4]

    Apaydınlı, K. 2019. Intelligent Tutoring Systems in Music Education . In Proceedings of The 2nd International Conference on Future of Teaching and Education . GLOBALKS. ISBN 978-609-485-043-1

  5. [5]

    Benetos, E.; Klapuri, A.; and Dixon, S. 2012. Score-informed transcription for automatic piano tutoring. Proceedings of the European Signal Processing Conference (EUSIPCO), pp. 2153--2157

  6. [6]

    Britannica, T. E. o. E. 2007. Homophony

  7. [7]

    Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021. An Image is Worth 16x16 Words : Transformers for Image Recognition at Scale . In Proceedings of the International Conference on Learning Representations

  8. [8]

    Ewert, S.; Wang, S.; and Sandler, M. 2016. Score-informed Identification of Missing and Extra Notes in Piano Recordings . Proceedings of the International Conference on Music Information Retrieval (ISMIR)

Show all 32 references
  1. [9]

    Fradet, N.; Briot, J.-P.; and Chhel, F. 2021. MidiTok : A Python package for MIDI file tokenization. Extended Abstracts for the Late-Breaking Demo Session of the 22nd International Society for Music Information Retrieval Conference

  2. [10]

    Gardner, J.; Simon, I.; Manilow, E.; Hawthorne, C.; and Engel, J. 2022. MT3 : Multi - Task Multitrack Music Transcription . International Conference on Learning Representations

  3. [11]

    Gong, Y.; Chung, Y.-A.; and Glass, J. 2021. AST : Audio Spectrogram Transformer . In Interspeech 2021, 571--575. ISCA

  4. [12]

    H.; Harwath, D.; Karlinsky, L.; Kuehne, H.; and Glass, J

    Gong, Y.; Rouditchenko, A.; Liu, A. H.; Harwath, D.; Karlinsky, L.; Kuehne, H.; and Glass, J. 2023. Contrastive Audio - Visual Masked Autoencoder . In Proceedings of the International Conference on Learning Representations ( ICLR )

  5. [13]

    Hawthorne, C.; Simon, I.; Swavely, R.; Manilow, E.; and Engel, J. 2021. Sequence-to-sequence piano transcription with transformers. In Proceedings of the 22nd ISMIR Conference

  6. [14]

    A.; Dieleman, S.; Elsen, E.; Engel, J.; and Eck, D

    Hawthorne, C.; Stasyuk, A.; Roberts, A.; Simon, I.; Huang, C.-Z. A.; Dieleman, S.; Elsen, E.; Engel, J.; and Eck, D. 2018. Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset . In Proceedings of the International Conference on Learning Representations

  7. [15]

    Hsiao, W.-Y.; Liu, J.-Y.; Yeh, Y.-C.; and Yang, Y.-H. 2021. Compound Word Transformer : Learning to Compose Full - Song Music over Dynamic Directed Hypergraphs . Proceedings of the AAAI Conference on Artificial Intelligence, 35(1): 178--186

  8. [16]

    Huang, P.-Y.; Xu, H.; Li, J.; Baevski, A.; Auli, M.; Galuba, W.; Metze, F.; and Feichtenhofer, C. 2022. Masked Autoencoders that Listen . NeurIPS

  9. [17]

    Huang, Y.-S.; and Yang, Y.-H. 2020. Pop Music Transformer : Beat -based Modeling and Generation of Expressive Pop Piano Compositions . In Proceedings of the 28th ACM International Conference on Multimedia , 1180--1188. Seattle WA USA: ACM. ISBN 978-1-4503-7988-5

  10. [18]

    JoyTunes. 2024. Simply Piano

  11. [19]

    B.; McCormick, P.; Shepherd, J

    Morrison, R. B.; McCormick, P.; Shepherd, J. L.; and Cirillo, P. 2022. National Arts Education Status Report Summary 2019. Technical report, Arts Education Data Project, Quadrant Research, State Education Agency Directors of Arts Education

  12. [20]

    Nart, S. 2016. Music Software in the Technology Integrated Music Education . The Turkish Online Journal of Educational Technology, 15(2)

  13. [21]

    Oore, S.; Simon, I.; Dieleman, S.; Eck, D.; and Simonyan, K. 2020. This time with feeling: learning expressive musical performance. Neural Computing and Applications, 32(4): 955--967

  14. [22]

    J.; Salamon, J.; Nieto, O.; Liang, D.; and Ellis, D

    Raffel, C.; McFee, B.; Humphrey, E. J.; Salamon, J.; Nieto, O.; Liang, D.; and Ellis, D. P. W. 2014. A TRANSPARENT IMPLEMENTATION OF COMMON MIR METRICS . The International Society for Music Information Retrieval (ISMIR)

  15. [23]

    Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020. Exploring the Limits of Transfer Learning with a Unified Text -to- Text Transformer . Journal of Machine Learning Research

  16. [24]

    Sakoe, H.; and Chiba, S. 1978. Dynamic programming algorithm optimization for spoken word recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing, 26(1): 43--49

  17. [25]

    J.; Price, A.; and Taylor, J

    Tibshirani, R. J.; Price, A.; and Taylor, J. 2011. A statistician Plays Darts . Journal of the Royal Statistical Society Series A: Statistics in Society, 174(1): 213--226

  18. [26]

    T.; Landy, M

    Trommershäuser, J.; Gepshtein, S.; Maloney, L. T.; Landy, M. S.; and Banks, M. S. 2005. Optimal Compensation for Changes in Task - Relevant Movement Variability . The Journal of Neuroscience, 25(31): 7169--7178

  19. [27]

    N.; Kaiser, .; and Polosukhin, I

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention is All you Need . In Advances in Neural Information Processing Systems 30 ( NIPS 2017)

  20. [28]

    Wang, S.; Ewert, S.; and Dixon, S. 2017. Identifying Missing and Extra Notes in Piano Recordings Using Score - Informed Dictionary Learning . IEEE/ACM Transactions on Audio, Speech, and Language Processing, 25(10): 1877--1889

  21. [29]

    Wu, Y.; Gardner, J.; Manilow, E.; Simon, I.; Hawthorne, C.; and Engel, J. 2022 a . The Chamber Ensemble Generator : Limitless High - Quality MIR Data via Generative Modeling . ArXiv:2209.14458 [cs, eess]

  22. [30]

    A.; and Engel, J

    Wu, Y.; Manilow, E.; Deng, Y.; Swavely, R.; Kastner, K.; Cooijmans, T.; Courville, A.; Huang, C.-Z. A.; and Engel, J. 2022 b . MIDI - DDSP : Detailed Control of Musical Performance via Hierarchical Modeling . In Proceedings of the International Conference on Learning Representations

  23. [31]

    Ying, X. 2019. An Overview of Overfitting and its Solutions . Journal of Physics: Conference Series, 1168: 022022

  24. [32]

    Yousician. 2024. Yousician

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.