REVIEW 4 major objections 4 minor 40 references
RUMAA: Repeat-Aware Unified Music Audio Analysis for Score-Performance Alignment, Transcription, and Mistake Detection
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read RUMAA claims that repeat-aware score-performance alignment can hold at 98.4 F1 even when conventional symbolic alignment methods drop by up to 87 points, by jointly decoding performance, score, and edit-operation tokens in one transformer.
desk verdict Solid unified MIR architecture with genuinely new repeat handling, but the headline repeat advantage is built on an unfair baseline comparison and two test pieces. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the tri-stream token sequence generated by a six-block autoregressive transformer decoder, conditioned hierarchically with audio cross-attention first and then score cross-attention, on frozen pre-trained encoders: a bar-patched ABC score encoder and a 12-frames-per-second audio encoder. Three proxy tasks share one latent space: T1 transcribes the performance aligned to the score, T2 converts the score following performance order, and T3 tags each note as match, insert, delete, or repeat. The placeholder alignment token and the repeat token carry the argument, because they let the model represent a one-to-one path through the music even when the performance goes back and repeats a section.
What would settle it
Run RUMAA on a held-out set of real piano performances whose scores contain written repeats, with human-verified repeat-unfolded ground truth; if the F_align drops materially on repeated sections, or if human listeners disagree with the adapted metric's independent counting of repeated notes, the central repeat-handling claim is not general.
Extended reading notes
Core claim
RUMAA is built around a tri-stream decoder that simultaneously emits, for each aligned position, a performance token, a score token, and an edit-operation token chosen from <Match>, <Insert>, <Delete>, or <Repeat>, with a special placeholder for notes present in only one modality. This converts repeat handling from an external structure problem into a sequence problem: the score encoder sees MusicXML with repeat symbols and barlines, and the decoder learns, through cross-attention to audio and score, to follow the performed structure rather than the unfolded written order. The reported result is that alignment accuracy on notes is 98.4 F1 both with and without repeats in the Vienna piano corpus, while HMM, DTW, and GlueNote baselines fall to 36.4, 28.2, and 12.7 when repeats appear. The same unified model also outperforms prior NMF systems on detecting extra and missed notes, and score-informed transcription reaches a 99.1 onset F1 on the (n)ASAP test set.
Load-bearing premise
The model's repeat handling is learned entirely from artificially inserted repeat signs and repeated audio in 20% of training scores, and it must transfer to real performances with repeats; if real repeat execution differs enough, the alignment advantage on repeated scores collapses.
Editorial extensions
If this is right
- If the central claim holds, alignment systems no longer need repeat-unfolded score-MIDI; a score with repeat symbols can be used directly, which removes a manual preprocessing step.
- Because transcription and alignment share one decoder, score-informed transcription becomes a by-product of alignment, reaching near-perfect onset detection (99.1 F1) on (n)ASAP.
- Mistake detection is expressed as insert and delete edits in the same token stream, so it no longer requires a separate post-hoc method.
- The uniform 98.4 F1 on both non-repeat and repeat scores implies the model follows performed order locally rather than relying on a global unfolded structure.
Reading between the lines
- We infer that the repeat handling is only as general as the synthetic augmentation: repeat signs were added to 20% of training scores with audio literally repeated, so real performances with ornaments, tempo changes, or first and second endings are an untested and likely harder case.
- The adapted alignment metric counts repeated notes independently, which may systematically favor systems that always repeat; a human-listening evaluation on repeated sections would clarify whether the 98.4 reflects musical correspondence or a metric bias.
- A natural extension is to use the same edit-operation token stream for expressive performance annotation, such as tempo rubato, dynamics, or articulation deviations, since timing and velocity tokens already encode those dimensions.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. RUMAA is a transformer-based framework that unifies score-to-performance alignment, score-informed transcription, and mistake detection for piano music. It encodes MusicXML scores as bar-level ABC patches with a pre-trained M3 encoder, encodes audio with a frozen YourMT3+ pre-trained audio encoder, and uses a six-layer decoder that autoregressively emits three aligned token streams: performance tokens, score tokens, and edit-operation tokens. The decoder is trained on (n)ASAP with data augmentation that includes synthetic repeat insertion, and evaluated on the Vienna corpus for alignment, (n)ASAP/Maestro for transcription, and STPD for mistake detection. The main claims are that RUMAA matches state-of-the-art alignment methods on non-repeated scores, outperforms them on scores with repeat symbols without requiring manually unfolded score-MIDI, achieves near-perfect score-informed transcription onset F1, and improves mistake detection over prior NMF-based systems.
Significance. If the repeat-aware alignment result is confirmed, RUMAA addresses a real practical bottleneck in music performance analysis: conventional alignment tools require manually unfoldable score-MIDI with repeat structures pre-specified, while RUMAA processes MusicXML scores with repeats directly. The unified multi-task formulation is original and the use of pretrained, independently benchmarked encoders is a strength. The paper also avoids circularity by training on separate public corpora and holding out its evaluation sets. However, the paper's headline repeat advantage is currently supported by only two test pieces and an unfair baseline protocol, and the transfer from synthetic repeat augmentation to real repeated performances is unvalidated. Because the central advertised contribution rests on this comparison, the evidence as presented is not yet sufficient; with a corrected baseline protocol and a broader evaluation, the contribution would be solid and of clear value to the MIR community.
major comments (4)
- [Section 6.1, Table 2] The repeat-condition comparison is not apples-to-apples. The caption and text state that in the 'w/ repeat' setting all methods receive the original scores with repeat symbols, but Nakamura HMM, hDTW+sym, and GlueNote are symbolic aligners that require repeat-unfolded score-MIDI as input, and the paper itself notes that they 'cannot interpret repeat symbols.' Their collapses to 36.4, 28.2, and 12.7 may therefore reflect an unusable input format rather than inferior alignment quality. To support the abstract claim that RUMAA 'outperforms them on scores with repeats,' please add an oracle-unfolded condition in which these baselines are given manually repeat-unfolded score-MIDI on the same two pieces, and report F_align under that protocol. RUMAA's claimed advantage is only established if it remains competitive or superior when baselines receive the input representation they were designed to consume.
- [Section 5.1, Table 2] The repeat-aware result is statistically thin and its transfer is not demonstrated. Only two pieces are evaluated (Mozart K331 and Schubert D783), with no error bars, no per-piece results, and no significance testing. Moreover, the model's repeat behavior is learned solely from synthetic augmentation: repeat symbols are added to random bars in 20% of ABC scores lacking repeats, and the corresponding audio is repeated verbatim. Real repeated performances can contain ornaments, tempo changes, first/second endings, and expressive variation, none of which appear in this training signal. Please evaluate on a larger held-out set of pieces with real repeats, or at least report per-piece and bootstrapped intervals for the current two pieces, and include an ablation that withholds repeat-augmented training data to show that the repeat handling does not come from the score encoder alone.
- [Section 5.2] The adapted F_align metric is underspecified. The paper says repeated notes are 'redefined to be counted independently,' but this is precisely the point at which a repeat-aware model and a symbolic aligner can differ; without a precise formula for how repeated notes are matched and counted, it is unclear whether the reported 98.4 on repeat pieces reflects alignment quality or a scoring choice that favors RUMAA's token design. Please state the exact matching criterion for repeated notes and report both the original F_align and the adapted F_align on the same data.
- [Section 6.2, Table 3] The score-informed transcription claim is not supported by the current comparison. The 'Score-informed' row for RUMAA is compared only with score-free baselines (MT3, YourMT3+, hFT-T, and IS-CRF on Maestro or (n)ASAP); no score-informed baseline is included. The statement that RUMAA 'surpasses all baselines by a clear margin' therefore overstates what the table shows. Please add score-informed baselines or revise the claim to state that RUMAA improves over score-free models when score information is provided.
minor comments (4)
- [Figure 2] The label 'Charater-level Tokenizer' contains a typo and should read 'Character-level Tokenizer.'
- [Table 2] Please clarify whether the 'w/o repeat' row includes the two repeat-containing pieces in unfolded form or excludes them; the current description 'all songs from Vienna' is ambiguous and directly affects how the within-row comparison should be read.
- [Section 6.1, Table 2] The prose says that on repeated scores symbolic aligners 'drop by up to 87%' and that symbolic-audio aligners 'drop by up to 87%' and later 'by up to 70%; these numbers should be stated consistently and tied explicitly to the rows in Table 2.
- [Section 7] The limitation of one-minute audio segments is acknowledged, but since the repeat evaluation uses full movements such as Mozart K331, please describe how longer audio is chunked and how repeat handling is maintained across chunk boundaries; otherwise readers cannot judge whether the reported F_align is artificially segment-local.
Circularity Check
No significant circularity: RUMAA's central claims rest on held-out external benchmarks, and the only self-citation (YourMT3+ encoder) is component reuse with independent public validation, not a load-bearing derivation.
full rationale
The paper's claimed derivation chain is an empirical training and evaluation loop, not a formal derivation whose conclusion is embedded in its premises. RUMAA is post-trained on external datasets: the decoder is trained on (n)ASAP alignment, MIDI, and score data; score-free transcription is tested on Maestro; mistake detection is tested on STPD; and alignment is evaluated on the Vienna corpus. All evaluation datasets are isolated from training data, so the reported F_align results, including the 98.4 score on repeated pieces, are genuine held-out measurements rather than quantities reconstructed from training inputs. The repeat-aware behavior is learned from synthetic repeat augmentation, where repeat symbols are added to random bars in 20% of scores lacking repeats and the audio is repeated, and is then evaluated on real performances containing repeat signs. Training and test repeat data are not the same, so no fitted-input-called-prediction pattern arises. The reuse of the pre-trained audio encoder from the authors' YourMT3+ paper is the only notable self-citation, but it is employed as a frozen component with independent public benchmarks and is not optimized against this paper's test sets; it is therefore component transfer rather than load-bearing circular evidence. The adaptation of F_align to count repeated notes independently is a definitional metric choice, not a derivation that presupposes the target result; any concern about bias belongs to metric validity, not circularity. Similarly, the observation that MIDI-based baselines were given original scores with repeat symbols they cannot parse is an evaluation-protocol fairness issue, not a circularity in RUMAA's own derivation. No equation or architectural step defines the target output in terms of itself, and no uniqueness theorem or ansatz is imported from the authors' prior work to force the model choice. Accordingly, no circular step meeting the evidentiary standard is present; the score reflects only a minor, non-load-bearing self-citation.
Assumptions & free parameters
free parameters (3)
- Repeat augmentation ratio =
20%
- Note alteration ratio for mistake augmentation =
10%
- Audio pre-encoder frame rate =
12 Hz
assumptions (3)
- domain assumption (n)ASAP manual note alignments are reliable ground truth
- ad hoc to paper Synthetic repeat insertion simulates real repeated performances
- domain assumption Frozen pre-trained encoders provide sufficient representations for alignment and transcription
Cite this review
Pith. "Pith review of RUMAA: Repeat-Aware Unified Music Audio Analysis for Score-Performance Alignment, Transcription, and Mistake Detection." pith.science (2026). https://pith.science/paper/AD6ON2HK
@misc{pith2026250712175,
author = {Pith},
title = {Pith review of: RUMAA: Repeat-Aware Unified Music Audio Analysis for Score-Performance Alignment, Transcription, and Mistake Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/AD6ON2HK}},
note = {Machine review of arXiv:2507.12175}
}
read the original abstract
This study introduces RUMAA, a transformer-based framework for music performance analysis that unifies score-to-performance alignment, score-informed transcription, and mistake detection in a near end-to-end manner. Unlike prior methods addressing these tasks separately, RUMAA integrates them using pre-trained score and audio encoders and a novel tri-stream decoder capturing task interdependencies through proxy tasks. It aligns human-readable MusicXML scores with repeat symbols to full-length performance audio, overcoming traditional MIDI-based methods that rely on manually unfolded score-MIDI data with pre-specified repeat structures. RUMAA matches state-of-the-art alignment methods on non-repeated scores and outperforms them on scores with repeats in a public piano music dataset, while also delivering promising transcription and mistake detection results.
Reference graph
Works this paper leans on
-
[1]
Music performance analysis: A survey,
A. Lerch, C. Arthur, A. Pati, and S. Gururani, “Music performance analysis: A survey,” in International Society for Music Information Retrieval Conference (ISMIR) , 2019
work page 2019
-
[2]
How does the teacher rate? observations from the neuropiano dataset,
H. Zhang, V . Cheung, H. Nishioka, S. Dixon, and S. Furuya, “How does the teacher rate? observations from the neuropiano dataset,” in International Society for Music Information Retrieval Conference (ISMIR) – Late Breaking Demo , 2024
work page 2024
-
[3]
Performance error detection and post-processing for fast and accurate symbolic music alignment
E. Nakamura, K. Yoshii, and H. Katayose, “Performance error detection and post-processing for fast and accurate symbolic music alignment.” in International Society for Music Information Retrieval Conference (ISMIR), 2017
work page 2017
-
[4]
Score-informed transcription for automatic piano tutoring,
E. Benetos, A. Klapuri, and S. Dixon, “Score-informed transcription for automatic piano tutoring,” in IEEE European Signal Processing Conference (EUSIPCO), 2012, pp. 2153–2157
work page 2012
-
[5]
M ¨uller, Fundamentals of music processing: Audio, analysis, algo- rithms, applications
M. M ¨uller, Fundamentals of music processing: Audio, analysis, algo- rithms, applications. Springer, 2015, vol. 5
work page 2015
-
[6]
Cross-modal music retrieval and applications: An overview of key methodologies,
M. M ¨uller, A. Arzt, S. Balke, M. Dorfer, and G. Widmer, “Cross-modal music retrieval and applications: An overview of key methodologies,” IEEE Signal Processing Magazine , vol. 36, no. 1, pp. 52–62, 2018
work page 2018
-
[7]
Structure-aware audio-to-score alignment using progressively dilated convolutional neural networks,
R. Agrawal, D. Wolff, and S. Dixon, “Structure-aware audio-to-score alignment using progressively dilated convolutional neural networks,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021
work page 2021
-
[8]
Exact indexing of dynamic time warping,
E. Keogh and C. A. Ratanamahatana, “Exact indexing of dynamic time warping,” Knowledge and information systems , vol. 7, pp. 358–386, 2005
work page 2005
Show all 40 references
-
[9]
Audio-to-score alignment using deep automatic music transcription,
F. Simonetta, S. Ntalampiras, and F. Avanzini, “Audio-to-score alignment using deep automatic music transcription,” in 2021 IEEE 23rd Inter- national Workshop on Multimedia Signal Processing (MMSP) . IEEE, 2021, pp. 1–6
2021
-
[10]
Automatic note-level score- to-performance alignments in the asap dataset,
S. D. Peter, C. E. Cancino-Chac ´on, F. Foscarin, A. P. McLeod, F. Henkel, E. Karystinaios, and G. Widmer, “Automatic note-level score- to-performance alignments in the asap dataset,” Transactions of the International Society for Music Information Retrieval (TISMIR) , 2023
2023
-
[11]
Handling repeats and jumps in score-performance synchronization
C. Fremerey, M. M ¨uller, and M. Clausen, “Handling repeats and jumps in score-performance synchronization.” in International Society for Music Information Retrieval Conference (ISMIR) , 2010
2010
-
[12]
Improved handling of repeats and jumps in audio-sheet image synchronization,
M. Shan and T. J. Tsai, “Improved handling of repeats and jumps in audio-sheet image synchronization,” in International Society for Music Information Retrieval Conference (ISMIR) , 2020
2020
-
[13]
Transformer-xl: Attentive language models beyond a fixed-length context,
Z. Dai, Z. Yang, Y . Yang, J. Carbonell, Q. V . Le, and R. Salakhutdinov, “Transformer-xl: Attentive language models beyond a fixed-length context,” arXiv preprint arXiv:1901.02860 , 2019
1901 arXiv
-
[14]
Thegluenote: Learned representations for robust and flexible note alignment,
S. D. Peter and G. Widmer, “Thegluenote: Learned representations for robust and flexible note alignment,” in Proceedings of the 25th International Society for Music Information Retrieval Conference (ISMIR), 2024
2024
-
[15]
Detecting music performance errors with transformers,
B. S.-H. Chou, P. Jajal, N. J. Eliopoulos, T. Nadolsky, C.-Y . Yang, N. Ravi, J. C. Davis, K. Y .-J. Yun, and Y .-H. Lu, “Detecting music performance errors with transformers,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2025
2025
-
[16]
Identifying missing and extra notes in piano recordings using score-informed dictionary learning,
S. Wang, S. Ewert, and S. Dixon, “Identifying missing and extra notes in piano recordings using score-informed dictionary learning,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 10, pp. 1877–1889, 2017
2017
-
[17]
Compound word transformer: Learning to compose full-song music over dynamic directed hypergraphs,
W.-Y . Hsiao, J.-Y . Liu, Y .-C. Yeh, and Y .-H. Yang, “Compound word transformer: Learning to compose full-song music over dynamic directed hypergraphs,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 1, 2021, pp. 178–186
2021
-
[18]
Mt3: Multi-task multitrack music transcription,
J. Gardner, I. Simon, E. Manilow, C. Hawthorne, and J. Engel, “Mt3: Multi-task multitrack music transcription,” in International Conference on Learning Representations (ICLR) , 2021
2021
-
[19]
Improvnet: Generating controllable musical improvisations with iterative corruption refinement,
K. Bhandari, S. Chang, T. Lu, F. R. Enus, L. B. Bradshaw, D. Her- remans, and S. Colton, “Improvnet: Generating controllable musical improvisations with iterative corruption refinement,” arXiv preprint arXiv:2502.04522, 2025
2025 arXiv
-
[20]
Pertok: Expressive encoding and modeling of symbolic musical ideas and variations,
J. Lenz and A. Mani, “Pertok: Expressive encoding and modeling of symbolic musical ideas and variations,” in International Society for Music Information Retrieval Conference (ISMIR) , 2024
2024
-
[21]
Scaling self-supervised representation learning for symbolic piano performance,
L. Bradshaw, H. Fan, A. Spangher, S. Biderman, and S. Colton, “Scaling self-supervised representation learning for symbolic piano performance,” arXiv preprint arXiv:2506.23869 , 2025
2025 arXiv
-
[22]
Yourmt3+: Multi- instrument music transcription with enhanced transformer architectures and cross-dataset stem augmentation,
S. Chang, E. Benetos, H. Kirchhoff, and S. Dixon, “Yourmt3+: Multi- instrument music transcription with enhanced transformer architectures and cross-dataset stem augmentation,” in 2024 IEEE International Workshop on Machine Learning for Signal Processing (MLSP) , 2024
2024
-
[23]
Musicxml 4.0,
W. M. N. C. Group, “Musicxml 4.0,” May 2021, accessed: 2025-03-27. [Online]. Available: https://www.w3.org/2021/06/musicxml40/
2021
-
[24]
A statistical analysis of the abc music notation corpus: Exploring duplication,
C. Walshaw, “A statistical analysis of the abc music notation corpus: Exploring duplication,” in International Workshop on Folk Music Analysis, A. Holzapfel, Ed., 2014
2014
-
[25]
Clamp 2: Multimodal music information retrieval across 101 languages using large language models,
S. Wu, Y . Wang, R. Yuan, Z. Guo, X. Tan, G. Zhang, M. Zhou, J. Chen, X. Mu, Y . Gao et al., “Clamp 2: Multimodal music information retrieval across 101 languages using large language models,” arXiv preprint arXiv:2410.13267, 2024
-
[26]
Music2latent: Consistency autoencoders for latent audio compression,
M. Pasini, S. Lattner, and G. Fazekas, “Music2latent: Consistency autoencoders for latent audio compression,” in International Society for Music Information Retrieval Conference (ISMIR) , 2024
2024
-
[27]
TorchScale: Transformers at scale,
S. Ma, H. Wang, S. Huang, W. Wang, Z. Chi, L. Dong, A. Benhaim, B. Patra, V . Chaudhary, X. Song, and F. Wei, “TorchScale: Transformers at scale,” CoRR, vol. abs/2211.13184, 2022
2022 arXiv
-
[28]
Exploring the limits of transfer learning with a unified text-to-text transformer,
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of machine learning research , vol. 21, no. 140, pp. 1–67, 2020
2020
-
[29]
Longnet: Scaling transformers to 1,000,000,000 tokens,
J. Ding, S. Ma, L. Dong, X. Zhang, S. Huang, W. Wang, and F. Wei, “Longnet: Scaling transformers to 1,000,000,000 tokens,” in Proceedings of the 10th International Conference on Learning Representations , 2023
2023
-
[30]
Enabling factorized piano music modeling and generation with the MAESTRO dataset,
C. Hawthorne and et al., “Enabling factorized piano music modeling and generation with the MAESTRO dataset,” in International Conference on Learning Representations (ICLR) , 2019
2019
-
[31]
Simulating piano performance mistakes for music learning,
A. Morsi, H. Zhang, A. Maezawa, S. Dixon, X. Serra et al., “Simulating piano performance mistakes for music learning,” in Sound and Music Computing Conference, 2024
2024
-
[32]
Midi-ddsp: Detailed control of musical performance via hierarchical modeling,
Y . Wu, E. Manilow, Y . Deng, R. Swavely, K. Kastner, T. Cooijmans, A. Courville, C.-Z. A. Huang, and J. Engel, “Midi-ddsp: Detailed control of musical performance via hierarchical modeling,” arXiv preprint arXiv:2112.09312, 2021
2021 arXiv
-
[33]
Vienna 4x22 Piano Corpus, Rematched,
S. Gasser, M. Grachten, T. Gualtieri, and G. Widmer, “Vienna 4x22 Piano Corpus, Rematched,” https://github.com/OFAI/vienna4x22 rematched, 2023, accessed: 2025-03-20
2023
-
[34]
Nanot5: Fast & simple pre-training and fine-tuning of t5 models with limited resources,
P. Nawrot, “Nanot5: Fast & simple pre-training and fine-tuning of t5 models with limited resources,” in Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023) , 2023, pp. 95–101
2023
-
[35]
Mir eval: A transparent implementation of common mir metrics
C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang, and D. P. Ellis, “Mir eval: A transparent implementation of common mir metrics.” in International Society for Music Information Retrieval Conference (ISMIR), 2014
2014
-
[36]
Score-informed identification of missing and extra notes in piano recordings,
S. Ewert, S. Wang, M. M ¨uller, M. Sandler, S. Ewert, S. Wang, M. Muller et al., “Score-informed identification of missing and extra notes in piano recordings,” in International Society for Music Information Retrieval Conference (ISMIR), 2016
2016
-
[37]
Automatic piano transcription with hierarchical frequency-time transformer,
K. Toyama, T. Akama, Y . Ikemiya, Y . Takida, W.-H. Liao, and Y . Mit- sufuji, “Automatic piano transcription with hierarchical frequency-time transformer,” in International Society for Music Information Retrieval Conference (ISMIR), 2023
2023
-
[38]
Scoring time intervals using non-hierarchical transformer for automatic piano transcription,
Y . Yan and Z. Duan, “Scoring time intervals using non-hierarchical transformer for automatic piano transcription,” arXiv preprint arXiv:2404.09466, 2024
2024 arXiv
-
[39]
The magaloff project: An interim report,
S. Flossmann, W. Goebl, M. Grachten, B. Niedermayer, and G. Widmer, “The magaloff project: An interim report,”Journal of New Music Research, vol. 39, no. 4, pp. 363–377, 2010
2010
-
[40]
Neural turing machines,
A. Graves, G. Wayne, and I. Danihelka, “Neural turing machines,” arXiv preprint arXiv:1410.5401, 2014
2014 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.