Pith. sign in

REVIEW 4 major objections 5 minor 26 references

Fine-Tuning MIDI-to-Audio Alignment using a Neural Network on Piano Roll and CQT Representations

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a CRNN trained on piano-roll and constant-Q spectrogram inputs can fine-tune MIDI-to-audio alignment beyond Dynamic Time Warping, reaching 62% accuracy within 10 ms on real recordings when combined with DTW.

desk verdict Useful proof-of-concept for neural fine-tuning of MIDI-to-audio alignment, but the headline DTW-beating claim only holds in constant-tempo, synthetic-jitter settings. read the letter →

arxiv 2506.22237 v1 pith:LZOYNVOD submitted 2025-06-27 cs.SD cs.CLcs.MMeess.AS

Sebastian Murgul , Moritz Reiser , Michael Heizmann , Christoph Seibert This is my paper · ORCID
classification cs.SDcs.CLcs.MMeess.AS
keywords MusicInformationRetrievalMIDI-to-AudioAlignmentNeuralNetworkCRNNDynamicTimeWarpingPianoRollCQTOnsetAccuracy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a convolutional recurrent neural network can correct small timing errors between a piano performance recording and its loosely aligned MIDI file. Its aim is to improve on Dynamic Time Warping, the industry-standard method, especially at the strict 10 ms tolerance that matters for training transcription systems. On six real-world metronome-based recordings, the model alone reaches 53.20% accuracy at 10 ms versus DTW's 41.53%, and feeding DTW's output through the network raises that to 62.02% with a mean onset error of 12.33 ms. This matters because accurate automatic alignment could make manual annotation of real-world audio for transcription datasets largely unnecessary.

What carries the argument

The core object is the predicted aligned piano roll, a binary time-pitch activation map produced by a CRNN with two convolutional branches, one processing the unaligned piano roll and one processing a log-magnitude CQT spectrogram with the same 88-bin pitch layout. The extracted features are concatenated and passed through a bidirectional LSTM and a dense layer with sigmoid activation, trained with binary cross-entropy on synthetic 'humanized' MAPS data in which each note onset is shifted independently by up to 100 ms. Postprocessing thresholds the output at 0.5, segments note blocks per pitch, and matches them to the input MIDI notes by pitch and temporal order, so that missing detections fall back to the original unaligned notes.

What would settle it

A concrete falsification would be to evaluate the DTW+CRNN pipeline on a test set of expressive piano recordings with local tempo changes above 5%, for instance real rubato performances, and compare 10 ms-window accuracy against DTW; if the advantage shrinks or reverses relative to the 62% versus 41% gap seen on metronome recordings, the paper's central claim about general fine-tuning capability would be contradicted.

Watch

Extended reading notes

Core claim

The central discovery is that MIDI-to-audio alignment can be treated as a supervised translation task: the network takes an unaligned piano roll and the audio's log-magnitude CQT spectrogram as inputs and outputs an aligned piano roll. On the synthetic MAPS test split the model raises 10 ms-window alignment accuracy from 24.79% with DTW to 40.78% with the CRNN alone, and on real recordings from 41.53% to 53.20%. Applying the network to the output of DTW gives the best and most consistent results, reaching 62.02% accuracy at 10 ms and a mean absolute onset error of 12.33 ms on real recordings. The authors argue this works because the network corrects individual note onsets, including non-monotonic deviations, whereas DTW is constrained to a single monotonic warping path.

Load-bearing premise

The load-bearing premise is that realistic human timing errors are well approximated by independent per-note shifts of up to 100 ms with no tempo drift; if real performances are dominated by rubato or gradual tempo changes, the network's advantage over DTW is not guaranteed to transfer, since the paper's own tempo-change ablation shows large degradation.

Editorial extensions

If this is right

  • MIDI annotations for automatic transcription training can be fine-aligned automatically, reducing the need for manual correction of onset times.
  • A DTW-then-CRNN pipeline gives the most consistent alignment on real recordings, with a mean absolute onset error of 12.33 ms.
  • The network can correct per-note onset deviations that a global monotonic warping path cannot represent, which is why its largest gains appear at the strict 10 ms tolerance.
  • The method is positioned as a final fine-tuning stage after coarse alignment such as DTW, not as a replacement for global warping.
  • Introducing tempo scaling into the unaligned data sharply degrades alignment accuracy, so the approach is best suited to small, tempo-stable timing variations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that augmenting training data with tempo drift, rather than only independent onset shifts, is the natural next experiment; the tempo-change ablation in Section 4.2.5 suggests the model's current limit is tied to the training distribution, not to the architecture itself.
  • Beyond the paper, the DTW+CRNN cascade could serve as a data-cleaning step for piano transcription datasets, correcting residual jitter after warping and potentially improving onset precision in downstream transcription models.
  • A likely untested consequence is that the model's per-note correction can address systematic asynchrony between bass and treble notes in chords, a failure mode the authors cite for global DTW alignment.
  • For practical deployment, the advantage at 10 ms implies the method pays off mainly when onset precision below 25 ms is required; at coarser tolerances DTW already performs well, so the added complexity may not be worthwhile.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes a CRNN that takes a CQT spectrogram and a loosely aligned piano roll as inputs and outputs an aligned piano roll, intended as a fine-tuning step for MIDI-to-audio alignment, optionally applied after DTW. The model is trained on MAPS by shifting note onsets and offsets independently by up to 100 ms, and evaluated on the MAPS test split and six newly recorded metronome-based real piano performances. The authors compare against MrMsDTW with DLNCO features and report that the combined DTW+CRNN pipeline improves 10 ms alignment accuracy from 41.53% to 62.02% on the real-world set, while the CRNN alone improves it from 41.53% to 53.20%.

Significance. If the result generalizes, the proposed pipeline would be a practical tool for refining coarse MIDI-to-audio alignments to the 10 ms scale, with clear benefits for automatic dataset annotation and AMT training. The paper has notable strengths: the architecture is clearly described, evaluation uses standard toolboxes (mir_eval, synctoolbox), and the ablation study honestly exposes the model's behavior under tempo changes. However, the central claim of superiority over DTW is currently supported only by an in-distribution synthetic test and a very small constant-tempo real-world sample, so the contribution is better framed as a proof-of-concept than as a demonstrated general improvement.

major comments (4)
  1. [§4.2.5, Table 6] The tempo-change ablation directly undermines the abstract's general claim. With a maximum tempo factor of just 0.01, the mean absolute onset error rises from 21.36 ms to 32.77 ms, and at 0.05 it reaches 44.20 ms. Because real human piano performances routinely contain tempo drift and rubato of several percent, the reported superiority over DTW is not established for the general task of aligning human performances. The authors should either evaluate on recordings with tempo changes, include tempo augmentation in training, or explicitly and prominently scope the claim to constant-tempo fine-tuning.
  2. [§3.1.1, §4.1, Table 1] The MAPS test set is generated with the same augmentation procedure used for training: each note onset is shifted independently by up to 100 ms with no tempo change. These independent shifts can violate the monotonic warping assumption of DTW, so the MAPS comparison is in-distribution for the CRNN and structurally unfavorable to DTW. The poor DTW performance on MAPS (24.79% at 10 ms, worse than the unaligned baseline) is acknowledged in the text but should not be used as evidence of general superiority. An independent test set with realistic, correlated timing errors is needed.
  3. [§3.1.2, Table 1] The real-world evaluation consists of only six metronome-based recordings with no tempo variation, and no confidence intervals or significance tests are reported. Given the high per-note standard deviations (e.g., 13.56 ms for the best pipeline), differences between methods could be within noise. Moreover, Table 1 shows that the CRNN alone is worse than DTW at the 25 ms, 50 ms, and 100 ms windows on this real-world set, so the abstract's phrasing 'up to 20% higher alignment accuracy than DTW across various tolerance windows' is not supported for the model alone; it is only the DTW+CRNN combination that improves across most windows.
  4. [§4.2.4] The claim that 'experiments with real world musical pieces also yield an average human timing variation of 95 ms' is not substantiated by any measurement, table, or reference to this study's real-world recordings. Since this value is used to motivate the 100 ms augmentation range, the authors should either report how it was derived or remove it.
minor comments (5)
  1. [Abstract, §4.1] The abstract should specify that the 20% improvement is in absolute percentage points, applies mainly to the 10 ms tolerance window, and is achieved by the DTW+CRNN combination rather than by the CRNN alone.
  2. [Tables 2 and 3] The formatting of percentages is inconsistent: Table 2 uses numbers without '%' signs and with inconsistent spacing (e.g., '47 .17'), while Table 3 uses '%' signs. Please unify the style.
  3. [§3.1.2] The real-world dataset description should include the number of notes per piece, piece durations, and recording details; six recordings is very small, and more details would help readers judge the variability.
  4. [§3.2] Please report the exact training epochs used by early stopping, the early-stopping patience, and the number of training segments after splitting; this information is needed for reproducibility.
  5. [§4.1] The 'CRNN (Blind Transcription)' baseline is not described in the architecture section; it is unclear how the MIDI input branch is removed or replaced. Please specify this baseline's exact input.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central comparison to DTW rests on independent real-world recordings, and the only self-citation is a non-load-bearing architectural inspiration.

full rationale

The derivation chain is self-contained. The model is trained on augmented MAPS triplets in which the unaligned MIDI is generated by independently shifting onsets and offsets of the aligned MIDI, and the network is evaluated on held-out MAPS examples and on six metronome-based real-world recordings. The held-out MAPS test uses the same augmentation distribution as training, which is a standard in-distribution evaluation and not a fitted parameter being renamed as a prediction; the real-world recordings provide an independent benchmark for the headline improvement over DTW. The only self-citation is reference [17], used solely to say that the convolutional block architecture is 'inspired by' the authors' earlier singing-transcription model; that citation is architectural inspiration rather than a load-bearing theorem, fitted value, or uniqueness argument. Section 4.2.5 shows that the model degrades with tempo changes, but that is a generalization limitation, not circularity. The skeptic's concern about rubato and tempo drift is a domain-mismatch concern, not evidence that any prediction reduces to its inputs by construction. Accordingly, the paper merits a low circularity score reflecting only one minor self-citation that does not support the central claim.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method relies on a supervised training setup where the ground truth is the original aligned MIDI and the input unalignment is synthetic. The free parameters are mostly data-generation and postprocessing choices. No new physical or mathematical entities are introduced. The most load-bearing assumption is that independent per-note jitter captures human timing variability, which is only partially supported by the real-world experiments.

free parameters (3)
  • Maximum onset/offset timing variation = 100 ms
    Section 3.1.1: the authors choose 100 ms as the maximum shift for synthetic unalignment, based on literature but adjusted upward. This controls the training and test distribution and directly affects all reported accuracy numbers.
  • Temporal resolution = 100 frames per second
    Section 2.1 and Table 4: chosen so each frame is 10 ms; the ablation shows performance drops at lower resolutions and does not improve at 200 FPS.
  • Postprocessing binarization threshold = 0.5
    Section 2.3: estimated piano roll activations are thresholded at 0.5 to form note segments. The choice is not ablated.
assumptions (5)
  • domain assumption MAPS synthesized performances are perfectly aligned to their MIDI ground truth.
    Section 3.1.1: the synthesized MAPS audio is treated as temporally accurate, and its MIDI is used as the ground truth for training and evaluation.
  • domain assumption DAW-recorded digital piano MIDI is exactly aligned to its audio.
    Section 3.1.2: real-world labels are taken from a digital piano's MIDI output, which is assumed to be perfectly synchronized with the recorded audio.
  • domain assumption Independent per-note timing jitter up to 100 ms models human timing errors.
    Section 3.1.1: this is the core augmentation model. The paper assumes that shifting each note independently captures the relevant variation; the tempo-change ablation in Section 4.2.5 shows this assumption fails when global tempo drift is present.
  • domain assumption One-to-one correspondence between CQT bins and piano-roll pitches is sufficient for cross-modal comparison.
    Section 2.1: both inputs use 88 bins with 12 bins per octave starting at 27.5 Hz, which allows the convolutional branches to align features directly. This is a design choice, not a proven property.
  • domain assumption The intended output is a timing correction of the input note sequence, not a new note sequence.
    Section 2.3: the postprocessing matches predicted notes to input MIDI notes by pitch and temporal order, and falls back to the original unaligned note if no match is found. This fixes the note sequence and limits the method to timing refinement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fine-Tuning MIDI-to-Audio Alignment using a Neural Network on Piano Roll and CQT Representations." pith.science (2026). https://pith.science/paper/LZOYNVOD

@misc{pith2026250622237,
  author       = {Pith},
  title        = {Pith review of: Fine-Tuning MIDI-to-Audio Alignment using a Neural Network on Piano Roll and CQT Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LZOYNVOD}},
  note         = {Machine review of arXiv:2506.22237}
}
read the original abstract

In this paper, we present a neural network approach for synchronizing audio recordings of human piano performances with their corresponding loosely aligned MIDI files. The task is addressed using a Convolutional Recurrent Neural Network (CRNN) architecture, which effectively captures spectral and temporal features by processing an unaligned piano roll and a spectrogram as inputs to estimate the aligned piano roll. To train the network, we create a dataset of piano pieces with augmented MIDI files that simulate common human timing errors. The proposed model achieves up to 20% higher alignment accuracy than the industry-standard Dynamic Time Warping (DTW) method across various tolerance windows. Furthermore, integrating DTW with the CRNN yields additional improvements, offering enhanced robustness and consistency. These findings demonstrate the potential of neural networks in advancing state-of-the-art MIDI-to-audio alignment.

Figures

Figures reproduced from arXiv: 2506.22237 by the authors.

Figure 1
Figure 1. Architecture of the proposed CRNN model for MIDI-to-audio alignment. Two parallel convolutional branches [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Illustration of timing deviations in a human piano performance. The blue bars represent ideal note lengths (as [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Alignment output of the proposed CRNN model on a MAPS test excerpt. Red blocks indicate the predicted [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 22 canonical work pages

  1. [1]

    Automatic music transcription: An overview,

    E. Benetos, S. Dixon, Z. Duan, and S. Ewert, “Automatic music transcription: An overview,” IEEE Signal Processing Magazine, vol. 36, no. 1, pp. 20–30, 2019

    2019

  2. [2]

    Polyphonic audio matching for score following and intelligent audio editors,

    R. B. Dannenberg and N. Hu, “Polyphonic audio matching for score following and intelligent audio editors,” in Proc. of the International Computer Music Conference (ICMC) , San Francisco, USA, 2003, pp. 27–34

    work page 2003

  3. [3]

    Polyphonic audio matching and alignment for music retrieval,

    N. Hu, R. B. Dannenberg, and G. Tzanetakis, “Polyphonic audio matching and alignment for music retrieval,” in Proc. of the IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) , New Paltz, NY , USA, 2003, pp. 185–188

    work page 2003

  4. [4]

    High resolution audio synchronization using chroma onset features,

    S. Ewert, M. Müller, and P. Grosche, “High resolution audio synchronization using chroma onset features,” in Proc. of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) , Taipei, Taiwan, 2009, pp. 1869–1872

    work page 2009

  5. [5]

    Memory-restricted multiscale dynamic time warping,

    T. Prätzlich, J. Driedger, and M. Müller, “Memory-restricted multiscale dynamic time warping,” in2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2016, pp. 569–573

    work page 2016

  6. [6]

    Improved score-performance matching using both structural and temporal informa- tion from midi recordings,

    B. Gingras and S. McAdams, “Improved score-performance matching using both structural and temporal informa- tion from midi recordings,” Journal of New Music Research, vol. 40, no. 1, pp. 43–57, 2011

    work page 2011

  7. [7]

    Improved score-performance alignment algorithms on polyphonic music,

    C.-T. Chen, J.-S. R. Jang, and W. Liou, “Improved score-performance alignment algorithms on polyphonic music,” in 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2014, pp. 1365–1369

    work page 2014

  8. [8]

    Audio-to-score alignment of piano music using rnn-based automatic music transcription,

    T. Kwon, D. Jeong, and J. Nam, “Audio-to-score alignment of piano music using rnn-based automatic music transcription,” arXiv preprint arXiv:1711.04480, 2017

    arXiv 2017

Show all 26 references
  1. [9]

    An improvement on audio-to-midi alignment using triplet pair,

    Y . Wang, S. Liu, and L. Guo, “An improvement on audio-to-midi alignment using triplet pair,” in1st International Workshop on Multimodal Understanding and Learning for Embodied Applications , 2019, pp. 43–48

    2019

  2. [10]

    Learning frame similarity using siamese networks for audio-to-score alignment,

    R. Agrawal and S. Dixon, “Learning frame similarity using siamese networks for audio-to-score alignment,” in 2020 28th European Signal Processing Conference (EUSIPCO) . IEEE, 2021, pp. 141–145

    2020

  3. [11]

    High resolution guitar transcription via domain adaptation,

    X. Riley, D. Edwards, and S. Dixon, “High resolution guitar transcription via domain adaptation,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 1051–1055

    2024

  4. [12]

    Unaligned supervision for automatic music transcription in the wild,

    B. Maman and A. H. Bermano, “Unaligned supervision for automatic music transcription in the wild,” in International Conference on Machine Learning . PMLR, 2022, pp. 14 918–14 934

    2022

  5. [13]

    En- abling factorized piano music modeling and generation with the maestro dataset,

    C. Hawthorne, A. Stasyuk, A. Roberts, I. Simon, C.-Z. A. Huang, S. Dieleman, E. Elsen, J. Engel, and D. Eck, “En- abling factorized piano music modeling and generation with the maestro dataset,”arXiv preprint arXiv:1810.12247, 2018

    arXiv

  6. [14]

    Pop2piano: Pop audio-based piano cover generation,

    J. Choi and K. Lee, “Pop2piano: Pop audio-based piano cover generation,” in ICASSP 2023-2023 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2023, pp. 1–5

    2023

  7. [15]

    Multipitch estimation of piano sounds using a new probabilistic spectral smoothness principle,

    V . Emiya, R. Badeau, and B. David, “Multipitch estimation of piano sounds using a new probabilistic spectral smoothness principle,” IEEE Transactions on Audio, Speech and Language Processing , vol. 18, no. 6, pp. 1643–1654, 2010

    2010

  8. [16]

    librosa: Audio and music signal analysis in python,

    B. McFee, C. Raffel, D. Liang, D. P. Ellis, M. McVicar, E. Battenberg, and O. Nieto, “librosa: Audio and music signal analysis in python,” Proceedings of the 14th Python in Science Conference , vol. 8, pp. 18–25, 2015

    2015

  9. [17]

    Dual task monophonic singing transcription,

    M. Schwabe, S. Murgul, and M. Heizmann, “Dual task monophonic singing transcription,” J. Audio Eng. Soc, vol. 70, no. 12, pp. 1038–1047, 2022

    2022

  10. [18]

    Timing variations in music performance: Musical communication, perceptual compensa- tion, and/or motor control?

    A. Penel and C. Drake, “Timing variations in music performance: Musical communication, perceptual compensa- tion, and/or motor control?” Perception & Psychophysics, vol. 66, no. 4, pp. 545–562, 2004

    2004

  11. [19]

    Music performance,

    C. Palmer, “Music performance,” Annual Review of Psychology, vol. 48, no. 1, pp. 115–138, 1997

    1997

  12. [20]

    Estimating onset and offset asynchronies in polyphonic score-audio alignment,

    J. Devaney, “Estimating onset and offset asynchronies in polyphonic score-audio alignment,” Journal of New Music Research, vol. 43, no. 3, pp. 266–275, 2014

    2014

  13. [21]

    Audio-to-score alignment using deep automatic music transcription,

    F. Simonetta, S. Ntalampiras, and F. Avanzini, “Audio-to-score alignment using deep automatic music transcription,” 2021 IEEE 23rd International Workshop on Multimedia Signal Processing , pp. 1–6, 2021

    2021

  14. [22]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014. 8 Fine-Tuning MIDI-to-Audio Alignment

    2014 arXiv

  15. [23]

    mir_eval: A transparent implementation of common mir metrics

    C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang, and D. P. W. Ellis, “mir_eval: A transparent implementation of common mir metrics.” Proceedings of the 15th International Society for Music Information Retrieval Conference, vol. 10, pp. 367–372, 2014

    2014

  16. [24]

    Sync toolbox: A python package for efficient, robust, and accurate music synchronization,

    M. Müller, Y . Özer, M. Krause, T. Prätzlich, and J. Driedger, “Sync toolbox: A python package for efficient, robust, and accurate music synchronization,” Journal of Open Source Software, vol. 6, no. 64, p. 3434, 2021

    2021

  17. [25]

    Sources of timing variations in music performance: A psychological segmentation model,

    A. Penel and C. Drake, “Sources of timing variations in music performance: A psychological segmentation model,” Psychological research, vol. 61, pp. 12–32, 1998

    1998

  18. [26]

    Guitarset: A dataset for guitar transcription

    Q. Xi, R. M. Bittner, J. Pauwels, X. Ye, and J. P. Bello, “Guitarset: A dataset for guitar transcription.” inISMIR, 2018, pp. 453–460. 9

    2018

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.