Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

Scaling Self-Supervised Representation Learning for Symbolic Piano Performance

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 650-million-parameter transformer, pretrained on 60,000 hours of piano MIDI, beats prior symbolic generators and yields embeddings that lead music-classification probes.

desk verdict A genuinely useful open scaling study for symbolic piano, but the headline classification gains need a clear held-out split before they can be trusted. read the letter →

arxiv 2506.23869 v1 pith:E5PAMDLT submitted 2025-06-30 cs.SD cs.AIcs.LGeess.AS

Louis Bradshaw , Honglu Fan , Alexander Spangher , Stella Biderman , Simon Colton This is my paper · ORCID
classification cs.SDcs.AIcs.LGeess.AS
keywords self-supervisedlearningsymbolicmusicpianoperformanceMIDItokenizationcontrastiveembeddingsinformationretrievalautoregressivetransformertransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether the scaling recipe behind large language models—pretrain on huge unlabeled data, then adapt—works for symbolic piano music, a domain where hand-annotated data is scarce. The authors pretrain a 650-million-parameter autoregressive transformer on about 60,000 hours of automatically transcribed solo-piano MIDI, then reuse the same checkpoint for generation, classification, and embedding extraction. In human listening tests, continuations from the generative variant are preferred over those of a leading symbolic continuation model and are statistically indistinguishable from a proprietary audio generator and from human-composed ground truth. Frozen embeddings from a secondary contrastive finetuning stage outperform prior audio and symbolic baselines on composer, genre, period, and form classification, and direct finetuning needs only a few hundred labeled files to reach strong accuracy. The paper's key structural finding is that the contrastive stage only works when initialized from the pretrained model.

What carries the argument

The mechanism that carries the argument is the two-stage training design together with a custom MIDI tokenizer. Notes are tokenized as instrument-pitch-velocity, onset, and duration tokens, with onset times measured absolutely inside five-second segments bounded by a special segment token, so the model never has to sum relative time shifts. Pretraining is ordinary next-token prediction on concatenated 8192-token sequences; the contrastive phase then replaces the language-model head with an embedding head and optimizes a temperature-scaled contrastive loss over pairs of slices from the same file. This sequence—pretrain, then contrastively finetune, rather than train contrastively from scratch—is what the paper identifies as decisive for producing transferable file-level embeddings.

What would settle it

Compute, for each of the four new benchmark tasks, the fraction of test files whose composition (composer plus opus or piece identifier) also appears in the pretraining or contrastive-finetuning pools; if the overlap is high, re-run the linear probes on a strictly disjoint split to see whether the reported accuracy advantage survives.

Watch

Extended reading notes

Core claim

The central claim is that next-token pretraining on large-scale symbolic piano data produces a reusable foundation for music information retrieval and generation, and that the same checkpoint can be redirected without task-specific architecture changes. After pretraining, one branch is finetuned for continuation, one for supervised classification, and one for contrastive embeddings; the contrastive branch maps two random slices of the same file to close 512-dimensional vectors using a temperature-scaled cross-entropy loss, so file-level attributes such as composer emerge without labels. The resulting embedding model outperforms the compared audio and symbolic baselines in linear probe experiments on the four new benchmarks and on existing piano benchmarks, while the supervised branch reaches high accuracy with a few hundred labeled examples. This is presented as evidence that scaling self-supervised learning in the symbolic domain yields the same transfer benefits long seen in text and audio.

Load-bearing premise

The load-bearing assumption is that the new MIR evaluation splits are genuinely held out: the four classification benchmarks are drawn from the same Aria-MIDI corpus used for pretraining and contrastive finetuning, duplicate filtering was relaxed in the contrastive phase, and the paper does not state that the 1,000-file test sets are composition-disjoint from training.

Editorial extensions

If this is right

  • A single pretrained symbolic-music model can serve as the backbone for generation, classification, and retrieval, so downstream researchers do not need to train large models from scratch for each task.
  • Frozen contrastive embeddings transfer to composer, genre, period, and form classification with linear probes, which makes lightweight and label-efficient downstream classifiers feasible.
  • Direct finetuning reaches strong accuracy with only a few hundred labeled files, which would lower the cost of building new music-classification datasets.
  • The use of absolute onset tokens, rather than relative time shifts, is presented as a central design choice, so the results indirectly support timing representations that avoid cumulative arithmetic.
  • Human listeners could not statistically distinguish the generative model's continuations from a proprietary audio generator or from human-composed music, suggesting symbolic-only generation is competitive in the solo-piano niche.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the benchmark-split ambiguity were resolved in the paper's favor, the symbolic-MIDI route would become an inexpensive testbed for scaling research, because automatic transcription can generate unlabeled training data without copyright-intensive audio.
  • A direct way to test the tokenizer claim is to pretrain the same model with relative time-shift tokens and compare listening and probe scores; the paper motivates but does not run this ablation.
  • Relaxing duplicate filtering during contrastive finetuning may be part of why embeddings cluster by composer, since repeated performances of the same work become positive views; a clean evaluation would hold out whole compositions and test retrieval on new performances.
  • The same two-stage recipe could plausibly extend to multi-track and non-classical MIDI, since the released tokenizer is multi-track, but the paper only evaluates solo piano and does not claim those extensions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces Aria, a 650M-parameter autoregressive transformer pretrained on roughly 60,000 hours of Aria-MIDI solo-piano transcriptions, using a custom absolute-onset tokenization. It evaluates the model in two ways: generative continuation quality via human listening tests against Anticipatory Music Transformer, MusicGen, Suno 3.5, and human ground truth; and representation learning through linear probes and supervised finetuning on Pianist8, VG-MIDI, and four new benchmarks (Genre, Musical Period, Form, Composer) derived from Aria-MIDI metadata, compared against MERT, M3, and CLaMP3. The authors report that their generative model outperforms symbolic baselines, is statistically indistinguishable from Suno 3.5 and ground truth, and that their contrastive embedding model AriaEmb achieves state-of-the-art linear-probe accuracy, with ablations showing the contrastive objective is only effective when applied as a secondary finetuning phase after next-token pretraining.

Significance. If the results hold, the paper provides strong evidence that scaling self-supervised pretraining on symbolic piano data yields both competitive generation and transferable representations, a meaningful step for symbolic MIR. The work is commendable for open-sourcing the model and preprocessing library, for the human-listening evaluation with reported binomial p-values, and for the external Pianist8 and VG-MIDI benchmarks that independently support transferability. However, the central state-of-the-art representation-learning claim rests on four new Aria-MIDI-derived benchmarks for which the paper does not demonstrate that test files or compositions were held out from pretraining or contrastive finetuning. Since Section 3.3 explicitly relaxed compositional-duplicate filtering during contrastive training, the headline linear-probe numbers on these benchmarks may be inflated by near-duplicate recognition, making the SOTA claim conditional until the holdout is specified and enforced.

major comments (2)
  1. [§4.1, Table 2] The four new benchmarks (Genre, Form, Musical Period, Composer) are extracted from Aria-MIDI metadata, and the paper states only that label-balanced 10,000/1,000 file splits were taken; it does not state that these files, or the compositions they belong to, were excluded from the 60,473-hour pretraining corpus or from the contrastive finetuning stage. Section 3.3 explicitly says the contrastive stage 'relaxed the preprocessing constraints on compositional duplicates', so different performances of the same composition likely appear on both sides of the split. If so, the linear-probe results (e.g., Composer 90.5, Period 84.7) could reflect near-duplicate composition recognition rather than general representation quality, and the comparison against MERT, M3, and CLaMP3, which did not train on Aria-MIDI, would not be a fair state-of-the-art comparison. The authors must either state and enforce a composition-level holdout for the four new benchmarks or re-run them on a genuinely held-out set of compositions.
  2. [§3.2, §4.1] The pretraining deduplication rule (for composers with more than 250 tagged files, keep at most 10 instances per opus/piece-number pair and discard untagged files for those composers) is not a global exclusion rule; it does not guarantee that the 1,000 test files, or the compositions they contain, are absent from the pretraining or contrastive-finetuning corpora. The contrastive finetuning phase in Section 3.3 deliberately includes duplicate compositions, making the leakage scenario concrete. The paper should specify exactly how the test splits were constructed, including whether any composition in the test splits appears anywhere in the training corpora, and if it does, the affected Table 2 results should be removed or recomputed under a clean holdout.
minor comments (5)
  1. [Table 1] The table formatting is confusing: the rows for AM Transformer and Suno 3.5 read '9 .43e-7' and '7 .49e-1', which appear to be split p-values (9.43e-7 and 7.49e-1). Please align the columns and use a consistent decimal format.
  2. [§4.1] The listening test uses only five prompts, each with eight continuations. Although the binomial p-values are computed on non-tied comparisons, the small prompt sample limits the generality of the claims; please discuss prompt diversity and the stability of preferences across prompts.
  3. [§3.1, §3.2] The paper describes the tokenization and model architecture in detail but does not report the vocabulary size or the exact parameter count after the architectural modifications. Please provide these numbers for reproducibility.
  4. [§3.3] The contrastive setup states the temperature τ and slice length range, but not the batch size N used for in-batch negatives, which is known to affect contrastive learning. Table 2 ablates epochs and temperature but not batch size; please report the batch size.
  5. [§4.2] The claim that the contrastive approach is 'effective only when applied as a secondary finetuning phase' is based on the Aria†e=25 ablation, but the from-scratch model may need more than 25 epochs or a different learning rate to converge; please add a brief discussion or additional ablation to strengthen this conclusion.

Circularity Check

1 steps flagged · score 3.0 of 10

Four new MIR benchmarks are built from Aria-MIDI, the same corpus used for pretraining and contrastive finetuning, with duplicate compositions deliberately retained, so the claimed SOTA probe results are not an independent prediction.

  1. fitted input called prediction [Section 3.3 and Section 4.1 (Contrastive Representation Learning; Setup/Contrastive embeddings)]
    "In the contrastive finetuning stage ... We trained on the reduced finetuning dataset described in Section 3.2; however, we relaxed the preprocessing constraints on compositional duplicates to encourage the model to distinguish between different performances of popular compositions. ... We assess performance using established benchmarks, Pianist8 [68] and VG-MIDI [69], as well as new benchmarks we derive from Aria-MIDI metadata."

    The model is pretrained on roughly 60,473 hours of Aria-MIDI and then contrastively finetuned on the same collection, with the paper explicitly relaxing compositional-duplicate filtering during that finetuning. The four new benchmarks are extracted from Aria-MIDI metadata, and the paper does not state that the 1,000 test files or their compositions were excluded from either pretraining or contrastive finetuning. Consequently, a frozen embedding can solve Composer, Period, Form, and Genre by recognizing near-duplicate compositions already seen in training, rather than by learning generalizable musical representations.

full rationale

The paper is an empirical scaling study rather than a mathematical derivation, and most claims are independently checkable: the listening-test preferences, the Pianist8/VG-MIDI results, the from-scratch contrastive ablation, and the data-efficiency finetuning curves do not reduce to the paper's own inputs. The main concern is benchmark integrity: four of the six classification tasks are derived from Aria-MIDI metadata, the same collection as the pretraining and contrastive-finetuning corpora, and the contrastive phase deliberately retained multiple performances of the same compositions. The paper never states that the 1,000-file test splits or their compositions were held out from the 60,473-hour pretraining set or the 25-epoch contrastive phase, so the Composer 90.5, Period 84.7, Form 82.5, and Genre 92.4 numbers may partly reflect memorization of near-duplicate compositions. This is a data-overlap threat to the 'state-of-the-art' claim on those four tasks, but it is not a derivation that is equivalent to its inputs by construction, and the external benchmarks and generative evaluations provide independent evidence for the central scaling claims. The score is moderate rather than high because the circularity is localized to the self-derived benchmarks and does not invalidate the paper's other empirical contributions.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The ledger is light on invented entities: the paper introduces no new physical or conceptual objects, only tokenizer control tokens such as <T> and <D>. The main free parameters are hand-chosen tokenizer and training constants plus the contrastive temperature selected by ablation. The load-bearing assumptions are domain-level: transcription quality at scale, metadata label reliability, the SimCLR positive-pair assumption for slices of the same piece, the adequacy of five prompts and 46 listeners, and an implicit assumption that benchmark test files were excluded from training corpora, which the text never states explicitly.

free parameters (6)
  • NT-Xent temperature tau = 0.1
    Selected after sweeping 0.05, 0.1, and 0.5 in Section 4.2; the reported headline results use the best value from the sweep.
  • Segment duration for absolute-onset tokenization = 5000 ms
    Hand-chosen in Section 3.1 to keep the onset vocabulary bounded; central to the tokenizer design but not otherwise motivated.
  • Temporal resolution = 10 ms
    Hand-chosen onset and duration resolution for all tokens; quantizes expressive timing and limits sequence length.
  • Velocity bin count = 12
    Hand-chosen discretization of MIDI velocity in Section 3.1; no ablation is reported for this choice.
  • Contrastive slice length range = 100 to 650 notes
    Hand-chosen bounds for the two SimCLR views in Section 3.3; this range determines what semantic level the embeddings capture.
  • Deduplication thresholds for pretraining corpus = at most 10 files per opus/piece pair for composers with more than 250 tagged instances
    Hand-set curation rules in Section 3.2 that shape both the size and the composition diversity of the 60,473-hour corpus.
assumptions (5)
  • domain assumption AMT-transcribed YouTube piano audio in Aria-MIDI is musically accurate enough to serve as pretraining signal at 60,000-hour scale.
    Sections 1 and 3.2 rely on 'highly reliable note-identification accuracy' without auditing transcription errors across the 820,944-file corpus.
  • domain assumption Aria-MIDI metadata labels (composer, genre, period, form) are reliable enough to serve as ground truth for the four new benchmarks.
    Section 4.1 builds label-balanced splits from YouTube-sourced metadata with no manual verification or noise estimate.
  • domain assumption Two random contiguous slices of the same MIDI file share the composition-level attributes the embeddings are meant to encode.
    Section 3.3: this positive-pair assumption of the SimCLR adaptation is what forces composer/style-level rather than local features; it is not validated directly, for example by a slice-retrieval sanity check.
  • standard math Standard transformer training with next-token prediction works on concatenated MIDI sequences with absolute-onset tokens and segment markers.
    Sections 3.1 and 3.2 use conventional LLaMA-derived machinery with minimal modifications; this is background ML practice, not a contested claim.
  • domain assumption Forty-six musically trained participants and five prompts give sufficient power for pairwise continuation preferences.
    Section 4.1: the listening-test design; power is low for the ground-truth and Suno comparisons, as reflected in the 15-17 win-loss split against ground truth.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling Self-Supervised Representation Learning for Symbolic Piano Performance." pith.science (2026). https://pith.science/paper/E5PAMDLT

@misc{pith2026250623869,
  author       = {Pith},
  title        = {Pith review of: Scaling Self-Supervised Representation Learning for Symbolic Piano Performance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E5PAMDLT}},
  note         = {Machine review of arXiv:2506.23869}
}
read the original abstract

We study the capabilities of generative autoregressive transformer models trained on large amounts of symbolic solo-piano transcriptions. After first pretraining on approximately 60,000 hours of music, we use a comparatively smaller, high-quality subset, to finetune models to produce musical continuations, perform symbolic classification tasks, and produce general-purpose contrastive MIDI embeddings by adapting the SimCLR framework to symbolic music. When evaluating piano continuation coherence, our generative model outperforms leading symbolic generation techniques and remains competitive with proprietary audio generation models. On MIR classification benchmarks, frozen representations from our contrastive model achieve state-of-the-art results in linear probe experiments, while direct finetuning demonstrates the generalizability of pretrained representations, often requiring only a few hundred labeled examples to specialize to downstream tasks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RUMAA: Repeat-Aware Unified Music Audio Analysis for Score-Performance Alignment, Transcription, and Mistake Detection

    cs.SD 2025-07 conditional novelty 6.0 of 10

    A single transformer model aligns scores to performances, transcribes piano audio, and detects mistakes, including faithful handling of repeat sections without pre-unfolded scores.

Reference graph

Works this paper leans on

82 extracted references · 42 canonical work pages · cited by 1 Pith paper

  1. [1]

    Scaling Self-Supervised Representation Learn- ing for Symbolic Piano Performance

    INTRODUCTION Modern machine learning systems increasingly utilize self- supervised learning (SSL) as a core component of their training pipeline. In this paradigm, general-purpose repre- sentations are learned during an initial phase of self-guided learning, which can then be adapted to specialized tasks, often outperforming purely supervised approaches, ...

    arXiv 2025

  2. [2]

    We introduce and open-sourceAria 2 , a pretrained au- toregressive transformer model trained on transcrip- tions of piano recordings. Through human listening tests, we show it generates coherent continuations from short musical prompts, outperforming Anticipa- tory Music Transformer [29] and rivaling proprietary audio models like Suno 3.5 [30]

  3. [3]

    Critically, we show that this contrastive ap- proach is effective only when applied as a secondary finetuning phase

    We further demonstrate the effectiveness of large- scale pretrained representations for symbolic MIR through two approaches: (1) directly finetuning our model for classification tasks, achieving strong per- formance when labeled examples are extremely lim- ited, and (2) proposing a novel adaptation of con- trastive learning that produces an embedding mode...

  4. [4]

    In this section, we focus on related work specific to the subfield of symbolic music modeling

    RELATED WORK Our work relates to many sub-areas of computational music, generative modeling, and representation learning. In this section, we focus on related work specific to the subfield of symbolic music modeling. The field of symbolic music generation using neural networks has advanced rapidly. Prior to the introduction of transformers, models such as...

    work page

  5. [5]

    Our ex- periments show that this pretraining framework effectively adapts to generative modeling, MIDI-embedding genera- tion, and supervised MIR tasks

    CONCLUSION We introduce Aria, an autoregressive generative trans- former model designed to investigate the scalability of self- supervised learning for symbolic music modeling. Our ex- periments show that this pretraining framework effectively adapts to generative modeling, MIDI-embedding genera- tion, and supervised MIR tasks. Moreover, our findings sugg...

    work page

  6. [6]

    METHOD To explore the capabilities of large-scale self-supervised models for piano performance, we first pretrained an au- toregressive transformer model using next-token prediction on a refined subset of the Aria-MIDI dataset. We adopt this setup due to its versatility: next-token prediction has a proven track record in generative modeling for both sym- ...

    work page

  7. [7]

    (2006) IMSLP/Petrucci music library

    IMSLP. (2006) IMSLP/Petrucci music library. IMSLP. [Online]. Available: https://imslp.org

    work page 2006

  8. [8]

    EXPERIMENTS Having outlined our methodology, we evaluate the gener- ative capabilities of our model, as well as the contrastive representation learning framework, in the context of piano performance. To understand its capabilities in the wider area of models for generative music and MIR, we compare our approach to both symbolic and audio-based baselines, ...

    work page

Show all 82 references
  1. [9]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971 , 2023

    2023 arXiv

  2. [10]

    Additional support was provided by EleutherAI and StabilityAI, as well as a compute grant from the Ministry of Science and ICT of Korea and Gwangju Metropolitan City

    ACKNOWLEDGMENTS This work was supported by UKRI and EPSRC under grant EP/S022694/1. Additional support was provided by EleutherAI and StabilityAI, as well as a compute grant from the Ministry of Science and ICT of Korea and Gwangju Metropolitan City

  3. [11]

    A survey on self-supervised learning: Algo- rithms, applications, and future trends,

    J. Gui, T. Chen, J. Zhang, Q. Cao, Z. Sun, H. Luo, and D. Tao, “A survey on self-supervised learning: Algo- rithms, applications, and future trends,” IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 2024

    2024

  4. [12]

    Notagen: Advanc- ing musicality in symbolic music generation with large language model training paradigms,

    Y . Wang, S. Wu, J. Hu, X. Du, Y . Peng, Y . Huang, S. Fan, X. Li, F. Yu, and M. Sun, “Notagen: Advanc- ing musicality in symbolic music generation with large language model training paradigms,” arXiv preprint arXiv:2502.18008, 2025

    2025 arXiv

  5. [13]

    Mupt: A generative symbolic music pretrained transformer,

    X. Qu, Y . Bai, Y . Ma, Z. Zhou, K. M. Lo, J. Liu, R. Yuan, L. Min, X. Liu, T. Zhang et al., “Mupt: A generative symbolic music pretrained transformer,” arXiv preprint arXiv:2404.06393, 2024

    2024 arXiv

  6. [14]

    Musicbert: Symbolic music understanding with large- scale pre-training,

    M. Zeng, X. Tan, R. Wang, Z. Ju, T. Qin, and T.-Y . Liu, “Musicbert: Symbolic music understanding with large- scale pre-training,” arXiv preprint arXiv:2106.05630, 2021

    2021 arXiv

  7. [15]

    Clamp: Con- trastive language-music pre-training for cross-modal symbolic music information retrieval,

    S. Wu, D. Yu, X. Tan, and M. Sun, “Clamp: Con- trastive language-music pre-training for cross-modal symbolic music information retrieval,” arXiv preprint arXiv:2304.11029, 2023

    2023 arXiv

  8. [16]

    Learning-based methods for comparing se- quences, with applications to audio-to-midi alignment and matching,

    C. Raffel, “Learning-based methods for comparing se- quences, with applications to audio-to-midi alignment and matching,” Ph.D. dissertation, Columbia University, 2016

    2016

  9. [17]

    Simple and control- lable music generation,

    J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Syn- naeve, Y . Adi, and A. Défossez, “Simple and control- lable music generation,” Advances in Neural Informa- tion Processing Systems , vol. 36, pp. 47 704–47 720, 2023

    2023

  10. [18]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PmLR, 2021, pp. 8748–8763

    2021

  11. [19]

    Hubert: Self- supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self- supervised speech representation learning by masked prediction of hidden units,” 2021. [Online]. Available: https://arxiv.org/abs/2106.07447

    2021 arXiv

  12. [20]

    Lima: Less is more for alignment,

    C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y . Mao, X. Ma, A. Efrat, P. Yu, L. Yu et al., “Lima: Less is more for alignment,” Advances in Neural Information Processing Systems, vol. 36, pp. 55 006–55 021, 2023

    2023

  13. [21]

    Big transfer (bit): General visual representation learning,

    A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung, S. Gelly, and N. Houlsby, “Big transfer (bit): General visual representation learning,” in Computer Vision– ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16. Springer, 2020, p...

    2020

  14. [22]

    Audiogen: Textually guided audio generation,

    F. Kreuk, G. Synnaeve, A. Polyak, U. Singer, A. Dé- fossez, J. Copet, D. Parikh, Y . Taigman, and Y . Adi, “Audiogen: Textually guided audio generation,”arXiv preprint arXiv:2209.15352, 2022

    2022 arXiv

  15. [23]

    Audiolm: A language mod- eling approach to audio generation,

    Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grang- ier, M. Tagliasacchi et al., “Audiolm: A language mod- eling approach to audio generation,” IEEE/ACM trans- actions on audio, speech, and language processing , vol. 31, ...

    2023

  16. [24]

    High fidelity neural audio compression,

    A. Défossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,”arXiv preprint arXiv:2210.13438, 2022

    2022 arXiv

  17. [25]

    Soundstream: An end-to-end neu- ral audio codec,

    N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “Soundstream: An end-to-end neu- ral audio codec,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 30, pp. 495– 507, 2021

    2021

  18. [26]

    Musiclm: Generat- ing music from text,

    A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi et al., “Musiclm: Generat- ing music from text,” arXiv preprint arXiv:2301.11325, 2023

    2023 arXiv

  19. [27]

    Pijama: Piano jazz with automatic midi annotations,

    D. Edwards, S. Dixon, and E. Benetos, “Pijama: Piano jazz with automatic midi annotations,” Transactions of the International Society for Music Information Re- trieval, 2023

    2023

  20. [28]

    Soundstorm: Efficient parallel audio generation,

    Z. Borsos, M. Sharifi, D. Vincent, E. Kharitonov, N. Zeghidour, and M. Tagliasacchi, “Soundstorm: Efficient parallel audio generation,” arXiv preprint arXiv:2305.09636, 2023

    2023 arXiv

  21. [29]

    Anticipatory music transformer,

    J. Thickstun, D. Hall, C. Donahue, and P. Liang, “Anticipatory music transformer,” arXiv preprint arXiv:2306.08620, 2023

    2023 arXiv

  22. [30]

    wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural infor- mation processing systems, vol. 33, pp. 12 449–12 460, 2020

    2020

  23. [31]

    Auto- matic music transcription: An overview,

    E. Benetos, S. Dixon, Z. Duan, and S. Ewert, “Auto- matic music transcription: An overview,”IEEE Signal Processing Magazine, vol. 36, no. 1, pp. 20–30, 2018

    2018

  24. [32]

    The autoregressive paradigm for symbolic music generation, which models music as a stream of tokens, gained traction by adapting architectures from natural language processing [33]

    demonstrated that neural networks are effective tools for modeling musical harmonies in Baroque music. The autoregressive paradigm for symbolic music generation, which models music as a stream of tokens, gained traction by adapting architectures from natural language processin...

  25. [33]

    High- resolution piano transcription with pedals by regressing onset and offset times,

    Q. Kong, B. Li, X. Song, Y . Wan, and Y . Wang, “High- resolution piano transcription with pedals by regressing onset and offset times,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3707–3717, 2021

    2021

  26. [34]

    Automatic piano transcription with hierarchical frequency-time transformer,

    K. Toyama, T. Akama, Y . Ikemiya, Y . Takida, W.-H. Liao, and Y . Mitsufuji, “Automatic piano transcription with hierarchical frequency-time transformer,” arXiv preprint arXiv:2307.04305, 2023

    2023 arXiv

  27. [35]

    Scoring time intervals using non- hierarchical transformer for automatic piano transcrip- tion,

    Y . Yan and Z. Duan, “Scoring time intervals using non- hierarchical transformer for automatic piano transcrip- tion,” arXiv preprint arXiv:2404.09466, 2024

    2024 arXiv

  28. [36]

    Giantmidi- piano: A large-scale midi dataset for classical piano music,

    Q. Kong, B. Li, J. Chen, and Y . Wang, “Giantmidi- piano: A large-scale midi dataset for classical piano music,” arXiv preprint arXiv:2010.07061, 2020

    2010 arXiv

  29. [37]

    Atepp: A dataset of automatically transcribed expres- sive piano performance,

    H. Zhang, J. Tang, S. Rafee, S. Dixon, and G. Fazekas, “Atepp: A dataset of automatically transcribed expres- sive piano performance,” in Proceedings of the 23rd International Society for Music Information Retrieval Conference (ISMIR), 2022

    2022

  30. [38]

    Aria-midi: A dataset of piano midi files for symbolic music modeling,

    L. Bradshaw and S. Colton, “Aria-midi: A dataset of piano midi files for symbolic music modeling,” in International Conference on Learning Representations,

  31. [39]

    Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions,

    Y .-S. Huang and Y .-H. Yang, “Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions,” in Proceedings of the 28th ACM international conference on multimedia, 2020, pp. 1180– 1188

    2020

  32. [40]

    Museformer: Transformer with fine-and coarse-grained attention for music generation,

    B. Yu, P. Lu, R. Wang, W. Hu, X. Tan, W. Ye, S. Zhang, T. Qin, and T.-Y . Liu, “Museformer: Transformer with fine-and coarse-grained attention for music generation,” Advances in neural information processing systems , vol. 35, pp. 1376–1388, 2022

    2022

  33. [41]

    Suno AI v3.5,

    I. Suno, “Suno AI v3.5,” 2024, computer software. [Online]. Available: https://sunnoai.com/v3-5/

    2024

  34. [42]

    Deepbach: A steerable model for bach chorales generation,

    G. Hadjeres, F. Pachet, and F. Nielsen, “Deepbach: A steerable model for bach chorales generation,” in In- ternational conference on machine learning. PMLR, 2017, pp. 1362–1371

    2017

  35. [43]

    Counterpoint by convolution,

    C.-Z. A. Huang, T. Cooijmans, A. Roberts, A. Courville, and D. Eck, “Counterpoint by convolution,” arXiv preprint arXiv:1903.07227, 2019

    1903 arXiv

  36. [44]

    Automatic stylistic composition of bach chorales with deep lstm,

    F. T. Liang, M. Gotham, M. Johnson, and J. Shotton, “Automatic stylistic composition of bach chorales with deep lstm,” in ISMIR, 2017, pp. 449–456

    2017

  37. [45]

    This time with feeling: Learning expressive musical performance,

    S. Oore, I. Simon, S. Dieleman, D. Eck, and K. Si- monyan, “This time with feeling: Learning expressive musical performance,” Neural Computing and Applica- tions, vol. 32, pp. 955–967, 2020

    2020

  38. [46]

    Music transformer,

    C.-Z. A. Huang, A. Vaswani, J. Uszkoreit, N. Shazeer, I. Simon, C. Hawthorne, A. M. Dai, M. D. Hoffman, M. Dinculescu, and D. Eck, “Music transformer,” arXiv preprint arXiv:1809.04281, 2018

    2018 arXiv

  39. [47]

    Enabling factorized piano music modeling and generation with the maestro dataset,

    C. Hawthorne, A. Stasyuk, A. Roberts, I. Simon, C.- Z. A. Huang, S. Dieleman, E. Elsen, J. Engel, and D. Eck, “Enabling factorized piano music modeling and generation with the maestro dataset,”arXiv preprint arXiv:1810.12247, 2018

    arXiv

  40. [48]

    Generating piano music with transformer,

    I. Simon, C.-Z. A. Huang, J. Engel, C. Hawthorne, and M. Dinculescu, “Generating piano music with transformer,” https://magenta.tensorflow.org/ piano-transformer, September 2019, blog post. [Online]. Available: https://magenta.tensorflow.org/ piano-transformer

    2019

  41. [49]

    Musenet,

    C. Payne, “Musenet,” 2019, openAI, 25 Apr. 2019. [Online]. Available: https://openai.com/blog/musenet

    2019

  42. [50]

    MIDI specification,

    “MIDI specification,” 1996. [Online]. Available: https://midi.org/midi-1-0-detailed-specification

    1996

  43. [51]

    Sequence-to-sequence piano transcription with transformers,

    C. Hawthorne, I. Simon, R. Swavely, E. Manilow, and J. Engel, “Sequence-to-sequence piano transcription with transformers,” arXiv preprint arXiv:2107.09142, 2021

    2021 arXiv

  44. [52]

    Figaro: Generating symbolic music with fine-grained artistic control,

    D. von Rütte, L. Biggio, Y . Kilcher, and T. Hofmann, “Figaro: Generating symbolic music with fine-grained artistic control,” arXiv preprint arXiv:2201.10936 , 2022

    2022 arXiv

  45. [53]

    Musecoco: Generating symbolic music from text,

    P. Lu, X. Xu, C. Kang, B. Yu, C. Xing, X. Tan, and J. Bian, “Musecoco: Generating symbolic music from text,” arXiv preprint arXiv:2306.00110, 2023

    2023 arXiv

  46. [54]

    Abc notation,

    C. Walshaw, “Abc notation,” abcnotation.com, 2008, retrieved 1 March 2008

    2008

  47. [55]

    A hierarchical latent vector model for learning long-term structure in music,

    A. Roberts, J. Engel, C. Raffel, C. Hawthorne, and D. Eck, “A hierarchical latent vector model for learning long-term structure in music,” in International confer- ence on machine learning. PMLR, 2018, pp. 4364– 4373

    2018

  48. [56]

    Clamp 2: Multimodal music information retrieval across 101 lan- guages using large language models,

    S. Wu, Y . Wang, R. Yuan, Z. Guo, X. Tan, G. Zhang, M. Zhou, J. Chen, X. Mu, Y . Gao et al., “Clamp 2: Multimodal music information retrieval across 101 lan- guages using large language models,” arXiv preprint arXiv:2410.13267, 2024

    arXiv

  49. [57]

    Clamp 3: Universal music information retrieval across unaligned modalities and unseen languages,

    S. Wu, Z. Guo, R. Yuan, J. Jiang, S. Doh, G. Xia, J. Nam, X. Li, F. Yu, and M. Sun, “Clamp 3: Universal music information retrieval across unaligned modalities and unseen languages,” arXiv preprint arXiv:2502.10362, 2025

    2025 arXiv

  50. [58]

    Joint beat and downbeat tracking with recurrent neural networks

    S. Böck, F. Krebs, and G. Widmer, “Joint beat and downbeat tracking with recurrent neural networks.” in ISMIR. New York City, 2016, pp. 255–261

    2016

  51. [59]

    Exploring the limits of transfer learning with a unified text-to- text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to- text transformer,”Journal of machine learning research, vol. 21, no. 140, pp. 1–67, 2020

    2020

  52. [60]

    MusicXML: An internet-friendly format for sheet music,

    M. Good, “MusicXML: An internet-friendly format for sheet music,” in Proceedings of XML 2001 Conference , 2001. [Online]. Avail- able: https://michaelgood.info/publications/music/ musicxml-an-internet-friendly-format-for-sheet-music/

    2001

  53. [61]

    A simple framework for contrastive learning of visual rep- resentations,

    T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual rep- resentations,” in International conference on machine learning, 2020, pp. 1597–1607

    2020

  54. [62]

    Contrastive learning of musical representations,

    J. Spijkervet and J. A. Burgoyne, “Contrastive learning of musical representations,” arXiv preprint arXiv:2103.09410, 2021

    2021 arXiv

  55. [63]

    Teaching arithmetic to small transformers,

    N. Lee, K. Sreenivasan, J. D. Lee, K. Lee, and D. Papailiopoulos, “Teaching arithmetic to small transformers,” 2023. [Online]. Available: https: //arxiv.org/abs/2307.03381

    2023 arXiv

  56. [64]

    Transformers can do arithmetic with the right embeddings,

    S. McLeish, A. Bansal, A. Stein, N. Jain, J. Kirchenbauer, B. R. Bartoldson, B. Kailkhura, A. Bhatele, J. Geiping, A. Schwarzschild, and T. Goldstein, “Transformers can do arithmetic with the right embeddings,” 2024. [Online]. Available: https://arxiv.org/abs/2405.17399

    2024 arXiv

  57. [65]

    Compound word transformer: Learning to compose full-song music over dynamic directed hypergraphs,

    W.-Y . Hsiao, J.-Y . Liu, Y .-C. Yeh, and Y .-H. Yang, “Compound word transformer: Learning to compose full-song music over dynamic directed hypergraphs,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 1, 2021, pp. 178–186

    2021

  58. [66]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Ka- dian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  59. [67]

    Training compute- optimal large language models,

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hen- dricks, J. Welbl, A. Clark et al., “Training compute- optimal large language models,” arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  60. [68]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural informa- tion processing systems, vol. 30, 2017

    2017

  61. [69]

    Layer normal- ization,

    J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normal- ization,” arXiv preprint arXiv:1607.06450, 2016

    2016 arXiv

  62. [70]

    Gqa: Training generalized multi-query transformer models from multi-head check- points,

    J. Ainslie, J. Lee-Thorp, M. De Jong, Y . Zemlyanskiy, F. Lebrón, and S. Sanghai, “Gqa: Training generalized multi-query transformer models from multi-head check- points,” arXiv preprint arXiv:2305.13245, 2023

    2023 arXiv

  63. [71]

    Root mean square layer normalization,

    B. Zhang and R. Sennrich, “Root mean square layer normalization,” Advances in Neural Information Pro- cessing Systems, vol. 32, 2019

    2019

  64. [74]

    Towards proper contrastive self-supervised learning strategies for mu- sic audio representation,

    J. Choi, S. Jang, H. Cho, and S. Chung, “Towards proper contrastive self-supervised learning strategies for mu- sic audio representation,” in 2022 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2022, pp. 1–6

    2022

  65. [75]

    Simcse: Simple contrastive learning of sentence embeddings,

    T. Gao, X. Yao, and D. Chen, “Simcse: Simple contrastive learning of sentence embeddings,” arXiv preprint arXiv:2104.08821, 2021

    2021 arXiv

  66. [76]

    Improving text embeddings with large language models,

    L. Wang, N. Yang, X. Huang, L. Yang, R. Majumder, and F. Wei, “Improving text embeddings with large language models,” arXiv preprint arXiv:2401.00368 , 2023

    2023 arXiv

  67. [77]

    Mulan: A joint embedding of music audio and natural language,

    Q. Huang, A. Jansen, J. Lee, R. Ganti, J. Y . Li, and D. P. Ellis, “Mulan: A joint embedding of music audio and natural language,” arXiv preprint arXiv:2208.12415 , 2022

    2022 arXiv

  68. [78]

    Pianoteq,

    Modartt, “Pianoteq,” https://www.modartt.com/ pianoteq, accessed: 2025-03-28

    2025

  69. [79]

    Midibert-piano: Large-scale pre-training for symbolic music understanding,

    Y .-H. Chou, I. Chen, C.-J. Chang, J. Ching, Y .-H. Yang et al., “Midibert-piano: Large-scale pre-training for symbolic music understanding,” arXiv preprint arXiv:2107.05223, vol. 2, 2021

    2021 arXiv

  70. [80]

    Learning to generate music with sentiment,

    L. N. Ferreira and J. Whitehead, “Learning to generate music with sentiment,” arXiv preprint arXiv:2103.06125, 2021

    2021 arXiv

  71. [81]

    Mert: Acoustic music understanding model with large-scale self-supervised training,

    Y . Li, R. Yuan, G. Zhang, Y . Ma, X. Chen, H. Yin, C. Xiao, C. Lin, A. Ragni, E. Benetos et al. , “Mert: Acoustic music understanding model with large-scale self-supervised training,” arXiv preprint arXiv:2306.00107, 2023

    2023 arXiv

  72. [82]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” inProceedings of the 2019 conference of the North American chapter of the asso- ciation for computational linguistics: human language technolog...

    2019

  73. [1536]

    This decreased the parameter count by roughly half, balancing model capacity with computational efficiency for our dataset scale. Secondly, we simplified the architec- ture by opting for standard multi-head attention (with 24 heads) and layer normalization [57, 58], instead of...

    2000

  74. [2025]

    Available: https://openreview.net/ forum?id=X5hrhgndxW

    [Online]. Available: https://openreview.net/ forum?id=X5hrhgndxW

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.