Pith. sign in

REVIEW 3 major objections 5 minor 27 references

OMAR-RQ: Open Music Audio Representation Model Trained with Multi-Feature Masked Token Prediction

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OMAR-RQ, a fully open 580M-parameter music audio model trained with masked token prediction over multiple parallel codebooks and multiple audio features, claims to be the first open self-supervised model to reach state-of-the-art results…

desk verdict A genuinely useful open music representation model with a real internal inconsistency in its SOTA claim—worth engaging, but the abstract needs fixing. read the letter →

arxiv 2507.03482 v1 pith:2GJTXWLA submitted 2025-07-04 cs.SD eess.AS

Pablo Alonso-Jiménez , Pedro Ramoneda , R. Oguz Araz , Andrea Poltronieri , Dmitry Bogdanov This is my paper · ORCID
classification cs.SDeess.AS
keywords self-supervisedlearningmusicrepresentationmaskedtokenpredictioninformationretrievalaudioembeddingsmulti-featurequantizationopen-sourcemodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OMAR-RQ is an attempt to build a strong, fully open music audio representation model. The authors pre-train a Conformer encoder on 330,000 hours of music audio by masked token prediction, following the BEST-RQ recipe but extending it to several parallel random codebooks and several quantized input features (mel spectrograms, CQT, EnCodec embeddings, and waveform). The paper's headline claim is that on six music information retrieval tasks, the resulting 580M-parameter model reaches the best results among open self-supervised models. If the claim holds, the released weights and training code give researchers a reproducible representation that can be fine-tuned cheaply with small probes, while also offering a concrete recipe for future SSL audio models.

What carries the argument

The load-bearing mechanism is a masked token classification pre-training objective in the style of BEST-RQ, defined in one phrase as a random-projection quantizer with masked token prediction: a frozen random projection maps audio feature frames to a codebook by nearest-neighbor search, the model sees patches with 60% of 0.4-second chunks masked, and its job is to predict the quantized tokens of the masked regions. OMAR-RQ's contribution is to run this with K parallel codebooks (e.g., four codebooks of 8192 codewords each) over multiple feature streams (mel, CQT, EnCodec, waveform), turning the task into a multi-label classification over a union of codebooks, and optionally replacing the random codebook with finite scalar quantization. The multi-codebook design attacks codebook underutilization, and the multi-feature design attacks coverage of the target space, which together are what the paper credits for the gains.

What would settle it

Run each probe training five times with different random seeds for the best OMAR-RQ variant, MERT, and MusicFM on the same splits, and compute 95% confidence intervals; if the intervals for tagging mAP, pitch accuracy, and chord accuracy overlap substantially between the best OMAR-RQ variant and the runner-up, the claimed state-of-the-art is not established.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that enriching the target space of masked token prediction is what drives downstream quality in music SSL. Starting from BEST-RQ's single random codebook, the authors show that using four parallel codebooks raises auto-tagging mean average precision and pre-training validation accuracy, and that combining multiple features as targets systematically improves pitch and chord accuracy over single-feature settings. The high-rate FSQ variant reaches 0.940 pitch accuracy and 0.749 chord accuracy, the multi-codebook mel model reaches 0.488 mAP on MagnaTagATune tagging, and the base model reaches 1.65 mean squared error on piano difficulty, each the best or tied best among open models in the comparison table. The paper interprets these results as showing that a model with more diverse, higher-coverage quantization targets learns a representation that serves both high-level semantic tasks and low-level temporal tasks.

Load-bearing premise

The state-of-the-art claim assumes the single-run numbers in Table 4 are stable enough that small gaps between models reflect true ordering rather than random noise; if run-to-run variance approaches those gaps, the ranking could change.

Editorial extensions

If this is right

  • Any music information retrieval system can start from a 580M-parameter open representation and reach competitive results with 2-layer MLP probes, removing the need to train task-specific models from scratch.
  • The multi-feature target recipe gives a concrete design choice: mel-family targets for semantic tasks such as tagging and segmentation, CQT/EnCodec targets for pitch and chords, and higher-rate variants for beat tracking.
  • The open release of weights and of training/evaluation code means the reported numbers are independently reproducible and improvable, rather than locked inside a closed model.
  • The paper's two observed trends, that high-level semantic tasks favor mel targets and lower rates while low-level temporal tasks favor tonal targets and higher rates, give future SSL training a clear trade-off to balance.
  • Because closed systems still report higher numbers in undisclosed settings, the open-model state-of-the-art claim establishes a new public ceiling while leaving room for further gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An editorial reading of Table 4: MERT's beat F1 of 0.868 exceeds every OMAR-RQ variant (best 0.855), so the abstract's beat-tracking claim appears to depend on comparing the best overall model rather than the best row per task; confidence intervals from repeated runs would settle which reading is fair.
  • A natural scaling test the paper does not run is to train the same multi-codebook, multi-feature recipe on a smaller corpus (e.g., 30,000 hours); if the gains persist at lower data scale, the recipe becomes accessible beyond groups with hundreds of GPUs.
  • The distinction between mel targets (semantic tasks) and CQT/EnCodec targets (temporal/tonal tasks) suggests a future single model could expose multiple feature-specific readout heads instead of one averaged embedding.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents OMAR-RQ, a Conformer-based self-supervised music audio representation model pretrained on 330,000 hours of YouTube audio via masked token classification. Building on BEST-RQ, the authors propose multi-codebook random projection targets, multi-feature targets (mel, CQT, EnCodec, waveform), and finite scalar quantization alternatives. They release the pretrained weights, training code, and evaluation pipelines. The main empirical claim, stated in the abstract and conclusion, is that OMAR-RQ achieves state-of-the-art performance among open self-supervised models on music tagging, pitch estimation, chord recognition, beat tracking, segmentation, and difficulty estimation, based on the benchmark in Table 4.

Significance. If the SOTA claim is established, OMAR-RQ would be a valuable open resource for music information retrieval: the released code and weights, the large-scale pretraining corpus, and the systematic ablations on codebook count and target features are concrete strengths. The paper is also careful to report downstream probe setups (2-layer MLPs, standard datasets) and training details. However, the headline claim currently rests on single-run benchmark differences, several of which are small, and it is partly contradicted by the paper's own results for beat tracking. The contribution is therefore significant in potential but needs stronger or more carefully qualified empirical support.

major comments (3)
  1. [Abstract, Section 5, Table 4] The abstract's claim of state-of-the-art performance in beat tracking is internally inconsistent with Table 4 and Section 5: MERT achieves beat F1 0.868 while the best OMAR-RQ variant (high-rate FSQ) achieves 0.855, and Section 5 explicitly states that MERT performs best on beat tracking. The SOTA statement should be revised to exclude beat tracking or to define the comparison criterion precisely (e.g., average performance across tasks).
  2. [Section 5 and Table 4] The central SOTA claim rests entirely on Table 4, in which every metric is reported as a single run with no seeds, variance, or significance tests. Several differences underpinning the claim are small, for example tagging 0.488 vs 0.448 mAP, pitch 0.940 vs 0.922 accuracy, and chords 0.749 vs 0.644. Because the best OMAR-RQ variant is selected after evaluating on these same benchmarks, the reported ordering may reflect selection bias and run-to-run noise. The authors should provide repeated runs with confidence intervals, or explicitly weaken the SOTA wording to 'competitive' and identify the tasks where the gaps are large enough to be robust.
  3. [Section 5] The sentence 'the base model achieved the highest MSE value in difficulty estimation, following the same trend as auto-tagging' is incorrect as written: Table 4 shows the base model has the lowest MSE (1.65), which is the best result for an error metric, not the highest. This misreporting of a result needs correction.
minor comments (5)
  1. [Table 2 caption] The caption refers to 'CTQ' but the row label and the text use 'CQT'; this typo should be fixed.
  2. [Table 4 header] The column header 'RF' is not defined, and the units of the 'Rate' column are unclear from the table alone; a brief definition in the caption would improve readability.
  3. [Section 4.2] The phrase '8196 codewords' appears to be a typo for 8192 codewords, consistent with Table 1; please verify.
  4. [Section 1] The statement that recent systems 'lack openly distributed weights' should be checked for precision, since it could be read as applying to all systems cited in [25] and does not mention that MusicFM provides open weights for some versions, as stated later in Section 5.
  5. [Section 3.1] The definition of the structure segmentation task would be clearer if the boundary classifier's evaluation metric were specified, since the text only mentions frame-wise accuracy for the functional classifier.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: OMAR-RQ is an empirical benchmark with independent external evaluations, and the self-citations present are for dataset provenance, not load-bearing.

full rationale

The paper does not present an analytic derivation chain; its claims are empirical. The pre-training targets (mel-spectrograms, CQT, EnCodec embeddings, and waveform) are external features, and the downstream tasks use standard external datasets (MagnaTagATune, NSynth, Isophonics/Billboard/Uspop/RWC-Pop, HarmonixSet/GTZAN, and PianoSyllabus). Probes are trained only after pre-training and on held-out labels, so no fitted parameter is renamed as a prediction. The variant selection in Table 4 is empirical model selection, not a quantity that is equivalent to its inputs by construction. The self-citations in Section 4.1 ('Following previous work [1, 2, 13], we relied on the Discogs music database to create our dataset') point to the authors' earlier MAEST data-construction work, but the dataset is a released data resource, not a theorem assumed to prove the SOTA claim; it is therefore not load-bearing. The abstract's beat-tracking SOTA statement is internally inconsistent with Section 5 ('MERT performing the best') and Table 4 (MERT 0.868 vs. best OMAR-RQ 0.855), but that is a correctness/consistency issue, not circularity. Accordingly, no circular step can be quoted and no reduction-by-construction exists; score 0.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The free parameters are engineering choices that determine the final model; none are derived from first principles. The axioms are domain assumptions inherited from the pretraining and evaluation literature. No new particles, forces, dimensions, or theoretical entities are introduced.

free parameters (5)
  • Number of parallel codebooks and codewords = 1x32768, 4x8192, or 16x2048; final multi-codebook uses 4x8192
    Chosen from Table 1 based on pre-training validation accuracy and downstream tagging; not derived from theory.
  • Target feature combination = enc/mel/cqt/audio for multi-feature variants
    Selected from the Table 3 sweep; the claimed gains depend on this specific combination.
  • Masking ratio and chunk size = 60% of 30s segment masked in 0.4s chunks
    Inherited from BEST-RQ; not optimized for music, but affects pretraining difficulty and downstream performance.
  • FSQ quantization settings = 5 channels, 6 levels
    Reported for the high-rate FSQ variant without a published sweep; the best FSQ configuration is a hand-chosen setting.
  • Model size and training budget = 580M params, 400k steps, 330k hours, batch 256
    Chosen by the authors; represents the compute available and affects the SOTA comparison against models of different sizes.
assumptions (3)
  • domain assumption Masked token prediction with random-projection quantizers transfers to downstream music tasks.
    Adopted from BEST-RQ and MusicFM; the paper relies on this premise as the training objective without proving the transfer.
  • domain assumption The Discogs-filtered YouTube corpus is a large, diverse, high-quality music dataset.
    Section 4.1 states this but provides no explicit audio quality filters or genre balance analysis; the model's generalization depends on this premise.
  • domain assumption Probing frozen representations with 2-layer MLPs on standard datasets is a valid measure of representation quality.
    Section 3.1 defines this protocol; probe hyperparameters and splits can change rankings, and the paper gives no error bars.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OMAR-RQ: Open Music Audio Representation Model Trained with Multi-Feature Masked Token Prediction." pith.science (2026). https://pith.science/paper/2GJTXWLA

@misc{pith2026250703482,
  author       = {Pith},
  title        = {Pith review of: OMAR-RQ: Open Music Audio Representation Model Trained with Multi-Feature Masked Token Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2GJTXWLA}},
  note         = {Machine review of arXiv:2507.03482}
}
read the original abstract

Developing open-source foundation models is essential for advancing research in music audio understanding and ensuring access to powerful, multipurpose representations for music information retrieval. We present OMAR-RQ, a model trained with self-supervision via masked token classification methodologies using a large-scale dataset with over 330,000 hours of music audio. We experiment with different input features and quantization options, and achieve state-of-the-art performance in music tagging, pitch estimation, chord recognition, beat tracking, segmentation, and difficulty estimation among open self-supervised models. We open-source our training and evaluation pipelines and model weights, available at https://github.com/mtg/omar-rq.

Figures

Figures reproduced from arXiv: 2507.03482 by the authors.

Figure 1
Figure 1. Diagram of the OMAR-RQ architecture. and finite scale quantizer [19] settings to optimize performance. We train and benchmark variations of OMAR-RQ in various music analysis tasks. Our results show that the proposed model performs competitively with recent closed-source systems. To promote open￾source music audio representation models, we release all pretrained model weights and source code for training and evaluati… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 19 canonical work pages

  1. [1]

    Pablo Alonso-Jiménez, Xavier Serra, and Dmitry Bogdanov. 2023. Efficient Supervised Training of Audio Transformers for Music Representation Learning. In Intl. Society for Music Information Retrieval Conf. (ISMIR)

    work page 2023

  2. [2]

    Pablo Alonso-Jiménez, Xavier Serra, and Bogdanov Dmitry. 2022. Music Repre- sentation Learning Based on Editorial Metadata From Discogs. In Intl. Society for Music Information Retrieval Conf. (ISMIR)

    work page 2022

  3. [3]

    Adam Berenzweig, Beth Logan, Daniel PW Ellis, and Brian Whitman. 2004. A large-scale evaluation of acoustic and subjective music-similarity measures. Computer Music Journal (2004), 63–76

    work page 2004

  4. [4]

    John Ashley Burgoyne, Jonathan Wild, and Ichiro Fujinaga. 2011. An Expert Ground Truth Set for Audio Chord Recognition and Music Analysis. In Intl. Society for Music Information Retrieval Conf. (ISMIR) , Vol. 11. 633–638

    work page 2011

  5. [5]

    Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. 2022. Self- supervised learning with random-projection quantizer for speech recognition. In International Conference on Machine Learning . PMLR, 3915–3924

    work page 2022

  6. [6]

    Fu, Stefano Ermon, Atri Rudra, and Christopher R’e

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher R’e. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. ArXiv abs/2205.14135 (2022)

    arXiv 2022

  7. [7]

    Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. 2022. High fidelity neural audio compression. arXiv:2210.13438 (2022)

    arXiv 2022

  8. [8]

    Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Douglas Eck, Karen Simonyan, and Mohammad Norouzi. 2017. Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders. arXiv:1704.01279 (2017)

    arXiv 2017

Show all 27 references
  1. [9]

    Masataka Goto, Hiroki Hashiguchi, Takuichi Nishimura, and Ryuichi Oka. 2002. RWC Music Database: Popular, Classical and Jazz Music Databases. InIntl. Society for Music Information Retrieval Conf. (ISMIR) , Vol. 2. 287–288

    2002

  2. [10]

    Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. 2020. Conformer: Convolution-augmented transformer for speech recognition. arXiv:2005.08100 (2020)

    2020 arXiv

  3. [11]

    Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. 2022. Masked autoencoders that listen. Advances in Neural Information Processing Systems 35 (2022)

    2022

  4. [12]

    Adrian Łańcucki, Jan Chorowski, Guillaume Sanchez, Ricard Marxer, Nanxin Chen, Hans JGA Dolfing, Sameer Khurana, Tanel Alumäe, and Antoine Laurent

  5. [13]

    Luca Lanzendörfer, Florian Grötschla, Emil Funke, and Roger Wattenhofer. 2023. DISCO-10M: A large-scale music dataset. Advances in Neural Information Pro- cessing Systems 36 (2023), 54451–54471

    2023

  6. [14]

    Edith Law, Kris West, Michael I Mandel, Mert Bay, and J Stephen Downie. 2009. Evaluation of algorithms using games: The case of music tagging.. InIntl. Society for Music Information Retrieval Conf. (ISMIR) . 387–392

    2009

  7. [15]

    Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Cheng- hao Xiao, Chenghua Lin, Anton Ragni, Emmanouil Benetos, et al. 2023. Mert: Acoustic music understanding model with large-scale self-supervised training. arXiv:2306.00107 (2023)

    2023 arXiv

  8. [16]

    Ugo Marchand, Quentin Fresnel, and Geoffroy Peeters. 2015. Gtzan-rhythm: Extending the gtzan test-set with beat, downbeat and swing annotations. (2015)

    2015

  9. [17]

    Matthias Mauch, Chris Cannam, Matthew Davies, Simon Dixon, Christopher Harte, Sefki Kolozali, Dan Tidhar, and Mark Sandler. 2009. OMRAS2 metadata project 2009. In Intl. Society for Music Information Retrieval Conf. (ISMIR)

    2009

  10. [18]

    Matthew C McCallum, Filip Korzeniowski, Sergio Oramas, Fabien Gouyon, and Andreas F Ehmann. 2022. Supervised and Unsupervised Learning of Audio Representations for Music Understanding. In Intl. Society for Music Information Retrieval Conf. (ISMIR)

    2022

  11. [19]

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. 2023. Finite scalar quantization: Vq-vae made simple. arXiv:2309.15505 (2023)

    2023 arXiv

  12. [20]

    Oriol Nieto, Matthew C McCallum, Matthew EP Davies, Andrew Robertson, Adam M Stark, and Eran Egozy. 2019. The Harmonix Set: Beats, Downbeats, and Functional Segment Annotations of Western Popular Music.. In Intl. Society for Music Information Retrieval Conf. (ISMIR) . 565–572

    2019

  13. [21]

    Valero-Mas, and Xavier Serra

    Pedro Ramoneda, Minhee Lee, Dasaem Jeong, Jose J. Valero-Mas, and Xavier Serra. 2025. Can Audio Reveal Music Performance Difficulty? Insights From the Piano Syllabus Dataset. IEEE Transactions on Audio, Speech and Language Processing 33 (2025), 1129–1141

    2025

  14. [22]

    Janne Spijkervet and John Ashley Burgoyne. 2021. Contrastive Learning of Musical Representations. In Intl. Society for Music Information Retrieval Conf. (ISMIR)

    2021

  15. [23]

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing 568 (2024), 127063

    2024

  16. [24]

    Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. 2024. Deepnet: Scaling transformers to 1,000 layers. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    2024

  17. [25]

    Minz Won, Yun-Ning Hung, and Duc Le. 2024. A foundation model for music informatics. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 1226–1230

    2024

  18. [26]

    Yu Zhang, James Qin, Daniel S Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc V Le, and Yonghui Wu. 2020. Pushing the limits of semi-supervised learning for automatic speech recognition. arXiv:2010.10504 (2020)

    2020 arXiv

  19. [2020]

    In2020 International Joint Conference on Neural Networks (IJCNN)

    Robust training of vector quantized bottleneck models. In2020 International Joint Conference on Neural Networks (IJCNN) . IEEE, 1–7

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.