REVIEW 4 major objections 4 minor 26 references
ASTAR-NTU solution to AudioMOS Challenge 2025 Track1
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Frozen-encoder DORA-MOS with Gaussian-softened ordinal training achieves 0.991 MI and 0.952 TA system-level SRCC on the official test set, beating the challenge baseline by 21.21% and 31.47%.
desk verdict Credible challenge win with a simple ordinal-label trick; the missing σ and hyperparameters are the real soft spots, not the method. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing pieces are three. First, the dual-branch encoders: MuQ, a self-supervised music representation model with mel residual vector quantization, provides frozen audio features, and RoBERTa provides frozen text embeddings; no encoder is fine-tuned, and the model learns only the fusion and prediction heads. Second, cross-attention fusion: for text alignment the projected text embeddings serve as queries and the projected audio sequence as keys and values, so the model learns text-informed audio representations that encode prompt-audio coherence. Third, the Gaussian-softened ordinal classifier: instead of scalar regression or one-hot classification, the target is a distribution over $K = 20$ equal-width bins spanning 1 to 5, with bin probabilities proportional to $\exp(-(s-c_k)^2 / 2\sigma^2)$ for true score $s$ and bin center $c_k$, normalized to sum to one; this makes the training loss rank-aware, so misplacing a score far from its bin is penalized more than a near miss, matching the Spearman criterion used for evaluation.
What would settle it
Run the full pipeline with every decision, including the Gaussian width $\sigma$, the temporal model, the pooling method, and the ensemble meta-model, fixed by development-split performance on a fresh partition of MusicEval systems and prompts, then apply it exactly once to a held-out set: if MI SRCC falls materially below 0.991, or if the best configuration changes between splits, the headline numbers reflected test-informed selection or evaluation noise rather than the ordinal training itself. A narrower calculation: sweep $\sigma$ from 0.05 to 1 and plot test SRCC; a sharp peak at one unpublished value would show the gain depends on a precisely chosen softening strength.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that mean-opinion-score prediction for text-to-music evaluation is best treated as an ordinal classification problem rather than a regression problem, provided the labels are softened. The authors train a dual-branch model in which a frozen MuQ audio encoder and a frozen RoBERTa text encoder feed a task-specific fusion: a transformer layer with attention pooling for the music impression branch, and a cross-attention layer in which text features act as queries over audio key/value features for the text alignment branch. The training target is a 20-bin probability distribution over scores 1 to 5, with each bin weighted by a Gaussian centered on the true continuous score. This single model obtains system-level SRCC of 0.991 (MI) and 0.952 (TA) on the official test set, outperforming L1 regression and hard-label cross-entropy in controlled ablations, outperforming Mamba and Bi-LSTM temporal models when paired with attention pooling, and outperforming the joint pretrained CLAP and MuQ-Mulan features on both scores. The submitted stacking ensemble, which combines nine models' predicted 20-bin distributions through Ridge regression, was slightly worse than the single model on the test set while still placing first in 13 of the 16 official challenge metrics.
Load-bearing premise
The system's reported gains rest on the assumption that the stratified development split fairly represents the official test set, so that the architecture choices and the never-reported Gaussian width $\sigma$ were settled without test labels, and that the small test set is large enough to rank systems on differences of a few thousandths in Spearman correlation.
Editorial extensions
If this is right
- Reframing MOS prediction as classification over score bins with Gaussian-smoothed targets raises system-level SRCC over both L1 regression and hard-label cross-entropy for MI and TA alike, so the training objective, not the architecture alone, is what drives the gain.
- Cross-attention fusion of separately frozen audio and text encoders surpasses joint pretrained contrastive models (CLAP, MuQ-Mulan) on this task, meaning prompt-music alignment for scoring is better learned at the fusion stage than borrowed from semantic similarity pretraining.
- Transformer temporal modeling with learned attention pooling outperforms Mamba and Bi-LSTM on both targets, though mean pooling becomes competitive for the linear-time encoders.
- No external data or augmentation is needed for this in-domain task, and the high test scores suggest the dataset may be approaching saturation for system-level ranking.
- Ensembling predicted 20-bin distributions with Ridge regression is stable but does not beat the best single model on the official test set, so the single DORA-MOS model is the configuration the authors would defend as the core result.
Reading between the lines
- A direct consequence the authors do not pursue: if Gaussian-softened bin classification is a good surrogate for Spearman correlation, training with an explicit differentiable rank-correlation surrogate on the same architecture would test how much of the gain is due to the smoothing versus the rank alignment of the cross-entropy objective.
- The Gaussian width $\sigma$ in Equation (1) is never reported even though it controls how strongly the ordinal signal is softened; a sensitivity sweep over $\sigma$ would show whether the 0.991 MI figure is a plateau or a sharp peak, and connects to a testable claim that the optimal width scales with annotator disagreement.
- Given the ensemble slightly underperformed the single model and LightGBM overfit the meta-train split, the official test set may be too small to distinguish systems within about 0.003 SRCC; re-running the same pipeline on multiple random dev/test splits would show whether the reported orderings are stable.
- The authors' own proposal to model annotator disagreement (e.g., a Beta distribution over scores) is a natural extension of the same machinery: the Gaussian-softened head already produces a predictive distribution, so calibrating its spread against the observed expert variance is an incremental change that could also absorb the never-reported $\sigma$ as a learned parameter.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes DORA-MOS, the first-place entry in the AudioMOS 2025 Challenge Track 1. The system uses frozen MuQ and RoBERTa encoders, a cross-attention fusion module, and a classification head over 20 score bins with Gaussian-softened ordinal labels. The central claim is that a single DORA-MOS model achieves system-level SRCC of 0.991 for Music Impression and 0.952 for Text Alignment on the official test set, corresponding to relative improvements of 21.21% and 31.47% over the challenge baseline, and that the final stacked ensemble ranked first in 13 of 16 official metrics.
Significance. If the reported configuration is fully specified, the result is a useful, independently validated system description for a practical MOS-prediction task. The external challenge evaluation supports the headline result, and the paper includes controlled ablations of the training criterion, temporal modeling, and pooling method, with an honest discussion of dataset saturation and of the ensemble underperforming the single model. The main weakness is that the exact configuration behind the headline numbers is not reported, so the internal comparisons cannot be reproduced or audited.
major comments (4)
- [Section II-C, Eq. (1)] The Gaussian width sigma is never reported, although Table I attributes the entire improvement of the proposed criterion over cross-entropy (MI SRCC 0.991 vs. 0.973; TA SRCC 0.952 vs. 0.940) to this softening. Please report the value of sigma, the grid searched, and the criterion used to choose it, together with the corresponding stratified-dev results; without this, the central performance claim cannot be reproduced or checked for test-set leakage.
- [Section III-A, Tables I-II] All ablation tables report official test-set numbers, with only a sentence asserting that the same ranking held on the internal dev split. Because the authors state that hyperparameter optimization was performed on the stratified split, please provide the dev-split equivalents of Tables I and II, or a summary table, so readers can verify that model selection did not use test labels. This is load-bearing for the claim that Gaussian softening and the Transformer/attention-pooling choices generalize.
- [Tables II-III] The system-level rank metrics are computed over a small number of systems (31 in Table III, 9 in Table IV), all results are from single runs, and adjacent configurations in Table II differ by as little as 0.004–0.016 in SRCC. Please report results over multiple seeds with mean and standard deviation, or provide a significance or confidence statement, so that the superiority of DORA-MOS over the second-best configuration is not attributable to noise.
- [Sections II-B and III] The paper does not specify training hyperparameters (epochs, optimizer, learning rate, batch size, projection dimensions, number of attention heads, pooling details, bin placement) or provide code, and the value of sigma in Eq. (1) is absent. Please include a complete configuration table or release code, since the headline 0.991/0.952 numbers are otherwise not independently verifiable.
minor comments (4)
- [Section III-C vs. Table IV] The final submission ('Ours', 0.988/0.944) is the Ensemble Ridge model, whereas Table III highlights the single DORA-MOS (0.991/0.952); the captions and Section III-D should state explicitly which configuration is being ranked by the challenge.
- [Section II-C] The description of CORAL says K−1 cumulative probabilities P(Y>k) for all k in {1,...,5}, which is inconsistent with K=20 bins; please clarify the number of ordinal thresholds.
- [Section II-B] The sentence 'Two out of the three best-performing models follow the principle shown in Fig. 1' is unclear; please specify which models these are and what the third model does differently.
- [Section II-A] The statement that the original dev set is 'reserved solely for final unseen data evaluation' conflicts with the later reporting of official test-set numbers; please clarify the role of the original dev set.
Circularity Check
No significant circularity; the reported SRCC gains are supported by external test-set evaluation and not by construction.
full rationale
The paper reports a trained system and its measured performance on the official AudioMOS 2025 Track 1 test set. The central claim (MI SRCC 0.991 and TA SRCC 0.952 in Table III) is an empirical result on external challenge data, not a quantity derived from the model's own definitions. Equation (1) defines a Gaussian-softened target distribution for training, and the ablations in Tables I and II compare training criteria and architectures against each other; no equation in the paper reduces a fitted parameter to the reported SRCC. The paper explicitly states that hyperparameter optimization was performed on an internal stratified split, and the official challenge test set provides independent evaluation. The only self-citations appear in the future-work paragraph (references [22] and [23] about annotator disagreement) and are not load-bearing for the main results. Reproducibility gaps, such as the unreported value of sigma and missing training hyperparameters, are legitimate concerns about transparency and statistical reporting, but they are not circularity: they do not make the reported predictions equivalent to the model inputs by construction. System-level SRCC over a small number of systems is a stability concern, not a circularity concern. Therefore, the derivation chain is self-contained and no significant circularity is present.
Assumptions & free parameters
free parameters (3)
- Gaussian kernel width sigma =
not reported
- Number of score bins K =
20
- Architecture hyperparameters =
1 transformer layer, 4 attention heads, 2-layer MLP
assumptions (4)
- domain assumption MuQ frozen features encode sufficient information for MI and TA prediction
- domain assumption The test set is in-domain and representative of the training distribution
- domain assumption System-level SRCC over the test set is a stable ranking metric
- standard math Gaussian kernel normalization produces a valid probability distribution
Cite this review
Pith. "Pith review of ASTAR-NTU solution to AudioMOS Challenge 2025 Track1." pith.science (2026). https://pith.science/paper/I7JXVECC
@misc{pith2026250709904,
author = {Pith},
title = {Pith review of: ASTAR-NTU solution to AudioMOS Challenge 2025 Track1},
year = {2026},
howpublished = {\url{https://pith.science/paper/I7JXVECC}},
note = {Machine review of arXiv:2507.09904}
}
read the original abstract
Evaluation of text-to-music systems is constrained by the cost and availability of collecting experts for assessment. AudioMOS 2025 Challenge track 1 is created to automatically predict music impression (MI) as well as text alignment (TA) between the prompt and the generated musical piece. This paper reports our winning system, which uses a dual-branch architecture with pre-trained MuQ and RoBERTa models as audio and text encoders. A cross-attention mechanism fuses the audio and text representations. For training, we reframe the MI and TA prediction as a classification task. To incorporate the ordinal nature of MOS scores, one-hot labels are converted to a soft distribution using a Gaussian kernel. On the official test set, a single model trained with this method achieves a system-level Spearman's Rank Correlation Coefficient (SRCC) of 0.991 for MI and 0.952 for TA, corresponding to a relative improvement of 21.21\% in MI SRCC and 31.47\% in TA SRCC over the challenge baseline.
Figures
Reference graph
Works this paper leans on
-
[1]
Audioldm: Text-to-audio generation with latent diffu- sion models,
H. Liu, Z. Chen, Y . Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley, “Audioldm: Text-to-audio generation with latent diffu- sion models,” International Conference on Machine Learning (ICML) , 2023
work page 2023
-
[2]
Audioldm 2: Learning holistic audio generation with self-supervised pretraining,
H. Liu, Y . Yuan, X. Liu, X. Mei, Q. Kong, Q. Tian, Y . Wang, W. Wang, Y . Wang, and M. D. Plumbley, “Audioldm 2: Learning holistic audio generation with self-supervised pretraining,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , 2024
work page 2024
-
[3]
Audiox: Diffusion transformer for anything-to-audio generation,
Z. Tian, Y . Jin, Z. Liu, R. Yuan, X. Tan, Q. Chen, W. Xue, and Y .-T. Guo, “Audiox: Diffusion transformer for anything-to-audio generation,” ArXiv Preprint, 2025
work page 2025
-
[4]
Musiceval: A generative music corpus with expert ratings for automatic text-to-music evaluation,
C. Liu, H. Wang, J. Zhao, S. Zhao, H. Bu, X. Xu, J. Zhou, H. Sun, and Y . Qin, “Musiceval: A generative music corpus with expert ratings for automatic text-to-music evaluation,” ICASSP, 2025
work page 2025
-
[5]
Musiclm: Generating music from text,
A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi et al. , “Musiclm: Generating music from text,” arXiv preprint arXiv:2301.11325 , 2023
arXiv 2023
-
[6]
Simple and controllable music generation,
J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y . Adi, and A. Defossez, “Simple and controllable music generation,” in Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 47 704–47 720. [Online]. Available: https://proceedin...
work page 2023
-
[7]
Audiogen: Textually guided au- dio generation,
F. Kreuk, G. Synnaeve, A. Polyak, U. Singer, A. D ´efossez, J. Copet, D. Parikh, Y . Taigman, and Y . Adi, “Audiogen: Textually guided au- dio generation,” International Conference on Learning Representations (ICLR), 2023
work page 2023
-
[8]
Make-an-audio: Text-to-audio generation with prompt- enhanced diffusion models,
R. Huang, J. Huang, D. Yang, Y . Ren, L. Liu, M. Li, Z. Ye, J. Liu, X. Yin, and Z. Zhao, “Make-an-audio: Text-to-audio generation with prompt- enhanced diffusion models,” in International Conference on Machine Learning. PMLR, 2023, pp. 13 916–13 932
work page 2023
Show all 26 references
-
[9]
Noise2music: Text-conditioned music generation with diffusion models,
Q. Huang, D. S. Park, T. Wang, T. I. Denk, A. Ly, N. Chen, Z. Zhang, Z. Zhang, J. Yu, C. Frank et al., “Noise2music: Text-conditioned music generation with diffusion models,” arXiv preprint arXiv:2302.03917 , 2023
2023 arXiv
-
[10]
Musecoco: Generating symbolic music from text,
P. Lu, X. Xu, C. Kang, B. Yu, C. Xing, X. Tan, and J. Bian, “Musecoco: Generating symbolic music from text,” arXiv preprint arXiv:2306.00110, 2023
2023 arXiv
-
[11]
Ernie-music: Text-to-waveform music generation with diffusion models,
P. Zhu, C. Pang, Y . Chai, L. Li, S. Wang, Y . Sun, H. Tian, and H. Wu, “Ernie-music: Text-to-waveform music generation with diffusion models,” arXiv preprint arXiv:2302.04456 , 2023
2023 arXiv
-
[12]
Mo ˆusai: Text-to- music generation with long-context latent diffusion,
F. Schneider, O. Kamal, Z. Jin, and B. Sch ¨olkopf, “Mo ˆusai: Text-to- music generation with long-context latent diffusion,” 2023
2023
-
[13]
Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,
Y . Wu, K. Chen, T. Zhang, Y . Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2022
2022
-
[14]
Muq: Self-supervised music representation learning with mel residual vector quantization,
H. Zhu, Y . Zhou, H. Chen, J. Yu, Z. Ma, R. Gu, Y . Luo, W. Tan, and X. Chen, “Muq: Self-supervised music representation learning with mel residual vector quantization,” arXiv preprint arXiv:2501.01108 , 2025
2025 arXiv
-
[15]
Roberta: A robustly optimized bert pretraining approach,
Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692 , 2019
1907 arXiv
-
[16]
Rank consistent ordinal regres- sion for neural networks with application to age estimation,
W. Cao, V . Mirjalili, and S. Raschka, “Rank consistent ordinal regres- sion for neural networks with application to age estimation,” Pattern Recognition Letters , vol. 140, pp. 325–331, 2020
2020
-
[17]
Attention is all you need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garn...
2017
-
[18]
Mamba: Linear-time sequence modeling with selective state spaces,
A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” in First Conference on Language Modeling, 2024. [Online]. Available: https://openreview.net/forum?id=tEYskw1VY2
2024
-
[19]
Framewise phoneme classification with bidirectional lstm and other neural network architectures,
A. Graves and J. Schmidhuber, “Framewise phoneme classification with bidirectional lstm and other neural network architectures,” Neural Networks, vol. 18, no. 5, pp. 602–610, 2005, iJCNN 2005
2005
-
[20]
Ridge regression: Biased estimation for nonorthogonal problems,
A. E. Hoerl and R. W. Kennard, “Ridge regression: Biased estimation for nonorthogonal problems,” Technometrics, vol. 12, no. 1, pp. 55–67, 1970
1970
-
[21]
Lightgbm: A highly efficient gradient boosting decision tree,
G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y . Liu, “Lightgbm: A highly efficient gradient boosting decision tree,” in Advances in Neural Information Processing Systems , I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R....
2017
-
[22]
Meta- perser: Few-shot listener personalized speech emotion recognition via meta-learning,
L.-Y . Shen, S.-X. Fang, Y .-C. Lin, H.-C. Chou, and H. yi Lee, “Meta- perser: Few-shot listener personalized speech emotion recognition via meta-learning,” 2025. [Online]. Available: https://arxiv.org/abs/2505. 16220
2025
-
[23]
Modelling inter-rater uncertainty in spoken language assessment,
J. H. Wong, H. Zhang, and N. F. Chen, “Modelling inter-rater uncertainty in spoken language assessment,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 31, pp. 2886–2898, 2023
2023
-
[24]
Deepmos-b: Deep posterior mean-opinion-score using beta distribution,
X. Liang, F. Cumlin, V . Ungureanu, C. K. Reddy, C. Sch ¨uldt, and S. Chatterjee, “Deepmos-b: Deep posterior mean-opinion-score using beta distribution,” in 2024 32nd European Signal Processing Conference (EUSIPCO). IEEE, 2024, pp. 416–420
2024
-
[25]
800.2: Mean opinion score interpretation and report- ing,
I.-T. R. ITU-TP, “800.2: Mean opinion score interpretation and report- ing,” Tech. rep, Tech. Rep., 2016
2016
-
[26]
Alignnet: Learning dataset score alignment functions to enable better training of speech quality estimators,
J. Pieper and S. D. V oran, “Alignnet: Learning dataset score alignment functions to enable better training of speech quality estimators,” arXiv preprint arXiv:2406.10205, 2024
2024 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.