Pith. sign in

REVIEW 4 major objections 4 minor 26 references

ASTAR-NTU solution to AudioMOS Challenge 2025 Track1

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read Frozen-encoder DORA-MOS with Gaussian-softened ordinal training achieves 0.991 MI and 0.952 TA system-level SRCC on the official test set, beating the challenge baseline by 21.21% and 31.47%.

desk verdict Credible challenge win with a simple ordinal-label trick; the missing σ and hyperparameters are the real soft spots, not the method. read the letter →

arxiv 2507.09904 v1 pith:I7JXVECC submitted 2025-07-14 cs.SD eess.AS

classification cs.SDeess.AS
keywords audioqualityassessmentmeanopinionscorepredictiontext-to-musicevaluationordinalclassificationGaussianlabelsofteningcross-attentionfusionself-supervisedencodersSpearmanrankcorrelation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports the winning AudioMOS 2025 Track 1 system, DORA-MOS, whose goal is to predict two expert ratings for text-to-music output: music impression (intrinsic quality) and text alignment (how well the music matches the prompt). Its central claim is that neither expensive joint audio-text pretraining nor fine-tuning of large encoders is needed: freezing MuQ for audio and RoBERTa for text, aligning them with a cross-attention layer, and reframing MOS prediction as classification over 20 bins with Gaussian-softened labels is enough to reach system-level Spearman rank correlation of 0.991 for MI and 0.952 for TA on the official test set, relative gains of 21.21% and 31.47% over the challenge baseline. The paper argues the decisive ingredient is the training criterion: softening one-hot bin targets with a Gaussian kernel centered on the continuous score embeds ordinal information into a rank-aware learning signal, which aligns training with the rank-based evaluation metric. If the claim holds, automatic evaluation of generative music can be made cheap and scalable without retraining encoders.

What carries the argument

The load-bearing pieces are three. First, the dual-branch encoders: MuQ, a self-supervised music representation model with mel residual vector quantization, provides frozen audio features, and RoBERTa provides frozen text embeddings; no encoder is fine-tuned, and the model learns only the fusion and prediction heads. Second, cross-attention fusion: for text alignment the projected text embeddings serve as queries and the projected audio sequence as keys and values, so the model learns text-informed audio representations that encode prompt-audio coherence. Third, the Gaussian-softened ordinal classifier: instead of scalar regression or one-hot classification, the target is a distribution over $K = 20$ equal-width bins spanning 1 to 5, with bin probabilities proportional to $\exp(-(s-c_k)^2 / 2\sigma^2)$ for true score $s$ and bin center $c_k$, normalized to sum to one; this makes the training loss rank-aware, so misplacing a score far from its bin is penalized more than a near miss, matching the Spearman criterion used for evaluation.

What would settle it

Run the full pipeline with every decision, including the Gaussian width $\sigma$, the temporal model, the pooling method, and the ensemble meta-model, fixed by development-split performance on a fresh partition of MusicEval systems and prompts, then apply it exactly once to a held-out set: if MI SRCC falls materially below 0.991, or if the best configuration changes between splits, the headline numbers reflected test-informed selection or evaluation noise rather than the ordinal training itself. A narrower calculation: sweep $\sigma$ from 0.05 to 1 and plot test SRCC; a sharp peak at one unpublished value would show the gain depends on a precisely chosen softening strength.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that mean-opinion-score prediction for text-to-music evaluation is best treated as an ordinal classification problem rather than a regression problem, provided the labels are softened. The authors train a dual-branch model in which a frozen MuQ audio encoder and a frozen RoBERTa text encoder feed a task-specific fusion: a transformer layer with attention pooling for the music impression branch, and a cross-attention layer in which text features act as queries over audio key/value features for the text alignment branch. The training target is a 20-bin probability distribution over scores 1 to 5, with each bin weighted by a Gaussian centered on the true continuous score. This single model obtains system-level SRCC of 0.991 (MI) and 0.952 (TA) on the official test set, outperforming L1 regression and hard-label cross-entropy in controlled ablations, outperforming Mamba and Bi-LSTM temporal models when paired with attention pooling, and outperforming the joint pretrained CLAP and MuQ-Mulan features on both scores. The submitted stacking ensemble, which combines nine models' predicted 20-bin distributions through Ridge regression, was slightly worse than the single model on the test set while still placing first in 13 of the 16 official challenge metrics.

Load-bearing premise

The system's reported gains rest on the assumption that the stratified development split fairly represents the official test set, so that the architecture choices and the never-reported Gaussian width $\sigma$ were settled without test labels, and that the small test set is large enough to rank systems on differences of a few thousandths in Spearman correlation.

Editorial extensions

If this is right

  • Reframing MOS prediction as classification over score bins with Gaussian-smoothed targets raises system-level SRCC over both L1 regression and hard-label cross-entropy for MI and TA alike, so the training objective, not the architecture alone, is what drives the gain.
  • Cross-attention fusion of separately frozen audio and text encoders surpasses joint pretrained contrastive models (CLAP, MuQ-Mulan) on this task, meaning prompt-music alignment for scoring is better learned at the fusion stage than borrowed from semantic similarity pretraining.
  • Transformer temporal modeling with learned attention pooling outperforms Mamba and Bi-LSTM on both targets, though mean pooling becomes competitive for the linear-time encoders.
  • No external data or augmentation is needed for this in-domain task, and the high test scores suggest the dataset may be approaching saturation for system-level ranking.
  • Ensembling predicted 20-bin distributions with Ridge regression is stable but does not beat the best single model on the official test set, so the single DORA-MOS model is the configuration the authors would defend as the core result.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct consequence the authors do not pursue: if Gaussian-softened bin classification is a good surrogate for Spearman correlation, training with an explicit differentiable rank-correlation surrogate on the same architecture would test how much of the gain is due to the smoothing versus the rank alignment of the cross-entropy objective.
  • The Gaussian width $\sigma$ in Equation (1) is never reported even though it controls how strongly the ordinal signal is softened; a sensitivity sweep over $\sigma$ would show whether the 0.991 MI figure is a plateau or a sharp peak, and connects to a testable claim that the optimal width scales with annotator disagreement.
  • Given the ensemble slightly underperformed the single model and LightGBM overfit the meta-train split, the official test set may be too small to distinguish systems within about 0.003 SRCC; re-running the same pipeline on multiple random dev/test splits would show whether the reported orderings are stable.
  • The authors' own proposal to model annotator disagreement (e.g., a Beta distribution over scores) is a natural extension of the same machinery: the Gaussian-softened head already produces a predictive distribution, so calibrating its spread against the observed expert variance is an incremental change that could also absorb the never-reported $\sigma$ as a learned parameter.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper describes DORA-MOS, the first-place entry in the AudioMOS 2025 Challenge Track 1. The system uses frozen MuQ and RoBERTa encoders, a cross-attention fusion module, and a classification head over 20 score bins with Gaussian-softened ordinal labels. The central claim is that a single DORA-MOS model achieves system-level SRCC of 0.991 for Music Impression and 0.952 for Text Alignment on the official test set, corresponding to relative improvements of 21.21% and 31.47% over the challenge baseline, and that the final stacked ensemble ranked first in 13 of 16 official metrics.

Significance. If the reported configuration is fully specified, the result is a useful, independently validated system description for a practical MOS-prediction task. The external challenge evaluation supports the headline result, and the paper includes controlled ablations of the training criterion, temporal modeling, and pooling method, with an honest discussion of dataset saturation and of the ensemble underperforming the single model. The main weakness is that the exact configuration behind the headline numbers is not reported, so the internal comparisons cannot be reproduced or audited.

major comments (4)
  1. [Section II-C, Eq. (1)] The Gaussian width sigma is never reported, although Table I attributes the entire improvement of the proposed criterion over cross-entropy (MI SRCC 0.991 vs. 0.973; TA SRCC 0.952 vs. 0.940) to this softening. Please report the value of sigma, the grid searched, and the criterion used to choose it, together with the corresponding stratified-dev results; without this, the central performance claim cannot be reproduced or checked for test-set leakage.
  2. [Section III-A, Tables I-II] All ablation tables report official test-set numbers, with only a sentence asserting that the same ranking held on the internal dev split. Because the authors state that hyperparameter optimization was performed on the stratified split, please provide the dev-split equivalents of Tables I and II, or a summary table, so readers can verify that model selection did not use test labels. This is load-bearing for the claim that Gaussian softening and the Transformer/attention-pooling choices generalize.
  3. [Tables II-III] The system-level rank metrics are computed over a small number of systems (31 in Table III, 9 in Table IV), all results are from single runs, and adjacent configurations in Table II differ by as little as 0.004–0.016 in SRCC. Please report results over multiple seeds with mean and standard deviation, or provide a significance or confidence statement, so that the superiority of DORA-MOS over the second-best configuration is not attributable to noise.
  4. [Sections II-B and III] The paper does not specify training hyperparameters (epochs, optimizer, learning rate, batch size, projection dimensions, number of attention heads, pooling details, bin placement) or provide code, and the value of sigma in Eq. (1) is absent. Please include a complete configuration table or release code, since the headline 0.991/0.952 numbers are otherwise not independently verifiable.
minor comments (4)
  1. [Section III-C vs. Table IV] The final submission ('Ours', 0.988/0.944) is the Ensemble Ridge model, whereas Table III highlights the single DORA-MOS (0.991/0.952); the captions and Section III-D should state explicitly which configuration is being ranked by the challenge.
  2. [Section II-C] The description of CORAL says K−1 cumulative probabilities P(Y>k) for all k in {1,...,5}, which is inconsistent with K=20 bins; please clarify the number of ordinal thresholds.
  3. [Section II-B] The sentence 'Two out of the three best-performing models follow the principle shown in Fig. 1' is unclear; please specify which models these are and what the third model does differently.
  4. [Section II-A] The statement that the original dev set is 'reserved solely for final unseen data evaluation' conflicts with the later reporting of official test-set numbers; please clarify the role of the original dev set.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the reported SRCC gains are supported by external test-set evaluation and not by construction.

full rationale

The paper reports a trained system and its measured performance on the official AudioMOS 2025 Track 1 test set. The central claim (MI SRCC 0.991 and TA SRCC 0.952 in Table III) is an empirical result on external challenge data, not a quantity derived from the model's own definitions. Equation (1) defines a Gaussian-softened target distribution for training, and the ablations in Tables I and II compare training criteria and architectures against each other; no equation in the paper reduces a fitted parameter to the reported SRCC. The paper explicitly states that hyperparameter optimization was performed on an internal stratified split, and the official challenge test set provides independent evaluation. The only self-citations appear in the future-work paragraph (references [22] and [23] about annotator disagreement) and are not load-bearing for the main results. Reproducibility gaps, such as the unreported value of sigma and missing training hyperparameters, are legitimate concerns about transparency and statistical reporting, but they are not circularity: they do not make the reported predictions equivalent to the model inputs by construction. System-level SRCC over a small number of systems is a stability concern, not a circularity concern. Therefore, the derivation chain is self-contained and no significant circularity is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several unstated hyperparameters, mainly the Gaussian width sigma, and on domain assumptions about feature sufficiency and test representativeness. No new physical or mathematical entities are postulated.

free parameters (3)
  • Gaussian kernel width sigma = not reported
    Width of the Gaussian used to soften labels in Eq. 1. Chosen as a hyperparameter but its value is never stated; the ranking results depend on it.
  • Number of score bins K = 20
    The MOS 1-5 range is divided into 20 equal-width bins for classification. The choice of 20 is arbitrary and not justified; results depend on bin resolution.
  • Architecture hyperparameters = 1 transformer layer, 4 attention heads, 2-layer MLP
    The temporal modeling block and pooling are ablated in Table II, but the selected hyperparameters (heads, layers, MLP dims, projection dims) are from internal dev tuning and not fully specified.
assumptions (4)
  • domain assumption MuQ frozen features encode sufficient information for MI and TA prediction
    Section II-B selects MuQ as the audio encoder after evaluating CLAP, MERT, and MuQ-Mulan; the claim that these features are adequate is assumed.
  • domain assumption The test set is in-domain and representative of the training distribution
    Section II-A states the original test set contains no unseen models or genres, so the task is treated as in-domain. If the test set shifted, the reported SRCC would change.
  • domain assumption System-level SRCC over the test set is a stable ranking metric
    The challenge uses system-level SRCC as the primary metric, but the number of test systems is not reported, and high correlation over a small set can be noisy.
  • standard math Gaussian kernel normalization produces a valid probability distribution
    Eq. 1 uses a standard Gaussian kernel normalized to sum to one; this is standard math.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ASTAR-NTU solution to AudioMOS Challenge 2025 Track1." pith.science (2026). https://pith.science/paper/I7JXVECC

@misc{pith2026250709904,
  author       = {Pith},
  title        = {Pith review of: ASTAR-NTU solution to AudioMOS Challenge 2025 Track1},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I7JXVECC}},
  note         = {Machine review of arXiv:2507.09904}
}
read the original abstract

Evaluation of text-to-music systems is constrained by the cost and availability of collecting experts for assessment. AudioMOS 2025 Challenge track 1 is created to automatically predict music impression (MI) as well as text alignment (TA) between the prompt and the generated musical piece. This paper reports our winning system, which uses a dual-branch architecture with pre-trained MuQ and RoBERTa models as audio and text encoders. A cross-attention mechanism fuses the audio and text representations. For training, we reframe the MI and TA prediction as a classification task. To incorporate the ordinal nature of MOS scores, one-hot labels are converted to a soft distribution using a Gaussian kernel. On the official test set, a single model trained with this method achieves a system-level Spearman's Rank Correlation Coefficient (SRCC) of 0.991 for MI and 0.952 for TA, corresponding to a relative improvement of 21.21\% in MI SRCC and 31.47\% in TA SRCC over the challenge baseline.

Figures

Figures reproduced from arXiv: 2507.09904 by the authors.

Figure 1
Figure 1. Proposed dual-branch architecture for MI and TA prediction [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

26 extracted references · 19 canonical work pages

  1. [1]

    Audioldm: Text-to-audio generation with latent diffu- sion models,

    H. Liu, Z. Chen, Y . Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley, “Audioldm: Text-to-audio generation with latent diffu- sion models,” International Conference on Machine Learning (ICML) , 2023

  2. [2]

    Audioldm 2: Learning holistic audio generation with self-supervised pretraining,

    H. Liu, Y . Yuan, X. Liu, X. Mei, Q. Kong, Q. Tian, Y . Wang, W. Wang, Y . Wang, and M. D. Plumbley, “Audioldm 2: Learning holistic audio generation with self-supervised pretraining,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , 2024

  3. [3]

    Audiox: Diffusion transformer for anything-to-audio generation,

    Z. Tian, Y . Jin, Z. Liu, R. Yuan, X. Tan, Q. Chen, W. Xue, and Y .-T. Guo, “Audiox: Diffusion transformer for anything-to-audio generation,” ArXiv Preprint, 2025

  4. [4]

    Musiceval: A generative music corpus with expert ratings for automatic text-to-music evaluation,

    C. Liu, H. Wang, J. Zhao, S. Zhao, H. Bu, X. Xu, J. Zhou, H. Sun, and Y . Qin, “Musiceval: A generative music corpus with expert ratings for automatic text-to-music evaluation,” ICASSP, 2025

  5. [5]

    Musiclm: Generating music from text,

    A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi et al. , “Musiclm: Generating music from text,” arXiv preprint arXiv:2301.11325 , 2023

  6. [6]

    Simple and controllable music generation,

    J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y . Adi, and A. Defossez, “Simple and controllable music generation,” in Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 47 704–47 720. [Online]. Available: https://proceedin...

  7. [7]

    Audiogen: Textually guided au- dio generation,

    F. Kreuk, G. Synnaeve, A. Polyak, U. Singer, A. D ´efossez, J. Copet, D. Parikh, Y . Taigman, and Y . Adi, “Audiogen: Textually guided au- dio generation,” International Conference on Learning Representations (ICLR), 2023

  8. [8]

    Make-an-audio: Text-to-audio generation with prompt- enhanced diffusion models,

    R. Huang, J. Huang, D. Yang, Y . Ren, L. Liu, M. Li, Z. Ye, J. Liu, X. Yin, and Z. Zhao, “Make-an-audio: Text-to-audio generation with prompt- enhanced diffusion models,” in International Conference on Machine Learning. PMLR, 2023, pp. 13 916–13 932

Show all 26 references
  1. [9]

    Noise2music: Text-conditioned music generation with diffusion models,

    Q. Huang, D. S. Park, T. Wang, T. I. Denk, A. Ly, N. Chen, Z. Zhang, Z. Zhang, J. Yu, C. Frank et al., “Noise2music: Text-conditioned music generation with diffusion models,” arXiv preprint arXiv:2302.03917 , 2023

  2. [10]

    Musecoco: Generating symbolic music from text,

    P. Lu, X. Xu, C. Kang, B. Yu, C. Xing, X. Tan, and J. Bian, “Musecoco: Generating symbolic music from text,” arXiv preprint arXiv:2306.00110, 2023

  3. [11]

    Ernie-music: Text-to-waveform music generation with diffusion models,

    P. Zhu, C. Pang, Y . Chai, L. Li, S. Wang, Y . Sun, H. Tian, and H. Wu, “Ernie-music: Text-to-waveform music generation with diffusion models,” arXiv preprint arXiv:2302.04456 , 2023

  4. [12]

    Mo ˆusai: Text-to- music generation with long-context latent diffusion,

    F. Schneider, O. Kamal, Z. Jin, and B. Sch ¨olkopf, “Mo ˆusai: Text-to- music generation with long-context latent diffusion,” 2023

  5. [13]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    Y . Wu, K. Chen, T. Zhang, Y . Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2022

  6. [14]

    Muq: Self-supervised music representation learning with mel residual vector quantization,

    H. Zhu, Y . Zhou, H. Chen, J. Yu, Z. Ma, R. Gu, Y . Luo, W. Tan, and X. Chen, “Muq: Self-supervised music representation learning with mel residual vector quantization,” arXiv preprint arXiv:2501.01108 , 2025

  7. [15]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692 , 2019

  8. [16]

    Rank consistent ordinal regres- sion for neural networks with application to age estimation,

    W. Cao, V . Mirjalili, and S. Raschka, “Rank consistent ordinal regres- sion for neural networks with application to age estimation,” Pattern Recognition Letters , vol. 140, pp. 325–331, 2020

  9. [17]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garn...

  10. [18]

    Mamba: Linear-time sequence modeling with selective state spaces,

    A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” in First Conference on Language Modeling, 2024. [Online]. Available: https://openreview.net/forum?id=tEYskw1VY2

  11. [19]

    Framewise phoneme classification with bidirectional lstm and other neural network architectures,

    A. Graves and J. Schmidhuber, “Framewise phoneme classification with bidirectional lstm and other neural network architectures,” Neural Networks, vol. 18, no. 5, pp. 602–610, 2005, iJCNN 2005

  12. [20]

    Ridge regression: Biased estimation for nonorthogonal problems,

    A. E. Hoerl and R. W. Kennard, “Ridge regression: Biased estimation for nonorthogonal problems,” Technometrics, vol. 12, no. 1, pp. 55–67, 1970

  13. [21]

    Lightgbm: A highly efficient gradient boosting decision tree,

    G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y . Liu, “Lightgbm: A highly efficient gradient boosting decision tree,” in Advances in Neural Information Processing Systems , I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R....

  14. [22]

    Meta- perser: Few-shot listener personalized speech emotion recognition via meta-learning,

    L.-Y . Shen, S.-X. Fang, Y .-C. Lin, H.-C. Chou, and H. yi Lee, “Meta- perser: Few-shot listener personalized speech emotion recognition via meta-learning,” 2025. [Online]. Available: https://arxiv.org/abs/2505. 16220

  15. [23]

    Modelling inter-rater uncertainty in spoken language assessment,

    J. H. Wong, H. Zhang, and N. F. Chen, “Modelling inter-rater uncertainty in spoken language assessment,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 31, pp. 2886–2898, 2023

  16. [24]

    Deepmos-b: Deep posterior mean-opinion-score using beta distribution,

    X. Liang, F. Cumlin, V . Ungureanu, C. K. Reddy, C. Sch ¨uldt, and S. Chatterjee, “Deepmos-b: Deep posterior mean-opinion-score using beta distribution,” in 2024 32nd European Signal Processing Conference (EUSIPCO). IEEE, 2024, pp. 416–420

  17. [25]

    800.2: Mean opinion score interpretation and report- ing,

    I.-T. R. ITU-TP, “800.2: Mean opinion score interpretation and report- ing,” Tech. rep, Tech. Rep., 2016

  18. [26]

    Alignnet: Learning dataset score alignment functions to enable better training of speech quality estimators,

    J. Pieper and S. D. V oran, “Alignnet: Learning dataset score alignment functions to enable better training of speech quality estimators,” arXiv preprint arXiv:2406.10205, 2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.