REVIEW 4 major objections 4 minor 22 references
Improving Perceptual Audio Aesthetic Assessment via Triplet Loss and Self-Supervised Embeddings
T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read AESA-Net predicts four aesthetic axes of synthetic audio from natural-audio training alone, using triplet-structured BEATs embeddings.
desk verdict A plausible challenge system whose central triplet-loss claim is untested and whose reported MSEs are internally inconsistent; the empirical case doesn't hold together. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the triplet loss with buffer-based sampling, applied to intermediate embeddings. Given an anchor embedding, a positive embedding with similar normalized aesthetic score (|ya − yp| < epsilon) is pulled closer while a negative embedding with dissimilar score (|ya − yn| > epsilon) is pushed apart, subject to a margin; the total loss is LMSE + alpha times LTriplet. The FIFO buffer supplies recent embedding-score pairs so positives and negatives exist within a batch of size one. This structuring is what the paper credits for making a BEATs/LSTM predictor trained only on natural audio rank synthetic audio correctly.
What would settle it
Running the same training pipeline with triplet loss removed (and with all else fixed) and comparing SRCC/KTAU on the official synthetic evaluation set would settle the central attribution; a second check is to measure whether L2 distances between synthetic clips and natural anchors ordered by score separate synthetic clips by human-rated quality.
Extended reading notes
Core claim
The central claim is that structuring a self-supervised audio embedding with triplet loss makes multi-axis aesthetic scoring transfer from natural to synthetic audio. AESA-Net extracts BEATs hidden states, combines them through a learnable softmax-weighted sum, encodes with a two-layer BLSTM, and branches into four attention-based scoring heads for Production Quality, Production Complexity, Content Enjoyment, and Content Usefulness. Training minimizes MSE plus a triplet loss whose positives and negatives are drawn from a FIFO buffer of recent embedding-score pairs: a positive lies within epsilon of the anchor's normalized score, a negative lies farther than epsilon. Trained on 2,950 natural
Load-bearing premise
The system is trained only on natural audio, yet its embedding structure is assumed to organize synthetic audio by perceived quality; if score-similarity in BEATs embeddings does not carry across the natural-to-synthetic domain gap, the reported gains would not transfer.
Editorial extensions
If this is right
- If the central claim holds, subjective aesthetic evaluation of TTS, TTA, and TTM audio can be automated without collecting synthetic-audio labels, reducing reliance on expensive listening tests.
- Across all four axes, rank-based agreement (SRCC, KTAU) with human raters improves, so the model can order generated audio by perceived quality even where absolute-score calibration is imperfect.
- The joint multi-axis design means one BEATs/LSTM backbone supplies four perceptual scores, which can be used as a training signal or filter in generative-audio systems.
- Because BEATs is pretrained on general audio events, the same pipeline is reusable for speech, music, and general-audio quality monitoring without per-domain feature engineering.
Reading between the lines
- Editorial inference: because no ablation removes the triplet loss, the metric gains could partly come from the BEATs backbone or the multi-task heads; a triplet-loss-off run is needed to isolate the mechanism the paper credits.
- Editorial inference: the same score-thresholded buffer sampling could be transferred to other subjective ratings (e.g., listener MOS, intelligibility) or other modalities, since the only requirement is a paired embedding-score buffer.
- Editorial inference: a small synthetic labeled set, even a few dozen clips per domain, would make the natural-to-synthetic transfer testable in development and could reveal whether larger margins or domain-weighted sampling close the CE MSE gap.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes AESA-Net, a multi-axis perceptual audio aesthetic assessment system for AudioMOS Challenge 2025 Track 2. The system uses BEATs self-supervised features, a multi-branch BLSTM/attention architecture, and a per-axis triplet loss with buffer-based sampling. It is trained only on AES-natural audio and evaluated on the official synthetic evaluation set. The authors report system-level MSE, LCC, SRCC, and KTAU across PQ, PC, CE, and CU, and compare against the official AudioMOS baseline. The central claim, stated in the conclusion, is that including the triplet loss improves both rank-based and correlation-based performance metrics.
Significance. If the central claim were established, the work would be a useful contribution to cross-domain audio quality assessment: it targets a practical domain-shift problem (natural training data, synthetic evaluation data) and tackles four aesthetic axes jointly. The use of an external challenge evaluation set is a positive feature, and the architecture is clearly described. However, the paper does not provide a controlled experiment isolating the triplet loss, and the reported results are internally inconsistent. As it stands, the significance is limited because the headline contribution is not evidenced by the experiments presented.
major comments (4)
- [Section V, Eq. (3), Table II] The conclusion states that 'the inclusion of triplet loss enhances both rank-based and correlation-based performance metrics,' but no experiment supports this causal attribution. The only comparison is against the official baseline [22], which differs in architecture, feature extractor, training objective, and possibly evaluation protocol. An ablation with α=0 in Eq. (3) (or at least a sweep over α) is necessary to separate the triplet-loss contribution from all other design choices. Without it, the central claim is unsupported.
- [Tables I and II] The reported MSE values are internally inconsistent. Table I reports domain-level MSE for CE of 1.4739 (speech), 2.3145 (music), and 1.8942 (audio), while Table II reports 'Ours' CE MSE = 3.991. For CU, Table I reports 0.9539, 1.1404, and 1.9407, while Table II reports 0.533. If Table I is per-domain and Table II is an aggregate, the aggregation formula is missing and the resulting values appear incompatible with any simple average. If both are supposed to be system-level on the same evaluation set, at least one of the tables is wrong. This needs to be resolved before any reliable comparison can be made.
- [Section IV.C, Table II] The prose after Table II states that 'our model achieves higher rank-based metrics (SRCC and KTAU) across all four perceptual axes.' This is directly contradicted by the PC row: SRCC is 0.928 vs baseline 0.934, and KTAU is 0.781 vs baseline 0.800. Either the prose or the table must be corrected. This is not a cosmetic issue; it affects the claimed superiority of the proposed approach.
- [Sections II.A and III] The paper assumes that score-similarity structure learned on natural-audio embeddings transfers to synthetic audio. The triplet loss is trained on AES-natural labels, yet the evaluation is entirely on synthetic TTS/TTA/TTM clips. No synthetic validation set is used during development, and no analysis is provided showing that the triplet-structured embedding separates synthetic clips by perceived quality. A concrete test would be to evaluate embedding nearest-neighbor behavior or triplet accuracy on a held-out synthetic set. Without such evidence, the mechanism claimed for domain robustness remains unverified.
minor comments (4)
- [Table I caption] The caption says 'system-level results' but the table is organized by domain. Please clarify whether 'system-level' refers to individual generative systems, domain-level averages, or the entire evaluation set, and define the aggregation in Section IV.B.
- [Section IV.B] The evaluation metrics list uses both 'PCC' and later 'LCC' for the same metric. Please standardize the terminology.
- [Section III] The buffer-based sampling is attributed to references [20] and [21], but [20] is the FaceNet triplet-loss paper and [21] is about non-parametric instance discrimination. The citation-to-method mapping should be checked and corrected.
- [Abstract / Section IV.C] The abstract claims that the method 'improves embedding discriminability,' but no experiment or visualization directly measures embedding discriminability. This claim should either be supported or rephrased as an interpretation.
Circularity Check
No circular derivation: external test set grounds the evaluation; the self-cited HAAQI-Net architecture is not load-bearing and the triplet loss is a standard supervised regularizer.
full rationale
The paper's predictions are evaluated on the official AudioMOS Challenge 2025 evaluation set of 3,000 unseen synthetic samples, whose ground-truth labels are not used during training. The training objectives in Eqs. (1)-(3) use the normalized AES-natural labels as MSE targets and as triplet positive/negative selectors; this is standard supervised metric learning, not a self-fulfilling definition of the held-out prediction. The Table II comparison is against an external baseline ([22], from Meta), not against a re-imported self-citation. The only substantive self-citation is [19] (HAAQI-Net), used as architectural inspiration ('inspired by HAAQI-Net [19]', Sec. II.C); no theorem or forced choice is imported from it, so it is not load-bearing. Separately, I flag as evidence/correctness concerns (not circularity): (a) the causal claim that 'the inclusion of triplet loss enhances' performance (Sec. V) is not tested by an alpha=0 ablation, since the only comparison is to a different baseline system; and (b) Table I and Table II report mutually inconsistent MSE values (e.g., CE domain MSEs 1.4739/2.3145/1.8942 vs overall CE MSE 3.991; CU domain MSEs 0.9539/1.1404/1.9407 vs overall CU MSE 0.533). These undermine confidence in the reported results but do not constitute circular reasoning.
Assumptions & free parameters
free parameters (5)
- triplet loss weight alpha =
0.2 (stated as an example; actual value not reported)
- triplet margin
- positive/negative score threshold epsilon
- buffer capacity
- BEATs layer weights =
learned, softmax-normalized
assumptions (3)
- domain assumption Mean listener AES-natural scores normalized to zero mean and unit variance are reliable perceptual targets and comparable across speech, music, and general audio.
- domain assumption BEATs, pretrained on AudioSet natural audio, provides representations that cover synthetic TTS, TTA, and TTM content.
- ad hoc to paper L2 distance in the triplet-structured embedding is a valid proxy for perceptual score difference.
Cite this review
Pith. "Pith review of Improving Perceptual Audio Aesthetic Assessment via Triplet Loss and Self-Supervised Embeddings." pith.science (2026). https://pith.science/paper/PCWK5Z6N
@misc{pith2026250903292,
author = {Pith},
title = {Pith review of: Improving Perceptual Audio Aesthetic Assessment via Triplet Loss and Self-Supervised Embeddings},
year = {2026},
howpublished = {\url{https://pith.science/paper/PCWK5Z6N}},
note = {Machine review of arXiv:2509.03292}
}
read the original abstract
We present a system for automatic multi-axis perceptual quality prediction of generative audio, developed for Track 2 of the AudioMOS Challenge 2025. The task is to predict four Audio Aesthetic Scores--Production Quality, Production Complexity, Content Enjoyment, and Content Usefulness--for audio generated by text-to-speech (TTS), text-to-audio (TTA), and text-to-music (TTM) systems. A main challenge is the domain shift between natural training data and synthetic evaluation data. To address this, we combine BEATs, a pretrained transformer-based audio representation model, with a multi-branch long short-term memory (LSTM) predictor and use a triplet loss with buffer-based sampling to structure the embedding space by perceptual similarity. Our results show that this improves embedding discriminability and generalization, enabling domain-robust audio quality assessment without synthetic training data.
Figures
Reference graph
Works this paper leans on
-
[22]
Meta audiobox aesthet- ics: Unified automatic quality assessment for speech,
A. Tjandra, Y .-C. Wu, B. Guo, J. Hoffman, B. Ellis, A. Vyas, B. Shi, S. Chen, M. Le, N. Zacharov et al. , “Meta audiobox aesthet- ics: Unified automatic quality assessment for speech,” arXiv preprint arXiv:2502.05139, 2025
arXiv 2025
-
[1]
W.-C. Huang and E. Cooper and Y . Tsao and H.-M. Wang and T. Toda and J. Yamagishi, “The V oiceMOS Challenge 2022,” in Proc. Interspeech, 2022, pp. 4536–4540
work page 2022
-
[2]
The voicemos challenge 2023: Zero-shot subjective speech quality prediction for multiple domains,
E. Cooper, W.-C. Huang, Y . Tsao, H.-M. Wang, T. Toda, and J. Yamag- ishi, “The voicemos challenge 2023: Zero-shot subjective speech quality prediction for multiple domains,” in Proc. IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) , 2023, pp. 1–7
work page 2023
-
[3]
The voicemos challenge 2024: Beyond speech quality prediction,
W.-C. Huang, S.-W. Fu, E. Cooper, R. E. Zezario, T. Toda, H.-M. Wang, J. Yamagishi, and Y . Tsao, “The voicemos challenge 2024: Beyond speech quality prediction,” in Proc. IEEE Spoken Language Technology Workshop (SLT), 2024, pp. 803–810
work page 2024
-
[4]
Fusion of self-supervised learned models for MOS prediction,
Z. Yang, W. Zhou, C. Chu, S. Li, R. Dabre, R. Rubino, and Y . Zhao, “Fusion of self-supervised learned models for MOS prediction,” in Proc. Interspeech, 2022, pp. 5443–5447
work page 2022
-
[5]
UTMOS: UTokyo-SaruLab system for V oiceMOS Chal- lenge 2022,
T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari, “UTMOS: UTokyo-SaruLab system for V oiceMOS Chal- lenge 2022,” in Proc. Interspeech, 2022, pp. 4521–4525
work page 2022
-
[6]
A study on incorporating Whisper for robust speech assessment,
R. E. Zezario, Y .-W. Chen, S.-W. Fu, Y . Tsao, H.-M. Wang, and C.-S. Fuh, “A study on incorporating Whisper for robust speech assessment,” in Proc. International Conference on Multimedia and Expo (ICME) , 2024, pp. 1–6
work page 2024
-
[7]
Corn: Co-trained full- and no-reference speech quality assessment,
P. Manocha, D. Williamson, and A. Finkelstein, “Corn: Co-trained full- and no-reference speech quality assessment,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP) , 2024, pp. 376–380
work page 2024
Show all 22 references
-
[8]
Enabling auditory large language models for automatic speech quality evaluation,
S. Wang, W. Yu, Y . Yang, C. Tang, Y . Li, J. Zhuang, X. Chen, X. Tian, J. Zhang, G. Sun, L. Lu, and C. Zhang, “Enabling auditory large language models for automatic speech quality evaluation,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP) , 2025, pp. 1–5
2025
-
[9]
Speech foundation models on intelligibility prediction for hearing-impaired listeners,
S. Cuervo and R. Marxer, “Speech foundation models on intelligibility prediction for hearing-impaired listeners,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP) , 2024, pp. 1421–1425
2024
-
[10]
Non-intrusive speech intelligibility prediction for hearing- impaired users using intermediate ASR features and human memory models,
R. Mogridge, G. Close, R. Sutherland, T. Hain, J. Barker, S. Goetze, and A. Ragni, “Non-intrusive speech intelligibility prediction for hearing- impaired users using intermediate ASR features and human memory models,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (I...
2024
-
[11]
A review on subjective and objective evaluation of synthetic speech,
E. Cooper, W.-C. Huang, Y . Tsao, H.-M. Wang, T. Toda, and J. Ya- magishi, “A review on subjective and objective evaluation of synthetic speech,” Acoustical Science and Technology , pp. e24–12, 2024
2024
-
[12]
Self-supervised speech quality estimation and enhancement using only clean speech,
S.-W. Fu, K.-H. Hung, Y . Tsao, and Y .-C. F. Wang, “Self-supervised speech quality estimation and enhancement using only clean speech,” in Proc. International Conference on Learning Representations (ICLR) , 2024
2024
-
[13]
Generalization ability of MOS prediction networks,
E. Cooper, W.-H. Huang, T. Toda, and J. Yamagishi, “Generalization ability of MOS prediction networks,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP) , 2022
2022
-
[14]
Deep learning-based non-intrusive multi-objective speech assessment model with cross-domain features,
R. E. Zezario, S.-W. Fu, F. Chen, C.-S. Fuh, H.-M. Wang, and Y . Tsao, “Deep learning-based non-intrusive multi-objective speech assessment model with cross-domain features,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 31, pp. 54–70, 2023
2023
-
[15]
Speechlmscore: Evaluat- ing speech generation using speech language model,
S. Maiti, Y . Peng, T. Saeki, and S. Watanabe, “Speechlmscore: Evaluat- ing speech generation using speech language model,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP) 2023 , 2023, pp. 1–5
2023
-
[16]
BEATs: Audio pre-training with acoustic tokenizers,
S. Chen, Y . Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, W. Che, X. Yu, and F. Wei, “BEATs: Audio pre-training with acoustic tokenizers,” in Proc. International Conference on Machine Learning (ICML) , Honolulu, Hawaii, USA, 2023, pp. 5178–5193
2023
-
[17]
MBNet: MOS prediction for synthesized speech with mean-bias network,
Y . Leng, X. Tan, S. Zhao, F. Soong, X.-Y . Li, and T. Qin, “MBNet: MOS prediction for synthesized speech with mean-bias network,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP) , 2021, pp. 391–395
2021
-
[18]
LDNet: Unified listener dependent modeling in MOS prediction for synthetic speech,
W.-C. Huang, E. Cooper, J. Yamagishi, and T. Toda, “LDNet: Unified listener dependent modeling in MOS prediction for synthetic speech,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP) , 2022, pp. 896–900
2022
-
[19]
HAAQI- Net: A non-intrusive neural music audio quality assessment model for hearing aids,
D. Wisnu, R. E. Zezario, S. Rini, H.-M. Wang, and Y . Tsao, “HAAQI- Net: A non-intrusive neural music audio quality assessment model for hearing aids,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 33, pp. 1877–1892, 2025
2025
-
[20]
FaceNet: A unified embed- ding for face recognition and clustering,
F. Schroff, D. Kalenichenko, and J. Philbin, “FaceNet: A unified embed- ding for face recognition and clustering,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2015, pp. 815–823
2015
-
[21]
Unsupervised feature learning via non-parametric instance discrimination,
Z. Wu, Y . Xiong, S. X. Yu, and D. Lin, “Unsupervised feature learning via non-parametric instance discrimination,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018, pp. 3733– 3742
2018
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.