Pith. sign in

REVIEW 4 major objections 5 minor 27 references

AudioBERTScore: Objective Evaluation of Environmental Sound Synthesis Based on Similarity of Audio embedding Sequences

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training-free audio metric matches human ratings better than MCD and WARP-Q

desk verdict A plausible training-free TTA evaluation metric, but the headline correlations are unverified because hyperparameters were tuned on the same test set and the exact reported configuration is not shown. read the letter →

arxiv 2507.00475 v1 pith:ARUMOBNZ submitted 2025-07-01 cs.SD eess.AS

Minoru Kishi , Ryosuke Sakai , Shinnosuke Takamichi , Yusuke Kanamori , Yuki Okamoto This is my paper
classification cs.SDeess.AS
keywords AudioBERTScoretext-to-audioobjectiveevaluationsemanticsimilarityp-normenvironmentalsoundembeddingssubjectivecorrelation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to fix the weak correlation between existing objective metrics and human judgments when evaluating synthesized environmental audio. It proposes AudioBERTScore, a training-free metric that compares sequences of audio embeddings from a pretrained audio model and combines a conventional max-norm BERTScore with a p-norm similarity to capture both localized and spread-out sound structure. On the PAM test set, the proposed score correlates more strongly with human overall quality (OVL) and text relevance (REL) ratings than MCD and WARP-Q. The practical payoff is a cheap automatic proxy for subjective listening tests in text-to-audio development.

What carries the argument

The central object is the cross-embedding similarity matrix $M_{ij}=\mathrm{sim}(\tilde{s}_i,\tilde{r}_j)$ between frame-level embeddings of synthesized and reference audio. Precision, recall, and F1 are defined by averaging row-wise and column-wise p-norms, which interpolate between a pure average ($p=1$) and the max-norm ($p\to\infty$); the final score is an affine blend $\lambda\cdot\text{max}+(1-\lambda)\cdot\text{p-norm}$ that the paper allows to go negative. The feature extractor is a pretrained audio Transformer, with AST layer 13 chosen empirically, and the p-norm hyperparameter pair $p=10^6,\ \lambda=-3.5$ is selected by maximizing correlation on the PAM test set. The max-norm component inherits BERTScore's locality assumption, while the p-norm component is designed to reflect non-local environmental sound structure.

What would settle it

Take any held-out set of text-to-audio synthesized sounds with human OVL and REL ratings, compute AudioBERTScore with the published settings, and compare its LCC/SRCC against MCD and WARP-Q. If the gap shrinks to statistical non-significance, the PAM-tuned hyperparameters are overfit; a positive control would be showing the same ranking of extractors and norms on a second dataset.

Watch

Extended reading notes

Core claim

The central claim is that replacing the max-norm in BERTScore with an interpolation between max-norm and p-norm, allowing a negative interpolation weight, produces an objective audio evaluation metric whose scores track human OVL and REL judgments better than existing unsupervised reference-based metrics. Using the 13th layer of an AudioSet-fine-tuned AST model as the feature extractor and $p=10^6$, $\lambda=-3.5$, AudioBERTScore attains LCC/SRCC of 0.424/0.433 for OVL and 0.546/0.567 for REL, while MCD correlates near zero and WARP-Q around 0.2. In the REL comparison it also exceeds CLAPScore, and it trails PAM only on OVL. The authors interpret the improvement as evidence that environmental sounds have non-local structure that max-only similarity misses.

Load-bearing premise

The reported advantage rests on the hyperparameters $p=10^6$, $\lambda=-3.5$, and AST layer 13 being tuned on the PAM test set itself; if these choices do not transfer to other text-to-audio outputs, the measured correlations are not an unbiased estimate of metric quality.

Editorial extensions

If this is right

  • AudioBERTScore can serve as a drop-in, training-free replacement for MCD and WARP-Q when ranking or monitoring text-to-audio systems.
  • The p-norm extension makes the BERTScore family applicable to sounds whose relevant structure is distributed over time, not just localized events.
  • Pretrained Transformer embeddings fine-tuned on environmental audio (AST) give markedly better correlation than CNN-based embeddings (BYOL-A).
  • Because the score uses only audio references, it inherits the language independence of MCD and WARP-Q while improving correlation with subjective judgments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is whether the same $p$ and $\lambda$ values transfer to TTA models beyond the four generators in the PAM test set; the paper does not report cross-set validation.
  • The negative $\lambda$ implies the max-norm component is being subtracted in the blend, which is an unusual extrapolation; a per-category breakdown (impulsive vs. continuous sounds) could clarify why this helps.
  • Combining AudioBERTScore with a reference-free method like PAM may yield a metric that retains high OVL correlation while improving REL correlation, a direction the authors flag as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes AudioBERTScore, a reference-based, training-free objective metric for evaluating text-to-audio (TTA) synthesis. It extracts frame-level embeddings from synthesized and reference audio using pretrained models (AST, ATST-Frame, BYOL-A), computes a cosine similarity matrix, and aggregates it via BERTScore-style max-norm precision/recall/F1, extended with a p-norm and an interpolation weight lambda. Experiments on the PAM test set report LCC and SRCC against human OVL and REL scores, claiming that the best configuration (AST layer 13, p=10^6, lambda=-3.5) outperforms MCD, WARP-Q, and CLAPScore, and is competitive with PAM. The paper also releases code on GitHub.

Significance. If the reported correlations were unbiased, this would be a useful contribution: it offers a training-free, language-independent, reference-based metric that can be applied to arbitrary TTA outputs, and the open-source implementation is a definite strength. The p-norm aggregation is a reasonable extension of BERTScore-style metrics to non-local environmental sounds. However, the empirical evaluation as reported is not yet convincing because the hyperparameters and feature-extractor layer are selected on the same test set used for the final comparison, and the post-hoc exclusion of low-REL samples further compromises the estimates. The underlying method is plausible, but the specific numerical claims in Table 3 should be treated as development-set results until a held-out evaluation is provided.

major comments (4)
  1. [4.2.2 / Table 3] The final configuration (AST 13th layer, p=10^6, lambda=-3.5) is selected by maximizing correlation on the PAM test set in Figures 3, 5, and 6, and Table 3 then reports correlations computed on exactly that same test set. The reported LCC/SRCC values are therefore in-sample estimates, and the magnitude of selection-induced inflation is unknown. Because the central claim of Section 4.2.3 rests on these numbers, the authors should provide a held-out evaluation (e.g., a separate tuning split, cross-validation, or a second dataset) and report the performance of the chosen configuration on data not used for any selection.
  2. [4.1] The exclusion of 17 reference samples with REL < 3.5 and their 68 synthesized samples is post-hoc: it is motivated by the proposed method's aim rather than by a pre-specified criterion, and it changes the test distribution in a way that can inflate REL correlations if low-REL items are hard cases. Please report results on the full 100-reference/400-synthesized test set as well, or justify the exclusion with external evidence and show that the conclusions are robust to the exclusion.
  3. [Eq. (7) / Section 4.2.2] Equation (7) defines lambda in [0,1], yet Section 4.2.2 explores negative lambda and the final configuration uses lambda=-3.5. This is an extrapolation outside the defined interpolation range, and the paper gives no formal interpretation or justification for this extrapolation. Since this value is central to the reported gains, the authors should either extend the model definition to lambda in R with a clear interpretation, or restrict the final configuration to the originally defined range.
  4. [4.2.3] The claim that the proposed method 'significantly outperformed' MCD and WARP-Q is not supported by significance tests or confidence intervals. The differences between the proposed variants and PAM are small (e.g., REL LCC 0.546 vs. 0.529, REL SRCC 0.567 vs. 0.556), and Figures 5 and 6 have no error bars. Add bootstrap or permutation-based significance tests and confidence intervals for the LCC/SRCC comparisons, or soften the claim accordingly.
minor comments (5)
  1. [Figure 6] The x-axis labels in Figure 6 appear as '5 4 3 2 1 0' with no minus signs; since the text describes negative lambda values, the labels should be '-5 -4 -3 -2 -1 0'.
  2. [4.2.2] The statement 'Performance peaked at p = 100' is immediately followed by 'p = 10^6 slightly outperformed p = 100', which is contradictory; please clarify that p=10^6 was evaluated outside the range shown in Figure 5, and ensure the superscript notation is rendered correctly.
  3. [Table 1 / Section 2.1] Citation numbers for MCD and WARP-Q are inconsistent: MCD appears as [4] and [8], and WARP-Q appears as [9] and [24]; please use one consistent set of references.
  4. [4.2.1] The heading 'Correlation between correlation scores of OVL-REL' should be rephrased for readability, e.g., 'Relationship between OVL and REL correlations'.
  5. [Table 2 caption] The caption 'The score calculation based on the maximum value and the features extracted use the best settings' is ungrammatical; rephrase to something like 'Each row uses the best-performing layer and feature setting for that extractor.'

Circularity Check

1 steps flagged · score 6.0 of 10

Reported p-norm gains are selected on the test set, not predicted: Table 3 reports the in-sample maximum of the tuning procedure.

  1. fitted input called prediction [Section 4.2.2 (Figures 5 and 6) and Section 4.2.3 (Table 3)]
    "Performance peaked at p = 100, especially at λ = 0. Interestingly, p = 106 slightly outperformed p = 100. ... Figure 6 shows correlation improves until λ = −4, then drops at λ = −5. ... Results are shown in Table 3, using the best-performing configurations found in Sections 4.2.1 and 4.2.2, with p = 106 and λ = −3.5 as derived from Figure 6."

    The correlation coefficients in Table 3 for the p-norm variant are computed on the PAM test set after p and λ were chosen to maximize the same correlation on that same test set. Section 4.2.2 reports 'Performance peaked' and 'correlation improves until λ = −4' based on the test-set LCC values in Figures 5 and 6; Section 4.2.3 then presents the best configuration's Table 3 numbers as the method's performance. Thus the headline REL/OVL correlations for F1λ=−3.5,p=106 are the maximum of the grid search objective, not an unbiased estimate on new data. The F1max variant avoids p/λ tuning, but its AST 13th-layer choice was also selected from Figure 3 on the same test set, so even the untuned variant's reported advantage is optimistically biased.

full rationale

AudioBERTScore itself is not definitionally circular: it is a training-free, reference-based similarity metric that extends BERTScore/SpeechBERTScore by mixing max-norm and p-norm (Equations (1)-(9)), using externally pretrained encoders (AST, ATST-Frame, BYOL-A), and it does not use subjective scores as training targets. The circularity is in the evaluation protocol. The paper tunes p and λ by maximizing correlation on the PAM test set (Figures 5-6) and then reports correlations on the same test set (Table 3), so the claimed advantage of the tuned variants reduces to the fitting objective. Additionally, Section 4.1 excludes 17 reference samples with REL<3.5 and the corresponding synthesized samples before computing correlations, which is selection on the outcome variable and further reshapes the test distribution; no significance tests or confidence intervals are reported, so 'significantly outperformed' is unsupported. The underlying idea is plausible and the simpler F1max variant may still beat the baselines, but the specific Table 3 numbers cannot be read as predictions for unseen text-to-audio outputs. Score 6 reflects partial circularity: the central evaluation claim is statistically forced by in-sample selection, while the method's construction remains independent.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The metric introduces two tuned hyperparameters, p and lambda, and relies on pretrained embeddings plus the PAM subjective scores; it introduces no new physical or conceptual entities.

free parameters (2)
  • p (norm order) = 10^6
    Chosen from grid search {1, 5, 10, 50, 100, 150, 10^6} to maximize LCC on the PAM test set (Figure 5).
  • lambda (interpolation weight) = -3.5
    Chosen from negative-lambda search (Figure 6) to maximize LCC on the PAM test set; lies outside the intended [0,1] range.
assumptions (3)
  • domain assumption Cosine similarity between pretrained audio embeddings (Eq. 1) is a valid proxy for human-perceived quality and text relevance.
    The whole score inherits this from BERTScore; no theoretical justification is given for environmental sounds.
  • domain assumption Environmental sounds have non-local similarity structure that max-norm misses.
    Section 3.2.2 asserts this via the babbling brook example, but no quantitative evidence is provided.
  • domain assumption MOS scores from 10 raters on the PAM test set are reliable ground truth.
    The evaluation treats averaged 5-point ratings as the target without analyzing rater agreement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AudioBERTScore: Objective Evaluation of Environmental Sound Synthesis Based on Similarity of Audio embedding Sequences." pith.science (2026). https://pith.science/paper/ARUMOBNZ

@misc{pith2026250700475,
  author       = {Pith},
  title        = {Pith review of: AudioBERTScore: Objective Evaluation of Environmental Sound Synthesis Based on Similarity of Audio embedding Sequences},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ARUMOBNZ}},
  note         = {Machine review of arXiv:2507.00475}
}
abstract

We propose a novel objective evaluation metric for synthesized audio in text-to-audio (TTA), aiming to improve the performance of TTA models. In TTA, subjective evaluation of the synthesized sound is an important, but its implementation requires monetary costs. Therefore, objective evaluation such as mel-cepstral distortion are used, but the correlation between these objective metrics and subjective evaluation values is weak. Our proposed objective evaluation metric, AudioBERTScore, calculates the similarity between embedding of the synthesized and reference sounds. The method is based not only on the max-norm used in conventional BERTScore but also on the $p$-norm to reflect the non-local nature of environmental sounds. Experimental results show that scores obtained by the proposed method have a higher correlation with subjective evaluation values than conventional metrics.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 19 canonical work pages

  1. [1]

    Leveraging AI to Generate Audio for User-generated Content in Video Games

    T. Marrinan, P. Akram, O. Gurmessa, and A. Shishkin, “Leveraging ai to generate audio for user-generated content in video games,” arXiv preprint arXiv:2404.17018, 2024

    work page Pith review arXiv 2024

  2. [2]

    How should we evaluate synthesized environmental sounds,

    Y . Okamoto, K. Imoto, S. Takamichi, T. Fukumori, and Y . Yamashita, “How should we evaluate synthesized environmental sounds,” in Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) , 2022, pp. 307–312

    work page 2022

  3. [3]

    An effective quality evaluation protocol for speech enhancement algorithms,

    J. H. L. Hansen and B. L. Pellom, “An effective quality evaluation protocol for speech enhancement algorithms,” 5th International Conference on Spoken Language Processing, 1998

    work page 1998

  4. [4]

    Mel-cepstral distance measure for objective speech quality assessment,

    R. Kubichek, “Mel-cepstral distance measure for objective speech quality assessment,” in Proceedings of IEEE Pacific Rim Conference on Communications Computers and Signal Processing , vol. 1, 1993, pp. 125–128 vol.1

    work page 1993

  5. [5]

    Human-CLAP: Human-perception-based contrastive language-audio pretraining,

    T. Takano, Y . Okamoto, Y . Kanamori, Y . Saito, R. Nagase, and H. Saruwatari, “Human-CLAP: Human-perception-based contrastive language-audio pretraining,” arXiv preprint arXiv:2506.23553, 2025

    work page arXiv 2025

  6. [6]

    Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Tasks

    X. Li, N. Shao, and X. Li, “Self-supervised audio teacher-student transformer for both clip-level and frame-level tasks,” arXiv preprint arXiv:2306.04186, 2023

    work page Pith review arXiv 2023

  7. [7]

    Foley sound synthesis at the dcase 2023 challenge,

    K. Choi, J. Im, L. Heller, B. McFee, K. Imoto, Y . Okamoto, M. Lagrange, and S. Takamichi, “Foley sound synthesis at the dcase 2023 challenge,” arXiv preprint arXiv:2304.12521 , 2023

    arXiv 2023

  8. [8]

    Using dynamic time warping to find patterns in time series,

    D. J. Berndt and J. Clifford, “Using dynamic time warping to find patterns in time series,” in Proceedings of the 3rd International Conference on Knowledge Discovery and Data Mining , 1994, p. 359–370

    work page 1994

Show all 27 references
  1. [9]

    Dynamic time warping under subsequence,

    S. Han and L. Zhang, “Dynamic time warping under subsequence,” in 4th International Conference on Information Science, Electrical, and Automation Engineering, vol. 12257, 2022, p. 122571X

    2022

  2. [10]

    Relate: Subjective evaluation dataset for automatic evaluation of relevance between text and audio,

    Y . Kanamori, Y . Okamoto, T. Takano, S. Takamichi, Y . Saito, and H. Saruwatari, “Relate: Subjective evaluation dataset for automatic evaluation of relevance between text and audio,” in Proceedings of Interspeech, Aug. 2025

    2025

  3. [11]

    PAM: Prompting audio-language models for audio quality assessment,

    S. Deshmukh, D. Alharthi, B. Elizalde, H. Gamper, M. A. Ismail, R. Singh, B. Raj, and H. Wang, “PAM: Prompting audio-language models for audio quality assessment,” arXiv preprint arXiv:2402.00282 , 2023

    2023 arXiv

  4. [12]

    Speech- BERTScore: Reference-aware automatic evaluation of speech generation leveraging nlp evaluation metrics,

    T. Saeki, S. Maiti, S. Takamichi, S. Watanabe, and H. Saruwatari, “Speech- BERTScore: Reference-aware automatic evaluation of speech generation leveraging nlp evaluation metrics,” in Proceedings of Interspeech , 2024, pp. 4943–4947

    2024

  5. [13]

    BERTScore: Evaluating text generation with bert,

    T. Zhang*, V . Kishore*, F. Wu*, K. Q. Weinberger, and Y . Artzi, “BERTScore: Evaluating text generation with bert,” in Proceedings of the International Conference on Learning Representations (ICLR) , 2020

    2020

  6. [14]

    BERT: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  7. [15]

    HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 29, p. 3451–3460, Oct. 2021

    2021

  8. [16]

    AudioCaps: Generating captions for audios in the wild,

    C. D. Kim, B. Kim, H. Lee, and G. Kim, “AudioCaps: Generating captions for audios in the wild,” in Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019

    2019

  9. [17]

    AudioLDM 2: Learning holistic audio generation with self-supervised pretraining,

    H. Liu, Y . Yuan, X. Liu, X. Mei, Q. Kong, Q. Tian, Y . Wang, W. Wang, Y . Wang, and M. D. Plumbley, “AudioLDM 2: Learning holistic audio generation with self-supervised pretraining,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 32, pp. 2871–2883, 2024

    2024

  10. [18]

    AudioLDM: Text-to-audio generation with latent diffusion models,

    H. Liu, Z. Chen, Y . Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley, “AudioLDM: Text-to-audio generation with latent diffusion models,” Proceedings of the International Conference on Machine Learning (ICML) , pp. 21 450–21 474, 2023

    2023

  11. [19]

    AudioGen: Textually guided audio generation,

    F. Kreuk, G. Synnaeve, A. Polyak, U. Singer, A. D ´efossez, J. Copet, D. Parikh, Y . Taigman, and Y . Adi, “AudioGen: Textually guided audio generation,” arXiv preprint arXiv:2209.15352 , 2022

    2022 arXiv

  12. [20]

    BYOL for Audio: Exploring pre-trained general-purpose audio representations,

    D. Niizumi, D. Takeuchi, Y . Ohishi, N. Harada, and K. Kashino, “BYOL for Audio: Exploring pre-trained general-purpose audio representations,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 31, pp. 137–151, 2023

    2023

  13. [21]

    Backpropagation applied to handwritten zip code recognition,

    Y . LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural Computation, vol. 1, no. 4, pp. 541–551, 1989

    1989

  14. [22]

    Attention Is All You Need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention Is All You Need,” arXiv preprint arXiv:1706.03762, 2023

    2023 arXiv

  15. [23]

    AST: Audio spectrogram trans- former,

    Y . Gong, Y .-A. Chung, and J. Glass, “AST: Audio spectrogram trans- former,” in Proceedings of Interspeech , 2021, pp. 571–575

    2021

  16. [24]

    W ARP-Q: Quality prediction for generative neural speech codecs,

    W. A. Jassim, J. Skoglund, M. Chinen, and A. Hines, “W ARP-Q: Quality prediction for generative neural speech codecs,” in Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 401–405

    2021

  17. [25]

    Fr ´echet Audio Distance: A metric for evaluating music enhancement algorithms,

    K. Kilgour, M. Zuluaga, D. Roblek, and M. Sharifi, “Fr ´echet Audio Distance: A metric for evaluating music enhancement algorithms,” arXiv preprint arXiv:1812.08466, 2019

    2019 arXiv

  18. [26]

    A reference-free metric for language-queried audio source separation using contrastive language-audio pretraining,

    F. Xiao, J. Guan, Q. Zhu, X. Liu, W. Wang, S. Qi, K. Zhang, J. Sun, and W. Wang, “A reference-free metric for language-queried audio source separation using contrastive language-audio pretraining,” in Proceedings of Detection and Classification of Acoustic Scenes and Events (D...

    2024

  19. [27]

    Conformer: Local features coupling global representations for recognition and detection,

    Z. Peng, Z. Guo, W. Huang, Y . Wang, L. Xie, J. Jiao, Q. Tian, and Q. Ye, “Conformer: Local features coupling global representations for recognition and detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 8, pp. 9454–9468, 2023

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.