Pith. sign in

REVIEW 4 major objections 6 minor 19 references

Audio-Aware Large Language Models as Judges for Speaking Styles

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An audio-aware large language model can rate speaking style as consistently as human raters rate one another.

desk verdict A promising but under-powered demonstration that Gemini-2.5-Pro can replace human raters for speaking-style evaluation; the voice-style result is solid, the role-playing result is too noisy to carry the claim. read the letter →

arxiv 2506.05984 v1 pith:TNPZWODA submitted 2025-06-06 eess.AS cs.AIcs.CL

Cheng-Han Chiang , Xiaofei Wang , Chung-Ching Lin , Kevin Lin , Linjie Li , Radu Kopetz , Yao Qian , Zhendong Wang
show 3 more authors
Zhengyuan Yang Hung-yi Lee Lijuan Wang
This is my paper
classification eess.AScs.AIcs.CL
keywords audio-awarelargelanguagemodelsspokenspeakingstyleevaluationLLM-as-a-judgevoiceinstructionfollowingrole-playingdialoguespeechgenerationSet
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that an off-the-shelf audio-aware large language model, Gemini-2.5-Pro, can judge the speaking style of generated speech about as well as human raters agree with one another. Across two new tasks in the released StyleSet benchmark—voice style instruction following and role-playing—Gemini's scores correlate with human scores at $r = 0.640$ and $r = 0.319$, while the average pairwise human-human correlations are $r = 0.596$ and $r = 0.253$. The comparison is made against four human raters per task, and the same audio is scored by humans and by the audio-aware model using matching instructions. If the claim holds, spoken language models can be evaluated automatically on expressive, non-textual qualities instead of relying exclusively on paid human evaluation.

What carries the argument

StyleSet is the central object: two 20-instance tasks, voice style instruction following and role-playing, that elicit speaking styles including emotion, volume, pace, pitch, word emphasis, and non-verbal elements such as laughter and sighs. The argument is carried by a judge pipeline in which each speech sample is scored from a text prompt containing the target style or role context, with chain-of-thought reasoning (step-by-step reasoning before the numeric verdict), five sampled verdicts ensembled by self-consistency, and regular-expression extraction of the numeric score. The load-bearing measurement is the average pairwise human-human Pearson correlation, treated as the reliability ceiling that an automatic judge must approach; the authors compare each audio-aware judge's correlation with each human rater against that ceiling.

What would settle it

Recruit a much larger panel of human raters, say 20 to 50 per instance, for the same StyleSet audio, take the panel mean as the gold standard, and recompute Gemini's correlation; the central claim fails if Gemini's correlation against this more reliable standard drops below the pairwise human-human correlation measured on the same audio.

Watch

Extended reading notes

Core claim

The paper's central claim is that audio-aware large language models can assess non-textual speaking style, and that the agreement between Gemini-2.5-Pro and human evaluators is comparable to the agreement among human evaluators themselves. On voice style instruction following, the average pairwise human-human Pearson correlation is 0.596, while the average human-Gemini correlation is 0.640; on role-playing style, the corresponding values are 0.253 and 0.319. GPT-4o-audio as a judge reaches 0.355 on the first task and 0.305 on the second. The paper takes these numbers as evidence that a general-purpose audio-aware LLM can stand in for at least one human rater when scoring generated speech for style adherence, role-appropriate emotion, prosody, pace, volume, emphasis, and non-verbal elements.

Load-bearing premise

The claim rests on treating the average of just four human raters per task as reliable ground truth, even though those raters agree only weakly with each other on role-playing style (correlation 0.253); if the human yardstick is noisy, matching or beating it is only weak evidence that an audio-aware model is a good judge.

Editorial extensions

If this is right

  • Gemini-2.5-Pro can serve as a cheap, reproducible first-pass judge for voice style instruction following, replacing one human rater whose agreement with other humans would typically be lower or equal.
  • StyleSet can be run without hiring human evaluators, allowing repeated automatic comparison of spoken language model versions as they are updated.
  • Audio-aware judges and humans agree on the coarse verdict that GPT-4o-audio is the best of the four tested spoken language models, but none of the models fully masters style control: the best human-rated average is 3.65 out of 5 on voice style instruction following.
  • Human-rated realism of the best spoken language model role-play is 0.51, far below the 0.95 realism score of human-recorded dialogues, so dialogue naturalness remains a major open problem even if style judging becomes automatic.
  • Because Gemini-human agreement exceeds the human-human baseline on both tasks, the consistency of automatic style judging is not merely a copy of one rater's taste.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A practical downstream consequence the paper does not test: the four-rater human average is a thin anchor, and re-running the comparison with a larger rater pool would show whether Gemini's edge over human-human agreement survives a more reliable gold standard.
  • The same judge pipeline could be pointed at other speech attributes the paper explicitly leaves out, such as intelligibility, speaker similarity, or full-duplex dialogue, and the release of StyleSet makes such extensions directly testable.
  • Because the paper demonstrates correlation but not absolute-score calibration, practical use of audio-aware judges as accept or reject gatekeepers would require mapping automatic scores onto the human scale means, not just relying on rank agreement.
  • Translating StyleSet to non-English languages would test whether Gemini's judging advantage persists; the authors note that Step-Audio and Qwen-2.5-Omni may be stronger in Chinese, so language coverage could change both spoken model rankings and judge reliability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces StyleSet, two evaluation tasks for speaking-style controllability of spoken language models (SLMs): voice style instruction following and role-playing. Four SLMs generate speech, and two audio-aware large language models (ALLMs), GPT-4o-audio and Gemini-2.5-Pro, judge the generated speech on 5-point Likert scales (plus a binary realism rating for role-playing). The judgments are compared with four human raters using average Pearson correlations. The central claim is that Gemini-2.5-Pro agrees with human judges at a level comparable to human-human agreement (voice style IF: 0.640 vs. 0.596; role-playing style: 0.319 vs. 0.253), while GPT-4o-audio agrees less well (0.355 and 0.305). The paper also reports that current SLMs, including GPT-4o-audio, still struggle with fine-grained style control and realistic dialogue.

Significance. If the central claim survives proper statistical scrutiny, the paper provides a useful automatic evaluation tool for a speech attribute—speaking style—that is otherwise expensive and subjective to evaluate. The authors release two tasks that can be reused, compare ALLM judgments against a human-human baseline, and are candid about limitations. The use of a human-human agreement reference is conceptually appropriate, and the experiments cover multiple SLM families and both instruction-following and interactive settings. However, the support for the headline claim currently rests on point estimates of Pearson correlations without confidence intervals or significance tests, and on a role-playing task with very low human-human agreement. The significance of the contribution therefore depends on whether the inferential gaps can be closed.

major comments (4)
  1. [Section 4.2, Table 2] The headline claim that Gemini-human agreement is comparable to human-human agreement compares two point estimates, r=0.640 and r=0.596, with no confidence intervals or significance test. Moreover, the correlations are computed over 80 scores that are not independent, because the same 20 contexts are reused across four SLMs. A naive sample size of 80 overstates the precision of the estimates. I ask the authors to report cluster-bootstrapped confidence intervals (resampling by context, and also by SLM) or an equivalent uncertainty-aware analysis, and to state whether the difference is statistically detectable.
  2. [Section 4.3, Table 2] On the role-playing style task, the human-human Pearson correlation is r=0.253, which the paper itself characterizes as 'somewhat subjective.' The Gemini-human correlation is r=0.319, only 0.066 higher. Given only four raters and non-independent items, this difference is well within plausible sampling noise. The conclusion that 'evaluating role-playing with ALLM judges is at least as good as using human evaluators' is not supported without uncertainty quantification, and the low human-human baseline weakens the interpretation that an ALLM-human correlation of roughly 0.3 indicates reliable judging. Please provide confidence intervals, and ideally a comparison against a null distribution obtained by correlating one human rater's scores with the average of the remaining three raters on the same items.
  3. [Section 4.2, Table 2] The correlations pool 80 scores from four SLMs with substantially different mean quality. Pearson's r over such pooled data can be inflated by between-model separation, so the reported values may overstate instance-level agreement. To support the claim that ALLMs track human judgments on individual items, please report within-SLM correlations (or within-context correlations), and consider a variance decomposition that separates between-model from within-model agreement.
  4. [Section 4.2, Section 4.3, Table 1] GPT-4o-audio is used both as one of the four SLMs whose outputs are judged and as one of the two ALLM judges. The paper acknowledges potential self-enhancement bias, but because GPT-4o-audio is presented as a candidate judge, the human-4o correlation of 0.355 could be affected by this overlap. I ask the authors to analyze this confound concretely, for example by reporting per-SLM correlations or by checking whether the 4o judge's scores for its own outputs differ systematically from human scores in ways that the other judge does not exhibit.
minor comments (6)
  1. [Section 1] There is a typo in the introduction: 'Invoice style instruction following' should read 'Voice style instruction following.'
  2. [Throughout] The headings contain an extra space in 'V oice style IF' and similar phrases; please fix the spacing.
  3. [Appendix A.2.1] The prompt for SLM 2 reads 'pretend that you are [role_2] and I am [role_2]', which appears to be a typo; presumably the second role should be [role_1]. Also, 'dialoge_context' should be 'dialogue_context'.
  4. [Section 4.1 and Appendix C] The aggregation of the five sampled judge responses is described as 'ensemble the verdicts', but the exact aggregation rule (majority vote, average, or something else) is not specified. Please state it explicitly for reproducibility.
  5. [Table 2 caption] The caption should clarify that 'Human–4o' and 'Human–Gemini' are averaged correlations between the ALLM judge and each of the four human raters, while 'Human–Human' is the average pairwise correlation among human raters; this is implied in the text but should be stated in the table.
  6. [Section 4.2] The temperature-sensitivity analysis for Gemini (ranging from 0.640 to 0.649) is useful, but it quantifies sensitivity to decoding hyperparameters, not sampling variability due to items or raters; the paper should not present this as a substitute for confidence intervals.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the ALLM-judge comparison is an empirical evaluation against external human ratings, with no fitted parameters or self-definitional reduction.

full rationale

The paper makes no analytic derivation that could reduce to its own inputs. The central evidence is a correlation between ALLM verdicts and independently collected human ratings on 80 scored items per task (4 SLMs x 20 instances), plus human-rated IEMOCAP dialogues as a sanity check. No constant is fitted to the human data and then reported as a prediction; the Pearson correlations in Table 2 are computed directly from the two score arrays. The authors' use of their own prior work (Chiang and Lee 2023a,b) is limited to citing LLM-as-a-judge methodology and chain-of-thought prompting; those citations are not invoked to justify the conclusion that Gemini agrees with humans. The fact that GPT-4o-audio is both an SLM under test and an ALLM judge is a possible self-enhancement bias, and the paper explicitly flags this and relies on Gemini and human cross-checks; a bias in one judge is not a circular derivation. Statistical concerns, such as the lack of confidence intervals and the low human-human agreement on role-playing, are validity issues rather than circularity. Under the quoted-evidence standard, no step in the paper is equivalent by construction to its input, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

This is an empirical study without a mathematical derivation, so there are no free parameters fitted to data. The experimental choices (20 instances, 4 raters, temperature 1.0, top_p 0.9, 5-fold ensemble) are fixed hyperparameters, not fitted to make a claim work. The axioms listed are the load-bearing assumptions adopted from the evaluation literature.

assumptions (4)
  • domain assumption Human evaluation scores are treated as ground truth for speaking style quality.
    The paper validates ALLM judges by comparing them to human judges (Table 2), so the reliability of the human ratings is load-bearing; the paper itself reports low human-human correlation (r=0.253) for role-playing style in Section 4.3.
  • domain assumption Pearson's r computed over 80 scores (4 models x 20 instances) is an appropriate measure of judge agreement.
    The correlation mixes between-model differences with within-model instance-level agreement, and the paper does not report within-model correlations or confidence intervals (Section 4.2).
  • domain assumption The ALLM judges can actually perceive the non-textual audio properties they are asked to rate.
    The entire method assumes audio-aware models hear and interpret prosody, emotion, volume, pace, and non-verbal sounds; this is plausible but not independently verified in the paper (Sections 1 and 4.1).
  • domain assumption Concatenating turn-by-turn audio with two-second silences into one-minute clips preserves the speaking style information for judging.
    The evaluation inputs are artificial concatenations, and the paper acknowledges that identical voices for both roles and longer silences differ from natural dialogue (Appendix A.2.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Audio-Aware Large Language Models as Judges for Speaking Styles." pith.science (2026). https://pith.science/paper/TNPZWODA

@misc{pith2026250605984,
  author       = {Pith},
  title        = {Pith review of: Audio-Aware Large Language Models as Judges for Speaking Styles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TNPZWODA}},
  note         = {Machine review of arXiv:2506.05984}
}
read the original abstract

Audio-aware large language models (ALLMs) can understand the textual and non-textual information in the audio input. In this paper, we explore using ALLMs as an automatic judge to assess the speaking styles of speeches. We use ALLM judges to evaluate the speeches generated by SLMs on two tasks: voice style instruction following and role-playing. The speaking style we consider includes emotion, volume, speaking pace, word emphasis, pitch control, and non-verbal elements. We use four spoken language models (SLMs) to complete the two tasks and use humans and ALLMs to judge the SLMs' responses. We compare two ALLM judges, GPT-4o-audio and Gemini-2.5-pro, with human evaluation results and show that the agreement between Gemini and human judges is comparable to the agreement between human evaluators. These promising results show that ALLMs can be used as a judge to evaluate SLMs. Our results also reveal that current SLMs, even GPT-4o-audio, still have room for improvement in controlling the speaking style and generating natural dialogues.

Figures

Figures reproduced from arXiv: 2506.05984 by the authors.

Figure 2
Figure 2. The interface used to collect human evaluation results for the role-playing task. [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 17 canonical work pages

  1. [1]

    **Text Following**: Does the generated speech accurately follow the text it was instructed to say? - If the speech omits required words or includes additional content not present in the original text, it should be considered as **not following** the text

    work page

  2. [2]

    ### Scoring Rubric - **1**: The speech **does not follow** the required text, regardless of style

    **Style Following**: Does the generated speech follow the style specified in the instruction? - Style may include emotional tone, prosody, emphasis, or any specific expressive requirement. ### Scoring Rubric - **1**: The speech **does not follow** the required text, regardless of style. - **2**: The speech **follows the text** but **does not follow any** ...

    work page

  3. [3]

    Provide a brief analysis of how the speech does or does not meet each requirement

    **Evaluate Style Adherence** For each requirement you listed in Step 1, evaluate how well the speech satisfies it. Provide a brief analysis of how the speech does or does not meet each requirement

    work page

  4. [4]

    Follow the scoring rubric strictly

    **Assign a Style Score** Based on your analysis, assign a final score between **2 and 5** reflecting how well the speech follows the intended style. Follow the scoring rubric strictly

    work page

  5. [5]

    **Analyze the Instruction** Carefully read the style instruction and list all style-related requirements that the TTS-generated speech must satisfy

    work page

  6. [6]

    - If it does **not** match, assign a score of **1** and skip the style evaluation

    **Check Text Accuracy** Listen to the generated speech and check whether the spoken text exactly matches the given text. - If it does **not** match, assign a score of **1** and skip the style evaluation

    work page

  7. [9]

    Replace score with an integer in {1, 2, 3, 4, 5}

    **Report the Final Score** Conclude your evaluation with the following format: Final score: [[score]]. Replace score with an integer in {1, 2, 3, 4, 5}. Keep the brackets as shown. # Text for the TTS Model to Generate Speech {target_text} # Speaking Style Instructions to the TTS Model {speaking_style} # The Speech Generated by the TTS Model Table 3: The e...

    work page

  8. [10]

    **Content Semantics** This criterion evaluates the semantic quality of the spoken content. Consider whether the dialogue is aligned with the background context, whether the conversation includes repetitive or irrelevant information, and whether the wording is natural given the relationship between the two roles and the topic being discussed

    work page

Show all 19 references
  1. [11]

    **Speaking Style** This criterion evaluates the non-textual aspects of the spoken dialogue. Consider whether the emotion and speaking style are appropriate for the context, and whether non-verbal elements (e.g., sighs, laughter, short pauses) are used effectively to enhance na...

  2. [12]

    **Analyze the Role and Context** Carefully read the context and understand the two roles the voice assistant is expected to play

  3. [13]

    **Listen to the Role-Play Dialogue** Listen closely to the spoken dialogue generated by the voice assistant

  4. [14]

    Reference specific turns or moments in the dialogue to support and justify your evaluation

    **Analyze the Role-Play Dialogue Based on the Evaluation Criteria** Evaluate the quality of the role-play according to the criteria above. Reference specific turns or moments in the dialogue to support and justify your evaluation

  5. [15]

    Keep the double brackets as shown

    **Report the Final Score** Conclude your evaluation with the following format: Final score: [[score]] Replace score with an integer in {1, 2, 3, 4, 5}. Keep the double brackets as shown. # Context of the Role-Playing {role_play_context} **Note**: In the above context, the voic...

  6. [16]

    **Analyze the Role and Context** Carefully read the context and understand the two roles in the role-playing

  7. [17]

    **Listen to the Role-Play Dialogue** Listen closely to the spoken dialogue

  8. [18]

    **Analyze the Role-Play Dialogue** Evaluate the role-play according to whether it is likely generated by humans

  9. [19]

    Oh”, emphasize “exactly

    **Report the Final Score** Conclude your evaluation with the following format: Final score: [[score]] Replace score with an integer in {0, 1}. Keep the double brackets as shown. # Context of the Role-Playing {role_play_context} **Note**: In the above context, the participant i...

  10. [2023]

    Why we should report the details in subjective evaluation of tts more rigorously. InProc. Inter- speech 2023, pages 5551–5555. Cheng-Han Chiang and Hung-yi Lee. 2023a. Can large language models be an alternative to human evalua- tions? InProceedings of the 61st Annual Meeting ...

    2023 arXiv

  11. [2025]

    SLM 1" and the same SLM that plays the other role

    V oxeval: Benchmarking the knowledge under- standing capabilities of end-to-end spoken language models.arXiv preprint arXiv:2501.04962. 5 Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. 2024. Moshi: ...

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.