Pith. sign in

REVIEW 5 major objections 7 minor 1 cited by

An approach to measuring the performance of Automatic Speech Recognition (ASR) models in the context of Large Language Model (LLM) powered applications

T0 review · 5 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes Answer Error Rate (AER), a context-level metric for evaluating ASR in LLM applications, and argues it reflects downstream performance better than WER.

desk verdict A plausible but unvalidated evaluation metric; the core idea deserves referee attention, but the paper needs a control condition and external validation before the central claim can be trusted. read the letter →

arxiv 2507.16456 v1 pith:KJ3PBVED submitted 2025-07-22 eess.AS cs.SD

Sujith Pulikodan , Sahapthan K , Prasanta Kumar Ghosh , Visruth Sanka , Nihar Desai This is my paper · ORCID
classification eess.AScs.SD
keywords automaticspeechrecognitionworderrorrateanswerlargelanguagemodelsASRevaluationLLMcorrectioncontextpreservation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that Word Error Rate (WER), which counts word-level insertions, deletions, and substitutions, is the wrong yardstick for choosing an ASR system when the transcript is passed to a large language model. It proposes Answer Error Rate (AER): generate questions from the reference text, answer them from both the reference and the ASR transcript, and count the fraction whose answers an LLM judge considers contextually different. Across several Whisper and Wav2Vec2 systems on three English datasets, AER is consistently higher than WER and does not track WER rankings, so a lower-WER model is not necessarily better for LLM-powered applications. A sympathetic reader should care because model selection in voice AI is currently driven by WER, and this metric gives a meaning-based alternative.

What carries the argument

The load-bearing object is the Answer Error Rate (AER), defined as the fraction of automatically generated questions for which an LLM judge decides that the answer drawn from the ASR transcript differs contextually from the answer drawn from the reference. It is carried by a three-LLM pipeline: LLM1 generates questions from the reference, LLM2 answers them from each context, and LLM3 compares the two answer sets. The mechanism converts ASR evaluation from word-counting to context-preservation checking, so errors on content-bearing words count more than errors on words an LLM can infer or ignore.

What would settle it

Run the AER pipeline on a fixed set of utterances with two different judge LLMs and compare the resulting ASR rankings with human ratings of answer equivalence; if the AER ordering of models flips while human ratings stay fixed, the metric is measuring judge behavior rather than ASR quality.

Watch

Extended reading notes

Core claim

The paper's central claim is that for applications where an LLM consumes ASR output, the right question is not 'how many words match?' but 'does the transcript preserve the meaning the LLM acts on?' To make that question measurable, the paper introduces AER, computed by (1) using one LLM to generate as many questions as possible from the reference text, (2) using a second LLM to answer those questions from the reference and from the ASR transcript independently, and (3) using a third, more powerful LLM to judge each pair of answers as contextually equivalent or different. AER is the fraction of questions judged different, with higher values meaning more context lost. On their evaluation, AER is much larger than WER and behaves differently: for example, on LibriSpeech, the Wav2Vec2-large model has the best WER at 2.76% but an AER around 26.43%, nearly the same as a Whisper model with 8.72% WER. The paper reads this as evidence that relying on WER alone can be misleading for LLM-based systems, and that AER better reflects the ASR's ability to preserve meaningful context for downstream tasks.

Load-bearing premise

The approach assumes that an LLM's judgment that two answers are contextually the same is a faithful measure of whether the ASR preserved the meaning a real downstream application needs.

Editorial extensions

If this is right

  • Model selection by WER alone can pick an ASR engine that is worse for LLM-powered applications: a model with lower WER can have the same or higher AER than a simpler competitor.
  • LLM-based error correction reduces WER most when the ASR is weak, but can slightly hurt already-accurate transcripts, so the correction step is not universally beneficial.
  • AER and WER rank systems differently, so any benchmark that claims to predict end-to-end voice-LLM performance should include a context-level measure such as AER.
  • Because spontaneous speech (VoxPopuli) shows much higher AER than read speech, application domain matters: a model chosen on a read-speech benchmark may not preserve enough meaning for conversational LLM use.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not shown in the paper: whether AER rankings survive swapping the judge LLM; testing this would separate ASR quality from judge bias.
  • Not shown in the paper: AER's sensitivity to questions generated only from the reference; generating questions from both contexts could catch hallucinated divergences.
  • Not shown in the paper: a direct comparison of AER against task accuracy; such a comparison would be the cleanest test of the metric's premise.
  • Not shown in the paper: online use of AER against user task completion in a deployed voice assistant.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper addresses the evaluation of Automatic Speech Recognition (ASR) systems in the context of LLM-powered applications. It first assesses whether LLMs can correct ASR errors using a simple one-shot prompt, reporting WER before and after correction for several Whisper and Wav2Vec2 models on Fleurs, LibriSpeech, and VoxPopuli. It then proposes a new metric, Answer Error Rate (AER), which is computed by generating questions from the reference text, obtaining answers from both the reference and the ASR transcript using a second LLM, and using a third LLM to judge whether the answers are contextually equivalent. The paper reports AER values for the same models and datasets and argues that low WER does not necessarily translate to better performance in LLM-based systems, concluding that AER better reflects the ASR system's ability to capture meaningful context in downstream tasks.

Significance. If validated, the proposed AER metric would be a practically useful complement to WER for model selection in spoken-language applications that rely on LLMs. The paper addresses a genuine and current limitation of WER, and the experimental setup is transparent: it uses standard public datasets, names the exact models, and includes the prompts used, making the pipeline in principle reproducible. However, the central claim is not yet supported. AER is never validated against any external criterion such as human judgments or downstream task performance, and no control condition is reported to separate ASR-induced context loss from the stochastic noise of the LLM pipeline. As it stands, the paper presents an interesting idea and a set of descriptive numbers, but the proposed metric's reliability is unestablished.

major comments (5)
  1. [Section 3.2] The AER definition does not control for LLM stochasticity. LLM2 answers are generated separately from the reference and ASR contexts, and LLM3 is a fallible judge. Even if the two contexts are identical, sampling can yield different answers and judge errors, so AER can be nonzero. The manuscript never reports an identical-context control, leaving the AER floor unknown. Without this baseline, the AER values in Table 2 (14.77% to 47.39%) cannot be attributed to ASR quality. Please add a control condition in which the reference text is used in both branches, report the resulting AER, and provide confidence intervals or error bars for the main estimates.
  2. [Section 5, concluding paragraph] The claim that AER provides a better reflection of the ASR system's ability to capture meaningful context in downstream tasks is not validated by any external criterion. The paper does not compare AER with human judgments, with performance on a concrete downstream task, or with existing semantic metrics such as SemDist [3] or BERTScore [6]. The observed lack of correspondence between WER and AER is purely descriptive; no correlation coefficient, significance test, or effect-size measure is reported. Please add external validation or, at minimum, explicitly state that the metric is proposed without validation and frame the conclusions accordingly.
  3. [Table 1] The entry openai/whisper-large/LibriSpeech/Gemini 1.5 Flash reports 26.94% WER after correction, whereas the same model has 4.41% WER before correction and 5.20% after correction with GPT-4o-mini. This outlier is roughly 20 points above all neighboring values and strongly suggests a data or pipeline error, undermining confidence in the correction experiment. Several other cells also lack percent signs (e.g., 06.64, 05.34, 06.99), and no replication or variability information is provided, making it impossible to assess whether differences between models are meaningful.
  4. [Section 3.2] Because LLM1 generates questions only from the reference text, AER is blind to ASR errors in portions of the transcript that do not yield questions. The manuscript does not report the number of questions generated per utterance, the fraction of the reference text covered by the questions, or the sensitivity of AER to the choice of LLM1, LLM2, and LLM3. Only one configuration (Gemini 1.5 Flash, GPT-4o-mini, GPT-4o) is evaluated, so the robustness and generality of the metric are not established.
  5. [Section 5, Table 2] The observation that larger ASR models show only marginal AER gains is based on a single LLM configuration and small sample sizes; for example, whisper-large versus whisper-tiny on LibriSpeech gives AER 25.40% versus 30.09%, but with only 250 utterances the difference may be within sampling noise. No statistical analysis is provided for any of the comparisons in Table 2, so the paper's qualitative conclusions about model ranking are not supported.
minor comments (7)
  1. [Section 3, first paragraph] The sentence 'we propose an alternative metric WER' should read 'AER'; as written, it mistakenly suggests the paper is proposing WER.
  2. [Table 1] The table formatting is inconsistent: percent signs are missing in several cells (e.g., 06.64, 05.34, 06.99) and one cell has an errant space ('13.63 %').
  3. [Section 3.2, example prompt] The example answer-generation prompt contains typos ('Provide the the dictionary', 'don not provide') and the sample questions are ungrammatical ('is the speakers current status on making the powder?'); these should be cleaned up.
  4. [Section 2] In the Related Work section, the citation for R. Ma's exploration of decoding strategies appears as '[?]' and must be completed.
  5. [Throughout] The dataset name is inconsistently typeset as 'V oxPopuli' with a space; it should be 'VoxPopuli'.
  6. [Section 5 and Figure 2] Figure 2 is referenced in the text but does not appear in the manuscript; please ensure the figure is embedded and properly labeled.
  7. [Section 3.1] The approach is described as 'one-shot', but the provided prompt contains no example and the setting appears to be zero-shot; please clarify.

Circularity Check

1 steps flagged · score 4.0 of 10

AER is defined as the proportion of contextually differing answers, so the claim that it reflects context is true by construction; whether it is 'better' than WER for downstream tasks is asserted without independent validation.

  1. self definitional [Section 3.2 (definition of AER) and Section 5 (concluding interpretation)]
    "AER is defined as the ratio of the number of questions with contextually differing answers to the total number of questions. ... We argue that AER provides a better reflection of the ASR system's ability to capture meaningful context in downstream tasks."

    The metric's error event is defined as 'contextually differing answers' as judged by LLM3. Therefore the conclusion that AER reflects contextual meaning is entailed by the definition: any question flagged by the judge is, by definition, an AER error. The paper does not compare AER against human equivalence judgments or actual downstream task performance, so the stronger claim that AER is a better reflection of downstream context is an assumption baked into the metric rather than an empirically established result. The comparison to WER (rankings differ) is independent, which is why the circularity is only partial.

full rationale

The paper contains no fitted parameters, no self-citations, and no imported uniqueness theorems, so most circularity patterns do not apply. The central weakness is that AER's validity as a context metric is built into its definition: a question counts as an error exactly when an LLM judge says the answers to reference and ASR contexts are contextually different. This makes the observation that AER is context-sensitive tautological. However, the paper's empirical observation that AER rankings differ from WER rankings is not forced by the definition, and the LLM error-correction results in Table 1 are standard WER measurements independent of AER. The absence of an identical-context control and the lack of validation against human judgments or real task metrics are correctness/validity concerns rather than circularity, so they are noted but do not raise the score above 4.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper's central claim rests entirely on the assumption that the three-LLM pipeline (question generation, answering, judging) faithfully measures contextual understanding relevant to downstream LLM applications. This assumption is neither validated against human judgments nor against an external downstream task.

assumptions (4)
  • domain assumption Questions generated by LLM1 from the reference text are a valid, comprehensive probe of the context needed for downstream tasks.
    Section 3.2 assumes the generated questions cover key aspects of the context; no coverage analysis or human validation is provided.
  • domain assumption Answer equivalence judgments by LLM3 reliably reflect contextual differences relevant to LLM applications.
    The AER definition in Section 3.2 relies on LLM3 to classify answers as equivalent; the judge's agreement with humans is never measured.
  • domain assumption The small subsets of Fleurs, LibriSpeech, and VoxPopuli (645, 250, and 245 utterances) are representative of each dataset.
    Section 4 describes the subsets without specifying selection criteria or seeds.
  • domain assumption LLM2's answers to the same questions under different contexts are comparable and sensitive to ASR errors.
    The entire AER measurement assumes that context changes due to ASR errors will manifest as answer changes; no control experiment demonstrates this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An approach to measuring the performance of Automatic Speech Recognition (ASR) models in the context of Large Language Model (LLM) powered applications." pith.science (2026). https://pith.science/paper/KJ3PBVED

@misc{pith2026250716456,
  author       = {Pith},
  title        = {Pith review of: An approach to measuring the performance of Automatic Speech Recognition (ASR) models in the context of Large Language Model (LLM) powered applications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KJ3PBVED}},
  note         = {Machine review of arXiv:2507.16456}
}
read the original abstract

Automatic Speech Recognition (ASR) plays a crucial role in human-machine interaction and serves as an interface for a wide range of applications. Traditionally, ASR performance has been evaluated using Word Error Rate (WER), a metric that quantifies the number of insertions, deletions, and substitutions in the generated transcriptions. However, with the increasing adoption of large and powerful Large Language Models (LLMs) as the core processing component in various applications, the significance of different types of ASR errors in downstream tasks warrants further exploration. In this work, we analyze the capabilities of LLMs to correct errors introduced by ASRs and propose a new measure to evaluate ASR performance for LLM-powered applications.

Figures

Figures reproduced from arXiv: 2507.16456 by the authors.

Figure 1
Figure 1. Proposed Evaluation Framework. The experiment follows a question-and-answer framework. The process starts by generating as many meaningful questions as possible using an LLM(LLM1). These questions are derived from the reference (ground truth), and the LLM is prompted to generate numerous questions based on the provided context. The prompt is carefully designed to ensure that the questions are relevant to the context… view at source ↗
Figure 2
Figure 2. Comparison between WER and AER The performance comparison of the proposed AER is pre￾sented in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AgenticASR: Refining Speech Recognition in Real-World Scenarios via an Agentic Approach

    cs.AI 2026-07 conditional novelty 6.0 of 10

    An ASR–Refiner system emits and revises clean transcripts online over a bounded sliding context, outperforming offline spoken-to-written baselines on a new bilingual rubric benchmark.

Reference graph

Works this paper leans on

29 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [3]

    Our goal is to explore the potential of LLMs by systemati- cally evaluating multiple models across a diverse set of English- language datasets

    Our Approach In this work, we investigate the correction capabilities of LLMs using a one-shot approach, considering that in many use cases, the full n-best output from the ASR system may not be accessi- ble. Our goal is to explore the potential of LLMs by systemati- cally evaluating multiple models across a diverse set of English- language datasets. The ...

    work page

  2. [6]

    Extending this analysis to multiple LLMs would be useful to better understand the overall trends across different models

    Conclusion and Future works We can observe that LLMs have the ability to correct ASR out- put, and we found that a low WER does not necessarily translate to better performance in LLM-based systems. Extending this analysis to multiple LLMs would be useful to better understand the overall trends across different models

    work page

  3. [1]

    Significant ad- vancements in the field have led to increasingly accurate mod- els, some of which even surpass human-level performance [1]

    Introduction Automated Speech Recognition (ASR) has become a corner- stone of human-computer interaction, enabling more natural and seamless communication with machines. Significant ad- vancements in the field have led to increasingly accurate mod- els, some of which even surpass human-level performance [1]. However, these achievements are predominantly c...

    work page

  4. [2]

    Related work Several approaches have been explored to improve ASR perfor- mance by incorporating LLMs. S. Li et al. fine-tuned a multilin- gual LLM covering more than 100 languages to correct 1-best hypothesis errors from various speech foundation models [8]. Z. Ma et al. introduced SLAM-ASR, which features a frozen speech encoder and a frozen LLM, with a...

    arXiv 2025

  5. [4]

    Specifically, we used 645 ut- terances from Fleurs, 245 utterances from V oxPopuli, and 250 utterances from Librispeech, all in English

    Datasets and Models used For our experiment, we utilized three datasets: Fleurs [14], V ox- Populi [15], and Librispeech [16]. Specifically, we used 645 ut- terances from Fleurs, 245 utterances from V oxPopuli, and 250 utterances from Librispeech, all in English. The Fleurs dataset and LibriSpeech both consist of read speech, where speakers deliver pre-wr...

    work page

  6. [5]

    As expected, the performance of ASR improves with an increase in the number of parameters in models that use the same backend architec- ture

    Results The performance of ASRs, along with the error correction ap- proach of LLMs, is presented in Table 1. As expected, the performance of ASR improves with an increase in the number of parameters in models that use the same backend architec- ture. For datasets consisting of read speech, the performance is relatively higher, as anticipated. However, fo...

    work page

  7. [7]

    Acknowledgments We sincerely thank the leadership teams of the Indian Institute of Science (IISc) and ARTPARK for their invaluable support, which has enabled us to work on this problem

    work page

  8. [8]

    The microsoft 2017 conversational speech recognition system,

    W. Xiong, L. Wu, F. Alleva, J. Droppo, X. Huang, and A. Stolcke, “The microsoft 2017 conversational speech recognition system,” inInternational Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP). IEEE, 2018, pp. 5934–5938

    work page 2017

Show all 29 references
  1. [9]

    Improving automatic speech recogni- tion performance for low-resource languages with self-supervised models,

    J. Zhao and W.-Q. Zhang, “Improving automatic speech recogni- tion performance for low-resource languages with self-supervised models,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1227–1241, 2022

    2022

  2. [10]

    Semantic distance: A new metric for ASR perfor- mance analysis towards spoken language understanding,

    S. Kim, A. Arora, D. Le, C.-F. Yeh, C. Fuegen, O. Kalinli, and M. L. Seltzer, “Semantic distance: A new metric for ASR perfor- mance analysis towards spoken language understanding,”arXiv preprint arXiv:2104.02138, 2021

    2021 arXiv

  3. [11]

    Semantic word error rate for sentence similarity,

    C. Spiccia, A. Augello, G. Pilato, and G. Vassallo, “Semantic word error rate for sentence similarity,” inTenth International Conference on Semantic Computing (ICSC). IEEE, 2016, pp. 266–269

    2016

  4. [12]

    From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition

    A. C. Morris, V . Maier, and P. D. Green, “From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition.” inInterspeech, 2004, pp. 2765–2768

    2004

  5. [13]

    Bertscore: Evaluating text generation with bert,

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with bert,”arXiv preprint arXiv:1904.09675, 2019

    1904 arXiv

  6. [14]

    A comprehensive overview of large language models,

    H. Naveed, A. U. Khan, S. Qiu, M. Saqib, S. Anwar, M. Usman, N. Akhtar, N. Barnes, and A. Mian, “A comprehensive overview of large language models,”arXiv preprint arXiv:2307.06435, 2023

    2023 arXiv

  7. [15]

    Investigating ASR error correction with large language model and multilingual 1-best hypotheses,

    S. Li, C. Chen, C. Y . Kwok, C. Chu, E. S. Chng, and H. Kawai, “Investigating ASR error correction with large language model and multilingual 1-best hypotheses,” inProc. Interspeech, 2024, pp. 1315–1319

    2024

  8. [16]

    An embarrassingly sim- ple approach for LLM with strong ASR capacity,

    Z. Ma, G. Yang, Y . Yang, Z. Gao, J. Wang, Z. Du, F. Yu, Q. Chen, S. Zheng, S. Zhanget al., “An embarrassingly sim- ple approach for LLM with strong ASR capacity,”arXiv preprint arXiv:2402.08846, 2024

    2024 arXiv

  9. [17]

    Evolutionary prompt design for LLM-based post-ASR error correction,

    R. Sachdev, Z.-Q. Wang, and C.-H. H. Yang, “Evolutionary prompt design for LLM-based post-ASR error correction,”arXiv preprint arXiv:2407.16370, 2024

    2024 arXiv

  10. [18]

    Lexical error guard: Lever- aging large language models for enhanced ASR error correction,

    M. Si, O. Cobas, and M. Fababeir, “Lexical error guard: Lever- aging large language models for enhanced ASR error correction,” Machine Learning and Knowledge Extraction, vol. 6, no. 4, pp. 2435–2446, 2024

    2024

  11. [19]

    Can genera- tive large language models perform ASR error correction?

    R. Ma, M. Qian, P. Manakul, M. Gales, and K. Knill, “Can genera- tive large language models perform ASR error correction?”arXiv preprint arXiv:2307.04172, 2023

    2023 arXiv

  12. [20]

    Leveraging large language models for exploiting ASR un- certainty,

    P. Dighe, Y . Su, S. Zheng, Y . Liu, V . Garg, X. Niu, and A. Tew- fik, “Leveraging large language models for exploiting ASR un- certainty,” inInternational Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 12 231–12 235

    2024

  13. [21]

    Fleurs: Few-shot learning evaluation of universal representations of speech,

    A. Conneau, M. Ma, S. Khanuja, Y . Zhang, V . Axelrod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna, “Fleurs: Few-shot learning evaluation of universal representations of speech,” inSpoken Lan- guage Technology Workshop (SLT). IEEE, 2023, pp. 798–805

    2023

  14. [22]

    V oxpopuli: A large-scale multilingual speech corpus for representation learn- ing, semi-supervised learning and interpretation,

    C. Wang, M. Riviere, A. Lee, A. Wu, C. Talnikar, D. Haz- iza, M. WILliamson, J. Pino, and E. Dupoux, “V oxpopuli: A large-scale multilingual speech corpus for representation learn- ing, semi-supervised learning and interpretation,”arXiv preprint arXiv:2101.00390, 2021

    2021 arXiv

  15. [23]

    Lib- rispeech: an ASR corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: an ASR corpus based on public domain audio books,” inInternational Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP). IEEE, 2015, pp. 5206–5210

    2015

  16. [24]

    Whisper: A general-purpose speech recognition model,

    OpenAI, “Whisper: A general-purpose speech recognition model,” https://openai.com/research/whisper, 2022, accessed: 2024-12-17

    2022

  17. [25]

    wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” 2020. [Online]. Available: https://arxiv.org/abs/ 2006.11477

    2020 arXiv

  18. [26]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Ale- man, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  19. [27]

    Gemini 1.5: Un- locking multimodal understanding across millions of tokens of context,

    G. Team, P. Georgiev, V . I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wanget al., “Gemini 1.5: Un- locking multimodal understanding across millions of tokens of context,”arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  20. [28]

    Chatbot arena: An open platform for evaluating LLMs by human preference,

    W.-L. Chiang, L. Zheng, Y . Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, and I. Stoica, “Chatbot arena: An open platform for evaluating LLMs by human preference,” 2024. [Online]. Available: https://arxiv.org/abs/2403.04132

    2024 arXiv

  21. [29]

    Berkeley function calling leader- board,

    F. Yan, H. Mao, C. C.-J. Ji, T. Zhang, S. G. Patil, I. Sto- ica, and J. E. Gonzalez, “Berkeley function calling leader- board,” https://gorilla.cs.berkeley.edu/blogs/8 berkeley function calling leaderboard.html, 2024

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.