Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

CONFETTI: Conversational Function-Calling Evaluation Through Turn-Level Interactions

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces CONFETTI, a turn-level benchmark for conversational function calling, and shows that top LLMs reach only about 41% accuracy while chained calls remain largely unsolved.

desk verdict The benchmark is a real contribution; the chain-length headline is a derived product, not a measured success rate, so treat the paper as a useful resource in need of a major revision. read the letter →

arxiv 2506.01859 v1 pith:DFV5LZ4V submitted 2025-06-02 cs.CL

Tamer Alkhouli , Katerina Margatina , James Gung , Raphael Shu , Claudia Zaghi , Monica Sunkara , Yi Zhang This is my paper · ORCID
classification cs.CL
keywords conversationalfunctioncallingturn-levelevaluationbenchmarkdialogactstooluseASTsoftaccuracychainedcallsLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CONFETTI is a benchmark for measuring how well large language models call functions in the middle of real-style conversations, where users follow up, correct themselves, switch goals, or leave details implicit. The paper's central claim is that turn-level evaluation on human-simulated dialogues exposes a large gap between simple single-turn tool use and conversational tool use: the best model, Nova Pro, reaches 40.91% abstract-syntax-tree soft accuracy, and most models do worse. The benchmark also drills into what fails: chained function calls degrade sharply, with the best model at 17.71% accuracy on chains of length three, and increasing the number of available APIs hurts several otherwise strong models. This matters because real agents need to sustain tool use over many turns, and existing benchmarks do not measure that in a controlled way.

What carries the argument

The load-bearing mechanism is off-policy turn-level evaluation. Each example truncates a conversation at a function-calling turn and gives the model the gold-truth conversation up to that point, including prior agent turns and simulated function results, then asks for the next action or action chain. Function calls are scored by parsing them into abstract syntax trees and matching names, parameters, and values against a reference, with string values scored softly using AlignScore. Chained-call performance is computed by teacher-forcing: each action in the chain is scored conditioned on the gold previous actions and results, and the chain score is the product of per-action accuracies. Response quality is scored separately by classifying agent turns into dialog acts (inform, seek information, reject, function-call, other) with an LLM-based classifier.

What would settle it

Run the same conversations on-policy: let the model generate every agent turn and function call from the start, execute its calls against the simulator, and only then score the final outcome or the sequence of actions. If on-policy accuracy on later turns and on chains of length three is close to the reported off-policy numbers, the off-policy assumption holds; if it drops substantially, the benchmark's headline results overstate deployed agent capability.

Watch

Extended reading notes

Core claim

The paper claims that CONFETTI provides a valid, rich test bed for conversational function calling, and that on this test bed state-of-the-art LLMs are far from reliable. Its headline numbers: Nova Pro at 40.91% AST soft accuracy, Claude Sonnet v3.5 at 35.46%, and Llama 3.1 405B at 33.19%, with the rest below 32%. Chained actions are the weakest point: length-2 chains succeed at most 21.46%, length-3 chains at most 17.71%, and length-4+ chains essentially fail. The benchmark's design choices all serve this conclusion: conversations are written by humans to include thirteen conversational complexities, function results are simulated and fixed, and each turn is scored in isolation with the gold history as context.

Load-bearing premise

The entire evaluation assumes that showing a model the gold-truth conversation up to the current turn, including perfect function results, tells us how well the model would act if it had to generate its own previous turns; the paper itself acknowledges this can inflate later-turn scores through in-context learning.

Editorial extensions

If this is right

  • No evaluated model approaches reliable conversational function calling; the best AST soft accuracy is 40.91%.
  • Chained function calls are the clearest bottleneck: even the best model succeeds only about 18% of the time on chains of length three.
  • Scaling the number of available APIs to 20+ degrades performance for several models, while a few, including Nova Pro and Mistral Large, hold up.
  • Dialog act analysis shows models both over-trigger and under-trigger function calls, frequently responding with information or clarification when a call is needed.
  • Parameter validity is limited even for the best models: Nova Pro's predicted parameters are valid 68.8% of the time versus 83.1% for the gold references.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The off-policy setup likely overstates real interactive performance: because the model receives gold-truth agent turns and function results as context, its later predictions can learn in-context from the gold trajectory; an on-policy re-run with the model's own previous outputs would probably show larger gaps, especially on chains.
  • The 20+ API conclusion rests on only 15 examples, so the apparent difference between models that handle 20+ APIs and those that do not may not be stable; a larger sample could change that ranking.
  • Because string parameter values are scored with a soft alignment model, the AST soft metric rewards paraphrases; a stricter exact-match or schema-validity metric would likely lower the reported scores.
  • The dialog act classifier used for response quality has 67.2% multi-label precision, so the response-quality rankings are noisy; swapping in a higher-precision judge could reorder the lower half of the leaderboard.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces CONFETTI, a conversational function-calling benchmark made of 109 human-simulated conversations, 313 user turns, 86 APIs, and 506 turn-level examples. It evaluates a set of LLMs in an off-policy, teacher-forced setup using AST soft accuracy for function-calling turns, dialog-act classification for response quality, and an LLM-based parameter-hallucination check. The headline findings are that top models reach around 40% AST soft accuracy, that performance degrades for several models as the number of APIs or conversation length grows, and that chained function-calling performance is 'severely limited' across all models.

Significance. The dataset is potentially a valuable resource: it is human-authored, covers a defined complexity taxonomy, spans 86 APIs, and is accompanied by a public GitHub link and a data statement. The off-policy turn-level design is clearly described, and the paper is transparent in Section 7 about important limitations. However, the most striking conclusion—that chained function-calling is severely limited—rests on a derived product of teacher-forced per-step scores rather than on any observed full-chain prediction, and the response-quality metric uses a lenient at-least-one-label match rule. If the chain evaluation is redone or the claims are appropriately weakened, the benchmark can still be a useful addition to the field; in its current form, the quantitative conclusions outrun the measurements.

major comments (3)
  1. [§5.3.1, Figure 4; Appendix A, Tables 9 and 10] Section 5.3.1 defines the score of a length-J chain as the product of per-step AST soft accuracies, where each factor is conditioned on gold-truth previous actions and results, and Figure 4 is then used to conclude that 'the performance on chained function-calls is severely limited across the models.' This product is not a valid chain success rate. AST soft accuracy is a corpus-level average similarity score, not a per-instance probability; the factorization requires conditional independence of step errors, which is unlikely when chained calls share parameter values (airport codes, dates, IDs); and teacher-forcing means each factor conditions on the gold previous action rather than on the event that the model's own previous prediction was correct. In addition, the prompts in Tables 9 and 10 explicitly instruct the model to 'output the first function call in the chain only,' so no experiment actually asks the model for a complete multi-step chain. The reported 17.71% for Nova Pro at chain length 3 is therefore an indirect estimate with unstated independence and calibration assumptions, and it cannot support the abstract's headline claim. Please either run an on-policy chain evaluation (or at least report exact-match accuracy on full predicted chains under teacher forcing), or re-label these numbers as a derived per-step product and revise the abstract and conclusion accordingly.
  2. [Section 7; abstract; §5.2 and §5.3] Section 7 correctly states that the off-policy turn-level setup 'does not correspond directly to inference behavior' and can inflate later-turn scores via in-context learning. However, all headline results in the abstract, Section 5.2, and Section 5.3 are reported without this qualifier, so a reader will naturally interpret 'models are able to handle long conversations' and 'chained function-calling is severely limited' as statements about deployed interactive agents. Please propagate the off-policy caveat into the abstract and the results sections and, if possible, quantify the inflation by comparing early-turn and late-turn performance or by adding a small on-policy comparison.
  3. [§4.2, §5.3.2, Table 5] The response-quality evaluation relies on an LLM dialog-act classifier with multi-label precision of 67.2%, and the paper declares a match when at least one predicted label overlaps any gold label. This at-least-one rule makes the reported 92.8% 'single-label accuracy' on gold data, and hence the model DA accuracies in Table 6, difficult to interpret: with five coarse labels and multiple predicted labels, chance overlap is non-negligible, and the low precision means many false-positive labels are counted as correct. Please report exact multi-label agreement (or a stricter criterion) for both the classifier validation and the model responses, and discuss how the precision of the classifier affects the confusion-matrix analysis in Figure 5.
minor comments (6)
  1. [Figures 2 and 3 captions] The Figure 2 and Figure 3 captions describe counts that sum to 313, which equals the number of user turns in Table 2, not the 506 dataset examples; please correct the terminology or the counts.
  2. [Table 2] Table 2 reports '# total turns 506' while also reporting 313 user turns and 313 agent turns with actions; please clarify whether 506 is the number of action calls or the number of function-calling examples, as the current labeling is inconsistent.
  3. [Abstract and Table 4] The abstract reports Nova Pro as 40.01%, but Table 4 reports 40.91%; please align the numbers.
  4. [§5.3.1] The notation p(·) suggests a probability, but the quantity being multiplied is an AST soft accuracy averaged over examples; please use a different symbol or explicitly state that it is an average score rather than a probability.
  5. [Table 3] The category counts in Table 3 sum to 698 while the stated total is 663; because a turn can carry multiple dialog-act labels, please add a sentence clarifying that the category counts are label-level and may exceed the turn count.
  6. [Section 7, Data Statement] The data statement notes that the same annotator played both the user and the agent roles; this is a potential source of trajectory bias (e.g., unrealistically consistent gold responses) and should be discussed as a limitation, even if the off-policy evaluation mitigates its direct effect.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the benchmark results are empirical measurements against human-written labels; the chain-product analysis is a statistical-validity concern, not a circular derivation.

full rationale

CONFETTI's central claims are measurements of LLM outputs against independently authored gold function calls and dialog-act labels: no parameter is fitted and then renamed as a prediction, and no uniqueness theorem is imported from prior work. The only self-citation, Gung et al. (2023), is used as methodological context for conversation complexities and does not carry the paper's empirical conclusions; the overlap in authorship is not load-bearing. The auxiliary LLM judges (Claude Haiku for dialog acts, gpt-4o-mini for hallucination) are themselves evaluated against human annotations, and Section 7 explicitly acknowledges the limits of off-policy evaluation and LLM-judge error, which is an honest limitation rather than a hidden circularity. The chain-length result in Section 5.3.1 is the one place where a headline conclusion ('performance on chained function-calls is severely limited') rests on a constructed product of per-step AST soft accuracies conditioned on gold-truth prior actions, with the evaluation prompts instructing models to output only the first function call in a chain. This is not circular in the rubric sense: the per-step factors are empirically measured, not fitted to the conclusion, and the product is transparently defined. However, the product's validity as a chain success rate depends on unstated independence and calibration assumptions, so it is a correctness/validity concern rather than a self-referential derivation. Overall, the paper is self-contained against external benchmarks (e.g., BFCL v2, AlignScore) and does not reduce its predictions to its own inputs; score 1 reflects only the minor, non-load-bearing self-citation.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters or invented entities: CONFETTI is an empirical benchmark, not a theoretical model. The load-bearing premises are evaluation-design assumptions, especially the off-policy approximation and the reliability of LLM judges.

assumptions (4)
  • domain assumption Off-policy turn-level evaluation with gold-truth trajectories approximates online interactive performance.
    The paper's Section 7 explicitly states this does not correspond directly to inference behavior and can cause artificial inflation; all reported scores rely on this premise.
  • domain assumption An LLM-based dialog act classifier provides valid labels for response quality.
    Table 5 shows multi-label precision 67.2% and recall 90.3%; the paper uses at-least-one-match scoring, which inflates reported accuracy.
  • domain assumption Function results are simulated with fixed outputs for each conversation, giving a deterministic environment.
    Section 3.1 states APIs have no explicit implementation and use fixed outputs defined per conversation, removing execution errors from evaluation.
  • ad hoc to paper Human annotators who play both user and agent roles produce unbiased gold trajectories.
    The Data Statement in Section 7 says the same annotator played both roles, which can introduce systematic style or consistency bias into the gold data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CONFETTI: Conversational Function-Calling Evaluation Through Turn-Level Interactions." pith.science (2026). https://pith.science/paper/DFV5LZ4V

@misc{pith2026250601859,
  author       = {Pith},
  title        = {Pith review of: CONFETTI: Conversational Function-Calling Evaluation Through Turn-Level Interactions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DFV5LZ4V}},
  note         = {Machine review of arXiv:2506.01859}
}
read the original abstract

We introduce Conversational Function-Calling Evaluation Through Turn-Level Interactions (CONFETTI), a conversational benchmark1 designed to evaluate the function-calling capabilities and response quality of large language models (LLMs). Current benchmarks lack comprehensive assessment of LLMs in complex conversational scenarios. CONFETTI addresses this gap through 109 human-simulated conversations, comprising 313 user turns and covering 86 APIs. These conversations explicitly target various conversational complexities, such as follow-ups, goal correction and switching, ambiguous and implicit goals. We perform off-policy turn-level evaluation using this benchmark targeting function-calling. Our benchmark also incorporates dialog act annotations to assess agent responses. We evaluate a series of state-of-the-art LLMs and analyze their performance with respect to the number of available APIs, conversation lengths, and chained function calling. Our results reveal that while some models are able to handle long conversations, and leverage more than 20+ APIs successfully, other models struggle with longer context or when increasing the number of APIs. We also report that the performance on chained function-calls is severely limited across the models. Overall, the top performing models on CONFETTI are Nova Pro (40.01%), Claude Sonnet v3.5 (35.46%) and Llama 3.1 405B (33.19%) followed by command-r-plus (31.18%) and Mistral-Large-2407 (30.07%).

Figures

Figures reproduced from arXiv: 2506.01859 by the authors.

Figure 1
Figure 1. CONFETTI comprises multi-turn conversations between a user and an agent. The agent has access to a set of APIs (tools) to assist the user with their request. The agent can invoke actions and receive observations from the environment. Each agent turn is annotated with a dialog act tag used to evaluate response quality. The figure illustrates the flow of a conversation, including user inputs, agent responses, API call… view at source ↗
Figure 2
Figure 2. AST soft accuracy for different models across various number of APIs. The dataset has 144 examples [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. AST soft accuracy for different models across various context lengths. The dataset has 174 examples with [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: AST soft accuracy for different models across chain lengths. The dataset has 220 examples with chain [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Confusion matrix for dialog acts for (a) mistral-large-2407 (b) llama3-1-70b-instruct, (c) claude-sonnet-3- [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. XGrammar-2: Dynamic and Efficient Structured Generation Engine for Agentic LLMs

    cs.AI 2026-01 conditional novelty 6.0 of 10

    XGrammar-2 makes dynamic, tag-triggered structured generation fast by dispatching to cached substructure grammars with an Earley-based adaptive token-mask cache.

Reference graph

Works this paper leans on

29 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    Ali Ahmadvand, Jason Ingyu Choi, and Eugene Agichtein. 2019. https://doi.org/10.1145/3331184.3331375 Contextual dialogue act classification for open-domain conversational agents . In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'19, page 1273–1276, New York, NY, USA. Association for ...

    arXiv 2019

  2. [2]

    Layla El Asri, Jing He, and Kaheer Suleman. 2016. https://api.semanticscholar.org/CorpusID:15491802 A sequence-to-sequence model for user simulation in spoken dialogue systems . In Interspeech

    work page 2016

  3. [3]

    Bender and Batya Friedman

    Emily M. Bender and Batya Friedman. 2018. https://doi.org/10.1162/tacl_a_00041 Data statements for natural language processing: Toward mitigating system bias and enabling better science . Transactions of the Association for Computational Linguistics, 6:587--604

    doi:10.1162/tacl_a_00041 2018

  4. [4]

    Saurabh Chakravarty, Raja Venkata Satya Phanindra Chava, and Edward A. Fox. 2019. https://api.semanticscholar.org/CorpusID:195693905 Dialog acts classification for question-answer corpora . In ASAIL@ICAIL

    work page 2019

  5. [5]

    Nathan Duran, Steve Battle, and Jim Smith. 2023. https://doi.org/10.1017/S1351324921000310 Sentence encoding for dialogue act classification . Natural Language Engineering, 29(3):794–823

    work page doi:10.1017/s1351324921000310 2023

  6. [6]

    Nicholas Farn and Richard Shin. 2023. https://arxiv.org/abs/2311.10775 Tooltalk: Evaluating tool-usage in a conversational setting . Preprint, arXiv:2311.10775

    arXiv 2023

  7. [7]

    James Gung, Emily Moeng, Wesley Rose, Arshit Gupta, Yi Zhang, and Saab Mansour. 2023. https://doi.org/10.18653/v1/2023.findings-acl.613 N at CS : Eliciting natural customer support dialogues . In Findings of the Association for Computational Linguistics: ACL 2023, pages 9652--9677, Toronto, Canada. Association for Computational Linguistics

    doi:10.18653/v1/2023.findings-acl.613 2023

  8. [8]

    Minkyoung Kim and Harksoo Kim. 2018. https://doi.org/10.1109/BigComp.2018.00090 Dialogue act classification model based on deep neural networks for a natural language interface to databases in korean . In 2018 IEEE International Conference on Big Data and Smart Computing (BigComp), pages 537--540

    arXiv 2018

Show all 29 references
  1. [9]

    Chuyi Kong, Yaxin Fan, Xiang Wan, Feng Jiang, and Benyou Wang. 2024. https://doi.org/10.18653/v1/2024.acl-long.424 P lato LM : Teaching LLM s in multi-round dialogue via a user simulator . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisti...

    2024 doi

  2. [10]

    Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. https://arxiv.org/abs/2304.08244 Api-bank: A comprehensive benchmark for tool-augmented llms . Preprint, arXiv:2304.08244

    2023 arXiv

  3. [11]

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2023 a . ht...

    2023 arXiv

  4. [12]

    Yang Liu, Kun Han, Zhao Tan, and Yun Lei. 2017. https://doi.org/10.18653/v1/D17-1231 Using context information for dialog act classification in DNN framework . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2170--2178, Copenhag...

    2017 doi

  5. [13]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 b . https://doi.org/10.18653/v1/2023.emnlp-main.153 G -eval: NLG evaluation using gpt-4 with better human alignment . In Proceedings of the 2023 Conference on Empirical Methods in Natural Langua...

    2023 doi

  6. [14]

    Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. 2024. https://arxiv.org/abs/2408.04682 Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool ...

    2024 arXiv

  7. [15]

    Gonzalez, and Shishir G

    Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Jason Huang, Vishnu Suresh, Yixin Huang, Xiaowen Yu, Joseph E. Gonzalez, and Shishir G. Patil. 2024. Bfcl v3: Multi-turn & multi-step function calling evaluation. gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html

    2024

  8. [16]

    Qian Pan, Zahra Ashktorab, Michael Desmond, Mart \' n Santill \'a n Cooper, James Johnson, Rahul Nair, Elizabeth Daly, and Werner Geyer. 2024. https://doi.org/10.18653/v1/2024.hucllm-1.2 Human-centered design recommendations for LLM -as-a-judge . In Proceedings of the 1st Huma...

    2024 doi

  9. [17]

    Gonzalez

    Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2024. https://openreview.net/forum?id=tBRNC6YemY Gorilla: Large language model connected with massive API s . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  10. [18]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. https://openreview.net/forum?id=dHng2O0Jjr...

    2024

  11. [19]

    Maddison, and Tatsunori Hashimoto

    Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. 2024. https://arxiv.org/abs/2309.15817 Identifying the risks of lm agents with an lm-emulated sandbox . Preprint, arXiv:2309.15817

    2024 arXiv

  12. [20]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. https://openreview.net/forum?id=Yacmpz84TH Toolformer: Language models can teach themselves to use tools . In Thirty-seventh C...

    2023

  13. [21]

    Ivan Sekulić, Silvia Terragni, Victor Guimarães, Nghia Khau, Bruna Guedes, Modestas Filipavicius, André Ferreira Manso, and Roland Mathis. 2024. https://arxiv.org/abs/2402.13374 Reliable llm-based user simulator for task-oriented dialogue systems . Preprint, arXiv:2402.13374

    2024 arXiv

  14. [22]

    Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. https://aclanthology.org/J00-3003 Dialogue act modeling for automatic tagging and recognition of conversatio...

    2000

  15. [23]

    Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. https://doi.org/10.18653/v1/2023.newsum-1.1 Is C hat GPT a good NLG evaluator? a preliminary study . In Proceedings of the 4th New Frontiers in Summarizatio...

    2023 doi

  16. [24]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. https://openreview.net/forum?id=WE_vluYUL-X React: Synergizing reasoning and acting in language models . In The Eleventh International Conference on Learning Representations

    2023

  17. [25]

    Seeun Yoon, Zhankui He, Jessica Maria Echterhoff, and Julian McAuley. 2024. https://arxiv.org/abs/2403.09738 Evaluating large language models as generative user simulators for conversational recommendation . Preprint, arXiv:2403.09738

    2024 arXiv

  18. [26]

    Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. https://arxiv.org/abs/2305.16739 Alignscore: Evaluating factual consistency with a unified alignment function . Preprint, arXiv:2305.16739

    2023 arXiv

  19. [27]

    Xing, Haotong Zhang, Joseph Gonzalez, and Ion Stoica

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph Gonzalez, and Ion Stoica. 2023. https://api.semanticscholar.org/CorpusID:259129398 Judging llm-as-a-judge with mt-bench an...

    2023 arXiv

  20. [28]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  21. [29]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.