Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

A Practical Guide for Evaluating LLMs and LLM-Reliant Systems

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A practical framework ties LLM evaluation to real-world requirements.

desk verdict A clear, well-organized practical synthesis of LLM evaluation practices; no new method and no validation of its central efficacy claim, but still a useful guide for practitioners. read the letter →

arxiv 2506.13023 v2 pith:V4FHGP5V submitted 2025-06-16 cs.AI cs.LG

Ethan M. Rudd , Christopher Andrews , Philip Tully This is my paper
classification cs.AIcs.LG
keywords LLMevaluationframeworkdatasetsmetricsmethodologyhallucinationmeasurementautoraternon-determinism
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that synthetic benchmarks and de-facto metrics like ROUGE and BLEU often mislead when used to evaluate LLM-reliant systems in production. It proposes a structured framework organized around three pillars: datasets curated to be defined in scope, demonstrative of production usage, diverse, decontaminated, and dynamic; a balanced suite of metrics spanning term overlap, semantic similarity, NLI, and LLM autoraters; and a methodology that handles non-determinism, prompt sensitivity, grounding, and hallucination. The aim is to give practitioners an actionable, iterative evaluation design process that aligns with real-world requirements and user-facing needs. A sympathetic reader would care because the framework offers concrete guidance where most guidance is either benchmark-driven or ad hoc.

What carries the argument

The central mechanism is the three-pillar evaluation framework itself, summarized in a flowchart that guides decisions from objectives through datasets, metrics, and methodology. The dataset pillar is operationalized by the 5 D's—Defined Scope, Demonstrative of Production Usage, Diverse, Decontaminated, Dynamic—which translate high-level goals into concrete curation requirements. The metrics pillar is a balanced scorecard: term-overlap metrics (ROUGE for summarization, BLEU for translation), semantic similarity from embeddings, NLI/entailment for factuality, and LLM autoraters for nuanced point-wise or side-by-side judgement, with perplexity acknowledged as a weak proxy. The methodology pillar adds strategies such as repeated sampling to estimate system noise, prompt perturbation testing, grounding ablation, and hallucination and non-response probes. The pillars integrate into an iterative loop where evaluation results are treated as actionable guidance rather than a final grade.

What would settle it

A concrete test: run the framework's full recommended suite on a set of production prompts, obtain human expert ratings on each response, and compute the correlation between the composite metric score and the human ratings. If the suite's scores do not move with human judgments—for instance, if a response judged factually wrong by humans still receives a high composite score—then the framework's core claim that this combination yields meaningful evaluation is false.

Watch

Extended reading notes

Core claim

The central claim is that meaningful evaluation of LLM-reliant systems cannot rely on any single benchmark or metric. The paper proposes an evaluation design framework whose three pillars work together: datasets curated according to five principles (Defined Scope, Demonstrative of Production Usage, Diverse, Decontaminated, Dynamic); metrics chosen as a balanced scorecard that combines term-overlap measures (ROUGE and BLEU), semantic similarity via embeddings, NLI/entailment models, and LLM autoraters; and a methodology that addresses non-determinism through repeated evaluation and self-consistency, prompt sensitivity through perturbation testing, grounding quality through component-level checks, and hallucination and unhelpful non-response rates through targeted probes. Following the framework should produce evaluation results that better reflect real-world system performance and support iterative improvement.

Load-bearing premise

The framework assumes that its recommended combination of metrics and dataset principles actually tracks what human users care about in production; the paper does not test this.

Editorial extensions

If this is right

  • Teams that follow the framework would assemble a scorecard of complementary metrics instead of relying on any single de-facto metric, reducing the chance that a syntactically similar but factually wrong response is scored as good.
  • Curating datasets that are demonstrative, diverse, decontaminated, and dynamic would expose failures such as hallucinations and over-reliance on prior knowledge earlier in the development lifecycle.
  • Explicit handling of non-determinism via repeated evaluation and baselines with error bounds would let engineers distinguish genuine performance regressions from random variation.
  • Separate evaluation of grounding and component-level modules would isolate whether failures originate in retrieval, chained LLM calls, or the final response generation.
  • Measuring hallucination rates through fictitious-entity probes and unhelpful non-response rates with 'I don't know' classifiers would turn two vague failure modes into trackable metrics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the framework would be a production deployment where the recommended metric scorecard is correlated against user satisfaction or expert human ratings; if the scorecard fails to track those judgments, the framework's core premise would need revision.
  • The 5 D's could be developed into quantitative dataset-quality indices (embedding-based diversity scores, contamination overlap ratios) that teams tune alongside model performance.
  • Because autoraters have known biases toward verbosity, position, and style, an extension would be to ensemble multiple independent autoraters or calibrate them against human judges on a small gold set.
  • The framework's iterative loop suggests that evaluation suites themselves be version-controlled and treated as first-class artifacts, an organizational practice the paper gestures at but does not fully develop.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper presents a three-pillar framework for evaluating LLMs and LLM-reliant systems: datasets (the '5 D's' of Defined Scope, Demonstrative of Production Usage, Diverse, Decontaminated, Dynamic), metrics (term-overlap, semantic similarity, NLI/entailment, LLM autoraters, perplexity), and methodology (handling non-determinism, prompt sensitivity, grounding evaluation, hallucination measurement). It offers practical guidance on dataset curation, sample size determination, metric selection, and statistical testing for autorater comparisons. The paper is a guide rather than an empirical study; it does not provide experimental validation of the framework's effectiveness.

Significance. The manuscript is a useful and well-organized synthesis of widely used LLM evaluation practices. Its strengths include the clear structure, the Apollo 11 running example that effectively illustrates the limitations of term-overlap metrics, the correct sample size calculation in Eq. (1), and Table 2's practical mapping of autorater scenarios to statistical tests. The paper also explicitly acknowledges limitations of each metric family, which is commendable. However, the central claim that the framework leads to evaluations that better reflect real-world performance is not empirically demonstrated. There is no case study, no correlation with user satisfaction, and no comparison to a baseline evaluation method. As a result, the framework's actionable value is currently an assumption rather than a demonstrated outcome. If revised to include a validation protocol or a worked example, the paper could serve as a valuable practitioner reference.

major comments (3)
  1. [Section 3.6 and Section 2.2] The paper recommends tracking multiple metrics as a 'balanced scorecard' (Section 3.6) but does not specify how these metrics should be combined, weighted, or reconciled when they disagree (e.g., high NLI entailment with low ROUGE). Section 2.2 suggests correlating offline metrics with user satisfaction surveys but does not explain how to use such a correlation to validate or re-weight the scorecard. Without a concrete aggregation or decision rule, the framework's central claim that it yields evaluations that 'better reflect real-world performance' is untestable and unfalsifiable. Please add a validation procedure, a worked example of scorecard construction, or an explicit statement that the framework is a qualitative checklist rather than a prescriptive method.
  2. [Section 2 and Section 2.2] The 5 D's are introduced as core principles for dataset quality, but two of them are not sufficiently operationalized. 'Defined Scope' (Section 2.2, item 3) is exemplified by modular dataset design, but no procedure is given to define or measure scope adherence. 'Dynamic' (Section 2.2, item 5) is described as regular audit, update, and version control, but no cadence, triggers, or metrics are specified. Since the 5 D's are presented as jointly sufficient for dataset quality, the lack of operational guidance for these two principles weakens the paper's practicality claim. Please provide concrete operational definitions, measurement strategies, or at minimum a rubric for each of the 5 D's.
  3. [Introduction and Section 4] The paper's stated scope includes 'multi-turn agentic workflows' (Introduction, Section 1) and LLM-reliant systems generally, but the methodology pillar (Section 4) focuses on single-turn evaluation topics: non-determinism, prompt sensitivity, grounding ablation, hallucination in Q/A, and non-response rates. No concrete guidance is given for evaluating multi-turn trajectories, tool-use correctness, state tracking, or error propagation across agentic steps. This gap means the framework does not yet address a major class of systems it claims to cover. Please add a subsection on multi-turn/agentic evaluation or explicitly narrow the paper's scope to single-turn systems.
minor comments (7)
  1. [Section 2.1.3] The Evol-Instruct citation is incorrect. The method is from Xu et al. (2023) 'WizardLM: Empowering Large Language Models to Follow Complex Instructions,' not from Bai et al. (2022a), whose bibliography entry lists the author list of the Constitutional AI paper. Please correct the citation and the reference entry.
  2. [Section 3.1.1] The text states that 'ROUGE scores range from 0 to 1.2,' but the footnote refers to scores of 0 or 1 and standard ROUGE scores are typically bounded by 1. This appears to be a typo; please clarify.
  3. [Section 3.4] There is a duplicated word in 'Autoraters can can handle nuance'; please change to 'can handle.'
  4. [Section 4.4] The word 'amd' should be 'and' in the wrap-up paragraph.
  5. [Section 2.2] There is a duplicated '(e.g.,' in 'grouping similar prompts (e.g., (e.g., using LSH clustering (Indyk and Motwani, 1998) ...'; please remove the duplicate.
  6. [Section 4.1] A stray quotation mark appears at the end of the paragraph about non-determinism; please remove it.
  7. [Section 2.3] The sample size formula in Eq. (1) is for a binomial proportion. For non-binary metrics such as ROUGE or NLI scores, the variance-based formula may need adjustment or the metric should be dichotomized; please note this caveat when presenting the formula.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a practical guidance framework with no derivation that reduces to its own inputs, and its only self-citation is illustrative, not load-bearing.

full rationale

The paper proposes a three-pillar evaluation framework (datasets, metrics, methodology) and does not claim to derive any predictive result from fitted parameters. The only mathematical relation, the sample-size formula n = z^2 * m_hat * (1 - m_hat) / epsilon^2 in Section 2.3, is a standard statistical calculation explicitly cited to Singh and Masuku (2014); it is not derived from the paper's own framework and is not used to justify the framework's recommendations. The sole self-citation, Rudd et al. (2024), appears in Section 2.2 as one example of grouping similar prompts using embeddings to measure diversity; it supports an illustrative suggestion rather than the central claim. The paper's recommendations are grounded in external references for ROUGE, BLEU, NLI, autoraters, perplexity, self-consistency, and robustness testing. The skeptic's concern that the balanced scorecard lacks an aggregation or validation rule is a correctness or completeness limitation, not circularity: the framework's efficacy is untested, but no step in the paper equates a prediction to an input by construction. Accordingly, the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters or invented entities. The framework relies on unproven domain assumptions, listed above.

assumptions (4)
  • domain assumption The sample size formula (Eq. 1) yields roughly correct evaluation sizes when applied to LLM metrics.
    Section 2.3 applies a standard proportion-based sample size formula to LLM evaluation metrics, which may not be binomially distributed or independent across samples.
  • domain assumption Embedding models and NLI models used for semantic similarity and entailment scores capture the semantics relevant to the evaluation task.
    Sections 3.2 and 3.3 rely on the validity of these models without benchmarks, error analysis, or domain calibration.
  • domain assumption LLM autoraters provide reliable judgments of response quality when given rubrics or grounding context.
    Section 3.4 acknowledges biases (verbosity, position, self-enhancement) but still recommends autoraters as a core metric component without validating agreement with human raters.
  • ad hoc to paper The 5 D's (Defined Scope, Demonstrative of Production Usage, Diverse, Decontaminated, Dynamic) are jointly sufficient to ensure a high-quality evaluation dataset.
    This is the paper's own framework; no evidence is provided that meeting the 5 D's guarantees a dataset that supports meaningful evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Practical Guide for Evaluating LLMs and LLM-Reliant Systems." pith.science (2026). https://pith.science/paper/V4FHGP5V

@misc{pith2026250613023,
  author       = {Pith},
  title        = {Pith review of: A Practical Guide for Evaluating LLMs and LLM-Reliant Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V4FHGP5V}},
  note         = {Machine review of arXiv:2506.13023}
}
read the original abstract

Recent advances in generative AI have led to remarkable interest in using systems that rely on large language models (LLMs) for practical applications. However, meaningful evaluation of these systems in real-world scenarios comes with a distinct set of challenges, which are not well-addressed by synthetic benchmarks and de-facto metrics that are often seen in the literature. We present a practical evaluation framework which outlines how to proactively curate representative datasets, select meaningful evaluation metrics, and employ meaningful evaluation methodologies that integrate well with practical development and deployment of LLM-reliant systems that must adhere to real-world requirements and meet user-facing needs.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SAGE: Scalable AI Governance & Evaluation

    cs.IR 2026-02 conditional novelty 6.0 of 10

    SAGE co-evolves a relevance policy, expert-curated precedents, and a distilled LLM judge to grade search relevance at production scale, reporting 0.72–0.73 linear Cohen's kappa against humans and a 0.25% DAU lift at LinkedIn.

Reference graph

Works this paper leans on

47 extracted references · 27 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Md Tanvirul Alam, Dipkamal Bhusal, Le Nguyen, and Nidhi Rastogi. 2024. Ctibench: A benchmark for evaluating llms in cyber threat intelligence. arXiv preprint arXiv:2406.07599

    arXiv 2024

  4. [4]

    Amazon Web Services, Inc. 2024. Amazon sagemaker ground truth. https://aws.amazon.com/sagemaker/ground-truth/. Accessed: June 17, 2024

    work page 2024

  5. [5]

    Yizhong Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022 a . Evol-instruct: Instructing language models through evolution. arXiv preprint arXiv:2211.09436

    work page Pith review arXiv 2022

  6. [6]

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022 b . Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073

    arXiv 2022

  7. [7]

    Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65--72

    2005

  8. [8]

    Burton H Bloom. 1970. Space/time trade-offs in hash coding with allowable errors. Communications of the ACM, 13(7):422--426

    1970

Show all 47 references
  1. [9]

    Olivier Bousquet, Nathan Scales, Nathanael Sch \"a rli, and Ilya Tolstikhin. 2024. https://github.com/google-deepmind/onetwo O ne T wo: I nteracting with L arge M odels

    2024

  2. [10]

    Nicholas Carlini, Florian Tram \`e r, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, \'U lfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extra...

    2021

  3. [11]

    Harrison Chase. 2022. Langchain. https://github.com/hwchase17/langchain

    2022

  4. [12]

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90\ quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6

    2023

  5. [13]

    George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the second international conference on Human Language Technology Research, pages 138--145

    2002

  6. [14]

    Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2023. Ragas: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217

    2023 arXiv

  7. [15]

    Haoyu Guo, Tianle Wang, Xin Huang, Yue Ma, Yi Dai, Jianping Zhao, Maosong Sun, and Yang Liu. 2023. Owl: A large language model for it operations. arXiv preprint arXiv:2306.02291

    2023 arXiv

  8. [16]

    Aman Gupta, Anup Shirgaonkar, Angels de Luis Balaguer, Bruno Silva, Daniel Holstein, Dawei Li, Jennifer Marsman, Leonardo O Nunes, Mahsa Rouzbahman, Morris Sharp, et al. 2024. Rag vs fine-tuning: Pipelines, tradeoffs, and a case study on agriculture. arXiv preprint arXiv:2401.08406

    2024 arXiv

  9. [17]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  10. [18]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751

    2019 arXiv

  11. [19]

    Piotr Indyk and Rajeev Motwani. 1998. Approximate nearest neighbors: towards removing the curse of dimensionality. In Proceedings of the thirtieth annual ACM symposium on Theory of computing, pages 604--613

    1998

  12. [20]

    Frederick Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker. 1977. Perplexity—a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1):S63--S63

    1977

  13. [21]

    Labelbox, Inc. 2024. Labelbox: The leading data-centric ai platform. https://labelbox.com/. Accessed: June 17, 2024

    2024

  14. [22]

    Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, et al. 2024. Gecko: Versatile text embeddings distilled from large language models. arXiv preprint arXiv:2403.20327

    2024 arXiv

  15. [23]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Proc...

    2020

  16. [24]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

    2004

  17. [25]

    Chin-Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 human language technology conference of the North American chapter of the association for computational linguistics, pages 150--157

    2003

  18. [26]

    Jerry Liu. 2023. Llamaindex. https://github.com/jerryjliu/llama_index

    2023

  19. [27]

    Udi Manber and Gene Myers. 1993. Suffix arrays: a new method for on-line string searches. SIAM Journal on Computing, 22(5):935--948

    1993

  20. [28]

    Wes McKinney et al. 2011. pandas: a foundational python library for data analysis and statistics. Python for high performance and scientific computing, 14(9):1--9

    2011

  21. [29]

    Tomas Mikolov. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781

    2013 arXiv

  22. [30]

    Ines Montani and Matthew Honnibal. 2018. https://prodi.gy Prodigy: A modern annotation tool for creating training and evaluation data for machine learning models

    2018

  23. [31]

    Hiroki Nakayama, Takahiro Kubo, Junya Kamura, Yasufumi Taniguchi, and Xu Liang. 2018. https://github.com/doccano/doccano doccano : Text annotation tool for human . Software available from https://github.com/doccano/doccano

    2018

  24. [32]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

    2002

  25. [33]

    Matt Post. 2018. A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771

    2018 arXiv

  26. [34]

    James Pustejovsky, Manik Arora, Chitta Baral Lee, et al. 2023. Towards building a comprehensive cybersecurity knowledge base. In Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI-23)

    2023

  27. [35]

    Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. https://www.aclweb.org/anthology/2020.acl-main.442 Beyond accuracy: Behavioral testing of NLP models with C heck L ist . In Proceedings of the 58th Annual Meeting of the Association for Computational ...

    2020

  28. [36]

    Leonard Richardson. 2024. Beautiful soup. https://www.crummy.com/software/BeautifulSoup/. Accessed: June 2, 2025

    2024

  29. [37]

    Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at trec-3. Nist Special Publication Sp, 109:109

    1995

  30. [38]

    Ethan M Rudd, David Krisiloff, Scott Coull, Daniel Olszewski, Edward Raff, and James Holt. 2024. Efficient malware analysis using metric embeddings. Digital Threats: Research and Practice, 5(1):1--20

    2024

  31. [39]

    Gerard Salton, Anita Wong, and Chung-Shu Yang. 1975. A vector space model for automatic indexing. Communications of the ACM, 18(11):613--620

    1975

  32. [40]

    Scrapy Developers . 2024. Scrapy: A fast and powerful web crawling & scraping framework. https://scrapy.org/. Accessed: June 2, 2025

    2024

  33. [41]

    Selenium Project . 2024. Selenium webdriver. https://www.selenium.dev/. Accessed: June 2, 2025

    2024

  34. [42]

    Ajay S Singh and Micah B Masuku. 2014. Sampling techniques & determination of sample size in applied statistics research: An overview. International Journal of economics, commerce and management, 2(11):1--22

    2014

  35. [43]

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03...

    2023

  36. [44]

    Xiao Wang, Qin Liu, Tao Gui, Qi Zhang, Yicheng Zou, Xin Zhou, Jiacheng Ye, Yongxin Zhang, Rui Zheng, Zexiong Pang, et al. 2021. Textflint: Unified multilingual robustness evaluation toolkit for natural language processing. In Proceedings of the 59th Annual Meeting of the Assoc...

    2021

  37. [45]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

    2022 arXiv

  38. [46]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems

    2022

  39. [47]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, ...

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.