Pith. sign in

REVIEW 5 major objections 5 minor 3 cited by

Enterprise Large Language Model Evaluation Benchmark

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a 14-task, Bloom's Taxonomy-based benchmark reveals enterprise-specific LLM gaps that more general benchmarks miss.

desk verdict Useful enterprise benchmark blueprint, but GPT-4o serving as labeler, judge, and contestant plus no released data makes the rankings unreliable as-is. read the letter →

arxiv 2506.20274 v1 pith:NDZ3YSHT submitted 2025-06-25 cs.AI

Liya Wang , David Yi , Damien Jose , John Passarelli , James Gao , Jordan Leventis , Kang Li This is my paper
classification cs.AI
keywords LargeLanguageModels(LLMs)EvaluationBenchmarkBloom'sTaxonomyLLM-as-a-LabelerLLM-as-a-JudgeCorrectiveRetrieval-AugmentedGeneration(CRAG)enterpriseapplicationsmodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a 14-task benchmark for evaluating LLMs in enterprise settings, organized by Bloom's six cognitive levels from Remember to Create. It argues that general-purpose benchmarks such as MMLU do not capture the proprietary knowledge and judgment demands of real enterprise work, and that its roughly 9,700-sample dataset, built with LLM-generated labels, corrective retrieval-augmented generation, and G-Eval judging, exposes those gaps. Evaluating six models, it reports that open-source DeepSeek R1 is competitive with proprietary models on reasoning tasks but lags on judgment-based ones, which it attributes to overthinking. If the benchmark is sound, it gives enterprises a reusable instrument for model selection and a blueprint for tailoring evaluations to their own data.

What carries the argument

The central machinery is the task taxonomy paired with a data curation pipeline. The taxonomy maps 14 tasks onto six cognitive levels — Remember, Understand, Apply, Analyze, Evaluate, Create — each with a specific metric, such as G-Eval correctness for question-answering, exact match for named entity recognition, and Spearman's r for the judge-alignment task. The pipeline combines LLM-as-a-Labeler (GPT-4o) with corrective retrieval-augmented generation to reduce hallucination during annotation, then uses LLM-as-a-Judge through G-Eval in the DeepEval package to score outputs, with human review applied only to low-confidence labels. This machinery is what allows the benchmark to reach about 9,700 samples while claiming quality, and it also underpins the overthinking explanation, since reasoning models receive extra tokens for chain-of-thought yet still score lower on judgment tasks.

What would settle it

Re-label a random sample of the 9,700 benchmark items entirely by human experts and recompute every model's score on that sample; if the rankings diverge from the GPT-4o-judged rankings, the reported comparisons collapse. A cheaper first check is to rerun the G-Eval scoring with labels and judge both produced by a different model family.

Watch

Extended reading notes

Core claim

The central claim is that an enterprise evaluation benchmark grounded in Bloom's Taxonomy can differentiate LLM capabilities in ways existing benchmarks do not. The paper reports that all six evaluated models score below 0.30 in G-Eval correctness on acronym memorization and factual question-answering over internal Atlassian data, indicating a systematic lack of proprietary enterprise knowledge; that open-source models win eight, tie one, and lose five comparisons against proprietary models overall; and that DeepSeek R1 leads in summarization and content generation while GPT-4o achieves the highest Spearman's r (0.47) on the LLM-as-a-Judge task, with reasoning models lagging there, likely due to overthinking. These results are presented as evidence that the benchmark surfaces actionable performance gaps and offers a practical blueprint for enterprise LLM evaluation and post-training decisions.

Load-bearing premise

The load-bearing premise is that the labels produced by GPT-4o and the G-Eval scores it computes are accurate enough to serve as ground truth, even though GPT-4o is itself one of the evaluated models and only low-confidence labels receive human review.

Editorial extensions

If this is right

  • Enterprises should base model selection on task-level scores rather than overall averages, since no single model leads across all 14 tasks and the paper recommends against further post-training of Llama-3.2-3B-Instruct, the weakest performer.
  • Proprietary enterprise knowledge is a core bottleneck: all models score below 0.30 on acronym and factual question-answering tasks, supporting continuous pre-training on company-specific data.
  • The overthinking hypothesis implies that giving reasoning models more chain-of-thought tokens can hurt judgment-based performance, so deployments may need to constrain or calibrate reasoning budgets.
  • Open-source models are competitive enough on several tasks that enterprises can reduce reliance on proprietary APIs for those workloads, which the paper frames as a cost-saving opportunity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of judge bias would rerun the benchmark with labels and G-Eval scores produced by a non-GPT-4o judge; if GPT-4o's relative ranking drops, part of the reported judgment gap is an artifact of self-preference.
  • The paper validates only low-confidence labels by human review, so the accuracy of the majority of GPT-4o-generated labels remains unverified; a random fully human-labeled sample would quantify label reliability across all tasks.
  • Because the data come from a single company, the benchmark's transferability to other enterprises is open; porting the task templates to another organization's documents would test whether the Bloom's Taxonomy structure generalizes.
  • The overthinking explanation can be tested more directly by running DeepSeek R1 on the Evaluate-level task with chain-of-thought disabled or token-limited and checking whether its Spearman's r increases.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a 14-task enterprise LLM benchmark organized by Bloom's Taxonomy, drawing on roughly 9,700 samples from Atlassian Confluence, Rovo chat, customer feedback, Slack queries, and developer documentation. Labels for most tasks are generated by GPT-4o via an LLM-as-a-Labeler pipeline with CRAG, and evaluation uses G-Eval with GPT-4o as judge. Six models are evaluated: Llama 3.2 3B, Llama 3.3 70B, Llama 4 Scout, DeepSeek R1, DeepSeek Distilled Llama 3.3 70B, and GPT-4o-2024-11-20. The central claims are that the benchmark is robust, that open-source models rival proprietary ones in reasoning tasks, and that DeepSeek R1 lags in judgment-based scenarios, attributed to overthinking.

Significance. If the benchmark and its results were valid, the paper would provide a reusable enterprise evaluation instrument and concrete guidance for model selection, which is a practically important contribution. The task taxonomy based on Bloom's Taxonomy, the use of real enterprise data, and the incorporation of CRAG for labeling are commendable design choices. However, the current evidence does not support the robustness claim because the measurement and the measured subject are the same system: GPT-4o generates most ground truth, serves as the judge, and is also one of the evaluated models. The paper also reports no dataset release, no error bars, and no human validation for the majority of labels. These limitations directly undermine the trustworthiness of the reported rankings and the 'overthinking' explanation, so the significance is conditional on major rework.

major comments (5)
  1. [§3.2, §3.3, §4.1] The evaluation loop is circular: GPT-4o is used as the LLM-as-a-Labeler for most tasks (Section 3.2), as the judge in G-Eval for correctness, relevance, and coherence (Section 3.3), and it is also one of the evaluated models (Section 4.1). The paper acknowledges in Section 2.3 that LLM judges exhibit self-preference and positional bias, but reports no mitigation such as blinding the model identity or swapping response order. Consequently, GPT-4o's higher scores on open-ended tasks (1-1, 1-2, 2-5, 3-1, 3-2, 3-3, 6-1) may reflect agreement with its own labels and judging criteria rather than true capability. This confounds every cross-model comparison, so the reported rankings are not trustworthy as evidence for the paper's conclusions.
  2. [§3.2 Human Validation] The paper states that 'human experts reviewed a subset of the annotated data that received low confidence scores,' but provides no details about the size of this subset, the selection criteria, or inter-annotator agreement. Because the majority of labels were never human-checked, the claim of a 'robust 9,700-sample benchmark' is unsupported. The authors should report the proportion of labels that received low confidence, the human correction rate, and ideally a random sample of high-confidence labels to verify that the LLM-generated ground truth is accurate beyond the low-confidence tail.
  3. [§3.3 and Table 2] G-Eval scores are reported without any validation against human judgments for these specific tasks. Citing the general G-Eval paper (ref [63]) is not sufficient; the authors need to show that GPT-4o's relevance, coherence, and correctness scores correlate with human ratings on a sample of the benchmark data. Without such calibration, differences like 0.88 vs. 0.87 in relevance or 0.97 vs. 0.91 in coherence in Table 2 cannot be interpreted as meaningful performance gaps, and the derived conclusions about model superiority are not supported.
  4. [§3.1 Sample Size Claim] The paper claims that 'a sample size of approximately 600 provides stable evaluation results,' but no ablation study, confidence intervals, or error bars are presented anywhere. This claim is load-bearing because the benchmark's robustness argument rests on it, and two tasks (3-4 with 218 samples and 5-1 with 265 samples) fall well below that threshold. The authors should provide the ablation data, report variance estimates, and justify why the smaller manual-label tasks still yield stable comparisons.
  5. [§4.3 and Abstract] The explanation that DeepSeek R1 lags in judgment-based scenarios 'likely due to overthinking' is speculative and unsupported by the reported data. No analysis of reasoning token counts, no comparison of R1 with its non-reasoning counterpart on task 5-1, and no ablation are provided; the cited reference [93] concerns agentic tasks and does not directly support this conclusion. The claim should be either supported with direct evidence or removed from the abstract and conclusions.
minor comments (5)
  1. [§3.2] In the Data Curation section, 'similarly in Task 4-1 ("NL2JQL")' should reference Task 3-4, since NL2JQL is listed as 3-4 in Table 1.
  2. [§4.3] The text says 'seven popular LLMs' but only six models are listed in Section 4.1 and Table 2; the count should be corrected.
  3. [Table 1] For tasks 3-1 and 6-1, the labeling method is marked '-', and the text later says these tasks do not require ground truth; please clarify in the table caption or in Section 3.2 that 'no ground truth' is intended.
  4. [General] The paper does not include a data availability statement or a link to the benchmark dataset, which is a significant omission for a benchmark paper aiming to provide a 'blueprint for enterprises'; consider adding a repository or an explicit statement about proprietary data restrictions.
  5. [§4.2] The experimental setup reports temperature and top-p but does not specify the exact prompt templates or the G-Eval rubric details; providing these in an appendix or supplementary material would aid reproducibility.

Circularity Check

2 steps flagged · score 7.0 of 10

GPT-4o is the labeler, the judge, and one of the evaluated models, so the benchmark's accuracy and G-Eval scores partly measure GPT-4o's agreement with itself rather than independent capability.

  1. self definitional [Section 3.2, Data Curation; Table 1; Section 3.3]
    "LLM-as-a-Labeler: As manual labeling is costly, time-consuming, and non-scalable, we utilize GPT-4o [71] to assist with data annotation, thereby streamlining the process. ... Human Validation: To ensure quality, human experts reviewed a subset of the annotated data that received low confidence scores from previous stage."

    Table 1 assigns "LLM" as the labeling method for tasks 1-1, 1-2, 2-1, 2-2, 2-3, 2-4, 2-5, 3-2, 3-3, and 4-1, and this section identifies that LLM as GPT-4o. Accuracy and correctness for those tasks are therefore computed against GPT-4o-generated ground truth, so GPT-4o's own high scores (e.g., 97.2% on sentiment analysis) are agreement with its own labels. Human review covered only a subset of the annotated data that received low confidence scores, leaving the majority of labels unverified. Consequently, the GPT-4o rankings on those tasks are a self-consistency measurement, not an independent evaluation.

  2. self definitional [Section 4.1, Model; Section 3.3, Evaluation Metrics Calculation]
    "In addition, as for the evaluation metrics calculation stage, G-Eval uses the GPT-4o-2024-11-20 version."

    GPT-4o-2024-11-20 is also one of the six models being ranked in Section 4.1. The G-Eval metrics in Table 2 (correctness, relevance, coherence) are produced by this same model for all contestants, including itself. The paper cites known LLM-judge limitations and biases (refs [6], [58], [61]) but reports no blinding of model identity, no response-order swapping, and no human validation of the G-Eval scores. GPT-4o's scores therefore can encode self-preference, so cross-model comparisons on open-ended and correctness tasks are not established as independent measurements.

full rationale

The benchmark is not circular in every part: Task 3-4 and Task 5-1 use manually collected labels, Task 2-1 and 2-2 use reference-less toxicity and bias metrics, and the reported content-generation ranking actually favors DeepSeek R1 over GPT-4o. However, the central claim of a "robust 9,700-sample benchmark" and the headline comparisons (open-source models rival proprietary ones in reasoning but lag in judgment-based scenarios) rest substantially on GPT-4o-generated labels and GPT-4o-as-judge scores, with GPT-4o itself one of the evaluated systems. That is a by-construction self-measurement loop for the affected tasks: the ground truth is GPT-4o's output, and the scores are GPT-4o's judgments, so the affected rankings reduce to self-consistency rather than external quality. The paper provides no independent human rescoring of a representative sample and no mitigation of known LLM-judge self-preference. No self-citation chain is load-bearing, so the circularity is methodological rather than citation-based. Score 7 reflects partial circularity: some tasks are independently grounded, but several central results are forced by the identity of labeler, judge, and evaluated model.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central results rest on four unvalidated assumptions: that Bloom's Taxonomy is a meaningful organizer for LLM evaluation, that GPT-4o-generated labels are correct ground truth for most tasks, that GPT-4o-as-judge produces reliable scores, and that the internal Atlassian data sources represent enterprise tasks generally. Because the labeling and judging models are the same model family as one of the evaluated systems (GPT-4o), the benchmark has a strong self-referential component. No free parameters are fitted in the traditional sense, but the 'approximately 600 samples per task' is a hand-chosen design decision with no reported ablation details.

assumptions (5)
  • domain assumption Bloom's Taxonomy is a meaningful framework for organizing enterprise LLM tasks.
    Section 3.1: the paper maps tasks to Bloom's levels, but no evidence is given that this pedagogy-inspired hierarchy corresponds to a useful difficulty or capability ordering for LLMs.
  • domain assumption LLM-as-a-Labeler with GPT-4o produces accurate ground truth labels.
    Section 3.2: GPT-4o generates labels for most tasks; only low-confidence samples receive human review, so the accuracy of the majority of labels is unverified.
  • domain assumption G-Eval with GPT-4o as judge yields reliable evaluation scores.
    Section 3.3 and 4.2: the paper relies on G-Eval/DeepEval scores computed by GPT-4o without reporting alignment with human judgments on this specific benchmark.
  • domain assumption The internal Atlassian data sources (Confluence, Rovo, customer feedback, Slack) are representative of general enterprise tasks.
    Section 3.2: the data are from one company's internal systems, so generalizability to other enterprises is assumed rather than demonstrated.
  • ad hoc to paper A sample size of approximately 600 per task gives stable evaluation results.
    Section 3.1: the paper states 'our findings suggest' without presenting the sample size ablation study, so this is a hand-chosen design decision.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enterprise Large Language Model Evaluation Benchmark." pith.science (2026). https://pith.science/paper/NDZ3YSHT

@misc{pith2026250620274,
  author       = {Pith},
  title        = {Pith review of: Enterprise Large Language Model Evaluation Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NDZ3YSHT}},
  note         = {Machine review of arXiv:2506.20274}
}
read the original abstract

Large Language Models (LLMs) ) have demonstrated promise in boosting productivity across AI-powered tools, yet existing benchmarks like Massive Multitask Language Understanding (MMLU) inadequately assess enterprise-specific task complexities. We propose a 14-task framework grounded in Bloom's Taxonomy to holistically evaluate LLM capabilities in enterprise contexts. To address challenges of noisy data and costly annotation, we develop a scalable pipeline combining LLM-as-a-Labeler, LLM-as-a-Judge, and corrective retrieval-augmented generation (CRAG), curating a robust 9,700-sample benchmark. Evaluation of six leading models shows open-source contenders like DeepSeek R1 rival proprietary models in reasoning tasks but lag in judgment-based scenarios, likely due to overthinking. Our benchmark reveals critical enterprise performance gaps and offers actionable insights for model optimization. This work provides enterprises a blueprint for tailored evaluations and advances practical LLM deployment.

Figures

Figures reproduced from arXiv: 2506.20274 by the authors.

Figure 1
Figure 1. Evaluation pipeline [65] 3.1. Evaluation Tasks To evaluate LLMs effectively, it is crucial to assemble a diverse set of tasks. Our extensive understanding of enterprise use cases has allowed us to categorize them into 14 distinct tasks, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Bloom’s Taxonomy [67] [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Data curation pipeline [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Different methods for metric calculation [73] [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: LLMs performance comparison on our selected tasks [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: CRAG process [94] G-Eval As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: The framework of G-Eval [79] [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Frontier AI performance across the business disciplines: a case-grounded benchmark of knowledge work and analytical reasoning

    cs.CL 2026-07 conditional novelty 7.0 of 10

    On a new 615-question business-case benchmark graded by AI against instructor rubrics, frontier LLMs score 87-88% partial credit but complete only about half the questions.

  2. EvalLoop: A Methodology for Evaluation-Driven Iterative Improvement of Business AI Systems

    cs.SE 2026-07 conditional novelty 6.0 of 10

    EvalLoop improves business LLM systems by grouping metrics into dimensions, classifying failure modes, and iterating one system variable at a time, raising a sales briefing model from 82.6% to 94.6%.

  3. Fast and Accurate Contextual Knowledge Extraction Using Cascading Language Model Chains and Candidate Answers

    cs.CL 2025-07 conditional novelty 4.0 of 10

    Cascading cheap and expensive language models, with answers validated against regex-extracted candidate dates, improved speed and modestly improved accuracy when extracting dates of birth from medical documents.

Reference graph

Works this paper leans on

95 extracted references · 13 canonical work pages · cited by 3 Pith papers

  1. [63]

    G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment,

    Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu, “G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment,” May 23, 2023, arXiv: arXiv:2303.16634. Accessed: Oct. 15, 2024. [Online]. Available: http://arxiv.org/abs/2303.16634

    arXiv 2023

  2. [93]

    The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Tasks,

    A. Cuadron et al., “The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Tasks,” Feb. 12, 2025, arXiv: arXiv:2502.08235. doi: 10.48550/arXiv.2502.08235

    arXiv doi:10.48550/arxiv.2502.08235 2025

  3. [1]

    AutoBencher: Creating Salient, Novel, Difficult Datasets for LanguageModels,

    X. Li, Z. Liu, and T. Hashimoto, “AutoBencher: Creating Salient, Novel, Difficult Datasets for LanguageModels,”Jul.11,2024,arXiv:arXiv:2407.08351.Accessed:Oct.04,2024.[Online].Available: http://arxiv.org/abs/2407.08351

    arXiv 2024

  4. [2]

    Measuring Massive Multitask Language Understanding,

    D. Hendrycks et al., “Measuring Massive Multitask Language Understanding,” Jan. 12, 2021, arXiv: arXiv:2009.03300. doi: 10.48550/arXiv.2009.03300

    arXiv doi:10.48550/arxiv.2009.03300 2021

  5. [3]

    Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models,

    A. Srivastava et al., “Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models,” Jun. 12, 2023, arXiv: arXiv:2206.04615. doi: 10.48550/arXiv.2206.04615

    arXiv doi:10.48550/arxiv.2206.04615 2023

  6. [4]

    On the Measure of Intelligence,

    F. Chollet, “On the Measure of Intelligence,” Nov. 25, 2019, arXiv: arXiv:1911.01547. doi: 10.48550/arXiv.1911.01547

    arXiv doi:10.48550/arxiv.1911.01547 2019

  7. [5]

    Holistic Evaluation of Language Models,

    P. Liang et al., “Holistic Evaluation of Language Models,” Oct. 01, 2023, arXiv: arXiv:2211.09110. Accessed: Oct. 06, 2024. [Online]. Available: http://arxiv.org/abs/2211.09110

    arXiv 2023

  8. [6]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,

    L. Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” Dec. 24, 2023, arXiv: arXiv:2306.05685. Accessed: Oct. 16, 2024. [Online]. Available: http://arxiv.org/abs/2306.05685

    arXiv 2023

Show all 95 references
  1. [7]

    What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams,

    D. Jin, E. Pan, N. Oufattole, W.-H. Weng, H. Fang, and P. Szolovits, “What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams,” Sep. 28, 2020, arXiv: arXiv:2009.13081. doi: 10.48550/arXiv.2009.13081

    2020 doi arXiv

  2. [8]

    MedQA-CS: Benchmarking Large Language Models Clinical Skills Using an AI- SCE Framework,

    Z. Yao et al., “MedQA-CS: Benchmarking Large Language Models Clinical Skills Using an AI- SCE Framework,” Oct. 02, 2024, arXiv: arXiv:2410.01553. doi: 10.48550/arXiv.2410.01553

    2024 doi

  3. [9]

    MedMCQA :ALarge-scaleMulti-SubjectMulti- Choice Dataset for Medical domain Question Answering,

    A.Pal,L.K.Umapathi,andM.Sankarasubbu,“MedMCQA :ALarge-scaleMulti-SubjectMulti- Choice Dataset for Medical domain Question Answering,” Mar. 27, 2022, arXiv: arXiv:2203.14371. doi: 10.48550/arXiv.2203.14371

    2022 doi arXiv

  4. [10]

    WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation,

    J. Matos et al., “WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation,” Oct. 16, 2024, arXiv: arXiv:2410.12722. doi: 10.48550/arXiv.2410.12722

    2024 doi arXiv

  5. [11]

    PubMedQA:ADatasetforBiomedicalResearch Question Answering,

    Q.Jin,B.Dhingra,Z.Liu,W.Cohen,andX.Lu,“PubMedQA:ADatasetforBiomedicalResearch Question Answering,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processingandthe9thInternationalJointConferenceonNaturalLanguageProcessing(EMNLP-IJCNLP), K. Inui, ...

    2019 doi

  6. [12]

    CMB:AComprehensiveMedicalBenchmarkinChinese,

    X.Wangetal.,“CMB:AComprehensiveMedicalBenchmarkinChinese,”Apr.04,2024,arXiv: arXiv:2308.08833. doi: 10.48550/arXiv.2308.08833

    2024 doi arXiv

  7. [13]

    The FinBen: An Holistic Financial Benchmark for Large Language Models,

    Q. Xie et al., “The FinBen: An Holistic Financial Benchmark for Large Language Models,” Feb. 20, 2024, arXiv: arXiv:2402.12659. doi: 10.48550/arXiv.2402.12659

    2024 doi arXiv

  8. [14]

    FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models,

    X. Guo et al., “FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models,” Dec. 08, 2024, arXiv: arXiv:2308.09975. doi: 10.48550/arXiv.2308.09975. 14

    2024 doi arXiv

  9. [15]

    WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domain,

    R. S. Shah et al., “WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domain,” Oct. 31, 2022, arXiv: arXiv:2211.00083. doi: 10.48550/arXiv.2211.00083

    2022 doi arXiv

  10. [16]

    BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark,

    D. Lu et al., “BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark,” Feb. 26, 2023, arXiv: arXiv:2302.09432. doi: 10.48550/arXiv.2302.09432

    2023 doi arXiv

  11. [17]

    XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters,

    X. Zhang, Q. Yang, and D. Xu, “XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters,” May 19, 2023, arXiv: arXiv:2305.12002. doi: 10.48550/arXiv.2305.12002

    2023 doi arXiv

  12. [18]

    PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance,

    Q. Xie et al., “PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance,” Jun. 08, 2023, arXiv: arXiv:2306.05443. doi: 10.48550/arXiv.2306.05443

    2023 doi arXiv

  13. [19]

    LexGLUE: A Benchmark Dataset for Legal Language Understanding in English,

    I. Chalkidis et al., “LexGLUE: A Benchmark Dataset for Legal Language Understanding in English,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), S. Muresan, P. Nakov, and A. Villavicencio, Eds., Dublin, Irelan...

    2022 doi

  14. [20]

    A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Prediction,

    W. Hwang, D. Lee, K. Cho, H. Lee, and M. Seo, “A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Prediction,” Oct. 05, 2022, arXiv: arXiv:2206.05224. doi: 10.48550/arXiv.2206.05224

    2022 doi arXiv

  15. [21]

    LegalBench:ACollaborativelyBuiltBenchmarkforMeasuringLegalReasoning in Large Language Models,

    N.Guhaetal.,“LegalBench:ACollaborativelyBuiltBenchmarkforMeasuringLegalReasoning in Large Language Models,” Aug. 20, 2023, arXiv: arXiv:2308.11462. doi: 10.48550/arXiv.2308.11462

    2023 doi arXiv

  16. [22]

    LawBench: Benchmarking Legal Knowledge of Large Language Models,

    Z. Fei et al., “LawBench: Benchmarking Legal Knowledge of Large Language Models,” Sep. 28, 2023, arXiv: arXiv:2309.16289. Accessed: Oct. 20, 2024. [Online]. Available: http://arxiv.org/abs/2309.16289

    2023 arXiv

  17. [23]

    CMMLU: Measuring massive multitask language understanding in Chinese,

    H. Li et al., “CMMLU: Measuring massive multitask language understanding in Chinese,” Jan. 17, 2024, arXiv: arXiv:2306.09212. doi: 10.48550/arXiv.2306.09212

    2024 doi arXiv

  18. [24]

    Evaluating the Performance of Large Language Models on GAOKAO Benchmark,

    X. Zhang, C. Li, Y. Zong, Z. Ying, L. He, and X. Qiu, “Evaluating the Performance of Large Language Models on GAOKAO Benchmark,” Feb. 24, 2024, arXiv: arXiv:2305.12474. doi: 10.48550/arXiv.2305.12474

    2024 doi arXiv

  19. [25]

    C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models,

    Y. Huang et al., “C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models,” Nov. 06, 2023, arXiv: arXiv:2305.08322. doi: 10.48550/arXiv.2305.08322

    2023 doi arXiv

  20. [26]

    Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?,

    O. Ghahroodi et al., “Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?,” 2024

    2024

  21. [27]

    M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models,

    W. Zhang, S. M. Aljunied, C. Gao, Y. K. Chia, and L. Bing, “M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models,” Nov. 10, 2023, arXiv: arXiv:2306.05179. doi: 10.48550/arXiv.2306.05179

    2023 doi arXiv

  22. [28]

    AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models,

    W. Zhong et al., “AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models,” Sep. 18, 2023, arXiv: arXiv:2304.06364. doi: 10.48550/arXiv.2304.06364

    2023 doi arXiv

  23. [29]

    Large Language Models for Data Annotation and Synthesis: A Survey,

    Z. Tan et al., “Large Language Models for Data Annotation and Synthesis: A Survey,” Dec. 02, 2024, arXiv: arXiv:2402.13446. doi: 10.48550/arXiv.2402.13446

    2024 doi arXiv

  24. [30]

    On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey,

    L. Long et al., “On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey,” Jun. 14, 2024, arXiv: arXiv:2406.15126. doi: 10.48550/arXiv.2406.15126

    2024 doi arXiv

  25. [31]

    Comprehensive Exploration of Synthetic Data Generation: A Survey,

    A. Bauer et al., “Comprehensive Exploration of Synthetic Data Generation: A Survey,” Feb. 01, 2024, arXiv: arXiv:2401.02524. doi: 10.48550/arXiv.2401.02524

    2024 doi arXiv

  26. [32]

    Best Practices and Lessons Learned on Synthetic Data,

    R. Liu et al., “Best Practices and Lessons Learned on Synthetic Data,” Aug. 10, 2024, arXiv: arXiv:2404.07503. doi: 10.48550/arXiv.2404.07503

    2024 doi arXiv

  27. [33]

    Self-Consistency Improves Chain of Thought Reasoning in Language Models,

    X. Wang et al., “Self-Consistency Improves Chain of Thought Reasoning in Language Models,” Mar. 07, 2023, arXiv: arXiv:2203.11171. Accessed: Oct. 29, 2024. [Online]. Available: http://arxiv.org/abs/2203.11171 15

    2023 arXiv

  28. [34]

    Self-Prompting Large Language Models for Zero-Shot Open-Domain QA,

    J. Li, J. Wang, Z. Zhang, and H. Zhao, “Self-Prompting Large Language Models for Zero-Shot Open-Domain QA,” Mar. 28, 2024, arXiv: arXiv:2212.08635. doi: 10.48550/arXiv.2212.08635

    2024 doi arXiv

  29. [35]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning,

    DeepSeek-AI et al., “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning,” Jan. 22, 2025, arXiv: arXiv:2501.12948. doi: 10.48550/arXiv.2501.12948

    2025 doi arXiv

  30. [36]

    HuggingFace, Open-R1. (2025). [Online]. Available: https://github.com/huggingface/open-r1

    2025

  31. [37]

    I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self- Enhancement Paradigm,

    Y. Liang et al., “I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self- Enhancement Paradigm,” Aug. 27, 2024, arXiv: arXiv:2408.08072. doi: 10.48550/arXiv.2408.08072

    2024 doi arXiv

  32. [38]

    Self-Rewarding Language Models,

    W. Yuan et al., “Self-Rewarding Language Models,” Feb. 08, 2024, arXiv: arXiv:2401.10020. doi: 10.48550/arXiv.2401.10020

    2024 doi arXiv

  33. [39]

    West-of-N: Synthetic Preferences for Self-Improving Reward Models,

    A. Pace, J. Mallinson, E. Malmi, S. Krause, and A. Severyn, “West-of-N: Synthetic Preferences for Self-Improving Reward Models,” Oct. 25, 2024, arXiv: arXiv:2401.12086. doi: 10.48550/arXiv.2401.12086

    2024 doi arXiv

  34. [40]

    Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a- Meta-Judge,

    T. Wu et al., “Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a- Meta-Judge,” Jul. 29, 2024, arXiv: arXiv:2407.19594. Accessed: Sep. 30, 2024. [Online]. Available: http://arxiv.org/abs/2407.19594

    2024 arXiv

  35. [41]

    Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies,

    L. Pan, M. Saxon, W. Xu, D. Nathani, X. Wang, and W. Y. Wang, “Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies,” Aug. 30, 2023, arXiv: arXiv:2308.03188. doi: 10.48550/arXiv.2308.03188

    2023 doi arXiv

  36. [42]

    Leveraging Large Language Models for Multiple Choice Question Answering,

    J. Robinson, C. M. Rytting, and D. Wingate, “Leveraging Large Language Models for Multiple Choice Question Answering,” Mar. 17, 2023, arXiv: arXiv:2210.12353. doi: 10.48550/arXiv.2210.12353

    2023 doi arXiv

  37. [43]

    Teaching Models to Express Their Uncertainty in Words,

    S. Lin, J. Hilton, and O. Evans, “Teaching Models to Express Their Uncertainty in Words,” Jun. 13, 2022, arXiv: arXiv:2205.14334. doi: 10.48550/arXiv.2205.14334

    2022 doi arXiv

  38. [44]

    Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,

    P. Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” in Advances in Neural Information Processing Systems, Curran Associates, Inc., 2020, pp. 9459–9474. Accessed: Feb. 05, 2025. [Online]. Available: https://proceedings.neurips.cc/paper_files/pap...

    2020

  39. [45]

    Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely,

    S. Zhao, Y. Yang, Z. Wang, Z. He, L. K. Qiu, and L. Qiu, “Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely,” Sep. 23, 2024, arXiv: arXiv:2409.14924. doi: 10.48550/arXiv.2409.14924

    2024 doi arXiv

  40. [46]

    MemoRAG: Moving towards Next-Gen RAG Via Memory-Inspired Knowledge Discovery,

    H. Qian, P. Zhang, Z. Liu, K. Mao, and Z. Dou, “MemoRAG: Moving towards Next-Gen RAG Via Memory-Inspired Knowledge Discovery,” Sep. 10, 2024, arXiv: arXiv:2409.05591. doi: 10.48550/arXiv.2409.05591

    2024 doi arXiv

  41. [47]

    Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity,

    S. Jeong, J. Baek, S. Cho, S. J. Hwang, and J. C. Park, “Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity,” Mar. 28, 2024, arXiv: arXiv:2403.14403. doi: 10.48550/arXiv.2403.14403

    2024 doi arXiv

  42. [48]

    Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection,

    A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi, “Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection,” Oct. 17, 2023, arXiv: arXiv:2310.11511. doi: 10.48550/arXiv.2310.11511

    2023 doi arXiv

  43. [49]

    Self-Knowledge Guided Retrieval Augmentation for Large Language Models,

    Y. Wang, P. Li, M. Sun, and Y. Liu, “Self-Knowledge Guided Retrieval Augmentation for Large Language Models,” Oct. 08, 2023, arXiv: arXiv:2310.05002. doi: 10.48550/arXiv.2310.05002

    2023 doi arXiv

  44. [50]

    Making Retrieval-Augmented Language Models Robust to Irrelevant Context,

    O. Yoran, T. Wolfson, O. Ram, and J. Berant, “Making Retrieval-Augmented Language Models Robust to Irrelevant Context,” May 05, 2024, arXiv: arXiv:2310.01558. doi: 10.48550/arXiv.2310.01558

    2024 doi arXiv

  45. [51]

    Search Augmented Instruction Learning,

    H. Luo et al., “Search Augmented Instruction Learning,” in Findings of the Association for Computational Linguistics: EMNLP 2023, H. Bouamor, J. Pino, and K. Bali, Eds., Singapore: Association for Computational Linguistics, Dec. 2023, pp. 3717–3729. doi: 10.18653/v1/2023.findi...

    2023 doi

  46. [52]

    Corrective Retrieval Augmented Generation,

    S.-Q. Yan, J.-C. Gu, Y. Zhu, and Z.-H. Ling, “Corrective Retrieval Augmented Generation,” Oct. 07, 2024, arXiv: arXiv:2401.15884. doi: 10.48550/arXiv.2401.15884. 16

    2024 doi arXiv

  47. [53]

    Constructing Domain-Specific Evaluation Sets for LLM-as-a-judge,

    R. Raju, S. Jain, B. Li, J. Li, and U. Thakker, “Constructing Domain-Specific Evaluation Sets for LLM-as-a-judge,” Aug. 20, 2024, arXiv: arXiv:2408.08808. doi: 10.48550/arXiv.2408.08808

    2024 doi arXiv

  48. [54]

    LLM-as-a-Judge: Rethinking Model-Based Evaluations in Text Generation

    H. Lee, “LLM-as-a-Judge: Rethinking Model-Based Evaluations in Text Generation.” Accessed: Jan. 18, 2025. [Online]. Available: https://leehanchung.github.io/blogs/2024/08/11/llm-as-a-judge/

    2025

  49. [55]

    EvaluatingtheEffectivenessofLLM-Evaluators(akaLLM-as-Judge),

    E.Yan,“EvaluatingtheEffectivenessofLLM-Evaluators(akaLLM-as-Judge),”eugeneyan.com. Accessed: Oct. 12, 2024. [Online]. Available: https://eugeneyan.com/writing/llm-evaluators/

    2024

  50. [56]

    From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge,

    D. Li et al., “From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge,” Nov. 25, 2024, arXiv: arXiv:2411.16594. doi: 10.48550/arXiv.2411.16594. [57]. Aman Yugank, “Deep Dive into LLM-evaluators aka ‘LLM-as-a-Judge,’” Medium. Accessed: Dec. 19, 2024. [Onlin...

    2024 doi

  51. [58]

    Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges,

    A. S. Thakur, K. Choudhary, V. S. Ramayapally, S. Vaidyanathan, and D. Hupkes, “Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges,” Oct. 11, 2024, arXiv: arXiv:2406.12624. Accessed: Oct. 16, 2024. [Online]. Available: http://arxiv.org/abs/2406.12624

    2024 arXiv

  52. [59]

    PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization,

    Y. Wang et al., “PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization,” May 24, 2024, arXiv: arXiv:2306.05087. Accessed: Sep. 22, 2024. [Online]. Available: http://arxiv.org/abs/2306.05087

    2024 arXiv

  53. [60]

    Evaluating Large Language Models at Evaluating Instruction Following,

    Z. Zeng, J. Yu, T. Gao, Y. Meng, T. Goyal, and D. Chen, “Evaluating Large Language Models at Evaluating Instruction Following,” Apr. 16, 2024, arXiv: arXiv:2310.07641. doi: 10.48550/arXiv.2310.07641

    2024 doi arXiv

  54. [61]

    Large Language Models are not Fair Evaluators,

    P. Wang et al., “Large Language Models are not Fair Evaluators,” Aug. 30, 2023, arXiv: arXiv:2305.17926. doi: 10.48550/arXiv.2305.17926

    2023 doi arXiv

  55. [62]

    RevisEval: Improving LLM-as-a-Judge via Response-Adapted References,

    Q. Zhang et al., “RevisEval: Improving LLM-as-a-Judge via Response-Adapted References,” Oct. 07, 2024, arXiv: arXiv:2410.05193. doi: 10.48550/arXiv.2410.05193

    2024 doi arXiv

  56. [64]

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,

    J. Wei et al., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,” Jan. 10, 2023, arXiv: arXiv:2201.11903. doi: 10.48550/arXiv.2201.11903

    2023 doi arXiv

  57. [65]

    A Survey on Evaluation of Large Language Models,

    Y. Chang et al., “A Survey on Evaluation of Large Language Models,” Dec. 28, 2023, arXiv: arXiv:2307.03109. doi: 10.48550/arXiv.2307.03109

    2023 doi arXiv

  58. [66]

    A Revision of Bloom’s Taxonomy: An Overview,

    D. R. Krathwohl, “A Revision of Bloom’s Taxonomy: An Overview,” Theory Into Practice, vol. 41, no. 4, pp. 212–218, 2002

    2002

  59. [67]

    Bloom’s Taxonomy,

    P. Armstrong, “Bloom’s Taxonomy,” Vanderbilt University. Accessed: Dec. 29, 2024. [Online]. Available: https://cft.vanderbilt.edu/guides-sub-pages/blooms-taxonomy/

    2024

  60. [68]

    Confluence

    Atlassian, “Confluence.” [Online]. Available: https://www.atlassian.com/software/confluence/resources

  61. [69]

    Atlassian, Rovo Chat. (2024). Atlassian. [Online]. Available: https://www.atlassian.com/software/rovo

    2024

  62. [70]

    Paraphrase Mining — Sentence Transformers documentation

    SBERT, “Paraphrase Mining — Sentence Transformers documentation.” Accessed: Dec. 29,

  63. [71]

    GPT-4 Technical Report,

    J. Achiam et al., “GPT-4 Technical Report,” Mar. 04, 2024, arXiv: arXiv:2303.08774. doi: 10.48550/arXiv.2303.08774

    2024 doi arXiv

  64. [72]

    Confidential AI, DeepEval. (2024). Confident AI

    2024

  65. [73]

    LLMEvaluationMetrics:TheUltimateLLMEvaluationGuide-ConfidentAI

    J.Ip,“LLMEvaluationMetrics:TheUltimateLLMEvaluationGuide-ConfidentAI.”Accessed: Dec.29,2024.[Online].Available:https://www.confident-ai.com/blog/llm-evaluation-metrics-everything- you-need-for-llm-evaluation 17

    2024

  66. [74]

    Bleu: a Method for Automatic Evaluation of Machine Translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a Method for Automatic Evaluation of Machine Translation,” in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, P. Isabelle, E. Charniak, and D. Lin, Eds., Philadelphia, Pennsylvania,...

    2002

  67. [75]

    ROUGE:APackageforAutomaticEvaluationofSummaries,

    C.-Y.Lin,“ROUGE:APackageforAutomaticEvaluationofSummaries,”inTextSummarization Branches Out, Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, pp. 74–81. Accessed: Feb. 09, 2025. [Online]. Available: https://aclanthology.org/W04-1013/

    2004

  68. [76]

    BLEU is Not Suitable for the Evaluation of Text Simplification,

    E. Sulem, O. Abend, and A. Rappoport, “BLEU is Not Suitable for the Evaluation of Text Simplification,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, Eds., Brussels, Belgium: Asso...

    2018 doi

  69. [77]

    BERTScore: Evaluating Text Generation with BERT,

    T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “BERTScore: Evaluating Text Generation with BERT,” Feb. 24, 2020, arXiv: arXiv:1904.09675. doi: 10.48550/arXiv.1904.09675

    2020 doi arXiv

  70. [78]

    BARTScore: Evaluating Generated Text as Text Generation,

    W. Yuan, G. Neubig, and P. Liu, “BARTScore: Evaluating Generated Text as Text Generation,” Oct. 27, 2021, arXiv: arXiv:2106.11520. doi: 10.48550/arXiv.2106.11520

    2021 doi arXiv

  71. [79]

    LLM-based NLG Evaluation: Current Status and Challenges,

    M. Gao, X. Hu, J. Ruan, X. Pu, and X. Wan, “LLM-based NLG Evaluation: Current Status and Challenges,” Feb. 26, 2024, arXiv: arXiv:2402.01383. doi: 10.48550/arXiv.2402.01383

    2024 doi arXiv

  72. [80]

    ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning,

    P. Törnberg, “ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning,” Apr. 13, 2023, arXiv: arXiv:2304.06588. doi: 10.48550/arXiv.2304.06588

    2023 doi arXiv

  73. [81]

    ChatGPT Outperforms Crowd-Workers for Text- Annotation Tasks,

    F. Gilardi, M. Alizadeh, and M. Kubli, “ChatGPT Outperforms Crowd-Workers for Text- Annotation Tasks,” Proc. Natl. Acad. Sci. U.S.A., vol. 120, no. 30, p. e2305016120, Jul. 2023, doi: 10.1073/pnas.2305016120

    2023 doi

  74. [82]

    ChatGPT vs. Crowdsourcing vs. Experts: Annotating Open-Domain Conversations with Speech Functions,

    L. Ostyakova, V. Smilga, K. Petukhova, M. Molchanova, and D. Kornev, “ChatGPT vs. Crowdsourcing vs. Experts: Annotating Open-Domain Conversations with Speech Functions,” in Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, S. Stoya...

    2023 doi

  75. [83]

    ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustness,

    J. Cegin, J. Simko, and P. Brusilovsky, “ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustness,” Oct. 19, 2023, arXiv: arXiv:2305.12947. doi: 10.48550/arXiv.2305.12947

    2023 doi arXiv

  76. [84]

    30, 2024

    DeepEval, “G-Eval.” Accessed: Dec. 30, 2024. [Online]. Available: https://docs.confident- ai.com/docs/metrics-llm-evals

    2024

  77. [85]

    Toxicity

    DeepEval, “Toxicity.” Accessed: Dec. 29, 2024. [Online]. Available: https://docs.confident- ai.com/docs/metrics-toxicity

    2024

  78. [86]

    29, 2024

    DeepEval, “Bias.” Accessed: Dec. 29, 2024. [Online]. Available: https://docs.confident- ai.com/docs/metrics-bias

    2024

  79. [87]

    Hallucination

    DeepEval, “Hallucination.” Accessed: Dec. 29, 2024. [Online]. Available: https://docs.confident- ai.com/docs/metrics-hallucination

    2024

  80. [88]

    Spearman’s rank correlation coefficient,

    Wikipedia, “Spearman’s rank correlation coefficient,” Wikipedia. Dec. 27, 2024. Accessed: Feb. 09, 2025. [Online]. Available: https://en.wikipedia.org/w/index.php?title=Spearman%27s_rank_correlation_coefficient&oldid=1265551 920

    2024

  81. [89]

    Llama 3.2: Revolutionizing edge AI and vision with open, customizable models,

    Meta, “Llama 3.2: Revolutionizing edge AI and vision with open, customizable models,” Meta AI. Accessed: Feb. 02, 2025. [Online]. Available: https://ai.meta.com/blog/llama-3-2-connect-2024- vision-edge-mobile-devices/

    2025

  82. [90]

    LLaMA 3.3,

    Meta, “LLaMA 3.3,” LLaMA 3.3 Document. [Online]. Available: https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_3/ 18

  83. [91]

    The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation,

    Meta, “The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation,” Meta AI. Accessed: Jun. 05, 2025. [Online]. Available: https://ai.meta.com/blog/llama-4-multimodal- intelligence/

    2025

  84. [92]

    Qwen2.5 Technical Report,

    A. Yang et al., “Qwen2.5 Technical Report,” Dec. 19, 2024, arXiv: arXiv:2412.15115. doi: 10.48550/arXiv.2412.15115

    2024 doi arXiv

  85. [94]

    LangGraph, “CRAG.” [Online]. Available: https://github.com/langchain- ai/langgraph/blob/main/examples/rag/langgraph_crag.ipynb?ref=blog.langchain.dev APPENDIX A: ALGORITHMS USED IN THE WORK CRAG The corrective retrieval augmented generation (CRAG) [52] algorithm, depicted in F...

  86. [95]

    This method has been incorporated into the DeepEval package [72]

    a prompt outlining the evaluation task and the specific criteria to be used, 2) a LLM call to generate CoT consisting of intermediate instructions that detail the evaluation steps, and 3) a scoring function that utilizes the LLM to compute a score based on the probabilities of...

  87. [2024]

    Available: https://sbert.net/examples/applications/paraphrase-mining/README.html

    [Online]. Available: https://sbert.net/examples/applications/paraphrase-mining/README.html

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.