Pith. sign in

REVIEW 3 major objections 3 minor 40 references

SLMEval: Entropy-Based Calibration for Human-Aligned Evaluation of Large Language Models

T0 review · 3 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read On open-ended tasks, SLMEval claims a maximum-entropy reweighting of raw evaluator scores, fit on a small human-preference set, turns a 4-bit small model into a more human-aligned judge than GPT-4-based calibration, at 5–30x lower cost.

desk verdict The entropy-based reweighting idea is neat and cheap, but the headline correlations likely reflect the fitted human data rather than predictive alignment, and the paper needs a held-out split before the numbers can be believed. read the letter →

arxiv 2505.16003 v1 pith:QOBJOS5G submitted 2025-05-21 cs.CL cs.AI

Roland Daynauth , Christopher Clarke , Krisztian Flautner , Lingjia Tang , Jason Mars This is my paper · ORCID
classification cs.CLcs.AI
keywords LLM-as-a-Judgeentropymaximizationhuman-alignedevaluationcalibrationsmalllanguagemodelsopen-endedgenerationSpearmancorrelationreference-free
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the usual fix for LLM-as-a-Judge evaluation—calibrating scores with GPT-4, chain-of-thought prompts, or token-level probabilities—breaks down on subjective, open-ended tasks, where several strong evaluators show weak or negative correlation with human judgments. SLMEval is the proposed alternative: it uses a small amount of human preference data to fit a latent "strength" weight for each model under evaluation, choosing the weights by maximum entropy so that they satisfy the observed preferences while staying as unbiased as possible. Those weights are then multiplied into the raw evaluator scores before pairwise comparison. In the paper's results, this lifts Spearman correlation with human judgments to 0.48 on a daily-pep-talk task and 0.57 on a recommendation task, where G-Eval with chain-of-thought scores 0.41 and -0.55, and it does so with a 4-bit quantized small model at 5–30x lower estimated cost than GPT-4-based evaluators. The reason this matters is practical: if it holds, production teams can get human-aligned evaluation of creative or personalized outputs without expensive GPT-4 calls.

What carries the argument

The load-bearing object is the latent strength vector $p=(p_1,\ldots,p_n)$, a probability distribution over the models being rated. SLMEval fits $p$ by maximizing the Shannon entropy $H(p)=-\sum_i p_i \log p_i$ subject to normalization and to the inequality constraints $p_i \ge P(i>j)(p_i+p_j)$ for every human-annotated pair $(i,j)$, which relaxes the Bradley-Terry model $P(i>j)=p_i/(p_i+p_j)$ so that noisy and intransitive human data can be tolerated. The fitted weights are applied multiplicatively to the raw evaluator scores, so model $i$ beats model $j$ exactly when $p_i S_{ij} > p_j S_{ji}$, and models are ranked by aggregate calibrated win rate. The entropy objective is what keeps the inferred distribution maximally agnostic about everything except the observed preference constraints.

What would settle it

Run SLMEval with $p$ fit on one half of the human evaluations and the Spearman correlation computed only on the other half; if the held-out correlation drops toward zero or negative, the reported 0.57 does not represent predictive alignment. Bootstrap the ten-model ranking as well, since with $n=10$ a single model's position can move the correlation substantially.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is that the alignment failure of LLM-based evaluators on open-ended tasks is a calibration problem rather than a capability problem. SLMEval models each evaluated model $i$ with a latent strength $p_i$, estimates the vector $p$ by maximizing Shannon entropy subject to normalization and to relaxed Bradley-Terry-style constraints $p_i \ge P(i>j)(p_i+p_j)$ built from human preferences, and then redefines the pairwise win as $p_i S_{ij} > p_j S_{ji}$. With a 4-bit quantized LLaMA 3.1 as the scorer, the calibrated win-rate ranking reaches Spearman correlation 0.48 on the pep-talk task and 0.57 on the recommendation task against human judgments, compared with 0.41 and -0.55 for G-Eval with chain-of-thought; on the public FairEval benchmark it reaches 58.8% pairwise accuracy against MT-Bench human labels, approaching GPT-4 + BPC at 62.5%. The same single-pass, small-model design is what yields the 5–30x cost reduction.

Load-bearing premise

The load-bearing assumption is that the human preference data used to fit the strength weights $p$ is not the same data used to measure SLMEval's correlation with human judgments; the paper describes 360 evaluations per task but no hold-out split, so if the same annotations do both jobs, the reported alignment is partly a reflection of the fitting constraints.

Editorial extensions

If this is right

  • A small human-preference calibration set can make reference-free evaluation reliable on subjective tasks, where standard metrics and GPT-4-based calibrators currently fail.
  • Evaluation costs drop by an estimated 5–30x because the scorer is a 4-bit quantized small model making a single pass, making large-scale or iterative human-aligned evaluation affordable.
  • Performance on the public FairEval dataset suggests the calibration method transfers beyond the two in-house tasks, outperforming GPTScore and plain G-Eval and approaching GPT-4 + BPC.
  • The negative correlations of strong baselines on the recommendation task imply that production teams should not trust benchmark-validated evaluators without task-specific calibration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the paper never states a train/test split between the 360 human evaluations used to fit $p$ and the human judgments used to compute the Spearman correlations, so the reported 0.57 may partly reflect fitting; a held-out replication would settle it.
  • Editorial extension: a transfer test—fit $p$ on one task and apply it to another—would show whether per-task human annotation is avoidable or whether each subjective task needs its own calibration set.
  • Editorial extension: because the calibration only requires pairwise preference inequalities, it could in principle consume any preference signal, including implicit user behavior such as clicks or retention, rather than only curated human annotations.
  • Editorial extension: with only ten models in the ranking, the correlation estimates carry wide uncertainty; bootstrap confidence intervals would show whether the 0.57 gap over G-Eval is statistically meaningful.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes SLMEval, a calibration method for LLM-as-a-Judge evaluation. It estimates a latent strength vector p over the evaluated models by maximizing Shannon entropy subject to relaxed Bradley-Terry-style preference constraints derived from a small set of human pairwise comparisons, then reweights pairwise evaluator scores S_ij by p_i before aggregating win rates into a final ranking. Experiments on two internal production tasks (Peptalk and Recommendation) and on the FairEval public benchmark report that SLMEval improves Spearman correlation over baselines (0.48 and 0.57 on the two production tasks) and reduces API cost by an order of magnitude relative to GPT-4-based calibrated evaluators.

Significance. If the empirical claims hold, SLMEval would be a useful practical contribution: the maximum-entropy formulation is simple and transparent, it requires only a small amount of human preference data, and the use of a 4-bit quantized small model is attractive for cost-sensitive deployment. The comparison against G-Eval, GPTScore, and GPT-4+BPC on a public benchmark is also informative. However, the current validation has a potentially circular evaluation protocol and reports correlations on only ten models without significance testing, so the headline numbers should not be accepted at face value until the protocol is clarified and the results are re-analyzed with an appropriate held-out split.

major comments (3)
  1. [§4.3, §5.1, Appendix A.5] The paper never states whether the human preference data used to fit p via the constraints in Eq. (5) is disjoint from the human judgment data used to compute the Spearman correlations in Table 1. Section 4.3 describes collecting 360 Clickworker evaluations per task, and Section 5.1 reports correlations against human judgments, but no train/test split or cross-validation is described. If the same 360 annotations are used both to constrain p and to compute the reported ρ, then the values 0.48 and 0.57 measure in-sample calibration fit rather than predictive alignment with human judgment. The authors must specify the split or re-run the evaluation with a held-out set; without this, the central empirical claim is not supported.
  2. [§3.3, Eq. (5), Appendix A.5.1] The relaxed constraint pi ≥ P(i>j)(pi + pj) can be infeasible under intransitive human preferences, despite the paper's claim that the relaxation provides robustness to violations of transitivity. For example, if P(A>B)=P(B>C)=P(C>A)=0.6, the constraints imply pA ≥ 1.5 pB ≥ 2.25 pC ≥ 3.375 pA, which has no solution. The paper does not discuss infeasibility, add a feasibility fallback, or report that the empirical D_human satisfied the acyclicity needed for a feasible solution. Since the SLSQP solver would either fail or return an arbitrary point in such cases, this is a load-bearing gap in the method description.
  3. [Table 1, Table 4] All reported correlations are computed over only 10 models. For n=10, a Spearman correlation of 0.57 has a two-sided p-value of roughly 0.085, so the headline result is not statistically significant at the conventional 5% level; no confidence intervals or significance tests are reported. Given that several baselines also have large-magnitude but noisy correlations (e.g., GPTScore at 0.39 on Recommendation), the claim that SLMEval 'achieves strong correlation' needs additional support, such as resampling-based intervals or a larger model set.
minor comments (3)
  1. [§3.3, Eq. (5)] Equation (5) contains a typo: '∀ ∈ Dhuman' should be '∀(i,j) ∈ Dhuman'.
  2. [Table 1, Table 2] The baseline 'GPT-4 + BPC (k = 3)' is never defined; the acronym BPC and the calibration procedure should be described, since this is the strongest baseline on FairEval.
  3. [§5.2] The description of the FairEval evaluation is too terse to reproduce: the paper should state what 'accuracy' means, how MT-Bench serves as the human reference, how many examples are used, and how the cost estimates are computed.

Circularity Check

1 steps flagged · score 8.0 of 10

SLMEval fits its latent strength weights p to human preference constraints and then reports Spearman correlation against the same human judgments without any described holdout split, making the headline production correlation an in-sample fit.

  1. fitted input called prediction [Section 3.3 Eq. (5), Section 3.4 Eq. (3), Section 4.3, Section 5.1 Table 1, Appendix A.5]
    "pi ≥ P (i > j)(pi + pj), ∀ ∈ Dhuman (5) ... model i beats j ⇐ ⇒piSij > pjSji (3) ... We collect human preference data from 360 Clickworker annotators, each comparing paired model responses for randomly selected prompts from two use cases ... yielding 360 evaluations per task ... Table 1 reports the performance of automated evaluators on each task, measured by the Spearman rank correlation (ρ)."

    The entropy fit (A.5.1) solves for p subject to the human-preference constraints in Eq. (5). The final ranking is the sorted win_rate(i) = 1/(n-1) Σ_j 1[p_i S_ij > p_j S_ji], so p directly determines the order that is compared with human judgments. Section 4.3 describes only one set of 360 human evaluations per task, and no calibration/evaluation split is reported; Appendix A.5 says P(i > j) is computed 'based on the human-labeled dataset D_human.' Therefore, as written, the human labels used to constrain p are the same labels used to compute the Spearman ρ in Table 1. Under Eq. (5), a majority preference in D_human with P(i > j) > 0.5 forces p_i > p_j, so the fitted ranking is constrained to reproduce the very human preferences against which it is scored.

full rationale

Aside from this central issue, the method is otherwise self-contained: the entropy-maximization objective and SLSQP solver are standard, the evaluator is a quantized LLaMA 3.1 producing independent raw scores, and the FairEval comparison provides an external benchmark not obviously fit by the production p. Self-citations, e.g., Daynauth et al. (2024) for preference intransitivity and Irugalbandara et al. (2024) for cost methodology, are not load-bearing for the claimed result. However, the production-use-case headline (0.57 on Recommendation, 0.48 on Peptalk) depends on an unstated independence assumption that is contradicted by the paper's own data description: the calibration target and the evaluation target coincide. The score is 8 rather than 10 because raw evaluator scores S still contribute to the final win_rate and the FairEval result is externally grounded.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method introduces a latent strength distribution p as a free parameter vector fit to human preference data. The key domain assumptions are that the relaxed Bradley-Terry constraint set captures human preferences, that maximum entropy over that set yields a useful strength distribution, and crucially that the calibration data is disjoint from the evaluation data used to report correlations. The latter is not established, and the paper's evaluation appears circular under the most natural reading.

free parameters (2)
  • latent strength vector p = not reported (sum to 1, positive)
    Each model i gets a weight p_i fit to human preference data via maximum entropy subject to the relaxed Bradley-Terry inequalities. These weights are the core of the method and are fitted, not derived from first principles.
  • epsilon lower bound = 1e-8
    A small positive constant added as a bound constraint to avoid numerical instability. It is a hand-chosen parameter but has negligible influence on results.
assumptions (4)
  • domain assumption The relaxed Bradley-Terry inequality pi >= P(i>j)(pi+pj) adequately encodes observed human preferences.
    The paper uses this relaxation (Eq. 5) to convert pairwise human preference probabilities into linear constraints on p. If this encoding is wrong, the fit p may not reflect true qualitative strengths.
  • domain assumption The maximum entropy distribution among distributions satisfying the preference constraints is a meaningful representation of model quality.
    There is no theoretical or empirical justification given that maximizing entropy over these constraints produces weights that improve alignment with human judgment, beyond the constraints themselves.
  • domain assumption The human preference data used for calibration is independent of the human judgment data used to compute reported correlations.
    The paper never states that a held-out split exists. If the same annotations are used for both purposes, the reported correlations are circular. This assumption is load-bearing for the validity of the empirical claims.
  • domain assumption The raw evaluator scores Sij, after reweighting by p, retain enough signal to rank models beyond the ordering already enforced by the human preference constraints.
    The final ranking is based on pi*Sij > pj*Sji. If the p weights already encode the human ordering, the scores only affect ties or cases where scores contradict the weights. The paper does not demonstrate that the scores contribute meaningful information beyond the constraints.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SLMEval: Entropy-Based Calibration for Human-Aligned Evaluation of Large Language Models." pith.science (2026). https://pith.science/paper/QOBJOS5G

@misc{pith2026250516003,
  author       = {Pith},
  title        = {Pith review of: SLMEval: Entropy-Based Calibration for Human-Aligned Evaluation of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QOBJOS5G}},
  note         = {Machine review of arXiv:2505.16003}
}
read the original abstract

The LLM-as-a-Judge paradigm offers a scalable, reference-free approach for evaluating language models. Although several calibration techniques have been proposed to better align these evaluators with human judgment, prior studies focus primarily on narrow, well-structured benchmarks. As a result, it remains unclear whether such calibrations generalize to real-world, open-ended tasks. In this work, we show that SOTA calibrated evaluators often fail in these settings, exhibiting weak or even negative correlation with human judgments. To address this, we propose SLMEval, a novel and efficient calibration method based on entropy maximization over a small amount of human preference data. By estimating a latent distribution over model quality and reweighting evaluator scores accordingly, SLMEval achieves strong correlation with human evaluations across two real-world production use cases and the public benchmark. For example, on one such task, SLMEval achieves a Spearman correlation of 0.57 with human judgments, while G-Eval yields a negative correlation. In addition, SLMEval reduces evaluation costs by 5-30x compared to GPT-4-based calibrated evaluators such as G-eval.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 14 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861

    arXiv 2021

  4. [4]

    Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324--345

    1952

  5. [5]

    John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al

    Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175

    arXiv 2018

  6. [6]

    Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. 2023. Exploring the use of large language models for reference-free text quality evaluation: An empirical study. In Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings), pages 361--374

    2023

  7. [7]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. https://lmsys.org/blog/2023-03-30-vicuna/ Vicuna: An open-source chatbot impressing gpt-4 with 90\

    2023

  8. [8]

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132

    arXiv 2024

Show all 40 references
  1. [9]

    Roland Daynauth, Christopher Clarke, Krisztian Flautner, Lingjia Tang, and Jason Mars. 2024. Ranking unraveled: Recipes for llm rankings in head-to-head ai combat. arXiv preprint arXiv:2411.14483

    2024 arXiv

  2. [10]

    Yann Dubois, Bal \'a zs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475

    2024 arXiv

  3. [11]

    Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166

    2023 arXiv

  4. [12]

    Jinlan Fu, See Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024. Gptscore: Evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6556--6576

    2024

  5. [13]

    Chandra Irugalbandara, Ashish Mahendra, Roland Daynauth, Tharuka Kasthuri Arachchige, Jayanaka Dantanarayana, Krisztian Flautner, Lingjia Tang, Yiping Kang, and Jason Mars. 2024. Scaling down to scale up: A cost-benefit analysis of replacing openai's llm with open source slms ...

    2024

  6. [14]

    E. T. Jaynes. 1957. Information theory and statistical mechanics. Phys. Rev. (2), 106:620--630

    1957

  7. [15]

    Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Qiu, Boxun Li, and Yaodong Yang. 2024. Pku-saferlhf: Towards multi-level safety alignment for llms with human preference. arXiv preprint arXiv:2406.15513

    2024 arXiv

  8. [16]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  9. [17]

    Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193--203

    2023

  10. [18]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval

    2023

  11. [19]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

    2004

  12. [20]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 a . https://doi.org/10.18653/v1/2023.emnlp-main.153 G -eval: NLG evaluation using gpt-4 with better human alignment . In Proceedings of the 2023 Conference on Empirical Methods in Natural Langua...

    2023 doi

  13. [21]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 b . G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511--2522

    2023

  14. [22]

    Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. 2024. Calibrating llm-based evaluator. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation ...

    2024

  15. [23]

    Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agarwal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, Hamid Palangi, Guoqing Zheng, Corby Rosset, Hamed Khanpour, and Ahmed Awadallah. 2023. https://arxiv.org/abs/2311.110...

    2023 arXiv

  16. [24]

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. https://arxiv.org/abs/2306.02707 Orca: Progressive learning from complex explanation traces of gpt-4 . Preprint, arXiv:2306.02707

    2023 arXiv

  17. [25]

    OpenAI . 2024. https://openai.com/index/gpt-4/ Gpt-4 . Accessed: 2024-10-24

    2024

  18. [26]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

    2002

  19. [27]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

    2024

  20. [28]

    Anand Rajaraman and Jeffrey David Ullman. 2011. Mining of Massive Datasets. Cambridge University Press

    2011

  21. [29]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084

    2019 arXiv

  22. [30]

    Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. 2020. Bleurt: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

    2020

  23. [31]

    Intel Analytics Software. 2023. https://medium.com/intel-analytics-software/the-practice-of-supervised-finetuning-and-direct-preference-optimization-on-habana-gaudi2-a1197d8a3cd3 The practice of supervised finetuning and direct preference optimization on habana gaudi2 . Access...

    2023

  24. [32]

    Stability AI . 2024. https://stability.ai/news/stablelm-zephyr-3b-stability-llm Stablelm zephyr 3b: Stability's latest llm . Accessed: 2024-10-24

    2024

  25. [33]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  26. [34]

    Rush, and Thomas Wolf

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. 2023. https://arxiv.org/abs/2310.16944 Zeph...

    2023 arXiv

  27. [35]

    Guan Wang, Sijie Cheng, Qiying Yu, and Changling Liu. 2023. Openllms: Less is more for open-source models, july 2023a. URL https://github. com/imoneoi/openchat

    2023

  28. [36]

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. 2024. https://aclanthology.org/2024.acl-long.511 Large language models are not fair evaluators . In Proceedings of the 62nd Annual Meeting of the ...

    2024

  29. [37]

    Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Boyuan Pan, Heda Wang, and Kan Li. 2023. Batcheval: Towards human-like text evaluation. arXiv e-prints, pages arXiv--2401

    2023

  30. [38]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

    2020 arXiv

  31. [39]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th Internat...

    2024

  32. [40]

    Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. 2023. Starling-7b: Improving llm helpfulness and harmlessness with rlaif

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.