pith. machine review for the scientific record. sign in

arxiv: 2303.16634 · v3 · submitted 2023-03-29 · 💻 cs.CL · cs.AI

Recognition: 3 theorem links

· Lean Theorem

G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment

Chenguang Zhu, Dan Iter, Ruochen Xu, Shuohang Wang, Yang Liu, Yichong Xu

Pith reviewed 2026-05-12 22:50 UTC · model grok-4.3

classification 💻 cs.CL cs.AI
keywords NLG evaluationGPT-4chain-of-thoughtform fillinghuman alignmenttext summarizationdialogue generation
0
0 comments X

The pith

GPT-4 with chain-of-thought and form-filling evaluates generated text with higher human alignment than prior automatic metrics.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents G-Eval as a framework that prompts large language models to assess the quality of outputs from natural language generation systems. It demonstrates that combining chain-of-thought reasoning with a structured form-filling approach on GPT-4 produces scores that match human judgments more closely than reference-based metrics like BLEU and ROUGE or earlier neural evaluators. This matters for tasks such as summarization and dialogue generation, where reference texts are often unavailable and outputs vary in creativity. The work also observes that these evaluators show a bias favoring texts generated by similar large language models.

Core claim

G-Eval instructs a large language model to break down evaluation criteria through chain-of-thought steps and then fill a form with scores for dimensions including coherence, consistency, fluency, and relevance, and when using GPT-4 as the backbone this process reaches a Spearman correlation of 0.514 with human ratings on summarization while outperforming all previous automatic methods.

What carries the argument

The G-Eval framework, which applies chain-of-thought reasoning followed by form-filling to guide the language model in producing dimension-specific quality scores.

If this is right

  • The approach works without human reference texts, allowing evaluation on open-ended generation tasks.
  • It exceeds both conventional metrics and medium-sized neural evaluators on summarization and dialogue tasks.
  • LLM-based evaluators exhibit a measurable preference for outputs that resemble their own generation patterns.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Reliable automatic evaluation could speed up development cycles for new NLG models by supplying feedback that tracks human preferences more closely.
  • The observed bias toward LLM-style text suggests testing whether mixing multiple evaluator models or post-hoc calibration reduces favoritism.
  • The same prompting structure might transfer to judging other creative outputs such as story writing or question answering.

Load-bearing premise

That the quality signal extracted from GPT-4 via this prompting method reflects stable human preferences and is not mainly shaped by the model's own training data or output style.

What would settle it

If a new collection of human-annotated NLG outputs shows that G-Eval with GPT-4 yields lower Spearman correlation than the strongest prior method, the performance advantage would not hold.

read the original abstract

The quality of texts generated by natural language generation (NLG) systems is hard to measure automatically. Conventional reference-based metrics, such as BLEU and ROUGE, have been shown to have relatively low correlation with human judgments, especially for tasks that require creativity and diversity. Recent studies suggest using large language models (LLMs) as reference-free metrics for NLG evaluation, which have the benefit of being applicable to new tasks that lack human references. However, these LLM-based evaluators still have lower human correspondence than medium-size neural evaluators. In this work, we present G-Eval, a framework of using large language models with chain-of-thoughts (CoT) and a form-filling paradigm, to assess the quality of NLG outputs. We experiment with two generation tasks, text summarization and dialogue generation. We show that G-Eval with GPT-4 as the backbone model achieves a Spearman correlation of 0.514 with human on summarization task, outperforming all previous methods by a large margin. We also propose preliminary analysis on the behavior of LLM-based evaluators, and highlight the potential issue of LLM-based evaluators having a bias towards the LLM-generated texts. The code is at https://github.com/nlpyang/geval

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The manuscript introduces G-Eval, a framework for evaluating NLG outputs that uses LLMs (primarily GPT-4) with chain-of-thought reasoning and a form-filling paradigm. On the SummEval summarization task it reports a Spearman correlation of 0.514 with human judgments, outperforming prior reference-free and reference-based metrics; similar experiments are presented for dialogue generation. The work also includes preliminary analysis of biases in LLM evaluators toward LLM-generated text and releases code at https://github.com/nlpyang/geval.

Significance. If the reported correlation proves robust after isolating the acknowledged bias toward LLM-generated text and after full disclosure of prompts and statistical controls, G-Eval would constitute a meaningful advance in reference-free NLG evaluation. The open-source code is a clear strength for reproducibility. At present the central claim rests on an empirical result whose magnitude may be partly artifactual, so the significance is conditional on the requested clarifications and ablations.

major comments (3)
  1. [Abstract and §4] Abstract and §4: The claim that G-Eval with GPT-4 achieves a Spearman correlation of 0.514 and outperforms all previous methods is presented without the exact CoT prompt templates, the number of few-shot examples, the precise form-filling instructions, or any statistical significance test for the margin over baselines.
  2. [§4] §4 (bias analysis): The manuscript explicitly flags that LLM evaluators exhibit bias toward LLM-generated outputs, yet provides no ablation that isolates this effect on the reported correlation (e.g., performance on human-written vs. model-generated summary subsets, or substitution of a non-LLM backbone on identical prompts).
  3. [Experiments] Experiments section: No information is given on the exact composition of the SummEval test set (fraction of model-generated summaries), variance across prompt runs, or controls that would rule out style overlap between GPT-4’s training data and the evaluated outputs.
minor comments (2)
  1. [Experiments] A consolidated table comparing all baselines (including prompt-based and neural metrics) with the same correlation metric and test-set splits would improve readability.
  2. [Method] The description of the form-filling paradigm would benefit from an explicit example of the output JSON schema used for each evaluation dimension.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback. The comments highlight important aspects of reproducibility, robustness, and potential confounds that we will address in the revision. We respond to each major comment below.

read point-by-point responses

  1. Referee: [Abstract and §4] Abstract and §4: The claim that G-Eval with GPT-4 achieves a Spearman correlation of 0.514 and outperforms all previous methods is presented without the exact CoT prompt templates, the number of few-shot examples, the precise form-filling instructions, or any statistical significance test for the margin over baselines.

    Authors: We agree that these details are necessary for full reproducibility and verification of the reported 0.514 Spearman correlation. In the revised manuscript we will add the exact CoT prompt templates, the number of few-shot examples, the precise form-filling instructions, and statistical significance tests (including p-values) comparing G-Eval against baselines. These will be placed in a new appendix section. revision: yes

  2. Referee: [§4] §4 (bias analysis): The manuscript explicitly flags that LLM evaluators exhibit bias toward LLM-generated outputs, yet provides no ablation that isolates this effect on the reported correlation (e.g., performance on human-written vs. model-generated summary subsets, or substitution of a non-LLM backbone on identical prompts).

    Authors: The bias analysis in §4 is indeed preliminary. We will add an ablation reporting G-Eval performance separately on the human-written and model-generated subsets of SummEval. Substitution of a non-LLM backbone is not straightforward because the framework depends on LLM-scale CoT reasoning; we will discuss this limitation explicitly and note that smaller models yield substantially lower correlations in our preliminary tests. revision: partial

  3. Referee: [Experiments] Experiments section: No information is given on the exact composition of the SummEval test set (fraction of model-generated summaries), variance across prompt runs, or controls that would rule out style overlap between GPT-4’s training data and the evaluated outputs.

    Authors: SummEval is a public dataset; we will state its exact composition (including the fraction of model-generated summaries) in the revised Experiments section. We will also report variance across multiple independent prompt runs. Complete controls for style overlap with GPT-4 training data are not feasible without access to the training corpus; we will add a discussion of this potential confound and its implications for interpreting the human correlation results. revision: yes

Circularity Check

0 steps flagged

No circularity: G-Eval reports direct empirical correlation against independent human judgments

full rationale

The paper's central result is an observed Spearman correlation (0.514 on summarization) between G-Eval scores and external human annotations on standard benchmarks such as SummEval. This is computed from prompt-based evaluations of model outputs and compared to pre-existing human ratings; no equations, fitted parameters, or derivations are present that could reduce the reported metric to the method's own inputs. Preliminary bias analysis is flagged but remains observational and does not support the main claim. No self-citations, uniqueness theorems, or ansatzes are load-bearing. The evaluation is therefore falsifiable against external data and self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No mathematical axioms, free parameters, or newly postulated entities are introduced; the work is an empirical prompting framework built on existing LLM capabilities.

pith-pipeline@v0.9.0 · 5535 in / 1062 out tokens · 26988 ms · 2026-05-12T22:50:40.954428+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

  • Cost.FunctionalEquation washburn_uniqueness_aczel unclear

    G-EVAL with GPT-4 as the backbone model achieves a Spearman correlation of 0.514 with human on summarization task, outperforming all previous methods by a large margin.

  • Foundation.PhiForcing phi_equation unclear

    G-EVAL is a prompt-based evaluator with three main components: 1) a prompt that contains the definition of the evaluation task and the desired evaluation criteria, 2) a chain-of-thoughts (CoT) ...

Forward citations

Cited by 29 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Green Shielding: A User-Centric Approach Towards Trustworthy AI

    cs.CL 2026-04 unverdicted novelty 7.0

    Green Shielding introduces CUE criteria and the HCM-Dx benchmark to demonstrate that routine prompt variations systematically alter LLM diagnostic behavior along clinically relevant dimensions, producing Pareto-like t...

  2. Four-Axis Decision Alignment for Long-Horizon Enterprise AI Agents

    cs.AI 2026-04 unverdicted novelty 7.0

    Long-horizon enterprise AI agents' decisions decompose into four measurable axes, with benchmark experiments on six memory architectures revealing distinct weaknesses and reversing a pre-registered prediction on summa...

  3. BIASEDTALES-ML: A Multilingual Dataset for Analyzing Narrative Attribute Distributions in LLM-Generated Stories

    cs.CL 2026-04 unverdicted novelty 7.0

    BiasedTales-ML provides a parallel multilingual corpus of LLM-generated children's stories that reveals substantial cross-lingual differences in narrative attributes not captured by English-centric analyses.

  4. CompliBench: Benchmarking LLM Judges for Compliance Violation Detection in Dialogue Systems

    cs.CL 2026-04 unverdicted novelty 7.0

    CompliBench uses simulation and adversarial flaw injection to create labeled dialogue data showing that top proprietary LLMs perform poorly at spotting guideline violations while fine-tuned smaller models outperform t...

  5. TimeSeriesExamAgent: Creating Time Series Reasoning Benchmarks at Scale

    cs.AI 2026-04 conditional novelty 7.0

    TimeSeriesExamAgent combines templates and LLM agents to generate scalable time series reasoning benchmarks, demonstrating that current LLMs have limited performance on both abstract and domain-specific tasks.

  6. Rank, Don't Generate: Statement-level Ranking for Explainable Recommendation

    cs.IR 2026-04 unverdicted novelty 7.0

    The work reframes explainable recommendation as statement-level ranking, introduces the StaR benchmark from Amazon reviews, and finds popularity baselines outperforming SOTA models in item-level personalized ranking.

  7. MSD-Score: Multi-Scale Distributional Scoring for Reference-Free Image Caption Evaluation

    cs.CV 2026-05 unverdicted novelty 6.0

    MSD-Score introduces multi-scale distributional scoring on von Mises-Fisher mixtures to evaluate image captions without references and reports state-of-the-art correlation with human judgments.

  8. An explainable hypothesis-driven approach to Drug-Induced Liver Injury with HADES

    cs.AI 2026-05 unverdicted novelty 6.0

    HADES is an agentic AI system that generates mechanistic hypotheses for drug-induced liver injury using molecular, metabolite, and pathway evidence, outperforming prior binary classifiers on the new DILER benchmark wh...

  9. SurgCheck: Do Vision-Language Models Really Look at Images in Surgical VQA?

    cs.CV 2026-05 unverdicted novelty 6.0

    SurgCheck benchmark reveals that vision-language models for surgical VQA often depend on linguistic shortcuts rather than visual reasoning, shown by consistent performance drops on less-biased questions.

  10. Diversity in Large Language Models under Supervised Fine-Tuning

    cs.LG 2026-04 unverdicted novelty 6.0

    TOFU loss mitigates the narrowing of generative diversity in LLMs after supervised fine-tuning by addressing neglect of low-frequency patterns and forgetting of prior knowledge.

  11. DWTSumm: Discrete Wavelet Transform for Document Summarization

    cs.CL 2026-04 unverdicted novelty 6.0

    DWT decomposes sentence- or word-level embeddings into multi-resolution components that preserve semantics for direct or LLM-guided summarization, yielding up to 97% fidelity and gains in BERTScore and semantic metric...

  12. Proposing Topic Models and Evaluation Frameworks for Analyzing Associations with External Outcomes: An Application to Leadership Analysis Using Large-Scale Corporate Review Data

    cs.CL 2026-04 unverdicted novelty 6.0

    An LLM-based topic modeling method with a custom evaluation framework improves topic interpretability, specificity, and polarity consistency over prior approaches when linking corporate review text to external outcome...

  13. Learning to Control Summaries with Score Ranking

    cs.CL 2026-04 unverdicted novelty 6.0

    A score-ranking loss enables controllable summarization by aligning outputs to evaluation scores, matching SOTA performance with dimension-specific control on LLaMA, Qwen, and Mistral.

  14. Beyond RAG for Cyber Threat Intelligence: A Systematic Evaluation of Graph-Based and Agentic Retrieval

    cs.AI 2026-04 unverdicted novelty 6.0

    A hybrid graph-text retrieval system for cyber threat intelligence improves multi-hop question answering by up to 35% over vector-based RAG on a 3,300-question benchmark.

  15. Noise-Aware In-Context Learning for Hallucination Mitigation in ALLMs

    cs.SD 2026-04 unverdicted novelty 6.0

    NAICL reduces hallucination rates in ALLMs from 26.53% to 16.98% via noise priors in context and introduces the Clotho-1K benchmark with four hallucination types.

  16. ProMedical: Hierarchical Fine-Grained Criteria Modeling for Medical LLM Alignment via Explicit Injection

    cs.AI 2026-04 unverdicted novelty 6.0

    ProMedical builds a 50k preference dataset with fine-grained rubrics and a multi-dimensional reward model that disentangles safety from proficiency, yielding 22.3% accuracy and 21.7% safety gains on Qwen3-8B via GRPO ...

  17. Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge

    cs.AI 2026-04 unverdicted novelty 6.0

    Both humans and LLMs trust content more when labeled human-authored than AI-generated, with LLMs showing denser attention to labels and higher uncertainty under AI labels, mirroring human heuristic patterns.

  18. OpenCLAW-P2P v7.0-P2PCLAW: Resilient Multi-Layer Persistence, Live Reference Verification, and Production-Scale Evaluation of Decentralized AI Peer Review v7.0 -- Mathematical Corrections & Ecosystem Developments Edition

    cs.AI 2026-04 unverdicted novelty 6.0

    OpenCLAW-P2P v6.0 demonstrates a multi-tier persistent decentralized AI peer-review platform that runs 14 autonomous agents, scores over 50 papers, recovers lost data, and verifies references live with claimed >85% fa...

  19. Structured Multi-Criteria Evaluation of Large Language Models with Fuzzy Analytic Hierarchy Process and DualJudge

    cs.AI 2026-04 unverdicted novelty 6.0

    Fuzzy AHP and DualJudge deliver more stable and calibrated LLM evaluations than direct scoring by breaking assessments into explicit criteria and adaptively fusing intuitive and deliberative judgments.

  20. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection

    cs.CL 2023-10 unverdicted novelty 6.0

    Self-RAG trains LLMs to adaptively retrieve passages on demand and self-critique using reflection tokens, outperforming ChatGPT and retrieval-augmented Llama2 on QA, reasoning, and fact verification.

  21. Diversity in Large Language Models under Supervised Fine-Tuning

    cs.LG 2026-04 unverdicted novelty 5.0

    Supervised fine-tuning narrows LLM generative diversity through neglect of low-frequency patterns and knowledge forgetting, but the TOFU loss mitigates this effect across models and benchmarks.

  22. AFA: Identity-Aware Memory for Preventing Persona Confusion in Multi-User Dialogue

    cs.HC 2026-04 unverdicted novelty 5.0

    AFA with identity-aware routing raises persona attribution accuracy from 35.7% to 61.3% on a new synthetic multi-user dialogue dataset.

  23. STELLAR-E: a Synthetic, Tailored, End-to-end LLM Application Rigorous Evaluator

    cs.AI 2026-04 unverdicted novelty 5.0

    STELLAR-E modifies the TGRT Self-Instruct framework to produce tailored synthetic LLM evaluation datasets that score an average 5.7% higher on LLM-as-a-judge metrics than existing language-specific benchmarks.

  24. Calibrating Model-Based Evaluation Metrics for Summarization

    cs.CL 2026-04 unverdicted novelty 5.0

    A reference-free proxy scoring framework combined with GIRB calibration produces better-aligned evaluation metrics for summarization and outperforms baselines across seven datasets.

  25. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions

    cs.CL 2023-11 unverdicted novelty 5.0

    The paper surveys hallucination in LLMs with an innovative taxonomy, factors, detection methods, benchmarks, mitigation strategies, and open research directions.

  26. Localization Boosting for Growth Markets: Mitigating Cross-Locale Behavioral Bias in Learning-to-Rank

    cs.LG 2026-05 unverdicted novelty 4.0

    Multi-objective LTR combining clicks, VLM labels, and locale boosting improves relevance and local content visibility across five growth markets.

  27. A Community-Based Approach for Stance Distribution and Argument Organization

    cs.CL 2026-04 unverdicted novelty 4.0

    Unsupervised graph community detection organizes arguments to reveal stance distributions in debates.

  28. LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods

    cs.CL 2024-12 accept novelty 3.0

    A survey that organizes LLMs-as-judges research into functionality, methodology, applications, meta-evaluation, and limitations.

  29. OpenCLAW-P2P v7.0-P2PCLAW: Resilient Multi-Layer Persistence, Live Reference Verification, and Production-Scale Evaluation of Decentralized AI Peer Review v7.0 -- Mathematical Corrections & Ecosystem Developments Edition

    cs.AI 2026-04 unverdicted novelty 2.0

    OpenCLAW-P2P v7.0 adds mathematical corrections for dimensional consistency and range constraints plus ecosystem expansions including new 4B/9B parameter models to a decentralized AI peer-review platform that claims >...

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages · cited by 27 Pith papers

  1. [1]

    Gptscore: Evaluate as you desire.arXiv preprint arXiv:2302.04166,

    Sentence mover’s similarity: Automatic evalu- ation for multi-sentence texts. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics, pages 2748–2760. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedin...

    work page arXiv 2019

  2. [2]

    In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1797–1807

    Don’t give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1797–1807. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama...

    work page 2018

  3. [3]

    Advances in Neural Information Processing Systems, 35:27730–27744

    Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Ling...

    work page 2002

  4. [4]

    In Proceedings of the 6th international conference on Computational Linguis- tics and Intelligent Text Processing, pages 341–351

    Evaluating evaluation methods for generation in the presence of variation. In Proceedings of the 6th international conference on Computational Linguis- tics and Intelligent Text Processing, pages 341–351. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the fac- tual consistency of summaries. In Proceedings of the...

    work page 2020

  5. [5]

    Is chatgpt a good nlg evaluator? a preliminary study

    Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. Zheng Ye, Liucun Lu, Lishan Huang, Liang Lin, and Xiaodan Liang. 2021. ...

    work page arXiv 2022

  6. [6]

    Towards a unified multi-dimensional evaluator for text generation.arXiv preprint arXiv:2210.07197,

    Benchmarking large language models for news summarization. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Chris- tian M Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized em- beddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th Intern...

    work page arXiv 2019

  7. [7]

    Read the news article carefully and identify the main topic and key points

    work page

  8. [8]

    Check if the summary covers the main topic and key points of the news article, and if it presents them in a clear and logical order

    Read the summary and compare it to the news article. Check if the summary covers the main topic and key points of the news article, and if it presents them in a clear and logical order

    work page

  9. [9]

    Assign a score for coherence on a scale of 1 to 5, where 1 is the lowest and 5 is the highest based on the Evaluation Criteria. Example: Source Text: {{Document}} Summary: {{Summary}} Evaluation Form (scores ONLY): - Coherence: Evaluate Engagingness in the Dialogue Generation Task You will be given a conversation between two individuals. You will then be ...

    work page

  10. [10]

    Read the conversation, the corresponding fact and the response carefully

    work page

  11. [11]

    Rate the response on a scale of 1-3 for engagingness, according to the criteria above

    work page

  12. [12]

    Provide a brief explanation for your rating, referring to specific aspects of the response and the conversation. Example: Conversation History: {{Document}} Corresponding Fact: {{Fact}} Response: {{Response}} Evaluation Form (scores ONLY): - Engagingness: Evaluate Hallucinations Human Evaluation of Text Summarization Systems: Factual Consistency: Does the...

    work page