REVIEW 26 cited by
Is ChatGPT a Good NLG Evaluator? A Preliminary Study
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Recently, the emergence of ChatGPT has attracted wide attention from the computational linguistics community. Many prior studies have shown that ChatGPT achieves remarkable performance on various NLP tasks in terms of automatic evaluation metrics. However, the ability of ChatGPT to serve as an evaluation metric is still underexplored. Considering assessing the quality of natural language generation (NLG) models is an arduous task and NLG metrics notoriously show their poor correlation with human judgments, we wonder whether ChatGPT is a good NLG evaluation metric. In this report, we provide a preliminary meta-evaluation on ChatGPT to show its reliability as an NLG metric. In detail, we regard ChatGPT as a human evaluator and give task-specific (e.g., summarization) and aspect-specific (e.g., relevance) instruction to prompt ChatGPT to evaluate the generated results of NLG models. We conduct experiments on five NLG meta-evaluation datasets (including summarization, story generation and data-to-text tasks). Experimental results show that compared with previous automatic metrics, ChatGPT achieves state-of-the-art or competitive correlation with human judgments in most cases. In addition, we find that the effectiveness of the ChatGPT evaluator might be influenced by the creation method of the meta-evaluation datasets. For the meta-evaluation datasets which are created greatly depending on the reference and thus are biased, the ChatGPT evaluator might lose its effectiveness. We hope our preliminary study could prompt the emergence of a general-purposed reliable NLG metric.
Forward citations
Cited by 26 Pith papers
-
Paladin: Defending LLM-enabled Phishing Emails with a New Trigger-Tag Paradigm
A trigger-tag watermark embedded by fine-tuning lets modified LLMs mark their own phishing outputs for cheap detection.
-
Towards a Holistic and Automated Evaluation Framework for Multi-Level Comprehension of LLMs in Book-Length Contexts
HAMLET introduces a three-level key-fact hierarchy with automated query-focused summarization to evaluate LLM recall and faithfulness in book-length contexts, achieving over 90% agreement with experts.
-
TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation
Distilling agreement-filtered multi-LRM jury annotations into Gemma-3-12B improves MQM translation quality evaluation from 52.63% to 55.03% average segment-level accuracy, approaching closed LRMs.
-
RetiBridge: Bridging Quantitative Retinal Biomarkers and Qualitative Diagnosis with a Knowledge-Guided Multimodal Large Language Model
RetiBridge/GROK is a 7B multimodal LLM trained to turn quantitative retinal biomarkers into qualitative diagnoses, outperforming larger open models and OpenAI-o3 on its self-built, LLM-judged benchmark.
-
Generalised Probabilistic Modelling and Improved Uncertainty Estimation in Comparative LLM-as-a-judge
A generalised Product-of-Experts framework with a new 'probability of reordering' selection metric that reduces the number of LLM comparisons needed for ranking by about 50%.
-
R-TOFU: Unlearning in Large Reasoning Models
R-TOFU shows that answer-level unlearning is insufficient for large reasoning models because residual knowledge persists in chain-of-thought traces.
-
AutoMedEval: Harnessing Language Models for Automatic Medical Capability Evaluation
A medical evaluation LLM trained with curriculum instruction tuning and iterative knowledge introspection correlates with human judgments better than GPT-4 and other baselines on medical QA responses.
-
SAS-Bench: A Fine-Grained Benchmark for Evaluating Short Answer Scoring with Large Language Models
The paper introduces a 1,030-question benchmark of 4,109 expert-annotated student responses with step-wise scores and error causes, along with two consistency metrics for evaluating LLM short answer scoring.
-
Do Automatic Comment Generation Techniques Fall Short? Exploring the Influence of Method Dependencies on Code Understanding
Including the bodies of helper methods in the prompt enables large language models to generate better comments for dependent Java methods than standard comment generators.
-
Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forward
V2R-Bench shows that 21 large vision-language models are markedly less accurate on simple object and direction tasks when object position, scale, orientation, or context is varied, and attributes the failure to multim...
-
Evaluating Small Language Models for News Summarization: Implications and Factors Influencing Performance
A benchmark of 19 small language models on 2,000 news articles shows that the best small models nearly match 70B-parameter LLMs in summary quality while producing shorter summaries.
-
Optimization is Better than Generation: Optimizing Commit Message Leveraging Human-written Commit Message
A commit-message optimization method that starts from human-written messages and uses GPT-4 plus automated evaluators outperforms message generation and completion methods on three of four quality metrics.
-
Retrieval-Augmented Recommendation Explanation Generation with Hierarchical Aggregation
REXHA improves recommendation explanation generation by hierarchically summarizing all reviews into user and item profiles and using pseudo-document queries to retrieve relevant reviews for a language model.
-
$I^2G$: Generating Instructional Illustrations via Text-Conditioned Diffusion
A pairwise-conditioned diffusion model generates instructional illustrations from procedural text and is finetuned with a text-image alignment reward.
-
PLHF: Prompt Optimization with Few-Shot Human Feedback
PLHF uses a prompt-optimized LLM evaluator trained on a few human-labeled examples to serve as the scoring metric, and reports improved prompt optimization for generative tasks where no clear metric exists.
-
An Empirical Study of Evaluating Long-form Question Answering
In long-form QA, LLM-based evaluators correlate with human judgments better than ROUGE or BERTScore, but they are biased by answer length, question type, self-reinforcement, and rare-word usage, and fine-grained promp...
-
LLMs to Support a Domain Specific Knowledge Assistant
A synthetic QA dataset for IFRS sustainability reporting is created with LLMs and used to build and evaluate two QA pipelines, with the fully LLM-based pipeline scoring highest.
-
Reason4Rec: Deliberative User Preference Alignment of Large Language Models for Recommendation
An LLM recommender that decomposes preference reasoning into summarization, matching, and prediction steps, trained on review text, improves rating accuracy and explanation quality on Amazon and Yelp benchmarks.
-
Can LLMs Ask Good Questions?
LLM-generated questions mostly ask for descriptions and long answers, and they use the source text more evenly than human-authored questions.
-
Evaluate Summarization in Fine-Granularity: Auto Evaluation with LLM
SumAutoEval is an LLM-based entity-level summarization evaluator with four dimensions; its claimed human-correlation advantage is not consistently supported by the experiments.
-
Can Large Language Models Serve as Evaluators for Code Summarization?
An LLM prompt that makes the model role-play a code reviewer scores code summaries with 81.59% Spearman correlation with human judgment, outperforming BLEU and BERTScore on a 300-sample benchmark.
-
LPCAN: Lightweight Pyramid Cross-Attention Network for Rail Surface Defect Detection Using RGB-D Data
A lightweight RGB-D cross-attention network is proposed for rail defect detection, but the SOTA accuracy and generalization claims are internally inconsistent and the implementation is not public.
-
Knockout LLM Assessment: Using Large Language Models for Evaluations through Iterative Pairwise Comparisons
A knockout tournament of iterative pairwise LLM comparisons improves agreement with human expert scores by 0.07 Pearson on average across exam grading and MT evaluation.
-
CEC-Zero: Chinese Error Correction Solution Based on LLM
The authors claim that reinforcement learning with an embedding-clustering reward improves Chinese spelling correction and cross-domain generalization, but the evidence is missing key baselines and reproducibility artifacts.
-
The Performance of the LSTM-based Code Generated by Large Language Models (LLMs) in Forecasting Time Series Data
This paper reports that LLM-generated LSTM forecast code can reach RMSE values close to a generic manually built LSTM on 10 stock datasets, but the comparison is weakened by cherry-picked best results and no repeated ...
-
Knowledge-Embedded and Hypernetwork-Guided Few-Shot Substation Meter Defect Image Generation Method
Fine-tuning Stable Diffusion with DreamBooth-style knowledge and hypernetwork-guided crack control maps can synthesize substation meter defect images that boost a YOLOv8 defect detector's mAP when added to the training set.
Discussion (0). Continue with ORCID to comment.