REVIEW 11 cited by
Human-like Summarization Evaluation with ChatGPT
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Evaluating text summarization is a challenging problem, and existing evaluation metrics are far from satisfactory. In this study, we explored ChatGPT's ability to perform human-like summarization evaluation using four human evaluation methods on five datasets. We found that ChatGPT was able to complete annotations relatively smoothly using Likert scale scoring, pairwise comparison, Pyramid, and binary factuality evaluation. Additionally, it outperformed commonly used automatic evaluation metrics on some datasets. Furthermore, we discussed the impact of different prompts, compared its performance with that of human evaluation, and analyzed the generated explanations and invalid responses.
Forward citations
Cited by 11 Pith papers
-
Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory
An IRT-based two-phase diagnostic framework with four metrics (CV, ρ, θratio, DW) for measuring LLM-judge intrinsic consistency and human alignment.
-
Learning How to Use Tools, Not Just When: Pattern-Aware Tool-Integrated Reasoning
A two-stage pattern-aware tool-integrated reasoning method raises code usage and code-plus-correct metrics on math benchmarks, but the paper conflates Code@1 with problem-solving accuracy in its headline claims.
-
Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge
A regression controlling for human-rated quality detects positive self- and family-bias in several LLM judges, including GPT-4o and Claude 3.5 Sonnet.
-
From General to Targeted Rewards: Surpassing GPT-4 in Open-Ended Long-Context Generation
ProxyReward trains long-form generation models by rewarding how well an AI judge can answer generated yes/no questions about the response, improving open-source models on ProxyQA.
-
From Moderation to Mediation: Can LLMs Serve as Mediators in Online Flame Wars?
The paper proposes a judgment-and-steering mediation framework for LLMs in online flame wars and reports that API-based models outperform open-source ones on a principle-based LLM-judged evaluation, with the toxicity-...
-
AraHalluEval: A Fine-grained Hallucination Evaluation Framework for Arabic LLMs
AraHalluEval introduces a 12-indicator Arabic hallucination taxonomy and finds factual errors dominate, with Allam competitive against reasoning models.
-
Towards Personalized Explanations for Health Simulations: A Mixed-Methods Framework for Stakeholder-Centric Summarization
A mixed-methods framework that elicits stakeholder preferences and steers LLMs to generate tailored summaries of health simulations, presented without empirical validation.
-
User Perceptions of an LLM-Based Chatbot for Cognitive Reappraisal of Stress: Feasibility Study
A GPT-4o chatbot guiding employees through an 11-step reappraisal script was associated with small short-term reductions in self-reported stress and improved stress mindset in an uncontrolled feasibility study.
-
Byzantine-Robust Decentralized Coordination of LLM Agents
A leaderless, Byzantine-robust LLM agent coordination protocol selects answers by geometric-median aggregation of evaluator scores instead of leader-based quorum voting.
-
Reliable Annotations with Less Effort: Evaluating LLM-Human Collaboration in Search Clarifications
LLMs alone annotate search clarifications unreliably; adding confidence-based selective human review cuts effort 24-45% in simulation, but the evaluation is partly built from the ground truth it predicts.
-
Efficient Online RFT with Plug-and-Play LLM Judges: Unlocking State-of-the-Art Performance
A frozen 7B-8B LLM with a JSON rubric and a small LoRA adapter is claimed to outperform 27B-70B reward models and enable 92% GSM-8K exact match under online PPO.
Discussion (0). Continue with ORCID to comment.