Pith. sign in

REVIEW 25 cited by

Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.01535 v2 pith:GBGRXUQN submitted 2024-05-02 cs.CL

Seungone Kim , Juyoung Suk , Shayne Longpre , Bill Yuchen Lin , Jamin Shin , Sean Welleck , Graham Neubig , Moontae Lee
show 2 more authors
Kyungjae Lee Minjoon Seo
This is my paper · ORCID
classification cs.CL
keywords assessmentdirectevaluatoropenprometheusrankingtheybenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Proprietary LMs such as GPT-4 are often employed to assess the quality of responses from various LMs. However, concerns including transparency, controllability, and affordability strongly motivate the development of open-source LMs specialized in evaluations. On the other hand, existing open evaluator LMs exhibit critical shortcomings: 1) they issue scores that significantly diverge from those assigned by humans, and 2) they lack the flexibility to perform both direct assessment and pairwise ranking, the two most prevalent forms of assessment. Additionally, they do not possess the ability to evaluate based on custom evaluation criteria, focusing instead on general attributes like helpfulness and harmlessness. To address these issues, we introduce Prometheus 2, a more powerful evaluator LM than its predecessor that closely mirrors human and GPT-4 judgements. Moreover, it is capable of processing both direct assessment and pair-wise ranking formats grouped with a user-defined evaluation criteria. On four direct assessment benchmarks and four pairwise ranking benchmarks, Prometheus 2 scores the highest correlation and agreement with humans and proprietary LM judges among all tested open evaluator LMs. Our models, code, and data are all publicly available at https://github.com/prometheus-eval/prometheus-eval.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 25 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning

    cs.CL 2026-07 conditional novelty 7.0 of 10

    SERPO co-evolves per-question grading rubrics, Good-Normal-Bad response archives, and policy parameters so a language model can train itself at inference time without labels, gaining up to 20.6 points on open-ended me...

  2. Autoregressive Modeling of Film with Applications in Video Montage

    cs.CV 2026-07 conditional novelty 7.0 of 10

    An autoregressive transformer with an explicit cut token and footage-constrained decoding edits raw video into sequences that people rate as better than two prior automated editing methods.

  3. SkillTV-Bench: Benchmarking How Well Judges Perform on Skill-Augmented Agentic Execution

    cs.AI 2026-08 conditional novelty 6.0 of 10

    SkillTV-Bench provides a multi-domain, skill-aware trajectory verification benchmark, and SkillTV-Evolve's evolved JudgeSkill improves an agent judge's accuracy by 14.8 points on a held-out set.

  4. TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation

    cs.CL 2026-08 conditional novelty 6.0 of 10

    Distilling agreement-filtered multi-LRM jury annotations into Gemma-3-12B improves MQM translation quality evaluation from 52.63% to 55.03% average segment-level accuracy, approaching closed LRMs.

  5. Multi-Modal Requirements Data-based Acceptance Criteria Generation using LLMs

    cs.SE 2025-08 conditional novelty 6.0 of 10

    Multi-modal RAG (text plus UI screenshots) with reward-based polishing generates acceptance criteria from user stories that three industry experts rated near 4/5 on relevance, correctness, and understandability.

  6. Interpretable Mnemonic Generation for Kanji Learning via Expectation-Maximization

    cs.CL 2025-07 conditional novelty 6.0 of 10

    An EM-type algorithm jointly learns interpretable mnemonic rules and learner/kanji traits, yielding modest cold-start gains over fine-tuning for kanji mnemonic generation.

  7. Hierarchical Memory Organization for Wikipedia Generation

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Generating Wikipedia articles with factoid memory units organized into a hierarchical outline improves informativeness, verifiability, and citation coverage over RAG and STORM baselines.

  8. FairI Tales: Evaluation of Fairness in Indian Contexts with a Focus on Bias and Stereotypes

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A new India-focused benchmark shows that popular LLMs exhibit measurable negative bias against marginalized Indian identities and frequently reinforce caste, religion, region, and tribe stereotypes.

  9. Outside Knowledge Conversational Video (OKCV) Dataset -- Dialoguing over Videos

    cs.CV 2025-06 conditional novelty 6.0 of 10

    OKCV is a new human-annotated video dialogue dataset where answering questions requires both visual grounding in the video and external knowledge.

  10. Improving Fairness of Large Language Models in Multi-document Summarization

    cs.CL 2025-06 conditional novelty 6.0 of 10

    FairPO combines document-set perturbation with DPO-style preference tuning and corpus-level dynamic weighting to improve both summary-level and corpus-level fairness in multi-document summarization.

  11. DECASTE: Unveiling Caste Stereotypes in Large Language Models through Multi-Dimensional Bias Analysis

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Caste-based stereotypes are measurably present in widely used LLMs, with the largest bias appearing when Dalits and Shudras are compared with dominant castes.

  12. Safety Degradation in AI Agents

    cs.CY 2025-05 conditional novelty 6.0 of 10

    Adding retrieval to aligned LLMs degrades safety: refusal rates fall, bias and harmfulness rise, and prompt-based mitigation only partially restores alignment.

  13. Automatic Evaluation of Healthcare LLMs Beyond Question-Answering

    cs.CL 2025-02 reject novelty 6.0 of 10

    In healthcare LLM evaluation, multiple-choice accuracy and open-ended task scores correlate only weakly, and the paper's proposed Relaxed Perplexity metric aims to improve open-ended factuality scoring but rests on an...

  14. Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering

    cs.SE 2025-02 conditional novelty 6.0 of 10

    Output-based LLM-as-a-judge methods using large LLMs achieve near-human correlation with human scores for code translation and generation, but not for code summarization or pairwise comparisons.

  15. SedarEval: Automated Evaluation using Self-Adaptive Rubrics

    cs.CV 2025-01 reject novelty 6.0 of 10

    A benchmark and judge model that uses per-question custom rubrics to score LLM outputs, claiming better alignment with human grading than GPT-4.

  16. Summary of DCASE 2026 Task 5: Audio-Dependent Question Answering

    eess.AS 2026-07 conditional novelty 5.0 of 10

    On a new 3000-item audio-only benchmark stripped of text-solvable questions, the best AI system scores 58.33%, random guessing scores 25.46%, and all 36 competing systems fail the same 233 items.

  17. When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability

    cs.CL 2026-07 conditional novelty 5.0 of 10

    Judge upgrades are not interchangeable: only Qwen3 1.7B→4B yields robust adjacent gains, MiniMax adjacent releases do not, and stronger judges reduce but do not remove bias or correlated jury errors.

  18. Healthier LLMs: Retrieval-Augmented Generation for Public Health Question Answering

    cs.CL 2026-07 conditional novelty 5.0 of 10

    Hybrid RAG over UK public health guidance sharply raises MCQA accuracy and free-form faithfulness, letting smaller open models match larger closed models without retrieval.

  19. BACON: Budgeted Human Calibration for Modeling and Evaluation with Multiple AI Judges

    cs.LG 2026-06 conditional novelty 5.0 of 10

    BACON calibrates multiple AI judges against a small human-labeled sample, then uses cross-fitted outcome models and augmented estimating equations to produce calibrated summary estimates and item-level surrogate scores.

  20. A Survey of Automatic Evaluation Methods on Text, Visual and Speech Generations

    cs.CL 2025-06 conditional novelty 5.0 of 10

    A unified taxonomy and comparative meta-evaluation of automatic evaluation methods across text, vision, and speech generation, concluding that LLM-based evaluators dominate current practice.

  21. YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering

    cs.CL 2025-05 conditional novelty 5.0 of 10

    An open-source LLM judge for science answers becomes less optimistic after supervised fine-tuning plus contrastive reinforcement learning on benign and adversarially perturbed answers.

  22. Salamandra Technical Report

    cs.CL 2025-02 conditional novelty 5.0 of 10

    Salamandra is an open, from-scratch multilingual LLM family with 2B, 7B, and 40B checkpoints, instruction-tuned variants, a vision proof-of-concept, and detailed evaluations across Iberian and European languages.

  23. FHIR-RAG-MEDS: Integrating HL7 FHIR with Retrieval-Augmented Large Language Models for Enhanced Medical Decision Support

    cs.AI 2025-09 conditional novelty 4.0 of 10

    FHIR-RAG-MEDS integrates HL7 FHIR patient summaries into a RAG system and reports improved guideline-based recommendation quality over bare medical LLMs across four clinical domains.

  24. Atla Selene Mini: A General Purpose Evaluation Model

    cs.CL 2025-01 conditional novelty 4.0 of 10

    The paper presents Selene Mini, an 8B open-weights judge model that reports state-of-the-art average scores across 11 LLM evaluation benchmarks, with gains on medical and financial expert agreement.

  25. PoQ-Judge: A Multi-Architecture Evaluation Framework for Cost-Aware Proof-of-Quality in Decentralized LLM Inference

    cs.CL 2026-04 conditional novelty 3.0 of 10

    Trained reference-free judge models (10M–184M params) achieve 0.747 Pearson correlation with ground-truth quality on QA, matching or exceeding reference-based evaluators for decentralized LLM inference.

Pith tools