REVIEW 9 cited by
MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We present MultiChallenge, a pioneering benchmark evaluating large language models (LLMs) on conducting multi-turn conversations with human users, a crucial yet underexamined capability for their applications. MultiChallenge identifies four categories of challenges in multi-turn conversations that are not only common and realistic among current human-LLM interactions, but are also challenging to all current frontier LLMs. All 4 challenges require accurate instruction-following, context allocation, and in-context reasoning at the same time. We also develop LLM as judge with instance-level rubrics to facilitate an automatic evaluation method with fair agreement with experienced human raters. Despite achieving near-perfect scores on existing multi-turn evaluation benchmarks, all frontier models have less than 50% accuracy on MultiChallenge, with the top-performing Claude 3.5 Sonnet (June 2024) achieving just a 41.4% average accuracy.
Forward citations
Cited by 9 Pith papers
-
Self-Preference Bias in Rubric-Based Evaluation of Large Language Models
Self-preference bias persists in rubric-based LLM evaluation even with fully objective, programmatically verifiable rubrics, and can shift subjective medical-chat scores by up to ~10 points.
-
Another Turn, Better Output? A Turn-Wise Analysis of Iterative LLM Prompting
Iterative LLM refinement helps early in ideation and code, but in math only late under elaboration prompting; vague feedback tends to plateau or degrade quality.
-
TextQuests: How Good are LLMs at Text-Based Video Games?
Frontier LLMs complete few of 25 Infocom text adventures even when given the official hint booklets, revealing a weakness in sustained long-context reasoning.
-
lmgame-Bench: How Good are LLMs at Playing Games?
lmgame-Bench turns six classic games into a scaffolded LLM evaluation suite, ranks 13 models, detects contamination, and reports RL transfer from Sokoban or Tetris to unseen games and planning tasks.
-
Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Models
Across 23 reasoning models on 420 constrained math problems, stronger reasoning-oriented training and longer chains of thought are associated with worse adherence to user-specified constraints.
-
Echoes of Human Malice in Agents: Benchmarking LLMs for Multi-Turn Online Harassment Attacks
Fine-tuning an LLM on synthetic toxic dialogues makes it harass in 95–97% of multi-turn conversations in Llama and ~99% in Gemini; memory and planning attacks also raise closed-source vulnerability.
-
A Conceptual Framework for AI Capability Evaluations
A descriptive conceptual framework with seven elements (target, task, subject, inputs, instance, measurement, result analysis) for systematizing analysis of AI capability evaluations.
-
OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries
DR.INFO, a vendor-built RAG clinical assistant, is reported to beat frontier LLMs on OpenAI's HealthBench, but the paper's own scores contradict its abstract.
-
Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.
Discussion (0). Sign in to comment.