REVIEW 10 cited by
MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The advent of Large Language Models (LLMs) has drastically enhanced dialogue systems. However, comprehensively evaluating the dialogue abilities of LLMs remains a challenge. Previous benchmarks have primarily focused on single-turn dialogues or provided coarse-grained and incomplete assessments of multi-turn dialogues, overlooking the complexity and fine-grained nuances of real-life dialogues. To address this issue, we introduce MT-Bench-101, specifically designed to evaluate the fine-grained abilities of LLMs in multi-turn dialogues. By conducting a detailed analysis of real multi-turn dialogue data, we construct a three-tier hierarchical ability taxonomy comprising 4208 turns across 1388 multi-turn dialogues in 13 distinct tasks. We then evaluate 21 popular LLMs based on MT-Bench-101, conducting comprehensive analyses from both ability and task perspectives and observing differing trends in LLMs performance across dialogue turns within various tasks. Further analysis indicates that neither utilizing common alignment techniques nor chat-specific designs has led to obvious enhancements in the multi-turn abilities of LLMs. Extensive case studies suggest that our designed tasks accurately assess the corresponding multi-turn abilities. The data and code are available at \url{https://github.com/mtbench101/mt-bench-101}.
Forward citations
Cited by 10 Pith papers
-
Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging
A three-party-decoupled multi-turn chat benchmark finds closed and open models nearly tied on subjective empathy/persona scores but separated by up to 9× on objective intent tracking, with reasoning and persona format...
-
SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
SPEED-Bench is a new standardized benchmark for speculative decoding that supplies semantically diverse qualitative data and throughput-oriented splits across concurrency levels, integrated with vLLM and TensorRT-LLM.
-
VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents
A Chinese benchmark built on real human speech evaluates large audio language models across instruction following, knowledge, and robustness, revealing large performance gaps.
-
On Robustness and Reliability of Benchmark-Based Evaluation of LLMs
Paraphrasing benchmark questions keeps LLM rankings stable but reduces their accuracy, suggesting static benchmarks overestimate model robustness.
-
Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance
Falcon-H1 reports competitive benchmark scores for a 0.5B to 34B family of parallel hybrid attention/Mamba-2 models, claiming 2x to 4x parameter efficiency versus dense transformers.
-
CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards
A 7B judge model trained with verifiable reward signals and a margin contrastive loss matches the judgment accuracy of models tens of times larger, and a new benchmark JudgerBenchV2 standardizes judge evaluation.
-
Stateful Cooperative Agents Safeguarding LLMs Against Evolving Multi-Turn Attacks
CoopGuard's defer-tempt-analyze-coordinate agents cut reported jailbreak success and raise attacker token costs on the new EMRA benchmark, but the deceptive-rate metric is partly defined by the paper's own scoring rubric.
-
HEAL: A Hypothesis-Based Preference-Aware Analysis Framework
HEAL evaluates preference optimization by measuring ranking accuracy and strength correlation between model likelihoods and proxy reward scores over multi-response hypothesis spaces.
-
HAEPO: History-Aggregated Exploratory Policy Optimization
HAEPO weights each trajectory by its softmax-normalized cumulative log-likelihood, adds entropy and KL penalties, and matches or slightly surpasses PPO, GRPO, and DPO on small RL and summarization tasks.
-
Solution for Meta KDD Cup'25: A Comprehensive Three-Step Framework for Vision Question Answering
Using a single Llama-3.2-11B-Vision-Instruct model per task with RAG, reranking, multi-task fine-tuning, and refusal-data augmentation, the solution ranked 1st on Task3 and 3rd on Tasks 1 and 2 in the CRAG-MM challenge.
Discussion (0). Sign in to comment.