REVIEW 3 cited by
ClarQ-LLM: A Benchmark for Models Clarifying and Requesting Information in Task-Oriented Dialog
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We introduce ClarQ-LLM, an evaluation framework consisting of bilingual English-Chinese conversation tasks, conversational agents and evaluation metrics, designed to serve as a strong benchmark for assessing agents' ability to ask clarification questions in task-oriented dialogues. The benchmark includes 31 different task types, each with 10 unique dialogue scenarios between information seeker and provider agents. The scenarios require the seeker to ask questions to resolve uncertainty and gather necessary information to complete tasks. Unlike traditional benchmarks that evaluate agents based on fixed dialogue content, ClarQ-LLM includes a provider conversational agent to replicate the original human provider in the benchmark. This allows both current and future seeker agents to test their ability to complete information gathering tasks through dialogue by directly interacting with our provider agent. In tests, LLAMA3.1 405B seeker agent managed a maximum success rate of only 60.05\%, showing that ClarQ-LLM presents a strong challenge for future research.
Forward citations
Cited by 3 Pith papers
-
One More Turn, Less Regret: A Regret-Based Multi-Turn Benchmark for LLMs' Clarification Policies
RegretBench evaluates LLM clarification as a sequential policy under hidden intent, showing that final accuracy alone misses large differences in interaction efficiency and stopping quality.
-
LOGOS: A Living Logic for AI Agent Teams That Evolve With Humans
LOGOS makes multi-agent self-evolution governable by compiling inputs into versioned Agent Packs and promoting only candidates that pass held-out evidence, root policy, and human authorization.
-
Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questions
A fine-tuned classifier and question generator let a small coding assistant detect under-specified prompts and ask for clarification, which users rated better than a baseline in a small study.
Discussion (0). Continue with ORCID to comment.